Spark on Yarn開(kāi)發(fā)運(yùn)維過(guò)程中遇到的問(wèn)題匯總

啟動(dòng)nodemanager報(bào)錯(cuò) No space left on device

使用df -h命令判斷nodemanager運(yùn)行日志和啟動(dòng)日志磁盤空間是否足夠。

使用pyspark讀取kafka對(duì)應(yīng)topic數(shù)據(jù)報(bào)錯(cuò)java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

更改之前

./bin/spark-submit –jars lib/spark-streaming-kafka_2.10-1.6.1.jar,lib/kafka_2.10-0.8.2.1.jar,lib/metrics-core-2.2.0.jar –deploy-mode client ./PRoject/stream.py

更改之后

./bin/spark-submit –jars lib/spark-streaming-kafka_2.10-1.6.1.jar,lib/kafka_2.10-0.8.2.1.jar,lib/metrics-core-2.2.0.jar,lib/kafka-clients-0.8.2.1.jar –deploy-mode client ./project/stream.py