spark on yarn 資料插入mysql

破棉襖發表於2016-05-27
SparkYarnMySql

手動將jar包加在所有計算節點/opt/cloudera/parcels/CDH/lib/hadoop-yarn目錄下
例：scp /opt/jar/mysql-connector-java-5.1.39.jar root@CDH23:/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop-yarn/

spark-shell --driver-class-path /opt/jar/mysql-connector-java-5.1.39.jar

程式碼：
  1.       val aa =sqlContext.read.option("mergeSchema", "true").parquet("/monitor/parquet/a7/year=2016/month=05/day=11")
  2.       aa.registerTempTable("aa")
  3.       val url="jdbc:mysql://ip:port/dbName"
  4.       val prop = new java.util.Properties
  5.       prop.setProperty("user","myUserName")
  6.       prop.setProperty("password","myPassword")
  7.       import org.apache.spark.sql.SaveMode
  8.       
  9.       sqlContext.sql("select appId,count(appId) as number from aa group by appId").write.mode(SaveMode.Append).jdbc(url,"tableName",prop)
亂碼問題：

點選(此處)摺疊或開啟

  1. prop.setProperty("useUnicode","true")
  2. prop.setProperty("characterEncoding","utf8")


來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29754888/viewspace-2108092/，如需轉載，請註明出處，否則將追究法律責任。

相關文章