一.DStream 整合RDD

1.官網運算元

2.使用案例

生產中使用多的是一個檔案中有很多域名，另一箇中是黑名單，要進行剔除
資料一：日誌資訊    DStream
    domain,traffic
    xinlang.com
    xinlang.com
    baidu.com
資料二：已有的檔案  黑名單  RDD
    domain
    baidu.com

3.RDD實現上述需求

package sparkstreaming02
import org.apache.spark.{SparkConf, SparkContext}
import scala.collection.mutable.ListBuffer
object Demo1 {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("Demo1").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val input1 = new ListBuffer[(String,Long)]
    input1.append(("www.xinlang.com", 8888))
    input1.append(("www.xinalng.com", 9999))
    input1.append(("www.baidu.com", 7777))
    val data1 = sc.parallelize(input1)
    //進行join一定要是key,value形式的
    val input2 = new ListBuffer[(String,Boolean)]
    input2.append(("www.baidu.com",true))
    val data2 = sc.parallelize(input2)
    data1.leftOuterJoin(data2)
      .filter(x => {
        x._2._2.getOrElse(false) != true
      }).map(x => (x._1,x._2._1))
      .collect().foreach(println)
  }
}

4.SparkStreaming實現

package sparkstreaming02
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import scala.collection.mutable.ListBuffer
object Streaming {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("Streaming").setMaster("local[2]")
    val ssc = new StreamingContext(conf,Seconds(10))
    val lines = ssc.socketTextStream("s201",9999)
    // 資料二： rdd
    val input2 = new ListBuffer[(String,Boolean)]
    input2.append(("www.baidu.com",true))
    val data2 = ssc.sparkContext.parallelize(input2)
    lines.map(x=>(x.split(",")(0), x)).transform(
      rdd => {
        rdd.leftOuterJoin(data2)
          .filter(x => {
            x._2._2.getOrElse(false) != true //注意 join之後過濾
          }).map(x => (x._1,x._2._1))
      }
    ).print()
    ssc.start()
    ssc.awaitTermination()
  }
}

二.SparkStreaming插入外部資料來源

1.插入外部資料來源用的，但是使用這個有幾個坑

、

2.錯誤一官網例子

3.原因

connect 在Driver端建立，record在executor，發過去序列化錯誤

4.解決

解決：第一種把connect放到executor端
這樣弊端是每條記錄會生成一個connect太耗費資源
        words.foreachRDD { rdd =>
          rdd.foreach { record =>
            val connection = createConnection()  // executed at the driver
            val word = record._1
            val count = record._2.toInt
            val sql = s"insert into wc (wc,count) values($word,$count)"
           connection.createStatement().execute(sql)
         }

5.最終解決辦法

package sparkstreaming02
import java.sql.DriverManager
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
object MysqlStreaming {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[2]").setAppName("MysqlStreaming")
    val ssc = new StreamingContext(conf,Seconds(1))
    val lines = ssc.socketTextStream("s201",9999)
    val words = lines.flatMap(x => x.split(",")).map((_,1)).reduceByKey(_+_)
//    words.foreachRDD { rdd =>
//      val connection = createConnection()  // executed at the driver
//      rdd.foreach { record =>
//        val word = record._1
//        val count = record._2
//        val sql = s"insert into wc (word,count) values($word,$count)"
//        connection.createStatement().execute(sql)
//      }
//    }
//        words.foreachRDD { rdd =>
//          rdd.foreach { record =>
//            val connection = createConnection()  // executed at the driver
//            val word = record._1
//            val count = record._2.toInt
//            val sql = s"insert into wc (wc,count) values($word,$count)"
//            connection.createStatement().execute(sql)
//          }
//        }
    //最終的寫法
    words.foreachRDD { rdd =>
      rdd.foreachPartition { partitionOfRecords =>
        val connection = createConnection()
        partitionOfRecords.foreach(
          record =>{
        val word = record._1
        val count = record._2
        val sql = s"insert into wc (wc,count) values('$word',$count)"
        connection.createStatement().execute(sql)}
        )
      }
    }
    ssc.start()
    ssc.awaitTermination()
  }
  def createConnection() = {
    Class.forName("com.mysql.cj.jdbc.Driver")
    DriverManager.getConnection("jdbc:mysql://localhost:3306/hive?serverTimezone=UTC&useSSL=false","root","123456")
  }
}

6.出現問題

錯誤，插入資料庫時，你要插入字串要用''
例如：
val sql = s"insert into wc (wc,count) values($word,$count)"
word是字串，你要不加雙引號就報這個錯誤
正確
val sql = s"insert into wc (wc,count) values('$word',$count)"