Spark常用Transformations運算元(一)

weixin_33935777發表於2018-11-05
SparkORM

介紹以下Transformations運算元：
map
flatMap
mapPartitions
mapPartitionsWithIndex
filter
sample
union
intersection
sortBy
sortByKey
groupByKey
reduceByKey
distinct
coalesce
repartition

(1) map、mapPartitions、mapPartitionsWithIndex

  1. map以一條記錄為單位進行操作
val arr = Array("Tom","Bob","Tony","Jerry")

//把4條資料分到兩個分割槽中
val rdd = sc.parallelize(arr,2)

/*
 * 模擬把RDD中的元素寫入資料庫的過程
 */
rdd.map(x => {
  println("建立資料庫連線...")
  println("寫入資料庫...")
  println("關閉資料庫連線...")
  println()
}).count()

結果：

建立資料庫連線...
寫入資料庫...
關閉資料庫連線...

建立資料庫連線...
寫入資料庫...
關閉資料庫連線...

建立資料庫連線...
寫入資料庫...
關閉資料庫連線...

建立資料庫連線...
寫入資料庫...
關閉資料庫連線...
  1. mapPartitions以分割槽為單位進行操作
val arr = Array("Tom","Bob","Tony","Jerry")
//把4條資料分到兩個分割槽中
val rdd = sc.parallelize(arr,2)

/*
* 將RDD中的資料寫入到資料庫中，絕大部分使用mapPartitions運算元來實現
*/
rdd.mapPartitions(x => {
  println("建立資料庫連線...")
  val list = new ListBuffer[String]()
  while(x.hasNext) {
    // 模擬寫入資料庫
    list += x.next() + "寫入資料庫"
  }
  // 模擬執行SQL語句，批量插入
  list.iterator
}).foreach(println)

結果：

建立資料庫
Tom:寫入資料庫
Bob:寫入資料庫 
建立資料庫
Tony:寫入資料庫
Jerry:寫入資料庫
  1. mapPartitionsWithIndex
val dataArr = Array("Tom01","Tom02","Tom03"
                  ,"Tom04","Tom05","Tom06"
                  ,"Tom07","Tom08","Tom09"
                  ,"Tom10","Tom11","Tom12")
val rdd = sc.parallelize(dataArr, 3);
val result = rdd.mapPartitionsWithIndex((index,x) => {
    val list = ListBuffer[String]()
    while (x.hasNext) {
      list += "partition:"+ index + " content:" + x.next
    }
    list.iterator
})
println("分割槽數量:" + result.partitions.size)
val resultArr = result.collect()
for(x <- resultArr){
  println(x)
}

結果：

分割槽數量:3
partition:0 content:Tom01
partition:0 content:Tom02
partition:0 content:Tom03
partition:0 content:Tom04
partition:1 content:Tom05
partition:1 content:Tom06
partition:1 content:Tom07
partition:1 content:Tom08
partition:2 content:Tom09
partition:2 content:Tom10
partition:2 content:Tom11
partition:2 content:Tom12

(2) flatMap

val conf = new SparkConf().setAppName("FlatMapTest").setMaster("local")
val sc = new SparkContext(conf)
val data = Array("hello hadoop","hello hive", "hello spark")
val rdd = sc.makeRDD(data)

rdd.flatMap(_.split(" ")).foreach(println)
/*
結果：
hello
hadoop
hello
hive
hello
spark
*/
rdd.map(_.split(" ")).foreach(println)
/*
[Ljava.lang.String;@3c986196
[Ljava.lang.String;@113116a6
[Ljava.lang.String;@542d75a6
*/

map 和 flatMap的區別
map：輸入一條資料，返回一條資料
flatMap：輸入一條資料，可能返回多條資料

468490-46443985b7a2f968.png
image.png

以下scala程式可以說明map函式、flatMap函式和flatten函式的區別和聯絡：

scala> val arr = Array("hello hadoop","hello hive","hello spark")
arr: Array[String] = Array(hello hadoop, hello hive, hello spark)

scala> val map = arr.map(_.split(" "))
map: Array[Array[String]] = Array(Array(hello, hadoop), Array(hello, hive), Array(hello, spark))

scala> map.flatten
res1: Array[String] = Array(hello, hadoop, hello, hive, hello, spark)

scala> arr.flatMap(_.split(" "))
res2: Array[String] = Array(hello, hadoop, hello, hive, hello, spark)

(3) filter ：過濾

val rdd = sc.makeRDD(Array("hello","hello","hello","world"))
// filter(boolean) 返回的是判斷條件為true的記錄
rdd.filter(!_.contains("hello")).foreach(println)

結果：
world

(4) sample ：隨機抽樣

sample(withReplacement: Boolean, fraction: Double, seed: Long)  

withReplacement : 是否是放回式抽樣  
    true代表如果抽中A元素，之後還可以抽取A元素
    false代表如果抽中了A元素，之後都不在抽取A元素  
fraction : 抽樣的比例  
seed : 抽樣演算法的隨機數種子，不同的數值代表不同的抽樣規則，可以手動設定，預設為long的隨機數


val rdd = sc.makeRDD(Array(
  "hello1","hello2","hello3","hello4","hello5","hello6",
  "world1","world2","world3","world4"
))
rdd.sample(false, 0.3).foreach(println)

結果：理論上會隨機抽取30%的資料，但是在資料量不大的時候，不一定很準確

hello1
hello3
world3

(5) union：把兩個RDD進行邏輯上的合併

val rdd1 =sc.makeRDD(1 to 3)
val rdd2 = sc.parallelize(4 until 6)
rdd1.union(rdd2).foreach {println}

結果：

1
2
3
4
5

(6) intersection：求兩個RDD的交集

val rdd1 =sc.makeRDD(1 to 3)
val rdd2 = sc.parallelize(2 to 5)

rdd1.intersection(rdd2).foreach(println)

結果：
2
3

(7) sortBy和sortByKey

  1. sortBy：手動指定排序的欄位
val arr = Array(
        Tuple3(190,100,"Jed"),
        Tuple3(100,202,"Tom"),
        Tuple3(90,111,"Tony")
    )
val rdd = sc.parallelize(arr)
rdd.sortBy(_._1).foreach(println)
/* 按第一個元素排序
   (90,111,Tony)
   (100,202,Tom)
   (190,100,Jed)
 */

rdd.sortBy(_._2, false).foreach(println)
/* 按照第二個元素排序，降序
   (100,202,Tom)
   (90,111,Tony)
   (190,100,Jed)
 */

rdd.sortBy(_._3).foreach(println)
/* 按照第三個元素排序
   (190,100,Jed)
   (100,202,Tom)
   (90,111,Tony)
 */

}
  1. sortByKey：按key進行排序
val rdd = sc.makeRDD(Array(
      (5,"Tom"),(10,"Jed"),(3,"Tony"),(2,"Jack")
    ))
rdd.sortByKey().foreach(println)

結果：

(2,Jack)
(3,Tony)
(5,Tom)
(10,Jed)

(8) groupByKey和reduceByKey

val rdd = sc.makeRDD(Array(
      ("Tom",1),("Tom",1),("Tony",1),("Tony",1)
    ))

rdd.groupByKey().foreach(println)
/*
(Tom,CompactBuffer(1, 1))
(Tony,CompactBuffer(1, 1))
*/

rdd.reduceByKey(_+_).foreach(println)
/*
(Tom,2)
(Tony,2)
*/
468490-d766cb75dd561d5d.png
image.png

(9) distinct：去掉重複資料

val rdd = sc.makeRDD(Array(
      "hello",
      "hello",
      "hello",
      "world"
    ))

rdd.distinct().foreach {println}
/*
hello
world
*/

// dinstinct = map + reduceByKey + map
val distinctRDD = rdd
  .map {(_,1)}
  .reduceByKey(_+_)
  .map(_._1)
distinctRDD.foreach {println}
/*
hello
world
*/
468490-7e4b2ee0e7d95818.png
image.png

(10) coalesce、repartition：改變RDD分割槽數

  1. coalesce
/*
 * false:不產生shuffle
 * true:產生shuffle
 * 如果重分割槽的數量大於原來的分割槽數量,必須設定為true,否則分割槽數不變
 * 增加分割槽會把原來的分割槽中的資料隨機分配給設定的分割槽中
 * 預設為false
 */
object CoalesceTest {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("MapTest").setMaster("local")
    val sc = new SparkContext(conf)
    val arr = Array(
      "partition:0 content:Tom01",
      "partition:0 content:Tom02",
      "partition:0 content:Tom03",
      "partition:0 content:Tom04",
      "partition:1 content:Tom05",
      "partition:1 content:Tom06",
      "partition:1 content:Tom07",
      "partition:1 content:Tom08",
      "partition:2 content:Tom09",
      "partition:2 content:Tom10",
      "partition:2 content:Tom11",
      "partition:2 content:Tom12")

    val rdd = sc.parallelize(arr, 3);

    val coalesceRdd = rdd.coalesce(6,true)

    val results = coalesceRdd.mapPartitionsWithIndex((index,x) => {
      val list = ListBuffer[String]()
      while (x.hasNext) {
        list += "partition:"+ index + " content:[" + x.next + "]"
      }
      list.iterator
    })

    println("分割槽數量:" + results.partitions.size)
    results.foreach(println)
    /*
    分割槽數量:6
    partition:0 content:[partition:1 content:Tom07]
    partition:0 content:[partition:2 content:Tom10]
    partition:1 content:[partition:0 content:Tom01]
    partition:1 content:[partition:1 content:Tom08]
    partition:1 content:[partition:2 content:Tom11]
    partition:2 content:[partition:0 content:Tom02]
    partition:2 content:[partition:2 content:Tom12]
    partition:3 content:[partition:0 content:Tom03]
    partition:4 content:[partition:0 content:Tom04]
    partition:4 content:[partition:1 content:Tom05]
    partition:5 content:[partition:1 content:Tom06]
    partition:5 content:[partition:2 content:Tom09]
    */

    // 增加分割槽肯定會發生shuffle，如果設定為false，結果是不變的
    val coalesceRdd2 = rdd.coalesce(6,false)
    val results2 = coalesceRdd2.mapPartitionsWithIndex((index,x) => {
      val list = ListBuffer[String]()
      while (x.hasNext) {
        list += "partition:"+ index + " content:[" + x.next + "]"
      }
      list.iterator
    })

    println("分割槽數量:" + results2.partitions.size)
    results2.foreach(println)
    /*
    分割槽數量:3
    partition:0 content:[partition:0 content:Tom01]
    partition:0 content:[partition:0 content:Tom02]
    partition:0 content:[partition:0 content:Tom03]
    partition:0 content:[partition:0 content:Tom04]
    partition:1 content:[partition:1 content:Tom05]
    partition:1 content:[partition:1 content:Tom06]
    partition:1 content:[partition:1 content:Tom07]
    partition:1 content:[partition:1 content:Tom08]
    partition:2 content:[partition:2 content:Tom09]
    partition:2 content:[partition:2 content:Tom10]
    partition:2 content:[partition:2 content:Tom11]
    partition:2 content:[partition:2 content:Tom12]
    */

    val coalesceRdd3 = rdd.coalesce(2,true)
    val results3 = coalesceRdd3.mapPartitionsWithIndex((index,x) => {
      val list = ListBuffer[String]()
      while (x.hasNext) {
        list += "partition:"+ index + " content:[" + x.next + "]"
      }
      list.iterator
    })

    println("分割槽數量:" + results3.partitions.size)
    results3.foreach(println)
    /*
    分割槽數量:2
    partition:0 content:[partition:0 content:Tom01]
    partition:0 content:[partition:0 content:Tom03]
    partition:0 content:[partition:1 content:Tom05]
    partition:0 content:[partition:1 content:Tom07]
    partition:0 content:[partition:2 content:Tom09]
    partition:0 content:[partition:2 content:Tom11]
    partition:1 content:[partition:0 content:Tom02]
    partition:1 content:[partition:0 content:Tom04]
    partition:1 content:[partition:1 content:Tom06]
    partition:1 content:[partition:1 content:Tom08]
    partition:1 content:[partition:2 content:Tom10]
    partition:1 content:[partition:2 content:Tom12]
    */

    val coalesceRdd4 = rdd.coalesce(2,false)
    val results4 = coalesceRdd4.mapPartitionsWithIndex((index,x) => {
      val list = ListBuffer[String]()
      while (x.hasNext) {
        list += "partition:"+ index + " content:[" + x.next + "]"
      }
      list.iterator
    })

    println("分割槽數量:" + results4.partitions.size)
    results4.foreach(println)
    /*
    分割槽數量:2
    partition:0 content:[partition:0 content:Tom01]
    partition:0 content:[partition:0 content:Tom02]
    partition:0 content:[partition:0 content:Tom03]
    partition:0 content:[partition:0 content:Tom04]
    partition:1 content:[partition:1 content:Tom05]
    partition:1 content:[partition:1 content:Tom06]
    partition:1 content:[partition:1 content:Tom07]
    partition:1 content:[partition:1 content:Tom08]
    partition:1 content:[partition:2 content:Tom09]
    partition:1 content:[partition:2 content:Tom10]
    partition:1 content:[partition:2 content:Tom11]
    partition:1 content:[partition:2 content:Tom12]
    */
  }

}

以下圖片說明這些情況：

468490-8119a50569008ea0.png
image.png
  1. repartition
repartition(int n) = coalesce(int n, true)
  1. partitionBy：自定義分割槽器，重新分割槽
package com.aura.transformations

import org.apache.spark.{Partitioner, SparkConf, SparkContext}

/**
  * Author: Jed
  * Description: 自定義分割槽規則
  * Date: Create in 2018/1/12
  */
class MyPartition extends Partitioner {

  // 分割槽數量為2
  override def numPartitions: Int = 2

  // 自定義分割槽規則
  override def getPartition(key: Any): Int = {
    if(key.hashCode() % 2 == 0) {
      0
    }else {
      1
    }
  }
}

object PartitionByTest {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("PartitionByTest").setMaster("local")
    val sc = new SparkContext(conf)

    val arr = Array((1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9))
    val rdd = sc.makeRDD(arr,3)
    println("分割槽數量：" + rdd.partitions.length)
    rdd.foreachPartition(x => {
      println("*******")
      while(x.hasNext) {
        println(x.next())
      }
    })
    /*
    分割槽數量：3
    *******
    (1,1)
    (2,2)
    (3,3)
    *******
    (4,4)
    (5,5)
    (6,6)
    *******
    (7,7)
    (8,8)
    (9,9)
     */

    val partitionRDD = rdd.partitionBy(new MyPartition)
    println("分割槽數量：" + partitionRDD.partitions.length)
    partitionRDD.foreachPartition(x => {
      println("*******")
      while(x.hasNext) {
        println(x.next())
      }
    })
    /*
    分割槽數量：2
    *******
    (2,2)
    (4,4)
    (6,6)
    (8,8)
    *******
    (1,1)
    (3,3)
    (5,5)
    (7,7)
    (9,9)
     */
  }

}

相關文章