Spark/Scala實現推薦系統中的相似度演算法（歐幾里得距離、皮爾遜相關係數、餘弦相似度：附實現程式碼）

大資料學習與分享發表於2020-12-11
Spark演算法

在推薦系統中，協同過濾演算法是應用較多的，具體又主要劃分為基於使用者和基於物品的協同過濾演算法，核心點就是基於"一個人"或"一件物品"，根據這個人或物品所具有的屬性，比如對於人就是性別、年齡、工作、收入、喜好等，找出與這個人或物品相似的人或物，當然實際處理中參考的因子會複雜的多。

本篇文章不介紹相關數學概念，主要給出常用的相似度演算法程式碼實現，並且同一演算法有多種實現方式。

 

歐幾里得距離

def euclidean2(v1: Vector, v2: Vector): Double = {
    require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
      s"=${v2.size}.")

    val x = v1.toArray
    val y = v2.toArray

    euclidean(x, y)
  }

  def euclidean(x: Array[Double], y: Array[Double]): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
      s"=${y.length}.")

    math.sqrt(x.zip(y).map(p => p._1 - p._2).map(d => d * d).sum)
  }
  
  def euclidean(v1: Vector, v2: Vector): Double = {
    val sqdist = Vectors.sqdist(v1, v2)
    math.sqrt(sqdist)
  }

 

皮爾遜相關係數

def pearsonCorrelationSimilarity(arr1: Array[Double], arr2: Array[Double]): Double = {
    require(arr1.length == arr2.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${arr1.length} and Len(y)" +
      s"=${arr2.length}.")

    val sum_vec1 = arr1.sum
    val sum_vec2 = arr2.sum

    val square_sum_vec1 = arr1.map(x => x * x).sum
    val square_sum_vec2 = arr2.map(x => x * x).sum

    val zipVec = arr1.zip(arr2)

    val product = zipVec.map(x => x._1 * x._2).sum
    val numerator = product - (sum_vec1 * sum_vec2 / arr1.length)

    val dominator = math.pow((square_sum_vec1 - math.pow(sum_vec1, 2) / arr1.length) * (square_sum_vec2 - math.pow(sum_vec2, 2) / arr2.length), 0.5)
    if (dominator == 0) Double.NaN else numerator / (dominator * 1.0)
  }

 

餘弦相似度

/** jblas實現餘弦相似度 */
  def cosineSimilarity(v1: DoubleMatrix, v2: DoubleMatrix): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(v1)=${x.length} and Len(v2)" +
      s"=${y.length}.")
      
    v1.dot(v2) / (v1.norm2() * v2.norm2())
  }
  
def cosineSimilarity(v1: Vector, v2: Vector): Double = {
    require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
      s"=${v2.size}.")

    val x = v1.toArray
    val y = v2.toArray

    cosineSimilarity(x, y)
  }

  
  def cosineSimilarity(x: Array[Double], y: Array[Double]): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
      s"=${y.length}.")

    val member = x.zip(y).map(d => d._1 * d._2).sum
   
    val temp1 = math.sqrt(x.map(math.pow(_, 2)).sum)
    val temp2 = math.sqrt(y.map(math.pow(_, 2)).sum)

    val denominator = temp1 * temp2
    if (denominator == 0) Double.NaN else member / (denominator * 1.0)
  }

 

修正餘弦相似度

def adjustedCosineSimJblas(x: DoubleMatrix, y: DoubleMatrix): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:DoubleMatrix length do not match: Len(x)=${x.length} and Len(y)" +
      s"=${y.length}.")

    val avg = (x.sum() + y.sum()) / (x.length + y.length)
    val v1 = x.sub(avg)
    val v2 = y.sub(avg)
    v1.dot(v2) / (v1.norm2() * v2.norm2())
  }

 
  def adjustedCosineSimJblas(x: Array[Double], y: Array[Double]): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
      s"=${y.length}.")

    val v1 = new DoubleMatrix(x)
    val v2 = new DoubleMatrix(y)

    adjustedCosineSimJblas(v1, v2)
  }
  
  def adjustedCosineSimilarity(v1: Vector, v2: Vector): Double = {
    require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +
      s"=${v2.size}.")
    val x = v1.toArray
    val y = v2.toArray

    adjustedCosineSimilarity(x, y)
  }

  def adjustedCosineSimilarity(x: Array[Double], y: Array[Double]): Double = {
    require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +
      s"=${y.length}.")

    val avg = (x.sum + y.sum) / (x.length + y.length)

    val member = x.map(_ - avg).zip(y.map(_ - avg)).map(d => d._1 * d._2).sum

    val temp1 = math.sqrt(x.map(num => math.pow(num - avg, 2)).sum)
    val temp2 = math.sqrt(y.map(num => math.pow(num - avg, 2)).sum)

    val denominator = temp1 * temp2
    if (denominator == 0) Double.NaN else member / (denominator * 1.0)
  }

 

大家如果在實際業務處理中有相關需求，可以根據實際場景對上述程式碼進行優化或改造，當然很多演算法框架提供的一些演算法是對這些相似度演算法的封裝，底層還是依賴於這一套，也能幫助大家做更好的瞭解。比如Spark MLlib在KMeans演算法實現中，底層對歐幾里得距離的計算實現。

 

推薦文章：
重要 | Spark分割槽並行度決定機制
解析SparkStreaming和Kafka整合的兩種方式

關注微信公眾號：大資料學習與分享，獲取更對技術乾貨

相關文章