spark的pom.xml
一個spark任務提交前需要用idea打包程式碼,pom.xml設定
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>spark_submit</artifactId>
<version>1.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<properties>
<scala.version>2.11.8</scala.version>
<hadoop.version>2.7.5</hadoop.version>
<spark.version>2.2.0</spark.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.37</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass/>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
相關文章
- pom.xml詳解XML
- Maven中的pom.xml詳解MavenXML
- Maven的pom.xml檔案詳解MavenXML
- Maven中pom.xml解析MavenXML
- POM.xml 標籤詳解XML
- maven的pom.xml用解決版本問題MavenXML
- Python改寫maven的pom.xml檔案PythonMavenXML
- 【SSM整合】-Maven管理SSM框架的pom.xml配置SSMMaven框架XML
- Maven 教程之 pom.xml 詳解MavenXML
- spring boot 3.3.3 通用 pom.xmlSpring BootXML
- Maven pom.xml檔案配置詳解MavenXML
- spark 的方式Spark
- Spark SQL:4.對Spark SQL的理解SparkSQL
- 【Spark篇】---Spark中Shuffle檔案的定址Spark
- pom.xml 配置阿里雲映象倉庫XML阿里
- Spark之spark shellSpark
- 【Spark篇】---Spark初始Spark
- Spark入門(五)--Spark的reduce和reduceByKeySpark
- Spark入門(四)--Spark的map、flatMap、mapToPairSparkAPTAI
- Spark RDD的預設分割槽數:(spark 2.1.0)Spark
- Spark RDD在Spark中的地位和作用如何?Spark
- Spark面試題(八)——Spark的Shuffle配置調優Spark面試題
- Spark記錄(二):Spark程式的生命週期Spark
- 【Spark篇】--Spark中Standalone的兩種提交模式Spark模式
- Maven實戰:pom.xml與settings.xmlMavenXML
- Spark on Yarn 和Spark on MesosSparkYarn
- Spark系列 - (3) Spark SQLSparkSQL
- spark 的簡介Spark
- Spark的工作原理Spark
- Maven快速入門(四)Maven中的pom.xml檔案詳解MavenXML
- 如何在maven專案的pom.xml檔案中新增jar包MavenXMLJAR
- Spark in action on Kubernetes - Spark Operator的原理解析Spark
- Spark流教程 :使用 Apache Spark 的Twitter情緒分析SparkApache
- spark基礎之spark streaming的checkpoint機制Spark
- 【Spark篇】--Spark中的寬窄依賴和Stage的劃分Spark
- Spark學習進度-Spark環境搭建&Spark shellSpark
- 【Spark】Spark容錯機制Spark
- Spark導論(Spark自學一)Spark