如何在eclipse除錯storm程式
一、介紹
storm提供了兩種執行模式:本地模式和分散式模式。本地模式針對開發除錯storm topologies非常有用。
Storm has two modes of operation: local mode and distributed mode. In local mode, Storm executes completely in process by simulating worker nodes with threads. Local mode is useful for testing and development of topologies
因為多數程式開發者都是使用windows系統進行程式開發,如果在本機不安裝storm環境的情況下,開發、除錯storm程式。如果你正在為此問題而煩惱,請使用本文提供的方法。
二、實施步驟
如何基於eclipse+maven除錯storm程式,步驟如下:
1.搭建好開發環境(eclipse+maven,本人使用的是eclipse Kepler 與maven3.1.1)
2.建立maven專案,並修改pom.xml,內容如pom.xml(機器聯網,下載所需的依賴jar)
Github上的pom.xml,引入的依賴太多,有些不需要,詳細可以參考:
https://github.com/nathanmarz/storm-starter/blob/master/m2-pom.xml
3. 編寫storm程式,指定為本地模式執行。本文提供的程式是wordcount
重要的是LocalCluster cluster = new LocalCluster();這一句
Config conf = new Config();
conf.setDebug(true);
conf.setNumWorkers(2);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("test", conf, builder.createTopology());
Utils.sleep(10000);
cluster.killTopology("test");
cluster.shutdown();
pom.xml檔案
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>storm.starter</groupId>
<artifactId>storm-starter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
<repository>
<id>github-releases</id>
<url>http://oss.sonatype.org/content/repositories/github-releases/</url>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>storm</groupId>
<artifactId>storm</artifactId>
<version>0.9.0.1</version>
<!-- keep storm out of the jar-with-dependencies -->
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
</dependencies>
</project>
storm程式
package storm.starter;
import java.util.HashMap;
import java.util.Map;
import storm.starter.spout.RandomSentenceSpout;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
/**
* This topology demonstrates Storm`s stream groupings and multilang
* capabilities.
*/
public class WordCountTopology {
public static class SplitSentence extends BaseBasicBolt {
@Override
public void execute(Tuple input, BasicOutputCollector collector) {
try {
String msg = input.getString(0);
System.out.println(msg + "-------------------");
if (msg != null) {
String[] s = msg.split(" ");
for (String string : s) {
collector.emit(new Values(string));
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
public static class WordCount extends BaseBasicBolt {
Map<String, Integer> counts = new HashMap<String, Integer>();
@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String word = tuple.getString(0);
Integer count = counts.get(word);
if (count == null)
count = 0;
count++;
counts.put(word, count);
collector.emit(new Values(word, count));
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word", "count"));
}
}
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping(
"spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split",
new Fields("word"));
Config conf = new Config();
conf.setDebug(true);
if (args != null && args.length > 0) {
conf.setNumWorkers(3);
StormSubmitter.submitTopology(args[0], conf,
builder.createTopology());
} else {
conf.setMaxTaskParallelism(3);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
Thread.sleep(10000);
cluster.shutdown();
}
}
}
package storm.starter.spout;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichSpout;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;
import java.util.Map;
import java.util.Random;
public class RandomSentenceSpout extends BaseRichSpout {
SpoutOutputCollector _collector;
Random _rand;
@Override
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
_collector = collector;
_rand = new Random();
}
@Override
public void nextTuple() {
Utils.sleep(100);
String[] sentences = new String[]{ "the cow jumped over the moon", "an apple a day keeps the doctor away",
"four score and seven years ago", "snow white and the seven dwarfs", "i am at two with nature" };
String sentence = sentences[_rand.nextInt(sentences.length)];
_collector.emit(new Values(sentence));
}
@Override
public void ack(Object id) {
}
@Override
public void fail(Object id) {
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
三、參考資料
https://github.com/nathanmarz/storm/wiki/Tutorial
相關文章
- 淺談1——用Eclipse除錯JAVA程式Eclipse除錯Java
- IDEA、ECLIPSE遠端除錯IdeaEclipse除錯
- 如何在vscode中除錯程式碼?VSCode除錯
- 使用 Eclipse 遠端除錯 Java 應用程式(mark)Eclipse除錯Java
- 使用Eclipse除錯PHP應用Eclipse除錯PHP
- 如何在 VS Code 中除錯 LeetCode 程式碼?除錯LeetCode
- 如何在VS CODE除錯Angular除錯Angular
- Eclipse除錯找不到源的解決辦法Eclipse除錯
- Python 程式碼除錯—使用 pdb 除錯Python除錯
- 使用Eclipse進行PHP的伺服器端除錯EclipsePHP伺服器除錯
- Storm系列(三)java編寫第個storm程式ORMJava
- OrchardNoCMS vNext如何在VS2015下除錯除錯
- 如何在Visual Studio中除錯.NET原始碼除錯原始碼
- 如何在C#中除錯LINQ查詢C#除錯
- GDB 除錯程式碼除錯
- repr除錯python程式除錯Python
- vscode 除錯linux程式VSCode除錯Linux
- 反除錯&反反除錯 -- 利用sysctl檢測偵錯程式是否存在除錯
- 如何在VScode 中 除錯 ts 檔案 Unit testVSCode除錯
- Flutter 應用程式除錯Flutter除錯
- 用GDB除錯程式(六)除錯
- vue 程式碼除錯神器Vue除錯
- Java程式中除錯Python程式方法Java除錯Python
- 第十課 如何在Remix環境下進行Solidity程式碼單步除錯REMSolid除錯
- 如何在vscode裡面除錯js和node.jsVSCode除錯Node.js
- 除錯篇——除錯物件與除錯事件除錯物件事件
- 使用 VSCode 除錯 Electron 主程式程式碼VSCode除錯
- 反除錯 -- 利用ptrace阻止偵錯程式附加除錯
- Emacs 除錯祕籍之 GUD 偵錯程式Mac除錯
- 純正的nodejs程式除錯NodeJS除錯
- 使用 vscode 除錯前端程式碼VSCode除錯前端
- phpstorm 使用 Xdebug 除錯程式碼PHPORM除錯
- 除錯 Docker 容器內部程式除錯Docker
- Linux中使用GDB除錯程式Linux除錯
- 如何除錯程式的 Release 版本除錯
- VS Code除錯C程式碼除錯C程式
- Python之PySnooper程式碼除錯PythonOOP除錯
- Maven:如何在eclipse中使用mvn clean install?MavenEclipse
- 如何在Eclipse中配置安卓環境Eclipse安卓