【預研】Storm Ubuntu 12.04 64 bit 的單機版本的安裝和執行

基礎知識思考整理
http://blog.csdn.net/aganlengzi/article/details/53261391

主要是Storm流計算的簡單的環境的搭建和簡單的demo的執行。
當前系統和系統中已經預裝的軟體：
其中軟體包括jdk1.7.0_79、python2.7.6、zookeeper3.3.5

如果沒裝的話，可以大致按照[1]這個安裝這些軟體。

安裝ZeroMQ和jzmq

Storm底層是基於ZeroMQ訊息佇列的，所以還需要安裝ZeroMQ和其java版本，可以參照[6]進行。但是在編譯的過程中可能會遇到一些問題，主要是配置和編譯的問題，可以通過[2]中的方式解決，同時按照[3]來進行相關配置。

安裝Storm

參照[1][3]可以完成安裝，主要是下載合適版本和相應的檔案的配置。

測試安裝

以上完成整個單機版本Storm的環境的配置，下面是Storm自帶的examples中的storm-starter例子（也就是WordCountTopology）的執行。
啟動Storm
cd /path/to/zookeeper/bin/
./zkServer.sh start #啟動zookeeper
cd /path/to/storm/bin
./storm nimbus &
./storm supervisor &
./storm ui &
啟動完成後利用jps檢視後臺程式：

執行demo

確保可以成功啟動後來進入storm-starter進行WordCountTopology例子的執行：
實際下面的過程就是strom-starter中README.markdown中的說明和步驟：
前面熟悉平臺和相應例子程式碼的過程先略過，來看利用Maven執行storm-starter：
安裝Maven3.x
對Storm首先進行一個本地構建

# Must be run from the top-level directory of the Storm code repository
$ mvn clean install -DskipTests=true

上面的命令在本地構建Storm並將jar包安裝到$HOME/.m2/repository中，後面執行maven命令的時候，Maven能夠找到相應的Storm版本等資訊。
將storm-starter打包，便於提交和執行

$ mvn package

生成的包存於storm-starter的target目錄下，命名為storm-starter-版本.jar
後面進行提交的時候可以使用這個包進行提交。

# Example 1: Run the ExclamationTopology in local mode (LocalCluster)
$ storm jar target/storm-starter-*.jar org.apache.storm.starter.ExclamationTopology

# Example 2: Run the RollingTopWords in remote/cluster mode,
#            under the name "production-topology"
$ storm jar storm-starter-*.jar org.apache.storm.starter.RollingTopWords production-topology remote

執行wordCountTopology例子：

cd /path/to/storm/bin
./storm jar path/to/generated.jar org.apache.storm.starter.WordCountTopology

執行結果，輸出比較多，整理其中一個例子檢視：

可以看到這是“I am at two with nature”的一個統計詞頻的結果。

原始碼和相應的註釋如下：

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.starter;

import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.StormSubmitter;
import org.apache.storm.task.ShellBolt;
import org.apache.storm.topology.BasicOutputCollector;
import org.apache.storm.topology.IRichBolt;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.topology.base.BaseBasicBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import org.apache.storm.starter.spout.RandomSentenceSpout;

import java.util.HashMap;
import java.util.Map;

/**
 * This topology demonstrates Storm's stream groupings and multilang capabilities.
 */
public class WordCountTopology {
    //分詞 bolt
    //基於shellBolt實現的SplitSentence類 方便使用多語言 如 python 等
  public static class SplitSentence extends ShellBolt implements IRichBolt {

    //用pyton寫的一個Bolt 建構函式 呼叫其它語言寫的bolt
    public SplitSentence() {
      super("python", "splitsentence.py");
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word"));
    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
      return null;
    }
  }
    //詞頻統計 bolt
    //用java實現,所以基類用BaseBasicBolt就夠了
  public static class WordCount extends BaseBasicBolt {

    Map<String, Integer> counts = new HashMap<String, Integer>();

    @Override
    public void execute(Tuple tuple, BasicOutputCollector collector) {
      String word = tuple.getString(0);
      Integer count = counts.get(word);
      if (count == null)
        count = 0;
      count++;
      counts.put(word, count);
      //發射單詞和計數值
      collector.emit(new Values(word, count));
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word", "count"));
    }
  }

  //啟動main函式
  //
  public static void main(String[] args) throws Exception {
    //建立一個topology
    TopologyBuilder builder = new TopologyBuilder();
    //建立spout  RandomSentenceSpout是一個隨機句子產生器
    builder.setSpout("spout", new RandomSentenceSpout(), 5);
    //建立split  bolt
    builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
    //建立count  bolt
    builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));


    //提交執行
    Config conf = new Config();
    conf.setDebug(true);

    if (args != null && args.length > 0) {
      conf.setNumWorkers(3);
      StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
    }
    else {
      //最大並行度
      conf.setMaxTaskParallelism(3);
      //提交
      LocalCluster cluster = new LocalCluster();
      cluster.submitTopology("word-count", conf, builder.createTopology());
      Thread.sleep(10000);
      cluster.shutdown();
    }
  }
}

其中呼叫的python檔案如下：

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import storm

class SplitSentenceBolt(storm.BasicBolt):
    def process(self, tup):
        words = tup.values[0].split(" ")
        for word in words:
          storm.emit([word])

SplitSentenceBolt().run()

參考

[1] http://www.cnblogs.com/stone_w/p/4487897.html
[2] https://my.oschina.net/mingdongcheng/blog/43009
[3] http://blog.csdn.net/zz430581/article/details/37505707
[4] http://blog.csdn.net/nb_vol_1/article/details/46287625
[5] http://blog.csdn.net/nb_vol_1/article/details/46287625
[6] http://www.cnblogs.com/literoad/archive/2013/03/15/2961035.html