storm 問題日誌記錄
5個worker
1. Nimbus機器： work的記憶體設定太小，導致excutor無法啟動
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[12 12] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[13 13] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[14 14] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[15 15] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[16 16] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[17 17] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[18 18] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[19 19] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[20 20] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[21 21] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[22 22] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[23 23] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[24 24] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[25 25] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[1 1] not alive
2016-05-18T11:20:51.346+0800 b.s.s.EvenScheduler [INFO] Available slots: (["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6703] ["4af
b6058-f368-4bdf-8a49-40c8cc48f950" 6702] ["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6701] ["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6700] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6703] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6702] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6701] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6700] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6703] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6702] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6701] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6700] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6702] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6701] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6700] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6703] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6702] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6701] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6700] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6703] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6702] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6701] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6700] ["ac81b593-ffac-4d49-9624-8949d530c092" 6703] ["ac81b593-ffac-4d49-9624-8949d530c092" 6702] ["ac81b593-ffac-4d49-9624-8949d530c092" 6701] ["ac81b593-ffac-4d49-9624-8949d530c092" 6700] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6703] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6702] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6701] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6700] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6702] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6701] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6700] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6703] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6702] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6701] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6700] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6703] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6702] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6701] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6700] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6703] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6702] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6701] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6700] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6703] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6702] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6701] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6700] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6703] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6702] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6701] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6700] ["d136b9eb-c210-482f-a511-11c7102134ca" 6703] ["d136b9eb-c210-482f-a511-11c7102134ca" 6702] ["d136b9eb-c210-482f-a511-11c7102134ca" 6701] ["d136b9eb-c210-482f-a511-11c7102134ca" 6700] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6703] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6702] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6701] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6700] ["ee065631-6304-4582-9dce-a91fa930c227" 6702] ["ee065631-6304-4582-9dce-a91fa930c227" 6701] ["ee065631-6304-4582-9dce-a91fa930c227" 6700] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6703] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6702] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6701] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6700] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6703] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6702] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6701] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6700] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6703] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6702] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6701] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6700] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6702] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6701] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6700] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6702] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6701] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6700])2016-05-18T11:20:51.346+0800 b.s.d.nimbus [INFO] Reassigning wcd-3-1463540797
to 5 slots
2016-05-18T11:24:53.869+0800 b.s.d.nimbus [INFO] Reassigning wcd-3-1463540797 to 5 slots
2016-05-18T11:24:53.869+0800 b.s.d.nimbus [INFO] Reassign executors: [[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10]
[11 11] [12 12] [13 13] [14 14] [15 15] [16 16] [17 17] [18 18] [19 19] [20 20] [21 21] [22 22] [23 23] [24 24] [25 25] [1 1]]
2. m2mSpouce 啟動執行任務機器節點日誌(10.0.15.103)： worker啟動失敗，任務無法分派
2016-05-18T11:12:40.798+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:41.298+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:41.799+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:42.299+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:42.800+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:43.300+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:43.800+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:44.301+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:44.801+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:45.301+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:45.802+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:46.302+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:46.803+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:47.303+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:47.803+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:48.304+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:48.638+0800 b.s.d.supervisor [INFO] Removing code for storm id wcd-3-1463540797
2016-05-18T11:12:48.804+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:49.304+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:49.805+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:50.305+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:50.806+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:51.306+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:51.806+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:52.307+0800 b.s.d.supervisor [INFO] Worker 24d26d73-f53e-40d8-a509-c877784f57a4 failed to start
2016-05-18T11:12:52.308+0800 b.s.d.supervisor [INFO] Shutting down and clearing state for id 24d26d73-f53e-40d8-a509-c877784f
57a4. Current supervisor time: 1463541172. State: :not-started, Heartbeat: nil2016-05-18T11:12:52.308+0800 b.s.d.supervisor [INFO] Shutting down b1b8f40f-cd05-4c95-adb0-bd77946f2a18:24d26d73-f53e-40d8-a5
09-c877784f57a42016-05-18T11:12:52.311+0800 b.s.util [INFO] Error when trying to kill 3302. Process is probably already dead.
2016-05-18T11:12:53.313+0800 b.s.util [INFO] Error when trying to kill 3302. Process is probably already dead.
2016-05-18T11:12:53.314+0800 b.s.d.supervisor [INFO] Shut down b1b8f40f-cd05-4c95-adb0-bd77946f2a18:24d26d73-f53e-40d8-a509-c
877784f57a42016-05-18T11:14:48.645+0800 b.s.d.supervisor [INFO] Downloading code for storm id wcd-3-1463540797 from /tmp/storm-local/nim
bus/stormdist/wcd-3-14635407972016-05-18T11:14:53.594+0800 b.s.d.supervisor [INFO] Finished downloading code for storm id wcd-3-1463540797 from /tmp/storm-
local/nimbus/stormdist/wcd-3-14635407972016-05-18T11:14:53.598+0800 b.s.d.supervisor [INFO] Launching worker with assignment #backtype.storm.daemon.supervisor.Local
Assignment{:storm-id "wcd-3-1463540797", :executors ([6 6] [11 11] [16 16] [21 21] [1 1])} for this supervisor b1b8f40f-cd05-4c95-adb0-bd77946f2a18 on port 6703 with id 6f349316-52f2-4f8b-84e4-7529675eb65c2016-05-18T11:14:53.599+0800 b.s.d.supervisor [INFO] Launching worker with command: '/opt/jdk1.7.0_45/bin/java' '-server' '-X
mx1024m' '-Djava.library.path=/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker-6703.log' '-Dstorm.home=/opt/storm-0.9.5' '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/opt/storm-0.9.5/logs' '-Dlogback.configurationFile=/opt/storm-0.9.5/logback/cluster.xml' '-Dstorm.id=wcd-3-1463540797' '-Dworker.id=6f349316-52f2-4f8b-84e4-7529675eb65c' '-Dworker.port=6703' '-cp' '/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/slf4j-api-1.7.5.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar' 'backtype.storm.daemon.worker' 'wcd-3-1463540797' 'b1b8f40f-cd05-4c95-adb0-bd77946f2a18' '6703' '6f349316-52f2-4f8b-84e4-7529675eb65c'2016-05-18T11:14:53.600+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:54.100+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:54.601+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:55.101+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
3. m2mSplitBolt 執行任務機器節點日誌(10.0.10.118)：
2016-05-18T11:25:08.361+0800 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
09:09 GMT2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=cloud118
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/opt/jdk1.7.0_45/jre
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm-0.9.5/lib/slf4j-api-1.7.5
.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/tmp/storm-local/supervisor/stormd
ist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-279.el6.x86_64
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=root
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/root
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02
/20/2014 09:09 GMT2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=cloud118
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_45
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/opt/jdk1.7.0_45/jre
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/opt/storm-0.9.5/lib/slf4j-a
pi-1.7.5.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/tmp/storm-local/superviso
r/stormdist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-279.el6.x86_64
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/root
2016-05-18T11:25:08.744+0800 b.s.d.worker [INFO] Launching worker for wcd-3-1463540797 on 54874c13-dd62-4442-9dbc-007d42dac0a
3:6703 with id dc529c0a-e522-4281-844a-a570e68d3e81 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "cloud41", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["a" "b" "c" "d" "e"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx1024m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx1024m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx2048m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 30000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx2048m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}2016-05-18T11:25:08.753+0800 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [
30000] the maxRetries [5]2016-05-18T11:25:08.786+0800 o.a.s.c.CuratorZookeeperClient [WARN] session timeout [20000] is less than connection timeout [3
0000]2016-05-18T11:25:08.805+0800 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-05-18T11:25:08.807+0800 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=a:2181,b:2181,c:2181,d:2181
,e:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@634ec3902016-05-18T11:25:08.832+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server b/10.0.15.167:2181. Will not attem
pt to authenticate using SASL (unknown error)2016-05-18T11:25:08.837+0800 o.a.s.z.ClientCnxn [INFO] Socket connection established to b/10.0.15.167:2181, initiating sessio
n2016-05-18T11:25:08.850+0800 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server b/10.0.15.167:2181, sessionid
= 0x252a661fc6e1faf, negotiated timeout = 20000
2016-05-18T11:25:08.853+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:25:08.855+0800 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2016-05-18T11:25:09.873+0800 o.a.s.z.ZooKeeper [INFO] Session: 0x252a661fc6e1faf closed
2016-05-18T11:25:09.873+0800 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2016-05-18T11:25:09.875+0800 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [
30000] the maxRetries [5]2016-05-18T11:25:09.875+0800 o.a.s.c.CuratorZookeeperClient [WARN] session timeout [20000] is less than connection timeout [3
0000]2016-05-18T11:25:09.876+0800 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-05-18T11:25:09.877+0800 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=a:2181,b:2181,c:2181,d:2181
,e:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@2f15ec752016-05-18T11:25:09.880+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server a/10.0.15.166:2181. Will not attem
pt to authenticate using SASL (unknown error)2016-05-18T11:25:09.881+0800 o.a.s.z.ClientCnxn [INFO] Socket connection established to a/10.0.15.166:2181, initiating sessio
n2016-05-18T11:25:09.883+0800 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server a/10.0.15.166:2181, sessionid
= 0x154996679e789f1, negotiated timeout = 200002016-05-18T11:25:09.884+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:25:09.923+0800 b.s.d.worker [INFO] Reading Assignments.
2016-05-18T11:26:01.957+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14291ms for se
ssionid 0x154996679e789f1, closing socket connection and attempting reconnect2016-05-18T11:26:31.391+0800 b.s.d.worker [ERROR] Error on initialization of server mk-worker
java.lang.OutOfMemoryError: GC overhead limit exceeded
at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:214) ~[storm-core-0.9.5.jar
:0.9.5] at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:211) ~[storm-core-0.9.5.jar
:0.9.5] at com.lmax.disruptor.RingBuffer.fill(RingBuffer.java:84) ~[disruptor-2.10.1.jar:na]
at com.lmax.disruptor.RingBuffer.<init>(RingBuffer.java:51) ~[disruptor-2.10.1.jar:na]
at backtype.storm.utils.DisruptorQueue.<init>(DisruptorQueue.java:66) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.disruptor$disruptor_queue.doInvoke(disruptor.clj:51) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map$fn__6761.invoke(worker.clj:143) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.5.1.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6034.invoke(protocols.clj:146) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6005$G__6000__6014.invoke(protocols.clj:19) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map.invoke(worker.clj:141) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:188) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$fn__6959$exec_fn__1103__auto____6960.invoke(worker.clj:400) ~[storm-core-0.9.5.jar:0.
9.5] at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
2016-05-18T11:26:31.406+0800 b.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.5.jar:0.9.5]
2016-05-18T11:29:13.155+0800 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
09:09 GMT2016-05-18T11:29:13.157+0800 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=cloud118
2016-05-18T11:29:13.157+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2016-05-18T11:29:13.158+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2016-05-18T11:29:13.158+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/opt/jdk1.7.0_45/jre
4. m2mSplitBolt 執行任務機器節點日誌(10.0.10.108)：
2016-05-18T11:33:14.542+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:33:14.584+0800 b.s.d.worker [INFO] Reading Assignments.
2016-05-18T11:34:22.767+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14339ms for se
ssionid 0x4543cfa6c96f29c, closing socket connection and attempting reconnect2016-05-18T11:34:35.987+0800 b.s.d.worker [ERROR] Error on initialization of server mk-worker
java.lang.OutOfMemoryError: GC overhead limit exceeded
at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:214) ~[storm-core-0.9.5.jar
:0.9.5] at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:211) ~[storm-core-0.9.5.jar
:0.9.5] at com.lmax.disruptor.RingBuffer.fill(RingBuffer.java:84) ~[disruptor-2.10.1.jar:na]
at com.lmax.disruptor.RingBuffer.<init>(RingBuffer.java:51) ~[disruptor-2.10.1.jar:na]
at backtype.storm.utils.DisruptorQueue.<init>(DisruptorQueue.java:66) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.disruptor$disruptor_queue.doInvoke(disruptor.clj:51) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map$fn__6761.invoke(worker.clj:143) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.5.1.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6034.invoke(protocols.clj:146) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6005$G__6000__6014.invoke(protocols.clj:19) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map.invoke(worker.clj:141) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:188) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$fn__6959$exec_fn__1103__auto____6960.invoke(worker.clj:400) ~[storm-core-0.9.5.jar:0.
9.5] at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
2016-05-18T11:34:35.998+0800 b.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.5.jar:0.9.5]
最終解決方法：
調整如下引數
conf.put("topology.receiver.buffer.size", Integer.valueOf(Integer.parseInt(args[3])));
conf.put("topology.executor.receive.buffer.size", Integer.valueOf(Integer.parseInt(args[4])));
conf.put("topology.executor.send.buffer.size", Integer.valueOf(Integer.parseInt(args[5])));
conf.put("topology.transfer.buffer.size", Integer.valueOf(Integer.parseInt(args[6])));
conf.put("topology.message.timeout.secs", 180); //訊息處理超時時間
conf.put("topology.max.spout.pending", 100000); //spout最多快取的tuple數
1. Nimbus機器： work的記憶體設定太小，導致excutor無法啟動
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[12 12] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[13 13] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[14 14] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[15 15] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[16 16] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[17 17] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[18 18] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[19 19] not alive
2016-05-18T11:20:51.300+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[20 20] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[21 21] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[22 22] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[23 23] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[24 24] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[25 25] not alive
2016-05-18T11:20:51.301+0800 b.s.d.nimbus [INFO] Executor wcd-3-1463540797:[1 1] not alive
2016-05-18T11:20:51.346+0800 b.s.s.EvenScheduler [INFO] Available slots: (["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6703] ["4af
b6058-f368-4bdf-8a49-40c8cc48f950" 6702] ["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6701] ["4afb6058-f368-4bdf-8a49-40c8cc48f950" 6700] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6703] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6702] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6701] ["8babaa7a-ba25-45f8-a01e-ff40e259bc13" 6700] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6703] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6702] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6701] ["e8f6518a-1180-44a0-bc77-fa6f47224228" 6700] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6702] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6701] ["cc6786c9-be80-4480-8d8c-6bd8d96003c9" 6700] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6703] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6702] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6701] ["a33b3fdc-a0d6-45d1-b51c-b7e1e1d4d08e" 6700] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6703] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6702] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6701] ["54874c13-dd62-4442-9dbc-007d42dac0a3" 6700] ["ac81b593-ffac-4d49-9624-8949d530c092" 6703] ["ac81b593-ffac-4d49-9624-8949d530c092" 6702] ["ac81b593-ffac-4d49-9624-8949d530c092" 6701] ["ac81b593-ffac-4d49-9624-8949d530c092" 6700] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6703] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6702] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6701] ["f114fa32-0a49-4af5-a324-fa8f23534a54" 6700] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6702] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6701] ["c84f5a19-953f-436e-8c88-325d3cac1c47" 6700] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6703] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6702] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6701] ["95a1e155-c71b-42a6-aa86-d01cc79b15d7" 6700] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6703] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6702] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6701] ["8a4201b7-48a9-4e34-9dd4-5f019ff4236a" 6700] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6703] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6702] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6701] ["160d9fa9-dd91-4a12-8523-7fe6630309ed" 6700] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6703] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6702] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6701] ["4102ab69-c96d-4ea2-ae0d-0cd400c89f88" 6700] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6703] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6702] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6701] ["022f881f-483f-49bd-ba83-3ecd405fa1b0" 6700] ["d136b9eb-c210-482f-a511-11c7102134ca" 6703] ["d136b9eb-c210-482f-a511-11c7102134ca" 6702] ["d136b9eb-c210-482f-a511-11c7102134ca" 6701] ["d136b9eb-c210-482f-a511-11c7102134ca" 6700] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6703] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6702] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6701] ["9f88ff78-d64e-4399-93f3-d8f42716d0ea" 6700] ["ee065631-6304-4582-9dce-a91fa930c227" 6702] ["ee065631-6304-4582-9dce-a91fa930c227" 6701] ["ee065631-6304-4582-9dce-a91fa930c227" 6700] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6703] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6702] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6701] ["ed7c8c5e-b4ba-4d39-bd74-57a4138ae8ee" 6700] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6703] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6702] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6701] ["15e3b77e-6de2-4317-a099-86c6ee27955c" 6700] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6703] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6702] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6701] ["93f4b0b5-7beb-4011-a109-ffa5105ff469" 6700] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6702] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6701] ["b1b8f40f-cd05-4c95-adb0-bd77946f2a18" 6700] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6702] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6701] ["c8ca5a72-84ff-466a-9a1e-7d6924325546" 6700])2016-05-18T11:20:51.346+0800 b.s.d.nimbus [INFO] Reassigning wcd-3-1463540797
to 5 slots
2016-05-18T11:24:53.869+0800 b.s.d.nimbus [INFO] Reassigning wcd-3-1463540797 to 5 slots
2016-05-18T11:24:53.869+0800 b.s.d.nimbus [INFO] Reassign executors: [[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10]
[11 11] [12 12] [13 13] [14 14] [15 15] [16 16] [17 17] [18 18] [19 19] [20 20] [21 21] [22 22] [23 23] [24 24] [25 25] [1 1]]
2. m2mSpouce 啟動執行任務機器節點日誌(10.0.15.103)： worker啟動失敗，任務無法分派
2016-05-18T11:12:40.798+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:41.298+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:41.799+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:42.299+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:42.800+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:43.300+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:43.800+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:44.301+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:44.801+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:45.301+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:45.802+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:46.302+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:46.803+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:47.303+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:47.803+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:48.304+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:48.638+0800 b.s.d.supervisor [INFO] Removing code for storm id wcd-3-1463540797
2016-05-18T11:12:48.804+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:49.304+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:49.805+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:50.305+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:50.806+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:51.306+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:51.806+0800 b.s.d.supervisor [INFO] 24d26d73-f53e-40d8-a509-c877784f57a4 still hasn't started
2016-05-18T11:12:52.307+0800 b.s.d.supervisor [INFO] Worker 24d26d73-f53e-40d8-a509-c877784f57a4 failed to start
2016-05-18T11:12:52.308+0800 b.s.d.supervisor [INFO] Shutting down and clearing state for id 24d26d73-f53e-40d8-a509-c877784f
57a4. Current supervisor time: 1463541172. State: :not-started, Heartbeat: nil2016-05-18T11:12:52.308+0800 b.s.d.supervisor [INFO] Shutting down b1b8f40f-cd05-4c95-adb0-bd77946f2a18:24d26d73-f53e-40d8-a5
09-c877784f57a42016-05-18T11:12:52.311+0800 b.s.util [INFO] Error when trying to kill 3302. Process is probably already dead.
2016-05-18T11:12:53.313+0800 b.s.util [INFO] Error when trying to kill 3302. Process is probably already dead.
2016-05-18T11:12:53.314+0800 b.s.d.supervisor [INFO] Shut down b1b8f40f-cd05-4c95-adb0-bd77946f2a18:24d26d73-f53e-40d8-a509-c
877784f57a42016-05-18T11:14:48.645+0800 b.s.d.supervisor [INFO] Downloading code for storm id wcd-3-1463540797 from /tmp/storm-local/nim
bus/stormdist/wcd-3-14635407972016-05-18T11:14:53.594+0800 b.s.d.supervisor [INFO] Finished downloading code for storm id wcd-3-1463540797 from /tmp/storm-
local/nimbus/stormdist/wcd-3-14635407972016-05-18T11:14:53.598+0800 b.s.d.supervisor [INFO] Launching worker with assignment #backtype.storm.daemon.supervisor.Local
Assignment{:storm-id "wcd-3-1463540797", :executors ([6 6] [11 11] [16 16] [21 21] [1 1])} for this supervisor b1b8f40f-cd05-4c95-adb0-bd77946f2a18 on port 6703 with id 6f349316-52f2-4f8b-84e4-7529675eb65c2016-05-18T11:14:53.599+0800 b.s.d.supervisor [INFO] Launching worker with command: '/opt/jdk1.7.0_45/bin/java' '-server' '-X
mx1024m' '-Djava.library.path=/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker-6703.log' '-Dstorm.home=/opt/storm-0.9.5' '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/opt/storm-0.9.5/logs' '-Dlogback.configurationFile=/opt/storm-0.9.5/logback/cluster.xml' '-Dstorm.id=wcd-3-1463540797' '-Dworker.id=6f349316-52f2-4f8b-84e4-7529675eb65c' '-Dworker.port=6703' '-cp' '/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/slf4j-api-1.7.5.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar' 'backtype.storm.daemon.worker' 'wcd-3-1463540797' 'b1b8f40f-cd05-4c95-adb0-bd77946f2a18' '6703' '6f349316-52f2-4f8b-84e4-7529675eb65c'2016-05-18T11:14:53.600+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:54.100+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:54.601+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
2016-05-18T11:14:55.101+0800 b.s.d.supervisor [INFO] 6f349316-52f2-4f8b-84e4-7529675eb65c still hasn't started
3. m2mSplitBolt 執行任務機器節點日誌(10.0.10.118)：
2016-05-18T11:25:08.361+0800 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
09:09 GMT2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=cloud118
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/opt/jdk1.7.0_45/jre
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm-0.9.5/lib/slf4j-api-1.7.5
.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/tmp/storm-local/supervisor/stormd
ist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-279.el6.x86_64
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.name=root
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.home=/root
2016-05-18T11:25:08.364+0800 o.a.s.z.ZooKeeper [INFO] Client environment:user.dir=/root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02
/20/2014 09:09 GMT2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:host.name=cloud118
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_45
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.home=/opt/jdk1.7.0_45/jre
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/opt/storm-0.9.5/lib/slf4j-a
pi-1.7.5.jar:/opt/storm-0.9.5/lib/json-simple-1.1.jar:/opt/storm-0.9.5/lib/minlog-1.2.jar:/opt/storm-0.9.5/lib/ring-servlet-0.3.11.jar:/opt/storm-0.9.5/lib/tools.logging-0.2.3.jar:/opt/storm-0.9.5/lib/clojure-1.5.1.jar:/opt/storm-0.9.5/lib/clj-time-0.4.1.jar:/opt/storm-0.9.5/lib/compojure-1.1.3.jar:/opt/storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/opt/storm-0.9.5/lib/logback-classic-1.0.13.jar:/opt/storm-0.9.5/lib/jetty-util-6.1.26.jar:/opt/storm-0.9.5/lib/asm-4.0.jar:/opt/storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm-0.9.5/lib/ring-core-1.1.5.jar:/opt/storm-0.9.5/lib/kryo-2.21.jar:/opt/storm-0.9.5/lib/jetty-6.1.26.jar:/opt/storm-0.9.5/lib/core.incubator-0.1.0.jar:/opt/storm-0.9.5/lib/ring-devel-0.3.11.jar:/opt/storm-0.9.5/lib/jline-2.11.jar:/opt/storm-0.9.5/lib/objenesis-1.2.jar:/opt/storm-0.9.5/lib/hiccup-0.3.6.jar:/opt/storm-0.9.5/lib/tools.cli-0.2.4.jar:/opt/storm-0.9.5/lib/commons-logging-1.1.3.jar:/opt/storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/opt/storm-0.9.5/lib/commons-io-2.4.jar:/opt/storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/opt/storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/opt/storm-0.9.5/lib/disruptor-2.10.1.jar:/opt/storm-0.9.5/lib/servlet-api-2.5.jar:/opt/storm-0.9.5/lib/snakeyaml-1.11.jar:/opt/storm-0.9.5/lib/joda-time-2.0.jar:/opt/storm-0.9.5/lib/commons-codec-1.6.jar:/opt/storm-0.9.5/lib/chill-java-0.3.5.jar:/opt/storm-0.9.5/lib/logback-core-1.0.13.jar:/opt/storm-0.9.5/lib/commons-exec-1.1.jar:/opt/storm-0.9.5/lib/storm-core-0.9.5.jar:/opt/storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm-0.9.5/lib/clout-1.0.1.jar:/opt/storm-0.9.5/lib/carbonite-1.4.0.jar:/opt/storm-0.9.5/lib/commons-lang-2.5.jar:/opt/storm-0.9.5/lib/tools.macro-0.1.0.jar:/opt/storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/opt/storm-0.9.5/conf:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/stormjar.jar2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/tmp/storm-local/superviso
r/stormdist/wcd-3-1463540797/resources/Linux-amd64:/tmp/storm-local/supervisor/stormdist/wcd-3-1463540797/resources:/usr/local/lib:/opt/local/lib:/usr/lib2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-279.el6.x86_64
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.name=root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.home=/root
2016-05-18T11:25:08.375+0800 o.a.s.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/root
2016-05-18T11:25:08.744+0800 b.s.d.worker [INFO] Launching worker for wcd-3-1463540797 on 54874c13-dd62-4442-9dbc-007d42dac0a
3:6703 with id dc529c0a-e522-4281-844a-a570e68d3e81 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/tmp/storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "cloud41", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["a" "b" "c" "d" "e"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx1024m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx1024m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx2048m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 30000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx2048m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}2016-05-18T11:25:08.753+0800 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [
30000] the maxRetries [5]2016-05-18T11:25:08.786+0800 o.a.s.c.CuratorZookeeperClient [WARN] session timeout [20000] is less than connection timeout [3
0000]2016-05-18T11:25:08.805+0800 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-05-18T11:25:08.807+0800 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=a:2181,b:2181,c:2181,d:2181
,e:2181 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@634ec3902016-05-18T11:25:08.832+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server b/10.0.15.167:2181. Will not attem
pt to authenticate using SASL (unknown error)2016-05-18T11:25:08.837+0800 o.a.s.z.ClientCnxn [INFO] Socket connection established to b/10.0.15.167:2181, initiating sessio
n2016-05-18T11:25:08.850+0800 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server b/10.0.15.167:2181, sessionid
= 0x252a661fc6e1faf, negotiated timeout = 20000
2016-05-18T11:25:08.853+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:25:08.855+0800 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2016-05-18T11:25:09.873+0800 o.a.s.z.ZooKeeper [INFO] Session: 0x252a661fc6e1faf closed
2016-05-18T11:25:09.873+0800 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2016-05-18T11:25:09.875+0800 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [
30000] the maxRetries [5]2016-05-18T11:25:09.875+0800 o.a.s.c.CuratorZookeeperClient [WARN] session timeout [20000] is less than connection timeout [3
0000]2016-05-18T11:25:09.876+0800 o.a.s.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-05-18T11:25:09.877+0800 o.a.s.z.ZooKeeper [INFO] Initiating client connection, connectString=a:2181,b:2181,c:2181,d:2181
,e:2181/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@2f15ec752016-05-18T11:25:09.880+0800 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server a/10.0.15.166:2181. Will not attem
pt to authenticate using SASL (unknown error)2016-05-18T11:25:09.881+0800 o.a.s.z.ClientCnxn [INFO] Socket connection established to a/10.0.15.166:2181, initiating sessio
n2016-05-18T11:25:09.883+0800 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server a/10.0.15.166:2181, sessionid
= 0x154996679e789f1, negotiated timeout = 200002016-05-18T11:25:09.884+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:25:09.923+0800 b.s.d.worker [INFO] Reading Assignments.
2016-05-18T11:26:01.957+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14291ms for se
ssionid 0x154996679e789f1, closing socket connection and attempting reconnect2016-05-18T11:26:31.391+0800 b.s.d.worker [ERROR] Error on initialization of server mk-worker
java.lang.OutOfMemoryError: GC overhead limit exceeded
at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:214) ~[storm-core-0.9.5.jar
:0.9.5] at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:211) ~[storm-core-0.9.5.jar
:0.9.5] at com.lmax.disruptor.RingBuffer.fill(RingBuffer.java:84) ~[disruptor-2.10.1.jar:na]
at com.lmax.disruptor.RingBuffer.<init>(RingBuffer.java:51) ~[disruptor-2.10.1.jar:na]
at backtype.storm.utils.DisruptorQueue.<init>(DisruptorQueue.java:66) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.disruptor$disruptor_queue.doInvoke(disruptor.clj:51) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map$fn__6761.invoke(worker.clj:143) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.5.1.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6034.invoke(protocols.clj:146) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6005$G__6000__6014.invoke(protocols.clj:19) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map.invoke(worker.clj:141) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:188) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$fn__6959$exec_fn__1103__auto____6960.invoke(worker.clj:400) ~[storm-core-0.9.5.jar:0.
9.5] at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
2016-05-18T11:26:31.406+0800 b.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.5.jar:0.9.5]
2016-05-18T11:29:13.155+0800 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
09:09 GMT2016-05-18T11:29:13.157+0800 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=cloud118
2016-05-18T11:29:13.157+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_45
2016-05-18T11:29:13.158+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2016-05-18T11:29:13.158+0800 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/opt/jdk1.7.0_45/jre
4. m2mSplitBolt 執行任務機器節點日誌(10.0.10.108)：
2016-05-18T11:33:14.542+0800 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-05-18T11:33:14.584+0800 b.s.d.worker [INFO] Reading Assignments.
2016-05-18T11:34:22.767+0800 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14339ms for se
ssionid 0x4543cfa6c96f29c, closing socket connection and attempting reconnect2016-05-18T11:34:35.987+0800 b.s.d.worker [ERROR] Error on initialization of server mk-worker
java.lang.OutOfMemoryError: GC overhead limit exceeded
at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:214) ~[storm-core-0.9.5.jar
:0.9.5] at backtype.storm.utils.DisruptorQueue$ObjectEventFactory.newInstance(DisruptorQueue.java:211) ~[storm-core-0.9.5.jar
:0.9.5] at com.lmax.disruptor.RingBuffer.fill(RingBuffer.java:84) ~[disruptor-2.10.1.jar:na]
at com.lmax.disruptor.RingBuffer.<init>(RingBuffer.java:51) ~[disruptor-2.10.1.jar:na]
at backtype.storm.utils.DisruptorQueue.<init>(DisruptorQueue.java:66) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.disruptor$disruptor_queue.doInvoke(disruptor.clj:51) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map$fn__6761.invoke(worker.clj:143) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.core$map$fn__4207.invoke(core.clj:2487) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.5.1.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6034.invoke(protocols.clj:146) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6005$G__6000__6014.invoke(protocols.clj:19) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$mk_receive_queue_map.invoke(worker.clj:141) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:188) ~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.daemon.worker$fn__6959$exec_fn__1103__auto____6960.invoke(worker.clj:400) ~[storm-core-0.9.5.jar:0.
9.5] at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) ~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.5.1.jar:na]
2016-05-18T11:34:35.998+0800 b.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:502) [storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.5.jar:0.9.5]
最終解決方法：
調整如下引數
conf.put("topology.receiver.buffer.size", Integer.valueOf(Integer.parseInt(args[3])));
conf.put("topology.executor.receive.buffer.size", Integer.valueOf(Integer.parseInt(args[4])));
conf.put("topology.executor.send.buffer.size", Integer.valueOf(Integer.parseInt(args[5])));
conf.put("topology.transfer.buffer.size", Integer.valueOf(Integer.parseInt(args[6])));
conf.put("topology.message.timeout.secs", 180); //訊息處理超時時間
conf.put("topology.max.spout.pending", 100000); //spout最多快取的tuple數
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31347383/viewspace-2109862/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- php日誌,記錄日誌PHP
- 日誌記錄器
- [提問交流]請問如何關掉日誌記錄
- log4jnet不記錄日誌的問題解決
- Laravel sql 日誌記錄LaravelSQL
- secureCRT記錄操作日誌Securecrt
- 記錄日誌檔案
- PHP日誌記錄方法PHP
- oracle日誌操作記錄Oracle
- mysql audit-訪問日誌記錄應用MySql
- 日誌記錄 PHP下往linux目錄下寫日誌PHPLinux
- Oracle補充日誌及日誌記錄規則Oracle
- Nginx訪問日誌、Nginx日誌切割、靜態檔案不記錄日誌和過期時間Nginx
- db2不記錄日誌插入記錄DB2
- 記錄騰訊雲使用日誌
- 記錄 | 實習日誌 9
- mysql開啟日誌記錄MySql
- 記錄 sql 查詢日誌SQL
- SecureCRT自動記錄日誌Securecrt
- Appfuse：記錄操作日誌APP
- 第11章 日誌記錄
- Storm模擬分析電話日誌ORM
- 解決記錄日誌導致VS2013緩慢的問題
- 怎樣設定不讓JBOSS記錄訪問日誌？？？
- Oracle寫日誌問題Oracle
- 日誌檔案問題
- monolog 日誌記錄器解析Mono
- iptables防火牆如何記錄日誌防火牆
- 使用Redis記錄系統日誌Redis
- log4net日誌記錄
- 使用Rsyslog記錄Apache日誌Apache
- secureCRT自動記錄log日誌Securecrt
- 使用 .NET Core 的日誌記錄
- Python多程式記錄日誌Python
- 駭客攻擊日誌記錄(轉)
- Rust 實現日誌記錄功能Rust
- Laravel 指定日誌檔案記錄任意日誌Laravel
- 檢視mysql日誌及日誌編碼問題MySql