高寫入負載情況cassandra當機故障解決

oxoxooxx發表於2011-09-27
cassandra問題:
下面程式併發300執行緒向cassandra中灌資料:
python cassandra_stress.py -p 300 -n 1000000

資料估算:
三個節點,每個節點上期一個程式,每個程式300個併發執行緒向cassandra中寫資料,每個key的大小為1.916KB,Write Latency: 0.014 ms.
單個節點的資料寫入速率:
--&gt300*(1*1000/0.014)*1.916KB=39.16GB/s

結果資料蒐集,驗證:
當前的commitlog檔案大小:
/cassandra/data # du -sh *
34G commitlog
53G data
20K saved_caches

統計commitlog檔案數:
>cassandra/data/commitlog # ls -al |grep .log |grep -v .header|wc -l
268

引入的問題:
1.當大量的資料寫入請求寫記憶體,memtable不能及時的重新整理到sstable,導致commitlog不能清楚,不斷累積,一度達到268個,導致當前節點當機。
?明確一下具體的當機條件?

2.當commitlog目錄積累大量commitlog時啟動cassandra,報告如下虛擬記憶體錯誤。
--解決方法:
ulimit -v unlimited

# /opt/galax/gcs/watchdog/script/shell/Cassandra-oper.sh start
user is root
Begin to start cassandra service.
Begin to check system environment
Cassandra home is: /opt/cassandra
Java home is: /usr/java/jdk1.6.0_24
Check system environment end.
cassandra is starting...
Exception encountered during startup.
java.io.IOError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:172)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:149)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:326)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:191)
at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:226)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:472)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:453)
at org.apache.cassandra.db.Table.initCf(Table.java:317)
at org.apache.cassandra.db.Table.(Table.java:254)
at org.apache.cassandra.db.Table.open(Table.java:110)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:160)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:164)
... 12 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 13 more
cassandra is starting...
Result:failed[@more@]

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/23937368/viewspace-1055368/,如需轉載,請註明出處,否則將追究法律責任。

相關文章