Cassandra не запускается: SSTableBatchOpen IllegalArgumentException

Cassandra не запускается: SSTableBatchOpen IllegalArgumentExceptionCassandra не запускается: SSTableBatchOpen IllegalArgumentException

Экспериментальная Cassandra установка работает в Ubuntu 16.04 VM был сбой питания и после этого Cassandra не запускается.

Все работало нормально до отказа питания. Я не знаю других причин, по которым установка не может быть запущена.

Cassandra system.log здесь:

cat /var/log/cassandra/system.log 

INFO [main] 2016-11-29 21:55:25,307 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml 
INFO [main] 2016-11-29 21:55:26,099 DatabaseDescriptor.java:317 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 
INFO [main] 2016-11-29 21:55:26,407 DatabaseDescriptor.java:437 - Global memtable on-heap threshold is enabled at 609MB 
INFO [main] 2016-11-29 21:55:26,408 DatabaseDescriptor.java:441 - Global memtable off-heap threshold is enabled at 609MB 
WARN [main] 2016-11-29 21:55:26,538 DatabaseDescriptor.java:643 - Only 30.073GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots 
INFO [main] 2016-11-29 21:55:26,646 CassandraDaemon.java:465 - Hostname: tiira 
INFO [main] 2016-11-29 21:55:26,647 CassandraDaemon.java:472 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_111 
INFO [main] 2016-11-29 21:55:26,647 CassandraDaemon.java:473 - Heap size: 2.379GiB/2.379GiB 
INFO [main] 2016-11-29 21:55:26,648 CassandraDaemon.java:478 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4315904(4214K) committed = 4390912(4288K) max = 251658240(245760K) 
INFO [main] 2016-11-29 21:55:26,649 CassandraDaemon.java:478 - Metaspace Non-heap memory: init = 0(0K) used = 16194936(15815K) committed = 16777216(16384K) max = -1(-1K) 
INFO [main] 2016-11-29 21:55:26,649 CassandraDaemon.java:478 - Compressed Class Space Non-heap memory: init = 0(0K) used = 1950560(1904K) committed = 2097152(2048K) max = 1073741824(1048576K) 
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - Par Eden Space Heap memory: init = 503316480(491520K) used = 110748120(108152K) committed = 503316480(491520K) max = 503316480(491520K) 
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - Par Survivor Space Heap memory: init = 62914560(61440K) used = 0(0K) committed = 62914560(61440K) max = 62914560(61440K) 
INFO [main] 2016-11-29 21:55:26,650 CassandraDaemon.java:478 - CMS Old Gen Heap memory: init = 1988100096(1941504K) used = 0(0K) committed = 1988100096(1941504K) max = 1988100096(1941504K) 
INFO [main] 2016-11-29 21:55:26,888 CLibrary.java:150 - JNA mlockall successful 
WARN [main] 2016-11-29 21:55:26,891 StartupChecks.java:123 - jemalloc shared library could not be preloaded to speed up memory allocations 
WARN [main] 2016-11-29 21:55:26,891 StartupChecks.java:156 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. 
WARN [main] 2016-11-29 21:55:26,892 StartupChecks.java:193 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release 
INFO [main] 2016-11-29 21:55:26,898 SigarLibrary.java:44 - Initializing SIGAR library 
WARN [main] 2016-11-29 21:55:26,942 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true 
INFO [main] 2016-11-29 21:55:27,366 QueryProcessor.java:113 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift) 
INFO [main] 2016-11-29 21:55:28,411 ColumnFamilyStore.java:412 - Initializing system.IndexInfo 
INFO [SSTableBatchOpen:1] 2016-11-29 21:55:29,537 BufferPool.java:229 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap 
INFO [main] 2016-11-29 21:55:30,040 CacheService.java:113 - Initializing key cache with capacity of 100 MBs. 
INFO [main] 2016-11-29 21:55:30,058 CacheService.java:135 - Initializing row cache with capacity of 0 MBs 
INFO [main] 2016-11-29 21:55:30,064 CacheService.java:164 - Initializing counter cache with capacity of 50 MBs 
INFO [main] 2016-11-29 21:55:30,067 CacheService.java:175 - Scheduling counter cache save to every 7200 seconds (going to save all keys). 
INFO [main] 2016-11-29 21:55:30,152 ColumnFamilyStore.java:412 - Initializing system.batches 
...deleted... 
INFO [main] 2016-11-29 21:55:32,490 ColumnFamilyStore.java:412 - Initializing system_auth.roles 
INFO [main] 2016-11-29 21:55:32,512 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized 
INFO [main] 2016-11-29 21:55:32,531 ColumnFamilyStore.java:412 - Initializing system_traces.events 
INFO [main] 2016-11-29 21:55:32,575 ColumnFamilyStore.java:412 - Initializing system_traces.sessions 
INFO [main] 2016-11-29 21:55:32,626 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized 
INFO [main] 2016-11-29 21:55:32,638 ColumnFamilyStore.java:412 - Initializing my_keyspace.user 
INFO [main] 2016-11-29 21:55:32,667 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_address_idx 
ERROR [SSTableBatchOpen:1] 2016-11-29 21:55:32,679 CassandraDaemon.java:226 - Exception in thread Thread[SSTableBatchOpen:1,5,main] 
java.lang.IllegalArgumentException: null 
    at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111] 
    at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619) 
    at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122) 
    at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:159) 
    at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139) 
    at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120) 
    at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
    at org.apache.cassandra.io.sstable.format.SSTableReader.validate(SSTableReader.java:1190) 
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:499) 
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:375) 
    at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:536) 
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111] 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111] 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111] 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] 
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] 
INFO [main] 2016-11-29 21:55:32,703 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_emails_idx 
INFO [SecondaryIndexManagement:1] 2016-11-29 21:55:32,707 CassandraIndex.java:707 - Submitting index build of user_address_idx for data in BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db') 
INFO [main] 2016-11-29 21:55:32,827 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_last_name_idx 
INFO [main] 2016-11-29 21:55:32,945 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_phone_idx 
INFO [main] 2016-11-29 21:55:33,028 ViewManager.java:137 - Not submitting build tasks for views in keyspace my_keyspace as storage service is not initialized 
INFO [pool-4-thread-1] 2016-11-29 21:55:33,037 AutoSavingCache.java:196 - reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db 
INFO [pool-4-thread-1] 2016-11-29 21:55:33,065 AutoSavingCache.java:172 - Completed loading (32 ms; 76 keys) KeyCache cache 
ERROR [CompactionExecutor:1] 2016-11-29 21:55:33,072 CassandraDaemon.java:226 - Exception in thread Thread[CompactionExecutor:1,1,main] 
java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx 
    at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316) 
    at org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:531) 
    at org.apache.cassandra.index.internal.CassandraIndex.access$100(CassandraIndex.java:72) 
    at org.apache.cassandra.index.internal.CassandraIndex$1.indexCell(CassandraIndex.java:444) 
    at org.apache.cassandra.index.internal.CassandraIndex$1.indexCells(CassandraIndex.java:436) 
Caused by: java.lang.IllegalArgumentException: null 
    at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111] 
    at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619) 
    at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122) 
    ... 16 common frames omitted 
ERROR [SecondaryIndexManagement:1] 2016-11-29 21:55:33,072 CassandraDaemon.java:226 - Exception in thread Thread[SecondaryIndexManagement:1,5,main] 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx 
    at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:403) 
    at org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:715) 
    at org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$5(CassandraIndex.java:685) 
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx 
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_111] 
    at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_111] 
    at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:399) 
    ... 6 common frames omitted 
Caused by: java.lang.RuntimeException: null for ks: my_keyspace, table: user.user_address_idx 
    at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1316) 
    at org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:531) 
    at org.apache.cassandra.index.internal.CassandraIndex.access$100(CassandraIndex.java:72) 
    ... 4 common frames omitted 
Caused by: java.lang.IllegalArgumentException: null 
    at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_111] 
    at org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:619) 
    at org.apache.cassandra.db.marshal.TupleType.compareCustom(TupleType.java:122) 
    ... 16 common frames omitted 
INFO [main] 2016-11-29 21:55:33,082 CommitLog.java:166 - Replaying /var/lib/cassandra/commitlog/CommitLog-6-1480266920262.log, /var/lib/cassandra/commitlog/CommitLog-6-1480266920263.log, /var/lib/cassandra/commitlog/CommitLog-6-1480428373824.log, /var/lib/cassandra/commitlog/CommitLog-6-1480430610402.log, /var/lib/cassandra/commitlog/CommitLog-6-1480446994136.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447217591.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447657364.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447697557.log, /var/lib/cassandra/commitlog/CommitLog-6-1480448913585.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449134204.log 
ERROR [main] 2016-11-29 21:55:33,315 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization. 
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation; saved to /tmp/mutation3325983627363773229dat. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street

Я пытался построить вторичный индекс:

nodetool rebuild_index my_keyspace user user_address_idx 
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'

Затем запустите sstablescrub, как Cassandra пользователь

[email protected]:/etc$ sstablescrub -v --debug my_keyspace user 
WARN 12:14:00 Only 29.952GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots 
Pre-scrub sstables snapshotted into snapshot pre-scrub-1480594443732 
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db') (0.372KiB) 
Reading row at 0 
row 48656c6c6f6132 is 0.022KiB 
Reading row at 32 
row 48656c6c6f33 is 0.053KiB 
Reading row at 94 
row 48656c6c6f is 0.138KiB 
Reading row at 242 
row 48656c6c6f32 is 0.128KiB 
Scrub of BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-5-big-Data.db') complete; looks like all 0 rows were tombstoned 
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db') (0.069KiB) 
Reading row at 0 
row 48656c6c6f is 0.063KiB 
WARNING: Error reading row (stacktrace follows): 
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db 
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$1Validator.validate(UnfilteredRowIterators.java:306) 
    at org.apache.cassandra.db.rows.UnfilteredRowIterators$1Validator.applyToRow(UnfilteredRowIterators.java:287) 
Caused by: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street 
    at org.apache.cassandra.db.marshal.UserType.validate(UserType.java:194) 
    at org.apache.cassandra.db.marshal.AbstractType.validateCellValue(AbstractType.java:186) 
    ... 9 more 
WARNING: Row starting at position 7 is unreadable; skipping to next 
WARNING: No valid rows found while scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-6-big-Data.db'); it is marked for deletion now. If you want to attempt manual recovery, you can find a copy in the pre-scrub snapshot 
[email protected]:/etc$ 
[email protected]:/etc$ 
[email protected]:/etc$ sstablescrub -v --debug my_keyspace user 
WARN 12:17:13 Only 29.953GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots 
Pre-scrub sstables snapshotted into snapshot pre-scrub-1480594637427 
Scrubbing BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-7-big-Data.db') (0.372KiB) 
Reading row at 0 
row 48656c6c6f6132 is 0.022KiB 
Reading row at 32 
row 48656c6c6f33 is 0.053KiB 
Reading row at 94 
row 48656c6c6f is 0.138KiB 
Reading row at 242 
row 48656c6c6f32 is 0.128KiB 
Scrub of BigTableReader(path='/var/lib/cassandra/data/my_keyspace/user-0d220340b4d711e6850931fba1c453e0/mc-7-big-Data.db') complete; looks like all 0 rows were tombstoned 
[email protected]:/etc$

Попытка начать Кассандру:

INFO [main] 2016-12-01 14:56:18,945 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml 
INFO [main] 2016-12-01 14:56:19,329 DatabaseDescriptor.java:317 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 
INFO [main] 2016-12-01 14:56:19,492 DatabaseDescriptor.java:437 - Global memtable on-heap threshold is enabled at 609MB 
INFO [main] 2016-12-01 14:56:19,493 DatabaseDescriptor.java:441 - Global memtable off-heap threshold is enabled at 609MB 
WARN [main] 2016-12-01 14:56:19,516 DatabaseDescriptor.java:643 - Only 29.949GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots 
INFO [main] 2016-12-01 14:56:19,590 CassandraDaemon.java:465 - Hostname: tiira 
INFO [main] 2016-12-01 14:56:19,591 CassandraDaemon.java:472 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_111 
INFO [main] 2016-12-01 14:56:19,591 CassandraDaemon.java:473 - Heap size: 2.379GiB/2.379GiB 
INFO [main] 2016-12-01 14:56:19,592 CassandraDaemon.java:478 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4371136(4268K) committed = 4390912(4288K) max = 251658240(245760K) 
INFO [main] 2016-12-01 14:56:19,592 CassandraDaemon.java:478 - Metaspace Non-heap memory: init = 0(0K) used = 15802936(15432K) committed = 16252928(15872K) max = -1(-1K) 
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Compressed Class Space Non-heap memory: init = 0(0K) used = 1910488(1865K) committed = 2097152(2048K) max = 1073741824(1048576K) 
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Par Eden Space Heap memory: init = 503316480(491520K) used = 100681584(98321K) committed = 503316480(491520K) max = 503316480(491520K) 
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - Par Survivor Space Heap memory: init = 62914560(61440K) used = 0(0K) committed = 62914560(61440K) max = 62914560(61440K) 
INFO [main] 2016-12-01 14:56:19,593 CassandraDaemon.java:478 - CMS Old Gen Heap memory: init = 1988100096(1941504K) used = 0(0K) committed = 1988100096(1941504K) max = 1988100096(1941504K) 
WARN [main] 2016-12-01 14:56:19,699 StartupChecks.java:123 - jemalloc shared library could not be preloaded to speed up memory allocations 
WARN [main] 2016-12-01 14:56:19,699 StartupChecks.java:156 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. 
WARN [main] 2016-12-01 14:56:19,700 StartupChecks.java:193 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release 
INFO [main] 2016-12-01 14:56:19,701 SigarLibrary.java:44 - Initializing SIGAR library 
WARN [main] 2016-12-01 14:56:19,729 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true 
INFO [main] 2016-12-01 14:56:19,893 QueryProcessor.java:113 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift) 
INFO [main] 2016-12-01 14:56:20,523 ColumnFamilyStore.java:412 - Initializing system.IndexInfo 
INFO [SSTableBatchOpen:1] 2016-12-01 14:56:21,449 BufferPool.java:229 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap 
INFO [main] 2016-12-01 14:56:21,790 CacheService.java:113 - Initializing key cache with capacity of 100 MBs. 
INFO [main] 2016-12-01 14:56:21,811 CacheService.java:135 - Initializing row cache with capacity of 0 MBs 
INFO [main] 2016-12-01 14:56:21,918 CacheService.java:164 - Initializing counter cache with capacity of 50 MBs 
INFO [main] 2016-12-01 14:56:21,922 CacheService.java:175 - Scheduling counter cache save to every 7200 seconds (going to save all keys). 
INFO [main] 2016-12-01 14:56:21,983 ColumnFamilyStore.java:412 - Initializing system.batches 
INFO [main] 2016-12-01 14:56:22,004 ColumnFamilyStore.java:412 - Initializing system.paxos 
INFO [main] 2016-12-01 14:56:22,035 ColumnFamilyStore.java:412 - Initializing system.local 
INFO [main] 2016-12-01 14:56:22,057 ColumnFamilyStore.java:412 - Initializing system.peers 
INFO [main] 2016-12-01 14:56:22,070 ColumnFamilyStore.java:412 - Initializing system.peer_events 
INFO [main] 2016-12-01 14:56:22,079 ColumnFamilyStore.java:412 - Initializing system.range_xfers 
INFO [main] 2016-12-01 14:56:22,089 ColumnFamilyStore.java:412 - Initializing system.compaction_history 
INFO [main] 2016-12-01 14:56:22,118 ColumnFamilyStore.java:412 - Initializing system.sstable_activity 
INFO [main] 2016-12-01 14:56:22,145 ColumnFamilyStore.java:412 - Initializing system.size_estimates 
INFO [main] 2016-12-01 14:56:22,173 ColumnFamilyStore.java:412 - Initializing system.available_ranges 
INFO [main] 2016-12-01 14:56:22,188 ColumnFamilyStore.java:412 - Initializing system.views_builds_in_progress 
INFO [main] 2016-12-01 14:56:22,218 ColumnFamilyStore.java:412 - Initializing system.built_views 
INFO [main] 2016-12-01 14:56:22,238 ColumnFamilyStore.java:412 - Initializing system.hints 
INFO [main] 2016-12-01 14:56:22,261 ColumnFamilyStore.java:412 - Initializing system.batchlog 
INFO [main] 2016-12-01 14:56:22,269 ColumnFamilyStore.java:412 - Initializing system.schema_keyspaces 
INFO [main] 2016-12-01 14:56:22,277 ColumnFamilyStore.java:412 - Initializing system.schema_columnfamilies 
INFO [main] 2016-12-01 14:56:22,284 ColumnFamilyStore.java:412 - Initializing system.schema_columns 
INFO [main] 2016-12-01 14:56:22,292 ColumnFamilyStore.java:412 - Initializing system.schema_triggers 
INFO [main] 2016-12-01 14:56:22,300 ColumnFamilyStore.java:412 - Initializing system.schema_usertypes 
INFO [main] 2016-12-01 14:56:22,308 ColumnFamilyStore.java:412 - Initializing system.schema_functions 
INFO [main] 2016-12-01 14:56:22,315 ColumnFamilyStore.java:412 - Initializing system.schema_aggregates 
INFO [main] 2016-12-01 14:56:22,317 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized 
INFO [main] 2016-12-01 14:56:22,700 JMXServerUtils.java:111 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi 
INFO [main] 2016-12-01 14:56:23,382 StorageService.java:535 - Populating token metadata from system tables 
INFO [main] 2016-12-01 14:56:23,406 StorageService.java:542 - Token metadata: Normal Tokens: 
INFO [main] 2016-12-01 14:56:23,422 ColumnFamilyStore.java:412 - Initializing system_schema.keyspaces 
INFO [main] 2016-12-01 14:56:23,470 ColumnFamilyStore.java:412 - Initializing system_schema.tables 
INFO [main] 2016-12-01 14:56:23,493 ColumnFamilyStore.java:412 - Initializing system_schema.columns 
INFO [main] 2016-12-01 14:56:23,512 ColumnFamilyStore.java:412 - Initializing system_schema.triggers 
INFO [main] 2016-12-01 14:56:23,529 ColumnFamilyStore.java:412 - Initializing system_schema.dropped_columns 
INFO [main] 2016-12-01 14:56:23,574 ColumnFamilyStore.java:412 - Initializing system_schema.views 
INFO [main] 2016-12-01 14:56:23,596 ColumnFamilyStore.java:412 - Initializing system_schema.types 
INFO [main] 2016-12-01 14:56:23,615 ColumnFamilyStore.java:412 - Initializing system_schema.functions 
INFO [main] 2016-12-01 14:56:23,626 ColumnFamilyStore.java:412 - Initializing system_schema.aggregates 
INFO [main] 2016-12-01 14:56:23,653 ColumnFamilyStore.java:412 - Initializing system_schema.indexes 
INFO [main] 2016-12-01 14:56:23,669 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized 
INFO [main] 2016-12-01 14:56:23,950 ColumnFamilyStore.java:412 - Initializing hotel.amenities_by_room 
INFO [main] 2016-12-01 14:56:23,956 ColumnFamilyStore.java:412 - Initializing hotel.available_rooms_by_hotel_date 
INFO [main] 2016-12-01 14:56:23,962 ColumnFamilyStore.java:412 - Initializing hotel.hotels 
INFO [main] 2016-12-01 14:56:23,967 ColumnFamilyStore.java:412 - Initializing hotel.hotels_by_poi 
INFO [main] 2016-12-01 14:56:23,973 ColumnFamilyStore.java:412 - Initializing hotel.pois_by_hotel 
INFO [main] 2016-12-01 14:56:23,974 ViewManager.java:137 - Not submitting build tasks for views in keyspace hotel as storage service is not initialized 
INFO [main] 2016-12-01 14:56:23,981 ColumnFamilyStore.java:412 - Initializing system_distributed.parent_repair_history 
INFO [main] 2016-12-01 14:56:23,986 ColumnFamilyStore.java:412 - Initializing system_distributed.repair_history 
INFO [main] 2016-12-01 14:56:23,992 ColumnFamilyStore.java:412 - Initializing system_distributed.view_build_status 
INFO [main] 2016-12-01 14:56:24,030 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized 
INFO [main] 2016-12-01 14:56:24,037 ColumnFamilyStore.java:412 - Initializing reservation.guests 
INFO [main] 2016-12-01 14:56:24,043 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_guest 
INFO [main] 2016-12-01 14:56:24,048 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_hotel_date 
INFO [main] 2016-12-01 14:56:24,052 ColumnFamilyStore.java:412 - Initializing reservation.reservations_by_confirmation 
INFO [main] 2016-12-01 14:56:24,054 ViewManager.java:137 - Not submitting build tasks for views in keyspace reservation as storage service is not initialized 
INFO [main] 2016-12-01 14:56:24,059 ColumnFamilyStore.java:412 - Initializing system_auth.resource_role_permissons_index 
INFO [main] 2016-12-01 14:56:24,064 ColumnFamilyStore.java:412 - Initializing system_auth.role_members 
INFO [main] 2016-12-01 14:56:24,069 ColumnFamilyStore.java:412 - Initializing system_auth.role_permissions 
INFO [main] 2016-12-01 14:56:24,074 ColumnFamilyStore.java:412 - Initializing system_auth.roles 
INFO [main] 2016-12-01 14:56:24,087 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized 
INFO [main] 2016-12-01 14:56:24,093 ColumnFamilyStore.java:412 - Initializing system_traces.events 
INFO [main] 2016-12-01 14:56:24,137 ColumnFamilyStore.java:412 - Initializing system_traces.sessions 
INFO [main] 2016-12-01 14:56:24,190 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized 
INFO [main] 2016-12-01 14:56:24,204 ColumnFamilyStore.java:412 - Initializing my_keyspace.user 
INFO [main] 2016-12-01 14:56:24,222 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_address_idx 
INFO [main] 2016-12-01 14:56:24,387 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_emails_idx 
INFO [main] 2016-12-01 14:56:24,448 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_last_name_idx 
INFO [main] 2016-12-01 14:56:24,501 ColumnFamilyStore.java:412 - Initializing my_keyspace.user.user_phone_idx 
INFO [main] 2016-12-01 14:56:24,556 ViewManager.java:137 - Not submitting build tasks for views in keyspace my_keyspace as storage service is not initialized 
INFO [pool-4-thread-1] 2016-12-01 14:56:24,561 AutoSavingCache.java:196 - reading saved cache /var/lib/cassandra/saved_caches/KeyCache-e.db 
INFO [pool-4-thread-1] 2016-12-01 14:56:24,575 AutoSavingCache.java:172 - Completed loading (15 ms; 43 keys) KeyCache cache 
INFO [main] 2016-12-01 14:56:24,592 CommitLog.java:166 - Replaying /var/lib/cassandra/commitlog/CommitLog-6-1480266920262.log, /var/lib/cassandra/commitlog/CommitLog-6-1480266920263.log, /var/lib/cassandra/commitlog/CommitLog-6-1480428373824.log, /var/lib/cassandra/commitlog/CommitLog-6-1480430610402.log, /var/lib/cassandra/commitlog/CommitLog-6-1480446994136.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447217591.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447657364.log, /var/lib/cassandra/commitlog/CommitLog-6-1480447697557.log, /var/lib/cassandra/commitlog/CommitLog-6-1480448913585.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449134204.log, /var/lib/cassandra/commitlog/CommitLog-6-1480449329313.log, /var/lib/cassandra/commitlog/CommitLog-6-1480527540752.log, /var/lib/cassandra/commitlog/CommitLog-6-1480584423323.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594224164.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594243517.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594321703.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594441672.log, /var/lib/cassandra/commitlog/CommitLog-6-1480594635525.log, /var/lib/cassandra/commitlog/CommitLog-6-1480596178586.log, /var/lib/cassandra/commitlog/CommitLog-6-1480596689253.log 
ERROR [main] 2016-12-01 14:56:24,878 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization. 
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Unexpected error deserializing mutation; saved to /tmp/mutation2085286393633679945dat. This may be caused by replaying a mutation against a table with the same name but incompatible schema. Exception follows: org.apache.cassandra.serializers.MarshalException: Not enough bytes to read 0th field street 
    at org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:410) 
    at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:343) 
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:202) 
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:85) 
[email protected]:~$

Как это исправить?

источник

2016-11-29 user4955663

Возможно, не связано непосредственно с вопросом, но для: «Только 30.073GiB бесплатно на всех томах данных», вам, вероятно, стоит посмотреть. Если вы используете STCS, вам нужны диски на 50% бесплатно. –

. Хороший вопрос, я не знал о 50-процентном ограничении. Эта виртуальная машина для SW dev env, содержимое Cassandra db будет довольно ограничено. Может быть, STCS - это не лучшая стратегия для этой виртуальной машины. Требует ли LCS меньше свободного места на диске, чем STCS? – user4955663

fwiw это не _actually_ требуется, но его безопаснее. Вы должны иметь возможность в абсолютном наихудшем случае удвоить размер данных, если это уплотнение, объединяющее все sstables. LCS действительно требует меньше, но его сложнее определить –

У вас есть поврежденный sstable, но с его вторичным индексом его довольно легко исправить. Вторичные индексы - это просто денормализация действительно локальных данных, поэтому все данные находятся в вашей таблице user, ничего не было потеряно.

Просто удалите sstables в каталоге данных user_address_idx и восстановите их с помощью nodetool rebuild_index my_keyspace user user_address_idx.

источник

2016-11-29 23:06:06

'mv /var/lib/cassandra/data/my_keyspace/user*/.user_address_idx/mc-* ~'. Всегда 'mv' вместо' rm' first;) –

Так он на самом деле пытается создать индекс при запуске. и что-то в пользовательской таблице выглядит поврежденным. Можете ли вы запустить 'sstablescrub' в таблице' my_keyspace.user'? (Https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSSTableScrub_t.html). Это потребовало бы ремонта после этого, но узлы были опущены дольше, чем полчаса, в этом случае я предполагаю, что так или иначе, так что это будет нужно в любом случае. –

запустить sstablescrub в качестве пользователя cassandra (или любого другого пользователя, с которым вы обычно запускаете C *). это было AccessDeniedException, на самом деле не существует –

Cassandra не запускается: SSTableBatchOpen IllegalArgumentException

ответ

Смежные вопросы