Database Administrators
Q&A for database professionals who wish to improve their database skills
Latest Questions
0
votes
0
answers
10
views
cassandra disk space not freeing up
I have Cassandra on K8s. It's being used as part of a temporal persistence store. I'm seeing that my disk space is ever growing. I'm deleting some completed workflows, but they still occupy disk space. I have set **gc_grace_period** to 6 hrs, and **LCS** is the compaction strategy being used. Forcin...
I have Cassandra on K8s. It's being used as part of a temporal persistence store. I'm seeing that my disk space is ever growing. I'm deleting some completed workflows, but they still occupy disk space.
I have set **gc_grace_period** to 6 hrs, and **LCS** is the compaction strategy being used. Forcing manual compaction will result in high disk I/O and CPU pressure, which can be problematic in my production environment. Is there any other way to solve this problem?
Mohit Agarwal
(1 rep)
Aug 3, 2025, 04:57 AM
• Last activity: Aug 3, 2025, 05:40 AM
1
votes
1
answers
375
views
Cassandra pool warning displaying continuously
I am using the [Cassandra driver for python][1] in Spyder. I am trying to fetch some data from Cassandra table. Here is my code: from cassandra.cluster import Cluster cluster=Cluster(['some_ip']) session=cluster.connect('some_key_space') df_filtered_10m=session.execute("some query") This is all work...
I am using the Cassandra driver for python in Spyder. I am trying to fetch some data from Cassandra table. Here is my code:
from cassandra.cluster import Cluster
cluster=Cluster(['some_ip'])
session=cluster.connect('some_key_space')
df_filtered_10m=session.execute("some query")
This is all working fine and I am getting the desired results. The problem is that in the console, this message is continuously popping up:
> WARNING:cassandra.pool:Error attempting to reconnect to 10.0.10.91,
> scheduling retry in 256.0 seconds: [Errno None] Tried connecting to
> [('10.0.10.91', 9042)]. Last error: timed out
I have tried
cluster.shutdown but it not working as well.
How to get rid of it?
Osama Dar
(111 rep)
Sep 12, 2018, 05:19 AM
• Last activity: Aug 2, 2025, 01:04 AM
1
votes
1
answers
910
views
Queries return ReadFailure exception, "Replica(s) failed to execute read"
I've a cluster consisting of 5 cassandra nodes. The cassandra version used is cassandra-3.11.3-1.noarch The `keyspace` strategy and is defined as follows: CREATE KEYSPACE my_keyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '2'} AND durable_writes = true; Now running qu...
I've a cluster consisting of 5 cassandra nodes. The cassandra version used is
cassandra-3.11.3-1.noarch
The
keyspace strategy and is defined as follows:
CREATE KEYSPACE my_keyspace
WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '2'}
AND durable_writes = true;
Now running queries on the cluster, the following issue occurs:
> ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
> message="Operation failed - received 0 responses and 1 failures"
> info={'consistency': 'LOCAL_ONE', 'received_responses': 0, 'required_responses': 1, 'failures': 1}
> info={'failures': None, 'consistency': 'Not Set', 'required_responses': None, 'received_responses': None}
Does anyone know what is causing this? If more information needed to better debug this problem, please let me know!
**UPDATE 1**
root# awk '!/#/' /etc/cassandra/conf/cassandra-rackdc.properties
dc=datacenter1
rack=rack1
Valentin Bajrami
(111 rep)
Aug 22, 2018, 08:29 AM
• Last activity: Jul 30, 2025, 05:07 AM
0
votes
2
answers
175
views
DSBulk count returns more rows than unloaded in CSV files
I'm doing `dsbulk unload` for one table, with one primary key field, without clustering key. At the end in console I see something like this ``` total | failed | rows/s | p50ms | p99ms | p999ms 174,971,236 | 0 | 1,946,689 | 148.95 | 285.21 | 400.56 ``` but when I counting number lines in all csv fil...
I'm doing
dsbulk unload for one table, with one primary key field, without clustering key.
At the end in console I see something like this
total | failed | rows/s | p50ms | p99ms | p999ms
174,971,236 | 0 | 1,946,689 | 148.95 | 285.21 | 400.56dsbulk count shows me same result as unload ~175M.
My general question why it happens? What is explanation? What real size of my table? and how to debug?
I already did dsbulk count with --dsbulk.engine.maxConcurrentQueries 1, --datastax-java-driver.basic.request.consistency LOCAL_QUORUM and number is same ~175M.
Viktor Tsymbaliuk
(25 rep)
Jun 12, 2024, 11:58 AM
• Last activity: Jul 29, 2025, 01:03 AM
0
votes
1
answers
150
views
sstableloader returns ExecutionException, StreamException with message "Stream failed"
While loading sstables through SSTableLoader command, I am frequently facing this issue. ``` java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) at org.apac...
While loading sstables through SSTableLoader command, I am frequently facing this issue.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:109)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:58)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:524)
at org.apache.cassandra.streaming.StreamSession.initiatorCompleteOrWait(StreamSession.java:920)
at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:874)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:614)
at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:76)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:124)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:58)
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:109)
... 1 more
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:243)
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:524)
at org.apache.cassandra.streaming.StreamSession.initiatorCompleteOrWait(StreamSession.java:920)
at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:874)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:614)
at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:76)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
prasanna
(1 rep)
Sep 18, 2023, 05:20 PM
• Last activity: Jul 26, 2025, 07:04 PM
0
votes
0
answers
13
views
SSTable increase a lot after repair
I'm trying repair in one of the node nodetool repair -full -st start_token -et end_token keyspace table_name I found the SSTable count of this table continusly increase and it's useless on compact commend nodetool compact keyspace table_name Eventually the node shutdown. Issue: 1. It seems the compa...
I'm trying repair in one of the node
nodetool repair -full -st start_token -et end_token keyspace table_name
I found the SSTable count of this table continusly increase and it's useless on compact commend
nodetool compact keyspace table_name
Eventually the node shutdown.
Issue:
1. It seems the compaction after repair or nodetool compact will fail, there's many error log as bellow
> ERROR [CompactionExecutor:3743] 2025-07-15 05:04:33,672 CassandraDaemon.java:119 - Exception in thread Thread[CompactionExecutor:3743,5,main]
java.lang.AssertionError: point2 should follow point1
2. When I try to start the node , it stops when initial because there're too many SSTable & metadata file. But the open file parameter is already the limit of my server.
> ERROR [SSTableBatchOpen:38] 2025-07-21 14:31:51,170 CassandraDaemon.java:119 - Exception in thread Thread[SSTableBatchOpen:38,5,main]
java.lang.RuntimeException: java.nio.file.FileSystemException: /db/cassandra/data/keyspace/table_name-ba0eccc225f811e9ae0c059871c9553d/aa-9273-bti-Partitions.db: Too many open files
Thanks for any suggestion.
cheeeeen
(11 rep)
Jul 25, 2025, 02:27 AM
1
votes
1
answers
146
views
How to reduce Backpressure rejection while receiving WRITES and increase throughput limit of cluster
After we rebuild a down node we could see some other nodes are going down and its prompting "Backpressure rejection while receiving WRITES". Nodes are behaving as down and its continuing for sometime. We want to reduce this behaviour and want to know how to redduce this. Its not an error but a sympt...
After we rebuild a down node we could see some other nodes are going down and its prompting "Backpressure rejection while receiving WRITES". Nodes are behaving as down and its continuing for sometime.
We want to reduce this behaviour and want to know how to redduce this. Its not an error but a symptom. How to deal with this.
Cassandra Thrift
(307 rep)
Apr 17, 2023, 12:18 PM
• Last activity: Jul 23, 2025, 07:04 PM
1
votes
1
answers
146
views
Cassandra can't start with an old backup of /var/lib/cassandra
I have an old Cassandra backup of `/var/lib/cassandra` on an external drive and I want to setup a new database with this backup. I install the same Cassandra version as then and in the yaml file I changed the `data_file_directories` paths into the new ones (external drive paths) but nothing. Any ide...
I have an old Cassandra backup of
/var/lib/cassandra on an external drive and I want to setup a new database with this backup.
I install the same Cassandra version as then and in the yaml file I changed the data_file_directories paths into the new ones (external drive paths) but nothing.
Any idea on how to fix it or what I am doing wrong?
Vasilis_Dimitrakopoulos
(23 rep)
Aug 30, 2022, 10:55 AM
• Last activity: Jul 19, 2025, 02:04 PM
0
votes
1
answers
147
views
Can I replace more than one node if RF=3 and two dead nodes?
Can I replace more than one dead node if ```RF=3``` and two dead nodes?
Can I replace more than one dead node if
=3
biendv
(21 rep)
Oct 17, 2022, 01:26 PM
• Last activity: Jul 19, 2025, 02:01 AM
1
votes
1
answers
164
views
Apache Cassandra node restarts randomly
I have a cluster of 11 nodes, `Apache Cassandra version 3.11.4`, where certain nodes are restarted at random times. Nodes characteristics: CPU(s): 32 Model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz RAM: 62G Disk: 42T Below, I am attaching system.log file of one node a.k.a node70 before restar...
I have a cluster of 11 nodes,
Apache Cassandra version 3.11.4, where certain nodes are restarted at random times.
Nodes characteristics:
CPU(s): 32
Model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
RAM: 62G
Disk: 42T
Below, I am attaching system.log file of one node a.k.a node70 before restart:
WARN [CompactionExecutor:47] 2024-01-07 21:07:17,332 BigTableWriter.java:211 - Writing large partition xxx/xxx:128274211 (203.247MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495387-big-Data.db
INFO [HintsDispatcher:7] 2024-01-07 21:07:26,969 HintsStore.java:133 - Deleted hint file d75b9d99-e44b-4b91-8ba1-2b60852699da-1704638444850-1.hints
INFO [HintsDispatcher:7] 2024-01-07 21:07:26,970 HintsDispatchExecutor.java:282 - Finished hinted handoff of file d75b9d99-e44b-4b91-8ba1-2b60852699da-1704638444850-1.hints to endpoint /10.147.49.69: d75b9d99-e44b-4b91-8ba1-2b60852699da
WARN [CompactionExecutor:36] 2024-01-07 21:07:33,280 BigTableWriter.java:211 - Writing large partition xxx/xxx:20755 (172.373MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495376-big-Data.db
WARN [CompactionExecutor:43] 2024-01-07 21:07:55,185 BigTableWriter.java:211 - Writing large partition xxx/xxx:128274211 (116.939MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495383-big-Data.db
...
I am leaving key logs, so you can follow up. Whole file contains a lot of WARNs about Writing large patition.
WARN [CompactionExecutor:47] 2024-01-07 21:17:45,303 BigTableWriter.java:211 - Writing large partition xxx/xxx:26300 (191.685MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495387-big-Data.db
WARN [CompactionExecutor:40] 2024-01-07 21:18:00,395 BigTableWriter.java:211 - Writing large partition xxx/xxx:26663181 (145.598MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495380-big-Data.db
WARN [CompactionExecutor:42] 2024-01-07 21:18:14,510 BigTableWriter.java:211 - Writing large partition xxx/xxx:39741453 (111.068MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495416-big-Data.db
INFO [ScheduledTasks:1] 2024-01-07 21:18:41,355 NoSpamLogger.java:91 - Some operations were slow, details available at debug level (debug.log)
WARN [CompactionExecutor:43] 2024-01-07 21:18:55,742 BigTableWriter.java:211 - Writing large partition xxx/xxx:332 (107.752MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495383-big-Data.db
WARN [CompactionExecutor:44] 2024-01-07 21:18:57,684 BigTableWriter.java:211 - Writing large partition xxx/xxx:770 (107.219MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495384-big-Data.db
Also here:
WARN [CompactionExecutor:44] 2024-01-07 21:22:19,816 BigTableWriter.java:211 - Writing large partition xxx/xxx:30096898 (103.169MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495384-big-Data.db
INFO [CompactionExecutor:42] 2024-01-07 21:22:26,206 NoSpamLogger.java:91 - Maximum memory usage reached (536870912), cannot allocate chunk of 1048576
WARN [CompactionExecutor:43] 2024-01-07 21:22:57,513 BigTableWriter.java:211 - Writing large partition xxx/xxx:26663181 (148.395MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495383-big-Data.db
INFO [IndexSummaryManager:1] 2024-01-07 21:22:59,163 IndexSummaryRedistribution.java:77 - Redistributing index summaries
WARN [CompactionExecutor:42] 2024-01-07 21:23:01,308 BigTableWriter.java:211 - Writing large partition xxx/xxx:20755 (350.542MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495416-big-Data.db
These kind of logs are repeating till **2024-01-7 22:49:00** and node just goes down and ups:
WARN [CompactionExecutor:43] 2024-01-07 22:49:26,242 BigTableWriter.java:211 - Writing large partition xxx/xxx:30269442 (109.637MiB) to sstable /home/user/apache-cassandra-3.11.4/data/data/xxx/xxx-5d4d2290f19811e99de8956ff675a115/md-1495383-big-Data.db
INFO [main] 2024-01-07 22:55:08,817 YamlConfigurationLoader.java:89 - Configuration location: file:/home/user/apache-cassandra-3.11.4/conf/cassandra.yaml
INFO [main] 2024-01-07 22:55:09,194 Config.java:496 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=; cluster_name=LBS Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=32; concurrent_compactors=16; concurrent_counter_writes=16; concurrent_materialized_view_writes=32; concurrent_reads=16; concurrent_replicates=null; concurrent_writes=16; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@7ec7ffd3; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=false; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=true; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.147.49.70; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=172800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=1024; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=10.147.49.70; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.147.49.69,10.147.49.70,10.147.49.71,10.147.49.74,10.147.49.75,10.147.49.73,10.147.49.72,10.147.49.67,10.147.49.68,10.147.49.64,10.147.49.65}; server_encryption_options=; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@5b239d7d; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO [main] 2024-01-07 22:55:09,195 DatabaseDescriptor.java:373 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2024-01-07 22:55:09,196 DatabaseDescriptor.java:431 - Global memtable on-heap threshold is enabled at 2508MB
INFO [main] 2024-01-07 22:55:09,196 DatabaseDescriptor.java:435 - Global memtable off-heap threshold is enabled at 2508MB
INFO [main] 2024-01-07 22:55:09,252 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO [main] 2024-01-07 22:55:09,252 DatabaseDescriptor.java:735 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
INFO [main] 2024-01-07 22:55:09,580 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
INFO [main] 2024-01-07 22:55:09,667 JMXServerUtils.java:246 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO [main] 2024-01-07 22:55:09,675 CassandraDaemon.java:476 - Hostname: server70
INFO [main] 2024-01-07 22:55:09,676 CassandraDaemon.java:483 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_222
INFO [main] 2024-01-07 22:55:09,677 CassandraDaemon.java:484 - Heap size: 9.800GiB/9.800GiB
INFO [main] 2024-01-07 22:55:09,678 CassandraDaemon.java:489 - Code Cache Non-heap memory: init = 2555904(2496K) used = 7550016(7373K) committed = 7602176(7424K) max = 251658240(245760K)
INFO [main] 2024-01-07 22:55:09,678 CassandraDaemon.java:489 - Metaspace Non-heap memory: init = 0(0K) used = 20864080(20375K) committed = 21495808(20992K) max = -1(-1K)
INFO [main] 2024-01-07 22:55:09,679 CassandraDaemon.java:489 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2402776(2346K) committed = 2621440(2560K) max = 1073741824(1048576K)
INFO [main] 2024-01-07 22:55:09,679 CassandraDaemon.java:489 - Par Eden Space Heap memory: init = 1718091776(1677824K) used = 481091008(469815K) committed = 1718091776(1677824K) max = 1718091776(1677824K)
INFO [main] 2024-01-07 22:55:09,679 CassandraDaemon.java:489 - Par Survivor Space Heap memory: init = 214695936(209664K) used = 0(0K) committed = 214695936(209664K) max = 214695936(209664K)
INFO [main] 2024-01-07 22:55:09,679 CassandraDaemon.java:489 - CMS Old Gen Heap memory: init = 8589934592(8388608K) used = 0(0K) committed = 8589934592(8388608K) max = 8589934592(8388608K)
**NB!**
When I want to check nodetool status in node70 it shows me: **Cassandra 3.0 and later require Java 8u40 or later**, but others can display even their versions are same.
Also, for some reason Snapshots does not work on this node, and data from 2020 and 2021 is stored there, although I configured it for two years.
I am also getting such logs in logs directory:
hs_err_pid1461756.log
hs_err_pid1502594.log
Here is my **gc.logs**:
2024-01-08T11:38:48.267+0600: 3533.985: Total time for which application threads were stopped: 0.0273467 seconds, Stopping threads took: 0.0017132 seconds
{Heap before GC invocations=5251 (full 22):
par new generation total 1887488K, used 1710170K [0x0000000540000000, 0x00000005c0000000, 0x00000005c0000000)
eden space 1677824K, 100% used [0x0000000540000000, 0x00000005a6680000, 0x00000005a6680000)
from space 209664K, 15% used [0x00000005b3340000, 0x00000005b52d68f0, 0x00000005c0000000)
to space 209664K, 0% used [0x00000005a6680000, 0x00000005a6680000, 0x00000005b3340000)
concurrent mark-sweep generation total 8388608K, used 3626506K [0x00000005c0000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 46509K, capacity 48029K, committed 48604K, reserved 1091584K
class space used 5645K, capacity 5969K, committed 6108K, reserved 1048576K
2024-01-08T11:38:48.753+0600: 3534.471: [GC (Allocation Failure) 2024-01-08T11:38:48.753+0600: 3534.471: [ParNew
Desired survivor size 107347968 bytes, new threshold 1 (max 1)
- age 1: 22025736 bytes, 22025736 total
: 1710170K->30151K(1887488K), 0.0262670 secs] 5336676K->3678067K(10276096K), 0.0266400 secs] [Times: user=0.46 sys=0.01, real=0.02 secs]
Heap after GC invocations=5252 (full 22):
par new generation total 1887488K, used 30151K [0x0000000540000000, 0x00000005c0000000, 0x00000005c0000000)
eden space 1677824K, 0% used [0x0000000540000000, 0x0000000540000000, 0x00000005a6680000)
from space 209664K, 14% used [0x00000005a6680000, 0x00000005a83f1f70, 0x00000005b3340000)
to space 209664K, 0% used [0x00000005b3340000, 0x00000005b3340000, 0x00000005c0000000)
concurrent mark-sweep generation total 8388608K, used 3647915K [0x00000005c0000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 46509K, capacity 48029K, committed 48604K, reserved 1091584K
class space used 5645K, capacity 5969K, committed 6108K, reserved 1048576K
}
2024-01-08T11:38:48.780+0600: 3534.498: Total time for which application threads were stopped: 0.0325483 seconds, Stopping threads took: 0.0013643 seconds
2024-01-08 11:38:49 GC log file has reached the maximum size. Saved as /home/user/apache-cassandra-3.11.4/logs/gc.log.0
Please, tell me if you need additional information. This is my first post in dba.stackexchange.com, so I may not take into account some forum rules
Thank you!
Midas Salimgali
(11 rep)
Jan 8, 2024, 07:31 AM
• Last activity: Jul 16, 2025, 07:05 AM
1
votes
1
answers
175
views
Cassandra System.Log shows nodtool tpstats output
We have a Docker Cassandra container running 6.8.21. At times when the container is restarted the the Logs show the usual startup activities and all just works well post that. But on a few occasions we observed that an output similar to that of nodetool tpstats keeps getting printed over and over ag...
We have a Docker Cassandra container running 6.8.21. At times when the container is restarted the the Logs show the usual startup activities and all just works well post that.
But on a few occasions we observed that an output similar to that of nodetool tpstats keeps getting printed over and over again. Posting a sample of the output below.
Need help:
1. Why is Cassandra Printing this? Is there some bottleneck that it has hit?
2. What does this log trying to tell?
3. What should be the next steps in our investigation?
Appreciate any help here.
Log:
INFO [OptionalTasks:1] 2022-08-11 01:33:55,678 StatusLogger.java:174 -
Pool Name Active Pending Backpressure Delayed Shared Stolen Completed Blocked All Time Blocked
AuthzStage 0 0 N/A N/A N/A N/A 0 0 0
LwtStage 0 0 N/A N/A N/A N/A 0 0 0
RequestResponseStage 0 0 N/A N/A N/A N/A 0 0 0
AntiCompactionExecutor 0 0 N/A N/A N/A N/A 0 0 0
AntiEntropyStage 0 0 N/A N/A N/A N/A 0 0 0
BackgroundIoStage 0 0 N/A N/A N/A N/A 0 0 0
CacheCleanupExecutor 0 0 N/A N/A N/A N/A 0 0 0
CompactionExecutor 0 0 N/A N/A N/A N/A 77 0 0
GossipStage 0 0 N/A N/A N/A N/A 0 0 0
HintsDispatcher 0 0 N/A N/A N/A N/A 0 0 0
InternalResponseStage 0 0 N/A N/A N/A N/A 0 0 0
MemtableFlushWriter 0 0 N/A N/A N/A N/A 78 0 0
MemtablePostFlush 0 0 N/A N/A N/A N/A 203 0 0
MemtableReclaimMemory 0 0 N/A N/A N/A N/A 78 0 0
MigrationStage 0 0 N/A N/A N/A N/A 0 0 0
MiscStage 0 0 N/A N/A N/A N/A 0 0 0
PendingRangeCalculator 0 0 N/A N/A N/A N/A 1 0 0
PerDiskMemtableFlushWriter_0 0 0 N/A N/A N/A N/A 39 0 0
Repair-Task 0 0 N/A N/A N/A N/A 0 0 0
Sampler 0 0 N/A N/A N/A N/A 0 0 0
SecondaryIndexManagement 0 0 N/A N/A N/A N/A 0 0 0
TPC/all N/A N/A N/A N/A N/A N/A N/A N/A N/A
TPC/all/AUTHENTICATION 0 N/A N/A N/A 0 0 8 N/A N/A
TPC/all/AUTHORIZATION 0 N/A N/A N/A 0 0 9 N/A N/A
TPC/all/BACKPRESSURED_MESSAGE_DECODE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/BACKPRESSURE_RESCHEDULE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/BACKUP_OPERATION 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/BATCH_REMOVE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/BATCH_REPLAY 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/BATCH_STORE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/BATCH_STORE_RESPONSE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/BLOCKING_OPERATION_RESTART 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/BUFFER_POOL_CLEANUP 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/CLEAN_BACKUPS_OPERATION 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/CONTINUOUS_PAGING 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/COUNTER_ACQUIRE_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/EXECUTE_STATEMENT 0 N/A N/A N/A 0 0 38 N/A N/A
TPC/all/FRAME_DECODE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/HINT_DISPATCH 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/HINT_RESPONSE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/INCREMENTAL_NODESYNC_MAINTENANCE 0 N/A N/A N/A 0 0 1271 N/A N/A
TPC/all/LWT_COMMIT 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/LWT_PREPARE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/LWT_PREPARE_AWAIT_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/LWT_PROPOSE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/LWT_PROPOSE_AWAIT_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/NETWORK_BACKPRESSURE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/NODESYNC_VALIDATION 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/OUTBOUND_FLUSH 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/OUTBOUND_SCHEDULED_FLUSH 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/PARALLEL_MESSAGE_DECODE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/POPULATE_VIRTUAL_TABLE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_DISK_ASYNC 0 N/A N/A N/A 0 0 36 N/A N/A
TPC/all/READ_DISK_RETRY 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_DISK_WOULD_BLOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_INTERNAL 0 N/A N/A N/A 0 0 13 N/A N/A
TPC/all/READ_LOCAL 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/READ_RANGE_INTERNAL 0 N/A N/A N/A 0 0 22 N/A N/A
TPC/all/READ_RANGE_LOCAL 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/READ_RANGE_NODESYNC 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/READ_RANGE_REMOTE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/READ_RANGE_RESPONSE 0 N/A N/A N/A 0 0 22 N/A N/A
TPC/all/READ_REMOTE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/READ_RESPONSE 0 N/A N/A N/A 0 0 13 N/A N/A
TPC/all/READ_SECONDARY_INDEX 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_SPECULATE 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_SWITCH_FOR_ITERATOR 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/READ_TIMEOUT 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/RESTORE_OPERATION 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/SAVE_LOCAL_NODE 0 N/A N/A N/A 0 0 7 N/A N/A
TPC/all/SAVE_PEER_NODES 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/SOLR_ACQUIRE_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/SOLR_INDEXING 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/SYSTEM_KEYSPACE_ACQUIRE_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/TIMED_TIMEOUT 0 N/A N/A N/A 0 0 2331 N/A N/A
TPC/all/TIMED_UNKNOWN 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/TRUNCATE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/UNKNOWN 0 N/A N/A N/A 0 0 49 N/A N/A
TPC/all/UPDATE_NOTIFICATION 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/VIEW_ACQUIRE_LOCK 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/WRITE_AWAIT_COMMITLOG_SEGMENT 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/WRITE_AWAIT_COMMITLOG_SYNC 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/WRITE_DEFRAGMENT 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/WRITE_INTERNAL 0 N/A N/A N/A 0 0 3 N/A N/A
TPC/all/WRITE_LOCAL 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/WRITE_MEMTABLE_FULL 0 N/A N/A N/A 0 0 0 N/A N/A
TPC/all/WRITE_REMOTE 0 0 N/A N/A 0 0 0 N/A 0
TPC/all/WRITE_RESPONSE 0 N/A N/A N/A 0 0 3 N/A N/A
TPC/all/WRITE_SWITCH_FOR_MEMTABLE 0 N/A N/A N/A 0 0 372 N/A N/A
ValidationExecutor 0 0 N/A N/A N/A N/A 0 0 0
ViewBuildExecutor 0 0 N/A N/A N/A N/A 0 0 0
CompactionManager 0 0
MessagingService n/a 0/0
Buffer pool size for cached reads: 4.000MiB, for direct reads: 4.000MiB
Cached reads buffer pool BufferPool for long lived buffers: Size-tiered from 4.000KiB to 4.000KiB buffers, using 1024 buffers per slab.
Sub pools:
Buffer size 4.000KiB: 584.000KiB used, 4.000MiB allocated, 1 slabs.
Overflow: 0.000KiB
Direct reads buffer pool BufferPool for temporary buffers: allocated 4.000MiB, used 0.000KiB, overflow 0.000KiB, overflow allocations mean rate 0.000000
Global memtable buffer pool size: onHeap = 3.402KiB, offHeap = 2.029KiB
Cache Type Size Capacity KeysToSave
KeyCache 0 104857600 all
RowCache 0 0 all
Table Memtable ops,data
system_schema.keyspaces 0,0
system_schema.hidden_columns 0,0
system_schema.edges 0,0
system_schema.views 0,0
system_schema.columns 0,0
system_schema.types 0,0
system_schema.indexes 0,0
system_schema.vertices 0,0
system_schema.dropped_columns 0,0
system_schema.aggregates 0,0
system_schema.triggers 0,0
system_schema.tables 0,0
system_schema.functions 0,0
system_auth.roles 2,262
system_auth.role_members 0,0
system_auth.role_permissions 1,199
dse_system_local.solr_resources 0,0
dse_system.encrypted_keys 0,0
dse_leases.logs 0,0
dse_leases.leases 0,0
solr_admin.solr_resources 0,0
dse_insights.insights_config 0,0
dse_insights.tokens 0,0
dse_insights_local.insights_config 0,0
system_distributed.nodesync_status 0,0
system_distributed.nodesync_user_validations 0,0
system_distributed.parent_repair_history 0,0
system_distributed.repair_history 0,0
system_distributed.restore_task_state 0,0
system_distributed.backups 0,0
system_distributed.backup_task_state 0,0
system_distributed.view_build_status 0,0
system_backups.backup_stores 0,0
system_backups.backup_configurations 0,0
dse_perf.node_slow_log 0,0
system_traces.sessions 0,0
system_traces.events 0,0
dse_security.digest_tokens 0,0
dse_security.role_options 0,0
dse_security.spark_security 0,0
axone.config_context 0,0
axone.test_hc 0,0
axone.config 0,0
axone.valid_context 0,0
system.compaction_history 1,254
system.IndexInfo 0,0
system.repairs 0,0
system.size_estimates 0,0
system.paxos 0,0
system.built_views 0,0
system.peer_events 0,0
system.range_xfers 0,0
system.peers 0,0
system.batches 0,0
system.transferred_ranges 0,0
system.view_builds_in_progress 0,0
system.local 0,0
system.sstable_activity 4,0
system.nodesync_checkpoints 0,0
system.available_ranges 0,0
system.prepared_statements 5,655
Sakiv
(11 rep)
Aug 21, 2022, 07:06 PM
• Last activity: Jul 15, 2025, 05:04 AM
1
votes
1
answers
166
views
nodetool rebuild returned "Unable to find sufficient sources for streaming range"
We are migrating self hosted cassandra 3.11.6 to self hosted cassandra with k8ssandra-operator using cassandra 3.11.7. At some point of the migration we had to stop it due to other unrelated incident. After resuming it and running rebuild on all the nodes 3 nodes started rebuilding data again. For 2...
We are migrating self hosted cassandra 3.11.6 to self hosted cassandra with k8ssandra-operator using cassandra 3.11.7.
At some point of the migration we had to stop it due to other unrelated incident. After resuming it and running rebuild on all the nodes 3 nodes started rebuilding data again.
For 2 remaining nodes I am getting this error:
Unable to find sufficient sources for streaming range (6296969270738431410,6299853300486273791] in keyspace system_distributed
System auth, traces, distributed were altered and are using NetworkTopologyStrategy with replication factors for dc1: 3 and dc2: 3.
Any idea why that could happen and how can I solve this issue? Why 3 nodes are working fine and only these 2 have problems? Thank you for help!
Mazurkim
(11 rep)
Mar 28, 2023, 04:03 AM
• Last activity: Jul 12, 2025, 06:07 PM
1
votes
1
answers
50
views
Cassandra cluster does not run in docker-compose after migration from version 4.1.9 to version 5.0.4
I had Cassandra cluster for tests that was run from docker-compose.yml. ```dockerfile services: cassandra1: image: cassandra:4.1.9 container_name: cassandra1 hostname: cassandra1 networks: - cassandra-net ports: - "7002:7000" - "7003:7001" - "7200:7199" - "9043:9042" - "9161:9160" environment: &cass...
I had Cassandra cluster for tests that was run from docker-compose.yml.
services:
cassandra1:
image: cassandra:4.1.9
container_name: cassandra1
hostname: cassandra1
networks:
- cassandra-net
ports:
- "7002:7000"
- "7003:7001"
- "7200:7199"
- "9043:9042"
- "9161:9160"
environment: &cassandra_config
CASSANDRA_START_RPC: true
CASSANDRA_RPC_ADDRESS: 0.0.0.0
CASSANDRA_LISTEN_ADDRESS: auto
CASSANDRA_SEEDS: cassandra1, cassandra2
CASSANDRA_CLUSTER_NAME: my-cluster
CASSANDRA_DC: datacenter1
CASSANDRA_RACK: rack1
CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
CASSANDRA_NUM_TOKENS: 16
volumes:
- cassandra-1:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: [ "CMD-SHELL", "nodetool status" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
cassandra2:
image: cassandra:4.1.9
container_name: cassandra2
hostname: cassandra2
networks:
- cassandra-net
ports:
- "7004:7000"
- "7005:7001"
- "7201:7199"
- "9044:9042"
- "9162:9160"
environment: *cassandra_config
depends_on:
cassandra1:
condition: service_healthy
volumes:
- cassandra-2:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: [ "CMD-SHELL", "nodetool status" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
cassandra3:
image: cassandra:4.1.9
container_name: cassandra3
hostname: cassandra3
networks:
- cassandra-net
ports:
- "7006:7000"
- "7007:7001"
- "7202:7199"
- "9045:9042"
- "9163:9160"
environment: *cassandra_config
depends_on:
cassandra2:
condition: service_healthy
volumes:
- cassandra-3:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: [ "CMD-SHELL", "nodetool status" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
volumes:
cassandra-1:
cassandra-2:
cassandra-3:
networks:
cassandra-net:
driver: bridgeservices:
cassandra1:
image: cassandra:5.0.4
container_name: cassandra1
hostname: cassandra1
networks:
- cassandra-net
ports:
- "7002:7000"
- "7003:7001"
- "7200:7199"
- "9043:9042"
- "9161:9160"
environment: &cassandra_config
CASSANDRA_START_RPC: true
CASSANDRA_RPC_ADDRESS: 0.0.0.0
CASSANDRA_LISTEN_ADDRESS: auto
CASSANDRA_BROADCAST_ADDRESS: auto
CASSANDRA_SEEDS: cassandra1, cassandra2
CASSANDRA_CLUSTER_NAME: my-cluster
CASSANDRA_DC: datacenter1
CASSANDRA_RACK: rack1
CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
CASSANDRA_NUM_TOKENS: 16
volumes:
- cassandra-1:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
cassandra2:
image: cassandra:5.0.4
container_name: cassandra2
hostname: cassandra2
networks:
- cassandra-net
ports:
- "7004:7000"
- "7005:7001"
- "7201:7199"
- "9044:9042"
- "9162:9160"
environment: *cassandra_config
depends_on:
cassandra1:
condition: service_healthy
volumes:
- cassandra-2:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
cassandra3:
image: cassandra:5.0.4
container_name: cassandra3
hostname: cassandra3
networks:
- cassandra-net
ports:
- "7006:7000"
- "7007:7001"
- "7202:7199"
- "9045:9042"
- "9163:9160"
environment: *cassandra_config
depends_on:
cassandra2:
condition: service_healthy
volumes:
- cassandra-3:/var/lib/cassandra:rw
restart: on-failure
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 2m
start_period: 2m
timeout: 10s
retries: 3
volumes:
cassandra-1:
cassandra-2:
cassandra-3:
networks:
cassandra-net:
driver: bridge...
2025-06-19 08:55:15.829 | WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-19 08:55:15.829 | WARNING: System::setSecurityManager has been called by org.apache.cassandra.security.ThreadAwareSecurityManager (file:/opt/cassandra/lib/apache-cassandra-5.0.4.jar)
2025-06-19 08:55:15.829 | WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.security.ThreadAwareSecurityManager
2025-06-19 08:55:15.829 | WARNING: System::setSecurityManager will be removed in a future release
2025-06-19 08:55:15.835 | INFO [main] 2025-06-19 05:55:15,834 CassandraDaemon.java:602 - Hostname: cassandra1:7000:7001
2025-06-19 08:55:15.835 | INFO [main] 2025-06-19 05:55:15,834 CassandraDaemon.java:609 - JVM vendor/version: OpenJDK 64-Bit Server VM/17.0.15
2025-06-19 08:55:15.836 | INFO [main] 2025-06-19 05:55:15,836 CassandraDaemon.java:610 - Heap size: 3.813GiB/3.813GiB
...
2025-06-19 08:55:15.843 | INFO [main] 2025-06-19 05:55:15,842 CassandraDaemon.java:617 - Classpath:
...
2025-06-19 08:55:44.422 | INFO [main] 2025-06-19 05:55:44,422 AbstractCryptoProvider.java:140 - AmazonCorrettoCryptoProvider health check OK.
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,422 DatabaseDescriptor.java:548 - DiskAccessMode is standard, indexAccessMode is mmap
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,422 DatabaseDescriptor.java:587 - Global memtable on-heap threshold is enabled at 976MiB
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,423 DatabaseDescriptor.java:591 - Global memtable off-heap threshold is enabled at 976MiB
2025-06-19 08:55:44.442 | INFO [main] 2025-06-19 05:55:44,442 DatabaseDescriptor.java:639 - commitlog_disk_access_mode resolved to: mmap
2025-06-19 08:55:44.443 | INFO [main] 2025-06-19 05:55:44,442 DatabaseDescriptor.java:659 - Native transport rate-limiting disabled.
2025-06-19 08:55:44.736 | INFO [main] 2025-06-19 05:55:44,735 GossipingPropertyFileSnitch.java:67 - Unable to load cassandra-topology.properties; compatibility mode disabled
2025-06-19 08:55:44.887 | INFO [main] 2025-06-19 05:55:44,886 JMXServerUtils.java:284 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
2025-06-19 08:55:44.890 | WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-19 08:55:44.890 | WARNING: System::setSecurityManager has been called by org.apache.cassandra.security.ThreadAwareSecurityManager (file:/opt/cassandra/lib/apache-cassandra-5.0.4.jar)
2025-06-19 08:55:44.890 | WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.security.ThreadAwareSecurityManager
2025-06-19 08:55:44.890 | WARNING: System::setSecurityManager will be removed in a future release
2025-06-19 08:55:44.897 | INFO [main] 2025-06-19 05:55:44,896 CassandraDaemon.java:602 - Hostname: cassandra1:7000:7001
2025-06-19 08:55:44.897 | INFO [main] 2025-06-19 05:55:44,897 CassandraDaemon.java:609 - JVM vendor/version: OpenJDK 64-Bit Server VM/17.0.15
2025-06-19 08:55:44.899 | INFO [main] 2025-06-19 05:55:44,898 CassandraDaemon.java:610 - Heap size: 3.813GiB/3.813GiB
......
2025-06-19 08:55:15.410 | INFO [main] 2025-06-19 05:55:15,409 AbstractCryptoProvider.java:140 - AmazonCorrettoCryptoProvider health check OK.
2025-06-19 08:55:15.410 | INFO [main] 2025-06-19 05:55:15,410 DatabaseDescriptor.java:548 - DiskAccessMode is standard, indexAccessMode is mmap
2025-06-19 08:55:15.411 | INFO [main] 2025-06-19 05:55:15,410 DatabaseDescriptor.java:587 - Global memtable on-heap threshold is enabled at 976MiB
2025-06-19 08:55:15.411 | INFO [main] 2025-06-19 05:55:15,411 DatabaseDescriptor.java:591 - Global memtable off-heap threshold is enabled at 976MiB
2025-06-19 08:55:15.426 | INFO [main] 2025-06-19 05:55:15,426 DatabaseDescriptor.java:639 - commitlog_disk_access_mode resolved to: mmap
2025-06-19 08:55:15.426 | INFO [main] 2025-06-19 05:55:15,426 DatabaseDescriptor.java:659 - Native transport rate-limiting disabled.
2025-06-19 08:55:15.698 | INFO [main] 2025-06-19 05:55:15,697 GossipingPropertyFileSnitch.java:67 - Unable to load cassandra-topology.properties; compatibility mode disabled
2025-06-19 08:55:15.818 | INFO [main] 2025-06-19 05:55:15,817 JMXServerUtils.java:284 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
2025-06-19 08:55:15.829 | WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-19 08:55:15.829 | WARNING: System::setSecurityManager has been called by org.apache.cassandra.security.ThreadAwareSecurityManager (file:/opt/cassandra/lib/apache-cassandra-5.0.4.jar)
2025-06-19 08:55:15.829 | WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.security.ThreadAwareSecurityManager
2025-06-19 08:55:15.829 | WARNING: System::setSecurityManager will be removed in a future release
2025-06-19 08:55:15.835 | INFO [main] 2025-06-19 05:55:15,834 CassandraDaemon.java:602 - Hostname: cassandra1:7000:7001
2025-06-19 08:55:15.835 | INFO [main] 2025-06-19 05:55:15,834 CassandraDaemon.java:609 - JVM vendor/version: OpenJDK 64-Bit Server VM/17.0.15
2025-06-19 08:55:15.836 | INFO [main] 2025-06-19 05:55:15,836 CassandraDaemon.java:610 - Heap size: 3.813GiB/3.813GiB
2025-06-19 08:55:15.838 | INFO [main] 2025-06-19 05:55:15,837 CassandraDaemon.java:615 - CodeHeap 'non-nmethods' Non-heap memory: init = 2555904(2496K) used = 1333632(1302K) committed = 2555904(2496K) max = 7606272(7428K)
2025-06-19 08:55:15.839 | INFO [main] 2025-06-19 05:55:15,838 CassandraDaemon.java:615 - Metaspace Non-heap memory: init = 0(0K) used = 21956984(21442K) committed = 22282240(21760K) max = -1(-1K)
2025-06-19 08:55:15.839 | INFO [main] 2025-06-19 05:55:15,838 CassandraDaemon.java:615 - CodeHeap 'profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 5859328(5722K) committed = 5898240(5760K) max = 122023936(119164K)
2025-06-19 08:55:15.839 | INFO [main] 2025-06-19 05:55:15,839 CassandraDaemon.java:615 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2654880(2592K) committed = 2818048(2752K) max = 1073741824(1048576K)
2025-06-19 08:55:15.841 | INFO [main] 2025-06-19 05:55:15,839 CassandraDaemon.java:615 - G1 Eden Space Heap memory: init = 2164260864(2113536K) used = 184549376(180224K) committed = 2164260864(2113536K) max = -1(-1K)
2025-06-19 08:55:15.842 | INFO [main] 2025-06-19 05:55:15,841 CassandraDaemon.java:615 - G1 Old Gen Heap memory: init = 1929379840(1884160K) used = 15577088(15212K) committed = 1929379840(1884160K) max = 4093640704(3997696K)
2025-06-19 08:55:15.842 | INFO [main] 2025-06-19 05:55:15,842 CassandraDaemon.java:615 - G1 Survivor Space Heap memory: init = 0(0K) used = 12166080(11880K) committed = 16777216(16384K) max = -1(-1K)
2025-06-19 08:55:15.842 | INFO [main] 2025-06-19 05:55:15,842 CassandraDaemon.java:615 - CodeHeap 'non-profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 1085824(1060K) committed = 2555904(2496K) max = 122028032(119168K)
2025-06-19 08:55:15.843 | INFO [main] 2025-06-19 05:55:15,842 CassandraDaemon.java:617 - Classpath:
...
2025-06-19 08:55:43.302 | CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom(Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents()Z bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop()V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition()V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush(I)V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow(JI)V bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly(I)J bool dontinline = true
2025-06-19 08:55:43.328 | CompileCommand: exclude org/apache/cassandra/utils/JVMStabilityInspector.forceHeapSpaceOomMaybe(Ljava/lang/OutOfMemoryError;)V bool exclude = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/db/NativeDecoratedKey.address()J bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/db/NativeDecoratedKey.length()I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/rows/SerializationHelper;Lorg/apache/cassandra/io/util/DataOutputPlus;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/io/util/Memory.checkBounds(JJ)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/io/util/SafeMemory.checkBounds(JJ)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/io/util/TrackedDataInputPlus.checkCanRead(I)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/net/FrameDecoderWith8bHeader.decode(Ljava/util/Collection;Lorg/apache/cassandra/net/ShareableBytes;I)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/service/reads/repair/RowIteratorMergeListener.applyToPartition(ILjava/util/function/Consumer;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/BloomFilter.indexes(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/BloomFilter.setIndexes(JJIJ[J)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare(Ljava/nio/ByteBuffer;[B)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare([BLjava/nio/ByteBuffer;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/lang/Object;JI)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/Awaitable$AsyncAwaitable.await(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/util/function/Predicate;Lorg/apache/cassandra/utils/concurrent/Awaitable;)Lorg/apache/cassandra/utils/concurrent/Awaitable; bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/Awaitable$AsyncAwaitable.awaitUntil(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/util/function/Predicate;Lorg/apache/cassandra/utils/concurrent/Awaitable;J)Z bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/Awaitable$AsyncAwaitable.register(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/util/function/Predicate;Lorg/apache/cassandra/utils/concurrent/Awaitable;)Lorg/apache/cassandra/utils/concurrent/WaitQueue/Signal; bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/Awaitable$AsyncAwaitable.signalAll(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Lorg/apache/cassandra/utils/concurrent/Awaitable;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/IntrusiveStack.push(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/lang/Object;Lorg/apache/cassandra/utils/concurrent/IntrusiveStack;)Lorg/apache/cassandra/utils/concurrent/IntrusiveStack; bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/IntrusiveStack.push(Ljava/util/function/Function;Lorg/apache/cassandra/utils/concurrent/IntrusiveStack/Setter;Ljava/lang/Object;Lorg/apache/cassandra/utils/concurrent/IntrusiveStack;)Lorg/apache/cassandra/utils/concurrent/IntrusiveStack; bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/ListenerList.push(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/lang/Object;Lorg/apache/cassandra/utils/concurrent/ListenerList;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/concurrent/ListenerList.pushExclusive(Ljava/util/concurrent/atomic/AtomicReferenceFieldUpdater;Ljava/lang/Object;Lorg/apache/cassandra/utils/concurrent/ListenerList;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/memory/BufferPool$LocalPool.tryGetInternal(IZ)Ljava/nio/ByteBuffer; bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI)[B bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/lang/Object;ILorg/apache/cassandra/db/marshal/ValueAccessor;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/nio/ByteBuffer;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLorg/apache/cassandra/io/util/DataOutputPlus;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt32(ILjava/lang/Object;ILorg/apache/cassandra/db/marshal/ValueAccessor;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt32(ILjava/nio/ByteBuffer;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLjava/lang/Object;ILorg/apache/cassandra/db/marshal/ValueAccessor;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLjava/nio/ByteBuffer;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLorg/apache/cassandra/io/util/DataOutputPlus;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt32(ILjava/lang/Object;ILorg/apache/cassandra/db/marshal/ValueAccessor;)I bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt32(ILjava/nio/ByteBuffer;)V bool inline = true
2025-06-19 08:55:43.328 | CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt32(ILorg/apache/cassandra/io/util/DataOutputPlus;)V bool inline = true
2025-06-19 08:55:43.884 | INFO [main] 2025-06-19 05:55:43,881 YamlConfigurationLoader.java:101 - Configuration location: file:/etc/cassandra/cassandra.yaml
2025-06-19 08:55:44.144 | INFO [main] 2025-06-19 05:55:44,144 Config.java:1355 - Node configuration: [...]
2025-06-19 08:55:44.180 | INFO [main] 2025-06-19 05:55:44,180 DatabaseDescriptor.java:1640 - Supported sstable formats are: big -> org.apache.cassandra.io.sstable.format.big.BigFormat with singleton components: [Data.db, Index.db, Statistics.db, CompressionInfo.db, Filter.db, Summary.db, Digest.crc32, CRC.db, TOC.txt], bti -> org.apache.cassandra.io.sstable.format.bti.BtiFormat with singleton components: [Data.db, Partitions.db, Rows.db, Statistics.db, CompressionInfo.db, Filter.db, Digest.crc32, CRC.db, TOC.txt]
2025-06-19 08:55:44.422 | INFO [main] 2025-06-19 05:55:44,422 AbstractCryptoProvider.java:140 - AmazonCorrettoCryptoProvider health check OK.
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,422 DatabaseDescriptor.java:548 - DiskAccessMode is standard, indexAccessMode is mmap
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,422 DatabaseDescriptor.java:587 - Global memtable on-heap threshold is enabled at 976MiB
2025-06-19 08:55:44.423 | INFO [main] 2025-06-19 05:55:44,423 DatabaseDescriptor.java:591 - Global memtable off-heap threshold is enabled at 976MiB
2025-06-19 08:55:44.442 | INFO [main] 2025-06-19 05:55:44,442 DatabaseDescriptor.java:639 - commitlog_disk_access_mode resolved to: mmap
2025-06-19 08:55:44.443 | INFO [main] 2025-06-19 05:55:44,442 DatabaseDescriptor.java:659 - Native transport rate-limiting disabled.
2025-06-19 08:55:44.736 | INFO [main] 2025-06-19 05:55:44,735 GossipingPropertyFileSnitch.java:67 - Unable to load cassandra-topology.properties; compatibility mode disabled
2025-06-19 08:55:44.887 | INFO [main] 2025-06-19 05:55:44,886 JMXServerUtils.java:284 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
2025-06-19 08:55:44.890 | WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-19 08:55:44.890 | WARNING: System::setSecurityManager has been called by org.apache.cassandra.security.ThreadAwareSecurityManager (file:/opt/cassandra/lib/apache-cassandra-5.0.4.jar)
2025-06-19 08:55:44.890 | WARNING: Please consider reporting this to the maintainers of org.apache.cassandra.security.ThreadAwareSecurityManager
2025-06-19 08:55:44.890 | WARNING: System::setSecurityManager will be removed in a future release
2025-06-19 08:55:44.897 | INFO [main] 2025-06-19 05:55:44,896 CassandraDaemon.java:602 - Hostname: cassandra1:7000:7001
2025-06-19 08:55:44.897 | INFO [main] 2025-06-19 05:55:44,897 CassandraDaemon.java:609 - JVM vendor/version: OpenJDK 64-Bit Server VM/17.0.15
2025-06-19 08:55:44.899 | INFO [main] 2025-06-19 05:55:44,898 CassandraDaemon.java:610 - Heap size: 3.813GiB/3.813GiB
2025-06-19 08:55:44.902 | INFO [main] 2025-06-19 05:55:44,900 CassandraDaemon.java:615 - CodeHeap 'non-nmethods' Non-heap memory: init = 2555904(2496K) used = 3094912(3022K) committed = 3145728(3072K) max = 7606272(7428K)
2025-06-19 08:55:44.904 | INFO [main] 2025-06-19 05:55:44,902 CassandraDaemon.java:615 - Metaspace Non-heap memory: init = 0(0K) used = 22420576(21895K) committed = 22806528(22272K) max = -1(-1K)
2025-06-19 08:55:44.904 | INFO [main] 2025-06-19 05:55:44,904 CassandraDaemon.java:615 - CodeHeap 'profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 6096000(5953K) committed = 6160384(6016K) max = 122023936(119164K)
2025-06-19 08:55:44.905 | INFO [main] 2025-06-19 05:55:44,904 CassandraDaemon.java:615 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2704512(2641K) committed = 2883584(2816K) max = 1073741824(1048576K)
2025-06-19 08:55:44.905 | INFO [main] 2025-06-19 05:55:44,904 CassandraDaemon.java:615 - G1 Eden Space Heap memory: init = 2164260864(2113536K) used = 209715200(204800K) committed = 2164260864(2113536K) max = -1(-1K)
2025-06-19 08:55:44.905 | INFO [main] 2025-06-19 05:55:44,904 CassandraDaemon.java:615 - G1 Old Gen Heap memory: init = 1929379840(1884160K) used = 15577088(15212K) committed = 1929379840(1884160K) max = 4093640704(3997696K)
2025-06-19 08:55:44.905 | INFO [main] 2025-06-19 05:55:44,905 CassandraDaemon.java:615 - G1 Survivor Space Heap memory: init = 0(0K) used = 12268536(11980K) committed = 16777216(16384K) max = -1(-1K)
2025-06-19 08:55:44.906 | INFO [main] 2025-06-19 05:55:44,905 CassandraDaemon.java:615 - CodeHeap 'non-profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 1383296(1350K) committed = 2555904(2496K) max = 122028032(119168K)
2025-06-19 08:55:44.906 | INFO [main] 2025-06-19 05:55:44,906 CassandraDaemon.java:617 - Classpath:
...
Victor Tokovenko
(31 rep)
Jun 19, 2025, 11:02 AM
• Last activity: Jul 9, 2025, 07:23 AM
0
votes
1
answers
160
views
Unable to insert large number of records results in timeouts, hung process
We are inserting large records of data into open-source Cassandra database one by one using asp.net core and it result into timeout and process hanged, so I want to know about bulk data insert process if it is supported in Cassandra. Kindly share some workable solution
We are inserting large records of data into open-source Cassandra database one by one using asp.net core and it result into timeout and process hanged, so I want to know about bulk data insert process if it is supported in Cassandra.
Kindly share some workable solution
Rohit Prakash
(1 rep)
May 26, 2023, 12:08 PM
• Last activity: Jul 9, 2025, 06:07 AM
0
votes
2
answers
191
views
Unable to start Cassandra after 3.11 to 4.1 upgrade
I'm trying to upgrade test cassandra cluster from 3.11.17 to 4.1.5, I wanted to do in rolling fashion so upgrade was done on the first node but the node is not is not starting up with the below error. Can you please let me know what is the issue here ``` INFO [Messaging-EventLoop-3-21] 2024-07-30 04...
I'm trying to upgrade test cassandra cluster from 3.11.17 to 4.1.5, I wanted to do in rolling fashion so upgrade was done on the first node but the node is not is not starting up with the below error. Can you please let me know what is the issue here
INFO [Messaging-EventLoop-3-21] 2024-07-30 04:46:26,203 NoSpamLogger.java:105 - /host1:7000->/host2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /host2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Solver
(1 rep)
Jul 30, 2024, 04:55 AM
• Last activity: Jul 8, 2025, 02:05 PM
0
votes
3
answers
194
views
How to restore snapshots to another table?
Someone accidentally deleted several records from Cassandra's table in the cluster. I want to restore the last snapshot to another copy table with the same structure within the cluster. What is the best way to do that? Cassandra v4.0.4
Someone accidentally deleted several records from Cassandra's table in the cluster.
I want to restore the last snapshot to another copy table with the same structure within the cluster.
What is the best way to do that?
Cassandra v4.0.4
EdiM
(51 rep)
May 13, 2024, 07:29 AM
• Last activity: Jul 5, 2025, 02:02 PM
0
votes
1
answers
164
views
How do I upgrade a DataStax Enterprise cluster to 6.8.25 with OpsCenter Lifecycle Manager?
What are the steps (or all the commands) to upgrade (just the 6.8.25 fix, I have 6.18.16 version now) all nodes in cluster with LifeCycle Manager? I've found this but it refers to 6.5 version of DSE: https://docs.datastax.com/eol/en/opscenter/6.5/opsc/LCM/upgradeDSEjob.html
What are the steps (or all the commands) to upgrade (just the 6.8.25 fix, I have 6.18.16 version now) all nodes in cluster with LifeCycle Manager?
I've found this but it refers to 6.5 version of DSE:
https://docs.datastax.com/eol/en/opscenter/6.5/opsc/LCM/upgradeDSEjob.html
dba_db2_dse
(21 rep)
Mar 6, 2023, 07:10 AM
• Last activity: Jul 5, 2025, 12:00 PM
0
votes
0
answers
25
views
Cassandra pod fails to start due to corrupted schema table
We’re currently facing an issue with one of our Cassandra pods in a K8ssandra cluster. One of the pods in the StatefulSet is unable to start due to what appears to be schema corruption. Here’s the error message we’re seeing in the logs: ``` No partition columns found for table ks_sv.test_test in sys...
We’re currently facing an issue with one of our Cassandra pods in a K8ssandra cluster. One of the pods in the StatefulSet is unable to start due to what appears to be schema corruption. Here’s the error message we’re seeing in the logs:
No partition columns found for table ks_sv.test_test in system_schema.columns. This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, restart cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and run the following query to cleanup:
"DELETE FROM system_schema.tables WHERE keyspace_name = 'ks_sv' AND table_name = 'test_test';
DELETE FROM system_schema.columns WHERE keyspace_name = 'ks_sv' AND table_name = 'test_test';"
If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.-Dcassandra.ignore_corrupted_schema_tables=true flag in a K8ssandra-managed environment and run the cleanup query as suggested by the error message. Or should we proceed to restore the entire cluster from a backup?
Any advice or steps from others who’ve encountered similar issues would be greatly appreciated. We’re trying to avoid data loss and ensure cluster stability.
Robbie
(1 rep)
Jul 1, 2025, 12:37 PM
• Last activity: Jul 1, 2025, 12:42 PM
1
votes
1
answers
33
views
What are recycled-commitlogs?
Cassandra/ScyllaDB I have a node with 51000+ recycled commitlogs and want to know if I can just blitz them. I am Linux not Scylla/Cassandra but these are filling up a datadisk. Normal commitlogs appear to be created and used/removed so I am assuming these are from some failed effort somewhere and ma...
Cassandra/ScyllaDB
I have a node with 51000+ recycled commitlogs and want to know if I can just blitz them.
I am Linux not Scylla/Cassandra but these are filling up a datadisk. Normal commitlogs appear to be created and used/removed so I am assuming these are from some failed effort somewhere and may be just copies of existing or commits already run. If these are for commits which have failed and need to be re-done would a restart of the service kickstart a purge?
I cannot find anything relating to recycled versions and am hoping some useful individual out there can shed some techlight.
Hpoonis
(11 rep)
Jun 24, 2025, 12:45 PM
• Last activity: Jun 30, 2025, 02:35 PM
0
votes
1
answers
200
views
Temporal service failed with "Persistent store operation failure", "Failed to update shard"
Hi i'm using Temporal service framework to orchestrate workflow process and cassandra database as datastore to record the workflow stateful process. My temporal is getting error when trying to update shard with detail as follow: ``` { "level": "error", "ts": "2024-02-02T02:54:57.744Z", "msg": "Persi...
Hi i'm using Temporal service framework to orchestrate workflow process and cassandra database as datastore to record the workflow stateful process.
My temporal is getting error when trying to update shard with detail as follow:
{
"level": "error",
"ts": "2024-02-02T02:54:57.744Z",
"msg": "Persistent store operation failure",
"service": "history",
"shard-id": 32,
"address": "10.59.68.74:7234",
"shard-item": "0xc00277ea00",
"store-operation": "update-shard",
"error": "Failed to update shard. previous_range_id: 30415, columns: (range_id=30414)",
"shard-range-id": 30416,
"previous-shard-range-id": 30415,
"logging-call-at": "context_impl.go:732"
}
Ahmad Ricky Nazarrudin
(1 rep)
Mar 26, 2024, 03:29 PM
• Last activity: Jun 27, 2025, 02:04 PM
Showing page 1 of 20 total questions