Sample Header Ad - 728x90

Cassandra 4.1.4 frequent timeouts/node failures

1 vote
0 answers
36 views
We have recently migrated our cassandra DB from one DC to another DC, and with this migration we upgraded the cassandra version too from 4.0.9 to 4.1.4. However, after the migration we are experiencing frequent timeouts with the nodes in the cluster. whenever the issue occurs the logs fill with the below errors and node is going into un responsive state. In the logs we found the below errors: - Operation timed out - received only 0 responses." while executing SELECT permissions FROM system_auth.role_permissions - 7000-SMALL_MESSAGES-49749d9d dropping message of type READ_REQ whose timeout expired before reaching the network - 7000-SMALL_MESSAGES-49749d9d dropping message of type MUTATION_REQ whose timeout expired before reaching the network But when we check the cassandra service status, it is showing active. Whenever this issue occurs, as a work around we are restarting the effected node and the issue is getting resolved. The mentioned timeout is occurring very randomly among all the nodes and at random timings. below is the DB config: Single cluster with 7 nodes total RAM - 128 GB allocated heap 64 GB CPU - 16core Please help us solving this issue.
Asked by Ram chandra (11 rep)
Apr 15, 2025, 02:48 PM
Last activity: Apr 15, 2025, 02:49 PM