Sample Header Ad - 728x90

MariaDB Galera Cluster (10.6.12) and Laravel WSREP issue

0 votes
1 answer
470 views
I'm administrating a galera cluster with 3 nodes, serving a Laravel application and HAProxy is the gateway to the cluster. We had an interesting issue, and I'm unable to figure out why was there an error message in the Laravel application. Here is what I know: At 13:25:36 there was a note on Node1, that ... connection to peer f3e3XXXX-XXX with addr tcp://X.X.X.X:4567 timed out, no messages seen in PT3S, ... This IP represents Node2. Roughly the same time 13:25:37, on Node2, the following message appeared in the logs ... WSREP: (f3e3XXXX-XXX, 'tcp://X.X.X.X:4567') turning message relay requesting on, nonlive peers: tcp://X.X.X.X:4567 ... This 2nd IP represent Node1. I'll post a more detailed log later in the post. So my understanding is there could have been a network hickup between the clusters and Node2 (and 3) had to rejoin the cluster. Meanwhile, someone triggered the Larave application that made a query towards the cluster. But this request ended up in an error message saying: [2023-11-07 13:25:46] production.ERROR: SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use ..... If my understanding how a cluster should work is correct, I have no clue what happened. Isn't the galera cluster supposed to be able to manage if a node disappears from the cluster and use an existing node to handle the incoming query? Was this a HAProxy issue maybe and it could not detect that Node2 and 3 are unavailable and sent the query there eitherway? This issue stayed for about 10 seconds and the cluster is working just fine ever since (I think... wsrep_cluster_size is 3 ATM so it must be OK). Now I don't want to "*spam*" this post with all the logs, so I'll insert the logs from Node1 and 2 into pastebin and URLs are provided. !!! Can anyone explain me what happened here in this situation exactly? Maybe a way to avoid this in the future? !!! Thank you in advance ! **MariaDB logs:** https://pastebin.com/SpDbe7Bb **Laravel logs:** ` [2023-11-07 13:25:46] production.ERROR: SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use (SQL: select * from .......) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 08S01): SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use (SQL: select * from .......) at /var/www/html/myproject/releases/184/vendor/laravel/framework/src/Illuminate/Database/Connection.php:671) `
Asked by Imre Bertalan (105 rep)
Nov 8, 2023, 11:09 AM
Last activity: Nov 8, 2023, 11:45 AM