mariadb galera : member count keeps changing between 5 and 3

0 votes
1 answer
231 views
                          I have a 5 nodes cluster, 2 nodes per DC, and an arbiter on a 3rd DC.

The cluster is running fine in the sense that data is synced, DBs don't have any errors.
The network is fine too.

However, in the logs, galera shows members leaving and rejoining the cluster every few hours. The same behaviour is observed on, it would be good to unuderstand.

    2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
    2022-10-10 11:33:39 2 [Note] WSREP: Server status change synced -> connected
    2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    2022-10-10 11:33:39 0 [Note] WSREP: view(view_id(NON_PRIM,599bc3d2-b9fe,13720) memb {
            599bc3d2-b9fe,0
    } joined {
    } left {
    } partitioned {
            37d9a4cb-997b,0
            47b6cf8d-9d2a,0
            5de8a52e-aa58,0
            d5464472-acc7,0
    })
    2022-10-10 11:33:39 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2022-10-10 11:33:39 0 [Note] WSREP: Flow-control interval: [16, 16]
    2022-10-10 11:33:39 0 [Note] WSREP: Received NON-PRIMARY.
    2022-10-10 11:33:39 2 [Note] WSREP: ================================================
    View:
      id: 2e8c4b3d-3d2c-11e9-ba33-9f9156834d83:50217575
      status: non-primary
      protocol_version: 4
      capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERE
    D, STREAMING, NBO
      final: no
      own_index: 0
      members(1):
            0: 599bc3d2-444c-11ed-b9fe-93ddbbf65238, p1mariadb1a
    =================================================
    2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
    2022-10-10 11:33:39 2 [Note] WSREP: Server status change connected -> connected

# 5 nodes coming together

    2022-10-10 11:33:40 0 [Note] WSREP: re-bootstrapping prim from partitioned components
    2022-10-10 11:33:40 0 [Note] WSREP: view(view_id(PRIM,37d9a4cb-997b,13721) memb {
            37d9a4cb-997b,0
            47b6cf8d-9d2a,0
            599bc3d2-b9fe,0
            5de8a52e-aa58,0
            d5464472-acc7,0
    } joined {
    } left {
    } partitioned {
    })

2022-10-10 11:33:40 0 [Warning] WSREP: Quorum: No node with complete state:
....
# cluster ok again:

    2022-10-10 11:33:40 0 [Note] WSREP: Full re-merge of primary xx found: 5 of 5.
    2022-10-10 11:33:40 0 [Note] WSREP: Quorum results:
            version    = 6,
            component  = PRIMARY,
            conf_id    = 7760,
            members    = 5/5 (joined/total),


sudo zgrep "members    =" /var/log/mysql/error.log-202210*

    /var/log/mysql/error.log-20221009.gz:   members    = 4/4 (joined/total),
    /var/log/mysql/error.log-20221009.gz:   members    = 4/5 (joined/total),
    /var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
    /var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
    /var/log/mysql/error.log-20221011.gz:   members    = 3/5 (joined/total),

show status where variable_name in ('wsrep_cluster_size','wsrep_cluster_status','wsrep_flow_control_paused','wsrep_ready','wsrep_connected','wsrep_local_state_comment');

    | Variable_name             | Value      |
    | wsrep_flow_control_paused | 0.00954013 |
    | wsrep_local_state_comment | Synced     |
    | wsrep_cluster_size        | 5          |
    | wsrep_cluster_status      | Primary    |
    | wsrep_connected           | ON         |
    | wsrep_ready               | ON         |



                        
Asked by boran (101 rep)
Oct 11, 2022, 06:52 AM
Last activity: Jun 13, 2025, 04:06 PM
mariadb galera : member count keeps changing between 5 and 3

Related Questions
