Sample Header Ad - 728x90

Metastore(Mysql) bottleneck for Hive

1 vote
1 answer
287 views
We have a hive installation that has MariaDB as metastore database. MariaDB has around ~250 GB metadata with ~100GB indexes. It becomes terribly slow during the peak load of 40-60K QPS. Looking from the community to share similar experiences if any and what they did to scale out the meta store or fix it? Some of the ideas i am looking at currently are: - Application Caching at HMS level: Didn't found out of box capability in my current v2.0.1. Is there support for it in higher versions? - Read replicas and routing SELECTS to it: Facing some failure if there is replication lag and i am trying to read back the value. - Horizontal sharding of Mysql: founding it way complex. Saw some recommendations of TiDB but not sure of its experience.
Asked by Shakti Garg (111 rep)
Feb 16, 2023, 02:48 PM
Last activity: Feb 18, 2023, 10:11 AM