There are chances I selected the wrong tool for this task, feel free to suggest if you know a better one.
The task: I need to store key-value pairs.
* Each pair is 60 bytes. The keys are IDs and values - both are pretty random or at least unstructured.
* There are ~2 milliards of such pairs which makes ~110 GB of pure data in total. Good chances for growth in the future.
* The write load is heavy, and the read one is rather light.
* It'd be nice to have a performance at 1K IOPS for writing, but maybe it's just a dream and I'll have to go with something slower but not so expensive.
* I can batch writing, but the keys won't be sequential (like
123,7,13565
instead of 1,2,3
)
* No fancy search is needed, just give the value for the complete key.
* I'm on AWS, if it matters, but can switch for a really good solution.
* Cost matters.
Redis is a key-value store, so I thought to use it but keeping such a big DB in memory is cost-prohibitive so I'd like to configure Redis in a way where it will take the data from memory as a cache, and when there's a cache miss - from the disk.
So, it is not about Redis's persistence as a backup.
Besides Redis I have tried:
* plain files in directory tree, like key='abcdef' => ab/cd/ef
. Ext2, BtrFS, tried to distribute writes across 16 partitions (terrible performance after ~0.5M pairs)
* MySQL (silently died)
Also I thought about AWS Elasticache Redis with Data Tiering, but the cheapest instance has insane cost for me (~$600/mo)
How can I achieve that?
Asked by Putnik
(295 rep)
Jul 26, 2023, 12:49 PM
Last activity: Jul 27, 2023, 12:51 PM
Last activity: Jul 27, 2023, 12:51 PM