Sample Header Ad - 728x90

How to make Redis work from disk? Not about persistence

1 vote
1 answer
741 views
There are chances I selected the wrong tool for this task, feel free to suggest if you know a better one. The task: I need to store key-value pairs. * Each pair is 60 bytes. The keys are IDs and values - both are pretty random or at least unstructured. * There are ~2 milliards of such pairs which makes ~110 GB of pure data in total. Good chances for growth in the future. * The write load is heavy, and the read one is rather light. * It'd be nice to have a performance at 1K IOPS for writing, but maybe it's just a dream and I'll have to go with something slower but not so expensive. * I can batch writing, but the keys won't be sequential (like 123,7,13565 instead of 1,2,3) * No fancy search is needed, just give the value for the complete key. * I'm on AWS, if it matters, but can switch for a really good solution. * Cost matters. Redis is a key-value store, so I thought to use it but keeping such a big DB in memory is cost-prohibitive so I'd like to configure Redis in a way where it will take the data from memory as a cache, and when there's a cache miss - from the disk. So, it is not about Redis's persistence as a backup. Besides Redis I have tried: * plain files in directory tree, like key='abcdef' => ab/cd/ef. Ext2, BtrFS, tried to distribute writes across 16 partitions (terrible performance after ~0.5M pairs) * MySQL (silently died) Also I thought about AWS Elasticache Redis with Data Tiering, but the cheapest instance has insane cost for me (~$600/mo) How can I achieve that?
Asked by Putnik (295 rep)
Jul 26, 2023, 12:49 PM
Last activity: Jul 27, 2023, 12:51 PM