Sample Header Ad - 728x90

How to handle ~1k inserts per second

1 vote
3 answers
14200 views
Assuming one has about 1k requests per second that require an insert. Now, there are a lot of answers to this on the internet... but they're technically wrong in this specific context. Yes, pretty much any RDBMS can handle 1k inserts per second on standard hardware but IF AND ONLY IF you drop ACID guarantees. It's surprising how many terrible answers there are on the internet. Such as "you can always scale up CPU and RAM" which is supposed to give you more inserts per second but that's not how it works. The limiting factor is disk speed or more precise: how many transactions you can actually flush/sync to disk. And this is the tricky bit. On decent "commodity hardware" (unless you invest into high performance SSDs) this is about what you can expect: * SQLite: 30 inserts/s * MySQL: 80 inserts/s This is the rate you can insert while maintaining ACID guarantees. This essentially means that if you have a forum with 100 posts per second... you can't handle that with such a setup. Read requests aren't the problem. You can have many thousands of read requests per second no problem but write requests are usually <100 per second. Thus, this question is specifically aimed at how one can handle 1k inserts per second while still maintaining ACID guarantees - assuming that a single node can handle about 80 transactions per second. One way I could see this working is if you buffer inserts somewhere in your application logic and submit them as larger transactions to the database (while keeping clients waiting until the transaction is over) which should work fine if you need single inserts only although it complicates the application logic quite a bit.
Asked by mroman (137 rep)
Jul 17, 2018, 03:15 PM
Last activity: Sep 10, 2023, 05:22 PM