Sample Header Ad - 728x90

pgpool-II connection pooling (only) but for long running transactions, can it even be done?

-1 votes
1 answer
121 views
I have a job queue management system where workers fetch a job from a queue and maintain that job record locked, they use save-points to make other database changes and, on error, roll back to that save-point still holding on to the locked job record. Then finally update the job result/status and commit. Since I sometimes need hundreds of workers, I am running into the problem of PgSQL database connection limits (I have 100 configured right now, but that scratches the limits of memory, etc.) So I thought to use pgpool-II to have more connections. Until I considered the fact that they all hold onto a locked record, or any transaction. And so the problem/question is, would pgpool-II even help here or would it not keep the connection locked while a transaction is running? In that case I would have to change my logic, not to lock the job records but update them. Right? Any other idea? Why does PgSQL require so much memory for each possible connection anyway, if my connections actually don't do very much most of the time?
Asked by Gunther Schadow (523 rep)
Mar 24, 2024, 11:47 PM
Last activity: Mar 25, 2024, 03:40 PM