Sample Header Ad - 728x90

what is a reliable way to push data from postgres into redis queue's?

1 vote
0 answers
870 views
We need a way to process the records in a table asynchronously, we were processing the records in a AFTER INSERT trigger but that is affecting the insert performance and the back pressure is affecting the entire system. So the initial solution we came up with was to use Postgres NOTIFY/LISTEN framework and push the record id's into a Redis queue to process them asynchronously, But we need some guarantee which doesn't seem possible with this, mainly the dropping of the notification if there are no active listeners, in case the listening process crashes for some reason the notifications will be lost, which is a critical problem for us. Periodically reading the table and inserting into redis is also not an option, as the events need to be processed as soon as they come in, and if the interval is too small, that might pit more load on the db. the other solution that could work was to use FDW, but currently available implementations of FDWs for Redis don't seem to support lists. Surprisingly there isn't as much information about Postgres to Redis interfacing, as I thought it would be. Is there a design which can guarantee no loss of events and asynchronous processing?.
Asked by Adithya Sama (111 rep)
Oct 13, 2022, 08:39 AM