Sample Header Ad - 728x90

Select query with 90+ parameters intermittently 20x slower on RDS PostgreSQL instance when using connection pool

0 votes
1 answer
188 views
I'm running a select statement on a postgresql table (on an RDS instance) in order to sum up values across 2 mil rows. The select statement has about 100 parameters (mostly different countries we want to sum across). When I run this query once from my application it'll execute in roughly 500ms. However when I execute it several times in a row, the query latency will suddenly increase by _10-20 times_ to roughly 5 to 10 seconds after 8 or 9 runs. (These measurements are server side from the RDS logs where I've enabled log_min_duration_statement). **What could possibly cause this jump in execution time?** -- I'd expect the query to be slower on the initial run, but I'm surprised to see it increase in latency after several consequtive runs. A few key observations: * **Connection Pool**: The exact connection pool doesn't seem to matter. The problem persists whether I use [HikariCP](https://github.com/brettwooldridge/HikariCP) or [C3P0](https://www.mchange.com/projects/c3p0/) * **Single Connection**: This issue _doesn't_ happen if I configure my connection pool to have only a single connection (so the application logic is not at fault) * **PSQL**: This issue also doesn't happen if I benchmark the query consequetively using psql to execute it. * **Query Parameters**: When I reduce the amount of query parameters to ~60 I don't see any issues. It's only when I query for 90+ territories that I run in to this behaviour. * **Pool Reset**: When I restart my application (and hence reset the connection pool), the queries go back to being fast (0.5s) for another 8-9 queries before being slow again (5s). I've tried to look out for the usual suspects like RDS IOPS Burst throttling, database load, and memory consumption, available connections, but nothing looks suspicious on this front. I've also made sure that the index the query is using and the rows returned from the table are cached. The query I'm running has the following shape. I've truncated some of the 90 country IDs:
SELECT d.primary_title_no,
       SUM(d.cume) AS lifetime
  FROM schema.my_data AS d 
 WHERE d.currency = 'USD'
   AND d.date >= '2020-01-01'::date + (7 * (d.week - 1))
   AND d.date   Sort  (cost=1218692.98..1218697.34 rows=1742 width=12) (actual time=369.145..369.154 rows=100 loops=1)                                                                                                                                                                                                                                                                           |
|         Sort Key: (sum(cume)) DESC                                                                                                                                                                                                                                                                                                                                                     |
|         Sort Method: top-N heapsort  Memory: 33kB                                                                                                                                                                                                                                                                                                                                      |
|         ->  GroupAggregate  (cost=1218595.34..1218626.41 rows=1742 width=12) (actual time=366.250..368.653 rows=1959 loops=1)                                                                                                                                                                                                                                                          |
|               Group Key: primary_title_no                                                                                                                                                                                                                                                                                                                                              |
|               ->  Sort  (cost=1218595.34..1218599.89 rows=1820 width=12) (actual time=366.240..366.844 rows=5739 loops=1)                                                                                                                                                                                                                                                              |
|                     Sort Key: primary_title_no                                                                                                                                                                                                                                                                                                                                         |
|                     Sort Method: quicksort  Memory: 462kB                                                                                                                                                                                                                                                                                                                              |
|                     ->  Index Scan using idx_dailies_currency_ter_id_primary_title_no_lifetime on dailies d  (cost=0.43..1218496.79 rows=1820 width=12) (actual time=1.093..364.406 rows=5739 loops=1)                                                                                                                                                                                 |
|                           Index Cond: (((currency)::text = 'USD'::text) AND ((ter_id)::text = ANY ('{AE,AM,AR,AT,AU,AZ,BA,BE,BG,BH,BO,BR,BY,CH,CL,CN,CO,CR,CU,CZ,DE,DK,DO,EC,EE,EG,ES,FI,FR,GE,GR,GT,HK,HN,HR,HU,IL,IN,IQ,IS,IT,JP,KG,KR,KW,KZ,LB,LT,LU,LV,MD,MX,MY,MZ,NI,NL,NO,NZ,OM,PA,PE,PH,PL,PT,PY,QA,RO,RS,RU,SA,SE,SG,SI,SK,SV,TH,TJ,TM,TR,TT,TW,UK,UP,UY,UZ,WA,ZA}'::text[]))) |
|                           Filter: (((date - (7 * (week - 1))) >= '2020-01-01'::date) AND ((date - (7 * (week - 1))) < '2020-06-01'::date))                                                                                                                                                                                                                                             |
|                           Rows Removed by Filter: 314494                                                                                                                                                                                                                                                                                                                               |
| Planning Time: 0.613 ms                                                                                                                                                                                                                                                                                                                                                                |
| Execution Time: 369.213 ms                                                                                                                                                                                                                                                                                                                                                             |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
For a case where the query executes slowly the explain analyze output is the same, but 20 times the compute is spent in the innermost Index Scan step. If anybody has any pointers to how I can troubleshoot this issue, I'd be all ears.
Asked by Arnfred (101 rep)
Feb 15, 2024, 06:01 PM
Last activity: Feb 19, 2024, 09:31 AM