Sample Header Ad - 728x90

Fast Select from a table with ~300 columns with about 200 mil static rows

0 votes
1 answer
394 views
I have a table with around 300+ columns which contain Demographics data with various attributes like Name, Address, State, Zipcode, Sex, education etc. The table has around 200mil rows. The table never gets updates or inserts and the data is always static unless it is refreshed, which happens once in 6 months, for which we delete the entire table and re-import it from AWS S3 bucket. Right now, the table is on Postgresql 13.6 which is launched on AWS RDS (Aurora Serverless). The only operation that is performed on this table/database is Select which is extremely slow (5-8 mins) when 4 or more conditions are applied. There is one index applied and the table is partitioned, by range, into 22 tables using Zip as range. All columns have datatype as text. i have tried using appropriate data types but there is no improvement in query time. Explain Analyse always shows sequential scan even when index scan can be used.
AWS Aurora size : Serverless v2 (30 - 128 ACUs)
Engine version : 13.6
All other AWS parameters set to default
I need help in determining if there would be a better approach to storing this data for milliseconds access and whether going with postgresql on AWS was the right choice. Unfortunately I cannot paste the full table structure or any example query. **Update** : Added table schema.I have removed some columns, but they are of the same type. please follow the below link (Thanks @Vérace) https://paste.depesz.com/s/bn Select count(*) from pubdata this query takes 11 seconds and uses index scan count : 255116641 the below query uses only 3 parallel workers and takes about 5 mins (which is unacceptable it seems) https://explain.depesz.com/s/1Gam **Update 2** I have made the following changes to PostgreSQL parameters
max_worker_processes : 20
max_parallel_workers : 10
max_parallel_workers_per_gather : 10
force_parallel_mode : true
I am still not looking at significant performance improvements. the RDS Aurora minimum config is 30ACUs (60 GiB RAM) and can go upto 128 ACUs (256 GiB)
Asked by Rohit Agre (101 rep)
Oct 30, 2022, 02:55 PM
Last activity: Feb 3, 2025, 11:07 PM