Configuring PostgreSQL for read performance
49
votes
2
answers
57385
views
Our system writes a lots of data (kind of Big Data system). Write performance is good enough for our needs but read performance is really too slow.
The primary key (constraint) structure is similar for all our tables:
timestamp(Timestamp) ; index(smallint) ; key(integer).
A table can have millions of rows, even billions of rows, and a read request is usually for a specific period (timestamp / index) and tag. It's common to have a query that returns around 200k lines. Currently, we can read about 15k lines per second but we need to be 10 times faster. Is this possible and if so, how?
**Note:** PostgreSQL is packaged with our software, so the hardware is different from one client to another.
It is a VM used for testing. The VM's host is Windows Server 2008 R2 x64 with 24.0 GB of RAM.
##Server Spec (Virtual Machine VMWare)
Server 2008 R2 x64
2.00 GB of memory
Intel Xeon W3520 @ 2.67GHz (2 cores)
###
postgresql.conf
optimisations
shared_buffers = 512MB (default: 32MB)
effective_cache_size = 1024MB (default: 128MB)
checkpoint_segment = 32 (default: 3)
checkpoint_completion_target = 0.9 (default: 0.5)
default_statistics_target = 1000 (default: 100)
work_mem = 100MB (default: 1MB)
maintainance_work_mem = 256MB (default: 16MB)
###Table Definition
CREATE TABLE "AnalogTransition"
(
"KeyTag" integer NOT NULL,
"Timestamp" timestamp with time zone NOT NULL,
"TimestampQuality" smallint,
"TimestampIndex" smallint NOT NULL,
"Value" numeric,
"Quality" boolean,
"QualityFlags" smallint,
"UpdateTimestamp" timestamp without time zone, -- (UTC)
CONSTRAINT "PK_AnalogTransition" PRIMARY KEY ("Timestamp" , "TimestampIndex" , "KeyTag" ),
CONSTRAINT "FK_AnalogTransition_Tag" FOREIGN KEY ("KeyTag")
REFERENCES "Tag" ("Key") MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
WITH (
OIDS=FALSE,
autovacuum_enabled=true
);
###Query
The query takes about 30 seconds to execute in pgAdmin3, but we would like to have the same result under 5 seconds if possible.
SELECT
"AnalogTransition"."KeyTag",
"AnalogTransition"."Timestamp" AT TIME ZONE 'UTC',
"AnalogTransition"."TimestampQuality",
"AnalogTransition"."TimestampIndex",
"AnalogTransition"."Value",
"AnalogTransition"."Quality",
"AnalogTransition"."QualityFlags",
"AnalogTransition"."UpdateTimestamp"
FROM "AnalogTransition"
WHERE "AnalogTransition"."Timestamp" >= '2013-05-16 00:00:00.000' AND "AnalogTransition"."Timestamp" Index Scan Backward using "PK_AnalogTransition" on "AnalogTransition" (cost=0.00..389244.53 rows=1548698 width=33) (actual time=2.187..1893.283 rows=500000 loops=1)"
" Index Cond: (("Timestamp" >= '2013-05-16 01:00:00-04'::timestamp with time zone) AND ("Timestamp" Index Scan using "PK_AnalogTransition" on "AnalogTransition" (cost=0.00..971400.46 rows=774511 width=35) (actual time=0.037..410088.960 rows=250001 loops=1)"
" Index Cond: (("Timestamp" >= '2013-05-22 20:00:00-04'::timestamp with time zone) AND ("Timestamp" <= '2013-05-24 20:00:00-04'::timestamp with time zone) AND ("KeyTag" = 16))"
"Total runtime: 411044.175 ms"
Asked by JPelletier
(593 rep)
May 15, 2013, 06:32 PM
Last activity: Apr 22, 2024, 05:21 PM
Last activity: Apr 22, 2024, 05:21 PM