Sample Header Ad - 728x90

Tuning UPSERT in PG14 w/ 130M records table

1 vote
2 answers
56 views
I'm tuning the server in order to speed up several SQL queries generated by a product and that can't be modified or tuned themselves. I'm stuck on a query that creates a CTO an then an INSERT / SELECT statement w/ a ON CONFLICT clause, this is the execution plan : Insert on data_location.md_associe_interlo_coord_red t (cost=0.57..592368.31 rows=0 width=0) (actual rows=0 loops=1) Conflict Resolution: UPDATE Conflict Arbiter Indexes: pkmd_associe_interlo_coord_red Conflict Filter: (t.b_batchid Index Scan using i2mi_associe_interlo_coord_red on data_location.mi_associe_interlo_coord_red s (cost=0.57..592368.31 rows=1343311 width=1187) (actual rows=1341390 loops=1) Output: s.b_pubid, s.b_sourceid, s.b_classname, s.b_batchid, (COALESCE(s.b_credate, '2025-08-01 08:43:28.853'::timestamp without time zone))::timestamp(6) without time zone, (COALESCE(s.b_upddate, '2025-08-01 08:43:28.853'::timestamp without time zone))::timestamp(6) without time zone, COALESCE(s.b_creator, 'semadmin'::character varying), COALESCE(s.b_updator, 'semadmin'::character varying), s.b_matchgrp, s.b_xgrp, s.b_confirmationstatus, NULL::numeric, NULL::character varying, s.b_hassuggmerge, s.b_suggmergeid, s.b_suggmergeconfscore, NULL::character varying, s.b_suggmergemasterscount, s.associe_interlo_coord_red, NULL::numeric, s.b_confirmedsdpk, s.num_compte_externe, s.siret, s.siren, s.nir, NULL::date, s.type_interlocuteur, NULL::integer, s.f_redevabilite, s.fp_redevabilite, s.fs_redevabilite, s.f_interlocuteur, s.fp_interlocuteur, s.fs_interlocuteur, s.f_coordonnee, s.fp_coordonnee, s.fs_coordonnee, s.id_golden_redevabilite, NULL::integer, s.dt_maj, s.code_organisme_source Index Cond: (s.b_batchid = '52615'::numeric) Filter: ((s.b_classname)::text = 'AssocieInterloCoordRede'::text) Buffers: shared hit=9 read=239313 written=57 Settings: effective_cache_size = '43GB', effective_io_concurrency = '50', max_parallel_workers = '4', random_page_cost = '2', search_path = '"$user", public, extensions', work_mem = '1GB' Table mi_associe_interlo_coord_red has +131M records (+140GB on disk) and md_associe_interlo_coord_red +129M records has (+81GB on disk). Colonne | Type | Collationnement | NULL-able | Par défaut | Stockage | Compression | Cible de statistiques | Description ---------------------------+--------------------------------+-----------------+-----------+------------+----------+-------------+-----------------------+------------- b_pubid | character varying(30) | | not null | | extended | | | b_sourceid | character varying(128) | | not null | | extended | | | b_classname | character varying(128) | | not null | | extended | | | b_batchid | numeric(38,0) | | not null | | main | | | b_credate | timestamp(6) without time zone | | | | plain | | | b_upddate | timestamp(6) without time zone | | | | plain | | | b_creator | character varying(128) | | | | extended | | | b_updator | character varying(128) | | | | extended | | | b_pendingactions | character varying(30) | | | | extended | | | b_matchgrp | numeric(38,0) | | | | main | | | b_oldmatchgrp | numeric(38,0) | | | | main | | | b_xgrp | uuid | | | | plain | | | b_confirmationstatus | character varying(30) | | | | extended | | | b_confscore | numeric(38,0) | | | | main | | | b_confscoretype | character varying(30) | | | | extended | | | b_hassuggmerge | boolean | | | | plain | | | b_suggmergeid | numeric(38,0) | | | | main | | | b_suggmergeconfscore | numeric(38,0) | | | | main | | | b_suggmergeconfscoretype | character varying(30) | | | | extended | | | b_suggmergemasterscount | numeric(38,0) | | | | main | | | associe_interlo_coord_red | numeric(38,0) | | | | main | | | b_oldsdpk | numeric(38,0) | | | | main | | | b_confirmedsdpk | numeric(38,0) | | | | main | | | num_compte_externe | character varying(18) | | | | extended | | | siret | character varying(14) | | | | extended | | | siren | character varying(9) | | | | extended | | | nir | character varying(13) | | | | extended | | | dtmaj | date | | | | plain | | | type_interlocuteur | character varying(128) | | | | extended | | | f_redevabilite | numeric(38,0) | | | | main | | | fp_redevabilite | character varying(30) | | | | extended | | | fs_redevabilite | character varying(128) | | | | extended | | | f_interlocuteur | numeric(38,0) | | | | main | | | fp_interlocuteur | character varying(30) | | | | extended | | | fs_interlocuteur | character varying(128) | | | | extended | | | f_coordonnee | numeric(38,0) | | | | main | | | fp_coordonnee | character varying(30) | | | | extended | | | fs_coordonnee | character varying(128) | | | | extended | | | id_golden_redevabilite | character varying(128) | | | | extended | | | dt_maj | date | | | | plain | | | code_organisme_source | character varying(3) | | | | extended | | | Index : "pkmi_associe_interlo_coord_red" PRIMARY KEY, btree (b_pubid, b_sourceid) "fkmi_associe_interlocuteur_re2" btree (fp_interlocuteur, fs_interlocuteur) "fkmi_associe_interlocuteur_re3" btree (fp_coordonnee, fs_coordonnee) "fkmi_associe_interlocuteur_red" btree (fp_redevabilite, fs_redevabilite) "i1mi_associe_interlo_coord_red" btree (associe_interlo_coord_red) "i2mi_associe_interlo_coord_red" btree (b_batchid) "i3mi_associe_interlo_coord_red" btree (b_matchgrp) "i4mi_associe_interlo_coord_red" btree (b_suggmergeid) "i5mi_associe_interlo_coord_red" btree (b_xgrp) "usr_idx_mi_associe_interlocuteur_red_b_pendingactions" btree (b_pendingactions) "usr_mi_associe_interlo_coord_red_b_batchid_idx" btree (b_batchid, b_classname, b_pubid, b_sourceid, fp_coordonnee, fs_coordonnee, fp_interlocuteur, fs_interlocuteur, fp_redevabilite, fs_redevabilite) "usr_mi_associe_interlo_coord_red_fp_fs_type" btree (fp_interlocuteur, fs_interlocuteur, type_interlocuteur) "usr_mi_associe_interlo_coord_red_id_golden_redevabilite_idx" btree (id_golden_redevabilite, type_interlocuteur) Méthode d'accès : heap Options: autovacuum_enabled=true, fillfactor=70 Colonne | Type | Collationnement | NULL-able | Par défaut | Stockage | Compression | Cible de statistiques | Description ---------------------------+--------------------------------+-----------------+-----------+------------+----------+-------------+-----------------------+------------- b_pubid | character varying(30) | | not null | | extended | | | b_sourceid | character varying(128) | | not null | | extended | | | b_classname | character varying(128) | | not null | | extended | | | b_batchid | numeric(38,0) | | not null | | main | | | b_credate | timestamp(6) without time zone | | | | plain | | | b_upddate | timestamp(6) without time zone | | | | plain | | | b_creator | character varying(128) | | | | extended | | | b_updator | character varying(128) | | | | extended | | | b_matchgrp | numeric(38,0) | | | | main | | | b_xgrp | uuid | | | | plain | | | b_confirmationstatus | character varying(30) | | | | extended | | | b_confscore | numeric(38,0) | | | | main | | | b_confscoretype | character varying(30) | | | | extended | | | b_hassuggmerge | boolean | | | | plain | | | b_suggmergeid | numeric(38,0) | | | | main | | | b_suggmergeconfscore | numeric(38,0) | | | | main | | | b_suggmergeconfscoretype | character varying(30) | | | | extended | | | b_suggmergemasterscount | numeric(38,0) | | | | main | | | associe_interlo_coord_red | numeric(38,0) | | not null | | main | | | b_oldsdpk | numeric(38,0) | | | | main | | | b_confirmedsdpk | numeric(38,0) | | | | main | | | num_compte_externe | character varying(18) | | | | extended | | | siret | character varying(14) | | | | extended | | | siren | character varying(9) | | | | extended | | | nir | character varying(13) | | | | extended | | | dtmaj | date | | | | plain | | | type_interlocuteur | character varying(128) | | | | extended | | | f_redevabilite | numeric(38,0) | | | | main | | | fp_redevabilite | character varying(30) | | | | extended | | | fs_redevabilite | character varying(128) | | | | extended | | | f_interlocuteur | numeric(38,0) | | | | main | | | fp_interlocuteur | character varying(30) | | | | extended | | | fs_interlocuteur | character varying(128) | | | | extended | | | f_coordonnee | numeric(38,0) | | | | main | | | fp_coordonnee | character varying(30) | | | | extended | | | fs_coordonnee | character varying(128) | | | | extended | | | id_golden_redevabilite | character varying(128) | | | | extended | | | dt_maj | date | | | | plain | | | code_organisme_source | character varying(3) | | | | extended | | | Index : "pkmd_associe_interlo_coord_red" PRIMARY KEY, btree (b_pubid, b_sourceid) "fkmd_associe_interlocuteur_re2" btree (fp_interlocuteur, fs_interlocuteur) "fkmd_associe_interlocuteur_re3" btree (fp_coordonnee, fs_coordonnee) "fkmd_associe_interlocuteur_red" btree (fp_redevabilite, fs_redevabilite) "i1md_associe_interlo_coord_red" btree (associe_interlo_coord_red) "i2md_associe_interlo_coord_red" btree (b_suggmergeid) "usr_idx_md_associe_interlocuteur_red_b_pendingactions" btree (b_batchid) Méthode d'accès : heap Server has 64GB RAM and 16 vCores, concurrency is low, typcally 3/4 postgres process running. Workload is mostly huge batches, and very little OLTP. Adding memory is not an option because of the company policies Create a covering index is not an option because of the limit of 27 for columns in INCLUDE clause. I tried to recreate the PK by including the b_batchid column (to avoid heap read while handling the conflict I thought) but it didn't help How does it handle the conflict ? why the cost of the upsert is 0?
Asked by Ivan Rododendro (31 rep)
Aug 7, 2025, 12:13 PM
Last activity: Aug 9, 2025, 05:49 PM