Sample Header Ad - 728x90

convert RDS csvlog into pgreplay-go compatible for replaying

0 votes
1 answer
322 views
Before upgrading AWS RDS PostgreSQL version at production, I want to replay 1-hour recorded logs at the test instance. Unfortunately [pgreplay](https://github.com/laurenz/pgreplay) is single-threaded application, limited by single CPU speed and its replays are failing with some weird errors after a few minutes working. On the other side, [pgreplay-go](https://github.com/gocardless/pgreplay-go) requires a different input log format: log_line_prefix='%m|%u|%d|%c| RDS csvlog sample:
2021-09-09 17:00:00.006 UTC,"user","database",27752,"172.30.1.2:34106",613a286d.6c68,13992,
"SELECT",2021-09-09 15:29:49 UTC,229/3866470,0,LOG,00000,
"execute : SELECT ""jobs"".* FROM ""jobs"" WHERE ""jobs"".""deleted_at"" IS NULL
AND ""jobs"".""user_id"" = $1","parameters: $1 = '124765'",,,,,,,,"bin/rails"
pgreplay-go log sample:
2010-12-31 10:59:57.870 UTC|postgres|postgres|4d1db7a8.4227|
LOG:  execute einf"ug: INSERT INTO runtest (id, c, t, b) VALUES ($1, $2, $3, $4)
2010-12-31 10:59:57.870 UTC|postgres|postgres|4d1db7a8.4227|
DETAIL:  parameters: $1 = '6', $2 = 'mit    Tabulator', $3 = '2050-03-31 22:00:00+00', $4 = NULL
Should I just convert one RDS csvlog line into two pgreplay-go lines and that's it? Log parsing logic in pgreplay-go is not trivial as you can see at https://github.com/gocardless/pgreplay-go/blob/master/pkg/pgreplay/parse.go#L220 So I'm not sure that just split of the one line into two lines will be sufficient. Should I add something else maybe?
Asked by mva (111 rep)
Sep 17, 2021, 05:56 AM
Last activity: Apr 28, 2025, 04:04 PM