Postgres - Extreme query planning times on simple query

On a 200mb table fetching entries takes forever due to high query planning times:

EXPLAIN (ANALYZE, BUFFERS, TIMING) 
SELECT id FROM tour_bookings WHERE id = 6311251;

| QUERY PLAN                                                                                                                            |
| ------------------------------------------------------------------------------------------------------------------------------------- |
| Index Only Scan using tour_bookings_pkey on tour_bookings  (cost=0.42..2.64 rows=1 width=8) (actual time=1.841..1.843 rows=1 loops=1) |
|   Index Cond: (id = 6311251)                                                                                                          |
|   Heap Fetches: 1                                                                                                                     |
|   Buffers: shared hit=4 read=3                                                                                                        |
| Planning:                                                                                                                             |
|   Buffers: shared hit=6088 read=17450                                                                                                 | 
| Planning Time: 4429.900 ms                                                                                                            |
| Execution Time: 2.973 ms                                                                                                              |

No success: - vacuuming - disabling triggers - disabling complex constraints - clustering pkey - analyse related tables A duplicated version of the table doesn't face those issues. Buffer access in planning seem extensive. How to find out what and why the query planner actually does?