Postgres slow query on large tables when doing a search radius search and joining on large tables with indices

I've been experiencing extremely slow queries despite having indices (hopefully the correct indices) on my tables. What I'm doing is searching for providers within a search radius as well as their associated insurances. Each table has 10 to 100s of millions of rows. Query (took over 8 minutes!!). After running it again

explain (analyze, buffers) SELECT
    p.id,
    p.npi,
    p.pqq_composite_score,
    jsonb_agg(pl.location_details)
    FROM provider.provider_location pl
    JOIN provider p on pl.provider_id = p.id
    JOIN provider_location_insurance pli ON pl.provider_id = pli.provider_id AND pl.location_id = pli.location_id
WHERE pli.insurance_id = 'c7d1b6aa-cb9a-44b9-8069-ce5a4476fdfa'
  AND pli.carrier_brand = 'some_brand_name'
  AND pl.confidence >= 3
  AND ST_DWithin(pl.geom, st_setsrid(ST_MakePoint(-87.6291, 41.8781), 4326), 1609.354 * 50)
group by p.id
limit 50

That query took about ~8.5 minutes

Limit  (cost=9626986.00..9626986.02 rows=1 width=67) (actual time=504568.135..504568.486 rows=50 loops=1)
"  Buffers: shared hit=10843973 read=3858153 dirtied=1838660 written=315789, temp read=32052 written=75066"
  I/O Timings: read=434866.551 write=3001.677
  ->  GroupAggregate  (cost=9626986.00..9626986.02 rows=1 width=67) (actual time=504568.134..504568.480 rows=50 loops=1)
        Group Key: p.id
"        Buffers: shared hit=10843973 read=3858153 dirtied=1838660 written=315789, temp read=32052 written=75066"
        I/O Timings: read=434866.551 write=3001.677
        ->  Sort  (cost=9626986.00..9626986.01 rows=1 width=588) (actual time=504568.088..504568.104 rows=63 loops=1)
              Sort Key: p.id
              Sort Method: external merge  Disk: 600504kB
"              Buffers: shared hit=10843973 read=3858153 dirtied=1838660 written=315789, temp read=32052 written=75066"
              I/O Timings: read=434866.551 write=3001.677
              ->  Nested Loop  (cost=20026.46..9626985.99 rows=1 width=588) (actual time=2945.990..501666.304 rows=895360 loops=1)
                    Join Filter: (pl.provider_id = p.id)
                    Buffers: shared hit=10843970 read=3858153 dirtied=1838660 written=315789
                    I/O Timings: read=434866.551 write=3001.677
                    ->  Nested Loop  (cost=20026.03..9626980.39 rows=1 width=585) (actual time=2945.398..404746.262 rows=895360 loops=1)
                          Buffers: shared hit=7739177 read=3333012 dirtied=1838660 written=273897
                          I/O Timings: read=347264.324 write=2578.856
"                          ->  Bitmap Heap Scan on ""provider_location_insurance_some_brand_name"" pli  (cost=20025.45..3845940.02 rows=1833404 width=32) (actual time=2941.360..115985.754 rows=1846415 loops=1)"
                                Recheck Cond: (insurance_id = 'c7d1b6aa-cb9a-44b9-8069-ce5a4476fdfa'::uuid)
                                Filter: (carrier_brand = 'some_brand_name'::text)
                                Heap Blocks: exact=1838660
                                Buffers: shared read=1840114 dirtied=1838660 written=154012
                                I/O Timings: read=86001.337 write=1467.246
"                                ->  Bitmap Index Scan on ""provider_location_insurance_Aetna_insurance_id_idx""  (cost=0.00..19567.10 rows=1833404 width=0) (actual time=2320.226..2320.226 rows=1846415 loops=1)"
                                      Index Cond: (insurance_id = 'c7d1b6aa-cb9a-44b9-8069-ce5a4476fdfa'::uuid)
                                      Buffers: shared read=1454
                                      I/O Timings: read=1858.023
                          ->  Memoize  (cost=0.58..31.29 rows=1 width=585) (actual time=0.155..0.155 rows=0 loops=1846415)
"                                Cache Key: pli.provider_id, pli.location_id"
                                Cache Mode: logical
                                Hits: 0  Misses: 1846415  Evictions: 1248029  Overflows: 0  Memory Usage: 256001kB
                                Buffers: shared hit=7739177 read=1492898 written=119885
                                I/O Timings: read=261262.987 write=1111.610
                                ->  Index Scan using provider_location_provider_id_location_id_key on provider_location pl  (cost=0.57..31.28 rows=1 width=585) (actual time=0.153..0.153 rows=0 loops=1846415)
                                      Index Cond: ((provider_id = pli.provider_id) AND (location_id = pli.location_id))
"                                      Filter: ((confidence >= 3) AND st_dwithin(geom, '0101000020E6100000787AA52C43E855C00E4FAF9465F04440'::geometry, '80467.7'::double precision))"
                                      Rows Removed by Filter: 1
                                      Buffers: shared hit=7739177 read=1492898 written=119885
                                      I/O Timings: read=261262.987 write=1111.610
                    ->  Index Scan using provider_id_idx on provider p  (cost=0.43..5.59 rows=1 width=35) (actual time=0.106..0.106 rows=1 loops=895360)
                          Index Cond: (id = pli.provider_id)
                          Buffers: shared hit=3104793 read=525141 written=41892
                          I/O Timings: read=87602.227 write=422.821
Planning:
  Buffers: shared hit=1049 read=43 dirtied=4
  I/O Timings: read=16.139
Planning Time: 21.215 ms
Execution Time: 504654.267 ms

Provider Location table definition which contains ~75 million rows:

Table "provider.provider_location"
         Column         |           Type           | Collation | Nullable |                                     Default
------------------------+--------------------------+-----------+----------+----------------------------------------------------------------------------------
 id                     | uuid                     |           | not null | gen_random_uuid()
 provider_id            | uuid                     |           | not null |
 location_id            | uuid                     |           | not null |
 created_at             | timestamp with time zone |           | not null | CURRENT_TIMESTAMP
 updated_at             | timestamp with time zone |           | not null | CURRENT_TIMESTAMP
 confidence             | integer                  |           |          |
 latitude               | double precision         |           |          |
 longitude              | double precision         |           |          |
 location_details       | jsonb                    |           |          |
 geom                   | geometry                 |           |          | generated always as (st_setsrid(st_makepoint(longitude, latitude), 4326)) stored
Indexes:
    "provider_location_pkey" PRIMARY KEY, btree (id)
    "provider_location_location_id_idx" hash (location_id)
    "provider_location_provider_id_idx" hash (provider_id)
    "provider_location_provider_id_location_id_key" UNIQUE CONSTRAINT, btree (provider_id, location_id
)
    "idx_provider_confidence_geom_idx" gist (geom, confidence)
    "idx_provider_location_geom" gist (geom) CLUSTER
    "provider_provider_location_confidence" btree (confidence)

And the Provider Location insurance table contains ~950 million rows. That table is partitioned by carrier_brand (a total of 135 partitions)

Partitioned table "provider.provider_location_insurance"
    Column     |           Type           | Collation | Nullable |      Default
---------------+--------------------------+-----------+----------+-------------------
 provider_id   | uuid                     |           | not null |
 location_id   | uuid                     |           | not null |
 insurance_id  | uuid                     |           | not null |
 carrier_brand | text                     |           | not null |
 created_at    | timestamp with time zone |           | not null | CURRENT_TIMESTAMP
 updated_at    | timestamp with time zone |           | not null | CURRENT_TIMESTAMP
Partition key: LIST (carrier_brand)
Indexes:
    "provider_location_insurance_pk" PRIMARY KEY, btree (provider_id, location_id, insurance_id, carrier_brand)
    "idx_provider_location_insurance_carrier" btree (carrier_brand)
    "idx_provider_location_insurance_insurance_id" btree (insurance_id)
    "idx_provider_location_insurance_location_id" btree (location_id)
    "idx_provider_location_insurance_provider_id" btree (provider_id)

My postgres instance has the following configs:

PostgreSQL version: 14
vCPU: 4
Memory 32GB

shared_buffers="25684MB"
work_mem="250MB"
max_parallel_workers_per_gather=2
effective_cache_size="26300784kB"

What am I doing wrong here? Do I need to create a separate "search" table that would avoid all of the joins on these big tables and then have the correct indices on each column(s)? From the plan it looks like the indices are being used. I've also CLUSTERED the geometry column to see if that would make a different but I'm starting to run out of ideas.