chrony takes many minutes to read refclocks after boot

I'm looking to provide precise time to a Raspberry Pi 4 without internet via a UBLOX GNSS module that sends NMEA sentences to /dev/serial0 and a pulse-per-second (PPS) signal to /dev/pps0. All this information arrives right after boot as verified with gpsmon -n and sudo ppscheck /dev/pps0. However, chrony takes many minutes to pick up the information. I have configured chrony by creating /etc/chrony/conf.d/gps.conf with the content:

refclock SHM 0 refid NMEA offset 0.1 precision 0.2 poll 0
refclock PPS /dev/pps0 refid PPS lock NMEA poll 0 trust prefer
log refclocks

I have experimented a lot with the options in the config file: delay, offset, precision, trust, noselect, prefer and lock, to no avail. I've left /etc/chrony/chrony.conf the way it always was:

# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usable directives.

# Include configuration files found in /etc/chrony/conf.d.
confdir /etc/chrony/conf.d

# Use Debian vendor zone.
pool 2.debian.pool.ntp.org iburst

# Use time sources from DHCP.
sourcedir /run/chrony-dhcp

# Use NTP sources found in /etc/chrony/sources.d.
sourcedir /etc/chrony/sources.d

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Uncomment the following line to turn logging on.
#log tracking measurements statistics

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1 3

# Get TAI-UTC offset and leap seconds from the system tz database.
# This directive must be commented out when using time sources serving
# leap-smeared time.
leapsectz right/UTC

The issue is: after rebooting, chrony doesn't receive data from gpsd or the PPS. I've verified this by running chronyc sources (It shows hyphens for both in the LastRx column.) and sudo tail -f /var/log/chrony/refclocks.log (no updates there). After several minutes (5-20) chrony starts getting time from gpsd and PPS and updates the system time a few seconds after that (verified the same way as explained in the previous paragraph). Sometimes it's different and chrony takes several more minutes to read the PPS after getting time from gpsd. I've tried rebooting many times and the result is always the same. Data from the GNSS module comes in immediately and can be accessed with gpsmon -n and ppscheck right after boot. **Then, why does chrony need over 5 minutes to get the same information?** By the way, after resetting the system time with sudo date +%T -s "12:00:00", and sudo systemctl reastart chrony there's no issue: chrony updates the system time within seconds. One thing I've tried for troubleshooting is to run sudo ntpshmmon. It shows NTP2 and NTP3 samples right from boot but NTP0 samples only show up at the same time as chrony starts connecting to the refclocks. That should help narrow down the cause but I have no idea how to continue debugging from there. This question has been adapted from my question on the Raspberry Pi Stack Exchange .