Why is this docker container process not triggering a mount for my systemd auto-mounted drive?
1
vote
1
answer
117
views
I've been struggling to make sense of something, so would appreciate some help.
I am mounting a remote NFS drive onto my Debian system with the following fstab entry which uses the systemd automounter, and is set to auto-unmount after 120 seconds of inactivity:
192.168.0.67:/mnt/SSD_240GB/backups/TIG_backups /mnt/nfs/SSD_240GB/backups/TIG_backups nfs auto,_netdev,bg,soft,x-systemd.automount,x-systemd.idle-timeout=120,timeo=14,nofail,noatime,nolock,tcp,actimeo=1800 0 0
Now on this Debian host system I am running a docker container (Telegraf ), to monitor some metrics of the Debian host. To facilitate this, I am bind-mounting the host filesystem and proc directory (as recommended here in the docs ), as well as bind-mounting the NFS drive. The docker run command looks like this:
docker run -d \
--name telegraf_container \
--user 1001:1001 \
--network docker_monitoring_network \
--mount type=bind,source=/,destination=/hostfs \
--mount type=bind,source=/mnt/nfs/SSD_240GB/backups/TIG_backups/telegraf_backups,destination=/mnt/nfs/SSD_240GB/backups/TIG_backups/telegraf_backups \
-e HOST_MOUNT_PREFIX=/hostfs \
-e HOST_PROC=/hostfs/proc \
telegraf:latest
I am using the Telegraf Disk Input plugin because I want to gather disk usage metrics once every hour for the NFS drive (used, free, total). The problem is that the disk is unmounted automatically 120s after system boot as expected, *but it is never remounted*.
I would expect the telegraf container to trigger an automount every hour. The reason I expect this is because the container is essentially running a .go program (as seen here in the source code) which is querying the filesystem. I believe under the hood it is calling some .go libraries (here and here ), which are essentially calling statfs(). I was under the impression that statfs() should trigger a systemd automount.
Here in the Debian host's logs, I can see the NFS drive mounting correctly at boot up, and then unmounting after a couple of minutes automatically (but then it never remounts):
root@docker-debian:/home/monitoring/docker_files/scripts# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.automount -b
Jun 05 13:54:12 docker-debian systemd[1] : Set up automount mnt-nfs-SSD_240GB-backups-TIG_backups.automount.
Jun 05 13:54:18 docker-debian systemd[1] : mnt-nfs-SSD_240GB-backups-TIG_backups.automount: Got automount request for /mnt/nfs/SSD_240GB/backups/TIG_backups, triggered by 893 (runc:[2:INIT])
root@docker-debian:/home/monitoring/docker_files/scripts# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.mount -b
Jun 05 13:54:18 docker-debian systemd[1] : Mounting mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups...
Jun 05 13:54:18 docker-debian systemd[1] : Mounted mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups.
Jun 05 13:57:39 docker-debian systemd[1] : Unmounting mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups...
Jun 05 13:57:39 docker-debian systemd[1] : mnt-nfs-SSD_240GB-backups-TIG_backups.mount: Deactivated successfully.
Jun 05 13:57:39 docker-debian systemd[1] : Unmounted mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups.
After the drive has auto-unmounted, it is missing from the host as expected:
monitoring@docker-debian:/$ df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 983908 0 983908 0% /dev
tmpfs 201420 816 200604 1% /run
/dev/sda1 15421320 4779404 9836748 33% /
tmpfs 1007084 0 1007084 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 201416 0 201416 0% /run/user/1001
But it is present in the container:
monitoring@docker-debian:/$ docker exec -it telegraf_container df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 15421320 4779404 9836748 33% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
/dev/sda1 15421320 4779404 9836748 33% /hostfs
udev 983908 0 983908 0% /hostfs/dev
tmpfs 1007084 0 1007084 0% /hostfs/dev/shm
tmpfs 201420 820 200600 1% /hostfs/run
tmpfs 5120 0 5120 0% /hostfs/run/lock
192.168.0.67:/mnt/SSD_240GB/backups/TIG_backups/telegraf_backups 229608448 42336256 175535104 20% /mnt/nfs/SSD_240GB/backups/TIG_backups/telegraf_backups
tmpfs 1007084 0 1007084 0% /proc/acpi
tmpfs 1007084 0 1007084 0% /sys/firmware
tmpfs 201416 0 201416 0% /hostfs/run/user/1001
In case it's relevant, the Telegraf config is here:
# GLOBAL SETTINGS
[agent]
hostname = "docker-debian"
flush_interval = "60m"
interval = "60m"
# COLLECT DISK USAGE OF THIS VM
[[inputs.disk]]
mount_points = ["/", "/mnt/nfs/SSD_240GB/backups/TIG_backups"] # Only these will be checked
fieldpass = [ "free", "total", "used", "used_percent" ]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
# VIEW COLLECTED METRICS
[[outputs.file]]
files = ["stdout"]
Why is the container not triggering an automount, which leads to it not being able to collect the metrics on the drive?
---
**EDIT**
After the answer from @grawity, I did a simpler check:
- I removed the idle timeout (by setting x-systemd.idle-timeout=0
)
- I removed explicit bind-mounts for these drives from the docker run command
In this situation, I found the following:
1) Immediately after boot, an automount is set up, but nothing triggered it yet, as expected:
root@docker-debian:# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.automount -b
Jun 06 12:22:20 docker-debian systemd[1] : Set up automount mnt-nfs-SSD_240GB-backups-TIG_backups.automount.
root@docker-debian:# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.mount -b
-- No entries --
2) I start a simple container up, with no explicit bind mounts for those drives (only the hostfs structure) :
docker run -d \
--name telegraf_container \
--mount type=bind,source=/,destination=/hostfs \
-e HOST_MOUNT_PREFIX=/hostfs \
-e HOST_PROC=/hostfs/proc \
telegraf:latest
This still does not trigger any automounts on the host.
3) Now I manually trigger an automount on the host by accessing the drive:
ls /mnt/nfs/SSD_240GB/backups/TIG_backups/
The automount is triggered and mounts the drive successfully:
root@docker-debian:# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.automount -b
Jun 06 12:22:20 docker-debian systemd[1] : Set up automount mnt-nfs-SSD_240GB-backups-TIG_backups.automount.
Jun 06 12:35:20 docker-debian systemd[1] : mnt-nfs-SSD_240GB-backups-TIG_backups.automount: Got automount request for /mnt/nfs/SSD_240GB/backups/TIG_backups, triggered by 936 (ls)
root@docker-debian:# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.mount -b
Jun 06 12:35:21 docker-debian systemd[1] : Mounting mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups...
Jun 06 12:35:21 docker-debian systemd[1] : Mounted mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups.
Interestingly, the mounted drive now *automatically* appears inside the container (even though no bind-mounts have been used), but it appears under /hostfs
instead:
monitoring@docker-debian:~$ docker exec -it telegraf_container df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 15421320 4686888 9929264 33% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
/dev/sda1 15421320 4686888 9929264 33% /hostfs
udev 983908 0 983908 0% /hostfs/dev
tmpfs 1007084 0 1007084 0% /hostfs/dev/shm
tmpfs 201420 656 200764 1% /hostfs/run
tmpfs 5120 0 5120 0% /hostfs/run/lock
tmpfs 201416 0 201416 0% /hostfs/run/user/1001
tmpfs 1007084 0 1007084 0% /proc/acpi
tmpfs 1007084 0 1007084 0% /sys/firmware
192.168.0.67:/mnt/SSD_240GB/backups/TIG_backups 16337920 5799936 9682944 38% /hostfs/mnt/nfs/SSD_240GB/backups/TIG_backups
If I unmount the drive directly on the host (using umount), then it disappears from the container again.
4) I repeated this but instead using an idle timeout of 2mins now. What I found was that having the docker container running *prevents* the autounmount after 2 mins from happening (even though the container does not explicitly bind-mount in the drive, but instead appears automatically in the container under /hostfs).
If I stop and remove the container, then the idle timeout unmounts the drive after the 2mins:
root@docker-debian:# journalctl -u mnt-nfs-SSD_240GB-backups-TIG_backups.mount -b
Jun 06 12:49:40 docker-debian systemd[1] : Mounting mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups...
Jun 06 12:49:41 docker-debian systemd[1] : Mounted mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups.
Jun 06 13:10:28 docker-debian systemd[1] : Unmounting mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups...
Jun 06 13:10:28 docker-debian systemd[1] : mnt-nfs-SSD_240GB-backups-TIG_backups.mount: Deactivated successfully.
Jun 06 13:10:28 docker-debian systemd[1] : Unmounted mnt-nfs-SSD_240GB-backups-TIG_backups.mount - /mnt/nfs/SSD_240GB/backups/TIG_backups.
This makes me think a couple of things:
- If I want to use telegraf to monitor drives that are mounted on the host, I don't need to bind mount them in explicitly, because they are present already due to the /hostfs bind-mount.
- I should never see what I was originally expecting - namely, a drive automatically unmounting due to the idle timeout, and then the container triggering a remount. Because I observed above that once a drive has been mounted in (in my case at /hostfs), the container actually prevents it from ever being auto-unmounted.
Asked by teeeeee
(305 rep)
Jun 5, 2025, 03:04 PM
Last activity: Jun 6, 2025, 01:08 PM
Last activity: Jun 6, 2025, 01:08 PM