Starting a systemd user service for a docker rootlesskit user

I run my docker stuff with a dedicated user, and installed the docker rootlesskit. I start docker with systemctl --user start docker.service. Everything related to docker, executed with that user, works. I am now installing a nostr relay. I followed the instructions and the thing actually works. But. I have to run it via tmux and then scripts/start_local. This script basically runs docker compose -f $DOCKER_FILE up -d. I tried setting up a systemd script for that. First, I tried the /etc/systemd/system/ location, with the correct USER variable and working directory. However, it would fail to start with the famous Cannot connect to the Docker daemon at unix:///home/me/.docker/run/docker.sock. Is the docker daemon running? error message. So I thought maybe it's because I should run the systemd script as the user. So I installed the script into $HOME/.config/systemd/user with a symlink to $HOME/.config/systemd/user/default.target.wants. Basically, the same way my docker service runs. To my surprise, I get the same error. Going back to running it in tmux, it just works. What is different here? For clarity, here is the service file I am using:

[Unit]
Description=Nostr TS Relay

[Service]
Type=simple
Restart=always
RestartSec=5
WorkingDirectory=/home/me/nostream
ExecStart=/home/me/nostream/scripts/start_local
ExecStop=/home/me/nostream/scripts/stop

[Install]
WantedBy=default.target

Note: it's not a big deal I can have it running with a tmux, but I'd prefer the systemd variant.