Unshare with overlayfs results in permission denied with su

I am trying to setup 'rootless' containers by hand, with just unshare and mounting overlayfs. Currently, I can unpack a rootfs tarball, setup a /tmp and /proc mount, and pivot_root/chroot into it without issue. I can also add adduser and su as that new user. However, when I setup an overlayfs mount, where the lower layer is the unpacked tarball, and the upper layer is a temp directory, repeating the above steps fails with a permission denied issue when I su as the new user. I suspect it has to do with the newly created user in adduser does not have permissions to read from the overlay filesystem, but I am not sure. I don't see any logs in demsg even with setting /proc/sys/kernel/printk to 6 # Reproduction Steps ## What Works (unpacked tarball) Unshare into a new mount and user namespace in a terminal

shell
unshare -pf --user --mount-proc --kill-child /bin/bash

In another terminal use newuidmap and newgidmap for the new process:

newuidmap $PID 0 1000 1 1 100000 65536
newgidmap $PID 0 1000 1 1 100000 65536

Back to the first terminal, in the namespace, unpack the rootfs, setup the mounts, and chroot into it and create a new user.

mkdir rootfs
tar -xvf alpine-minirootfs-3.15.3-x86.tar -C rootfs
mkdir mountpoint
mount --bind rootfs/ mountpoint/
mkdir -p mountpoint/tmp/
mkdir -p mountpoint/proc/
mount -t tmpfs none mountpoint/tmp/
mount -t proc none mountpoint/proc
cd mountpoint/
pivot_root . .
exec chroot . /bin/sh
# Create the new user and su as it
adduser -s /bin/sh -D newuser
# The below command works
su newuser -

The above steps work where I am able to su as the newuser. ## What doesn't work (overlayfs) Changing the mountpoint to be backed by overlay fs does not work. Repeating the steps from above to prepare the rootfs. Unshare into a new mount and user namespace in a terminal.

unshare -pf --user --mount-proc --kill-child /bin/bash

In another terminal use newuidmap and newgidmap for the new process:

newuidmap $PID 0 1000 1 1 100000 65536
newgidmap $PID 0 1000 1 1 100000 65536

Back to the first terminal, in the namespace, unpack the rootfs, setup an overlay mount, chroot into it and create a new user.

mkdir rootfs
tar -xvf alpine-minirootfs-3.15.3-x86.tar -C rootfs
mkdir mountpoint
# This is the key difference from above
mount -t overlay none -o lowerdir=$(realpath ./rootfs),upperdir=$(mktemp -d),workdir=$(mktemp -d) $(realpath ./mountpoint)
mkdir -p mountpoint/tmp/
mkdir -p mountpoint/proc
mount -t tmpfs none mountpoint/tmp/
mount -t proc none mountpoint/proc
cd mountpoint/
pivot_root . .
exec chroot . /bin/sh
# Create the new user and su as it
adduser -s /bin/sh -D newuser
# The below command fails with permission denied
su newuser -
#
# su: can't execute '/bin/sh': Permission denied

## Environment Info The above commands were run on Ubuntu Impish with kernel 5.13.0-37-generic