Deleting temporary direction via trap

I have a script which looks like this:

#!/bin/bash
set -e 

tmpdir=$(mktemp -d)
pushd $tmpdir
trap 'popd && rm -rf $tmpdir' EXIT

# Business logic which may succeed or fail...
/bin/false

When it exits, it popds just fine, but it doesn't remove the temporary directory. It seems that $tmpdir isn't resolved in the trap, but how can we overcome that? I verified that $tmpdir isn't resolved by rerunning the script without -rf in the rm. That exited with: rm: missing operand instead of the expected rm: cannot remove '/tmp/tmp.Y1SdoY5dSu/': Is a directory