How to run for loop over SSH on remote so that variables expand?

If I ls files without a for loop in a remote host everything is fine however, if I capture the ls in a variable and try to echo each file it fails as the variables don't expand/have a value. What I mean:

IFS=$'\n'
servers=(
  blue
  green
  red
)

for i in ${servers[@]}; do 
   ssh $i "
     ls /dir/xyz_${i}*/details.txt
   "
done
/dir/xyx_blue0/details.txt 
/dir/xyz_blue1/details.txt
/dir/xyx_green2/details.txt 
/dir/xyz_green4/details.txt
/dir/xyx_red1/details.txt 
/dir/xyz_red8/details.txt

But I actually need to loop through the output of ls so I can do things to the files, however the variable doesn't expand:

for i in ${servers[@]}; do 
   ssh $i "
     for i in $(ls /dir/xyz_${i}*/details.txt); do 
        echo $i
     done
   "
done
not found: /dir/xyx_blue*/details.txt 
not found: /dir/xyx_green*/details.txt 
not found: /dir/xyx_red*/details.txt

How can I get $i to expand when running a loop on the remote host?