fluxbox inside rc.local startup script sometimes does not spawn process and start

This issue is driving me crazy. I have no clue why it sometimes happens. Its totally feeling intermittent. I have a /etc/rc.local shell script that fires off on startup. I am using Ubuntu x64 19.10... For what its worth, I run this off a cloud droplet via Digital Ocean. Long story short... My script has at the beginning something like this...

echo "Starting virtual display session..."
export DISPLAY=:0
Xvfb :0 -screen 0 640x360x16 & 
DISPLAY=:0.0 fluxbox &
sleep 1
x11vnc -display :0.0 -usepw -forever -bg

All of this works most of the time.... However the fluxbox sometimes just will not load. I will end up with a X display session but no window manager. When I run ps aux | grep fluxbox I see nada. The issue is that it only happens sometimes. I can reboot the system and most of the time it will load, but on occasion it just totally does not start process. UPDATE: I can see the following error happen when this seems to occur:

Error: Couldn't connect to XServer:0.0

Why is this happening if I am setting the export of the display and also setting the display when I start the fluxbox process? Does anyone have any idea? Note: I only have the virtual frame buffer via xvfb and it always runs on DISPLAY=:0 Thank you! 🙏