Move between spaces on active display

I have two displays connected to my Mac. On both displays, I open Terminal in full-screen mode. Let's say display B is currently active. I then use Command+Tab to switch to Chrome, which is open on display A. So therefore, display A moves from Terminal to Chrome. At some point later on, I want to switch back to Terminal on display A. **I notice that if the mouse is present in display B, even when display A is the active display, Ctrl+ArrowKey gets applied to display B rather than display A.** Is there any way to change this behavior so that Ctrl+ArrowKey gets applied to the active display?