Universal Control: Mouse pointer keeps jumping back to computer 1 when working on computer 2

I've started using Universal Control to be able to switch between devices with just one mouse (MX Master 3) and without toggling on the mouse. This works fine as soon as I move the mouse pointer to the edge, it appears on the other computer. But if I work something with Safari (e.g. write this text) and then move the mouse again, the pointer is always on the other computer. So it's all unusable. Am I missing a setting?