Can't swipe between different work spaces on one out of two external monitors

When connecting two external monitors to my mac, sometimes 3-fingers swipe only works for one of them. In this case, even when the cursor is on one scree, 3-fingers swipe will lead to a work space change on the other. It's consistent and happens even when switching usb-c cables between ports, only one screen would swipe.