How can I stop my controller from moving my (mouse) pointer?

I'm using antix17, based on Debian. I'm using lightdm with the kde greeter, IceWM for window manager, and x display server. My controller is a PS2 knock off, I'm connecting it to the PC with a USB adapter. The adapter has two ports for two controllers, for this reason, the computer thinks there's two controllers connected at all times, regardless of how many are actually connected to the computer. Now, onto the issues. My controller can directly interact with my mouse. I can move it around with the joystick and I can press some buttons to simulate clicks. This is unwanted behavior because it causes issues when I actually want to game. After trying for some time, I managed to get RetroArch to run on this old PC, but it's riddles with controller issues. On the menus, I seem to be getting double input. When I enter a menu, I see a quick flash of what I presume to be the sub-menu I entered and then I'm back at where I started with the first option selected. This makes controller navigation on RetroArch impossible since I can't enter sub-menus. Also, the stick moving the mouse makes it so I can reach the bottom of the screen and render the taskbar on top of the games, making it lose focus. The biggest issue by far is when using two player mode. The two controllers seem to be linked in some way. For example, in a platformer, I pressed the jump button and that made both characters jump, same thing happened when I pressed jump on the other controller. I tried on another game, and this time only the second controller made both characters jump. The D-pad seems to be independent. I have no clue on how any of this works and I'm losing my mind. By the way, the driver on RetroArch is set to udev, I tried with many more and none removed any issues. I seem to be using eudev tho, whatever that is. Can somebody help me troubleshoot this? Controllers in *nix environments is something I should really learn how to configure.