Accessing USB Gadgetfs driver without sudo

I am working with an embedded machine (rock pi s) which can act as a USB device, communicating with a host device using the USB OTG port on the machine. I use the gadgetfs functionality of linux for this, specifically the driver /dev/gadget/ff400000.usb. It works, but only if I run the software as sudo because the device files are access protected. This causes headaches because I often need to debug various problems, but my remote debugging tools dot run the program as root, so basically I cannot use a debugger as it is now, slowing down development a lot. So how can I enable access to my usb gadgetfs driver without root? - I have tried simply temporarily chmod'ing the file, but it doesn't work because it also dynamically creates a bunch of secondary files for each usb endpoint, which also need sudo to access. - I have also attempted to use udev rules, but if I run "udevadm info -a -n /dev/gadget/ff400000.usb", it just says "Unknown device", which afaik means that udev cannot be used on this driver?