iCal invites sent to my Gmail address (which is also my Apple ID) get added to iCloud

My Apple ID is my Gmail address. I also use Google Calendar as my main calendar. But, when someone sends me a calendar invite from their Apple Calendar app, I don't get an email in Gmail, and it there's no invite on my Google Calendar. Instead, it gets added to the built in iCloud Calendar that I never use. This causes me to miss events that I need to be added to. I tested this on my own, and "invited" another Gmail account I have from the Apple Calendar app - it sent an email invite to that Gmail, which I could open, click add to calendar, and have on that calendar. This is what I want to happen when someone else with an iPhone and iCloud Calendar invites my main Gmail to a meeting. This never used to be an issue for me in the past when I didn't utilize an iPhone (but still had the same Apple ID), but now I am unable to receive any invite from an Apple user to that specific Gmail. It appears as though Apple recognizes the email address when someone tries to invite me, and instead of sending the invite to the email, just adds it to the iCloud calendar. Is there a way to turn off this iCloud calendar from accepting invites, so that these invites can be sent to my Gmail as a normal email invite? Unfortunately, switching to using the iCloud calendar is not an option. I know I can subscribe to my Google Calendar in the Apple Calendar app, and to my iCloud Calendar on Google Calendar, but I need these events to actually populate on my Google Calendar itself, not just subscribe so that I can "see them all in once place". Thank you!