Can we use Microsoft Teams without giving it access to all Android contacts?

I'm trying to prevent the Microsoft Teams app from accessing all contacts on my Android. I have gotten as far as asking somebody to share a link to their invite to teams. This way, this contact appears in my chats list, and the status of the contact is displayed (active, away, etc.), but I can't send or receive messages. Can we use Microsoft Teams without giving it access to all Android contacts?