Downside of decreasing audio latency

I see that with audio servers (in my case, pipewire) you can alter the "latency". (please forgive me, I am not very knowledgeable with these things.)

="128/48000"

The Arch Linux wiki described this as "request[ing] a custom buffer size". I was wondering, is there a "downside" to setting the latency really low. Is it simply more responsive audio a higher cost of resources?