Changing cluster size of SD card formatted as exFAT from 128KiB to some lower value?

I'm using an app that allows me to download data for offline usage. The app is setup to store the data on my 64GB SD card. This all works fine, however, when I downloaded a larger set of data, i.e. some 6GB of data, that data used up some 30GB of storage space, and filled the SD card. I was baffled in the first place, so I had a deeper look at the data. It turned out that the data consisted of some 120'000 files of 40KB in size, some larger, some smaller. I next had a look at the SD card and found it was formatted as exFAT and to use 128KiB blocksize (FX tells me so, I guess this should read cluster size). I'm thinking about reformatting the SD card to use a smaller cluster size, say 48KiB or 64KiB, which would be more suitable for this kind of data. Can this be done? Any inhibitors?