Encrypted LUKS fs inside a file: sparse or not?

I have a LUKS encrypted file filled with around 160 GB of data that I use a lot. For safety, I created the file with 400 GB.  That is, of course, a lot of wasted space.  So I switched to a sparse file, basically following the advice here , simply by creating the file with the seek option:

dd if=/dev/zero of= bs=1G count=0 seek=400

But then I thought: what happens if the file begins to be fragmented?  Usually this is not a problem since I don't have very big files, and when I do, they are media files that usually don't change.  But an encrypted thing that I use so frequently will probably get fragmented quite soon... So my question is: is there any true downside to using a sparse file instead of a fixed-sized file in my situation?  Is fragmenting really an issue? Do you have any further advice?