Why does unmounting of a VFAT filesystem result in reading the whole FAT?

Each time I unmount the filesystem of my SD card that I use on my DSLR, I have to wait several seconds for umount command to finish — even if I've just mounted the FS as read-only and immediately unmount it. The mount command is simple:

mount /dev/sdd1 /mnt/tmp -oro

I've tried setting vm.block_dump=1 to see what's going on, and here's what I saw when I issued umount /mnt/tmp:

$ tail -n9999 /var/log/kern.log | grep sdd1

Jul  7 16:56:42 IntegralH kernel: [143924.204781] umount(16203): READ block 1635 on sdd1 (1 sectors)
Jul  7 16:56:42 IntegralH kernel: [143924.210786] umount(16203): READ block 1636 on sdd1 (1 sectors)
Jul  7 16:56:42 IntegralH kernel: [143924.216805] umount(16203): READ block 1637 on sdd1 (1 sectors)
Jul  7 16:56:42 IntegralH kernel: [143924.222841] umount(16203): READ block 1638 on sdd1 (1 sectors)
Jul  7 16:56:42 IntegralH kernel: [143924.228867] umount(16203): READ block 1639 on sdd1 (1 sectors)

Jul  7 16:57:34 IntegralH kernel: [143976.105848] umount(16203): READ block 9006 on sdd1 (1 sectors)
Jul  7 16:57:34 IntegralH kernel: [143976.111932] umount(16203): READ block 9007 on sdd1 (1 sectors)
Jul  7 16:57:34 IntegralH kernel: [143976.117915] umount(16203): READ block 9008 on sdd1 (1 sectors)

The skipped part after block 1639 consists of single-block reads of every block between 1639 and 9006. To make sure that my suspicion is correct, I've checked the first sector of the FS:

$ sudo head -c 40 /dev/sdd1 | xxd -g1
0000000: eb 00 90 43 41 4e 4f 4e 45 4f 53 00 02 40 62 06  ...CANONEOS..@b.
0000010: 02 00 00 00 00 f8 00 00 3f 00 ff 00 00 20 00 00  ........?.... ..
0000020: 00 18 9a 03 cf 1c 00 00                          ........

You can see that the number of reserved sectors (which denotes the first sector of the FAT; 2 bytes at offset 0x0E), is 0x662, or 1634. And the size of the FAT (4 bytes at offset 0x24) is 0x1ccf, or 7375=9008+1-1634. I.e. the driver reads the whole FAT except its first sector. I wonder now, why does the driver need to read the whole FAT simply to unmount the filesystem? And how can I avoid this time-consuming operation?