cd into directory with git lfs files takes very long

I have a (data) repository with ~50GB of data, split into =60 seconds. 1. Why is that, given that I just cd into the repository? git lfs filter-process converts between pointers and actual data -> when I cd into the repository I should not trigger converting since I did not do a git lfs pull or something? 2. Even if it needs to check some pointers and convert, what could cause this process to take >=60 seconds? The system's M2 NVMe SSD is fine (benchmarked reading/writing speed -> above 3GB/s).