16GB M1 Mac running slow after couple days of usage

After a fresh restart, everything is smooth. During the next couple of days of regular usage of the system, a problem of responsiveness gradually increases, until you can count a full second between you swiping a desktop and the system actually responding to the gesture. I've tried reducing memory pressure by closing high memory apps, but it doesn't help at all. Video demonstration here .