I’ll throw one more possibility out there that’s a little less obvious.
Sometimes those little piles aren’t from the vacuum missing them, they’re from dirt blowing back out when suction drops mid-run. If your filter pressure is already 5 to 7 psi over clean pressure before you even start vacuuming, suction at the head is weaker than you think. The vacuum passes over the spot, stirs it up, then a bit of it settles back down once the flow shifts.
I started checking my clean pressure before every manual vac. Mine sits at 15 psi clean. If it’s 20 or higher, I rinse the cartridge first. Huge difference in how much gets left behind. Also, if you’re vacuuming to filter instead of to waste, fine silt can pass through sand or older cartridges and come right back into the pool until it clumps.
Another thing to test: move the vacuum slowly over one of those piles and hold it there for a few seconds. If it disappears completely, it’s a flow issue. If it kind of swirls and resettles, that’s ultra-fine dust and you may need tighter filtration or longer pump runtime afterward.
Vacuum speed, clean filter pressure, and return jet direction all play into it more than people think. What’s your clean filter pressure compared to when you’re seeing the leftover piles?