Last year I wanted to redo the pond after 1 year being up. So I empty the pond and start recovering up the 100kg of gravel I laid. Of course there was a lot of muck below these stones and the remaining water turn jet black. However it didn't smell bad, actually very earthy, like soil. So I assume the beneficial bacteria was doing its job and supporting a healthly system.
Two weeks later I had to retake out the liner I just placed cause it didn't fit. Anyways when retaking up the gravel again this time it had a really bad smell. So after that I decided never to disturb the gravel again, now over a year and no issues. Once you have a good filter, should get most of the dirt.
Now, I'm not saying this is way to go. It's just what I notice and seems best for me.