Little to no sludge and honestly my parameters where acceptable every time I testedSorry to hear about your troubles. Not sure from your pic on what type of worm. The stuff on the side I get also, assuming algae too. Part of the good algae growing on the sides of the pond. Typically get lots of worms, larvae in such in a pond, especially if lots of organic matter.
How was the sludge in the pond when your broke it down? Fin rot and bacterial infections rarely spread or cause problems in a pond with good water quality. Did the fish die? Usually they will chow down on worms.