The pond lily pads come up, then when they die away they just go to the bottom. Same with the cattails and irises. So I have all of this dead vegetation on the bottom, and a skimmer wouldn't really get that.
You may want to be more proactive with the plant grooming - pull or cut any dying or dead foliage that will end up in the bottom to avoid getting a lot of decaying matter build up.