should I try to keep them out
The only way you'd keep them out is with a net covering your pond. I've had a pair of ducks show up for years, even before I had a pond. Once I installed the pond, there was no keeping them out. Initially I would walk outside and they'd fly away, only to return a few hours later, but after awhile, they'd only fly over the fence, wait until they thought I was gone and fly or walk right back over. Now, I can walk right up to them and they won't do anything other than swim across the pond. I stopped chasing them away when they decided to nest in my pond.
They do tear the waterlilies, but other than that, any visual damage has been minimal. They can also harbor parasites, leeches, etc. I've had no issues with that. I know that
@RobAmy have ducks all the time. To my knowledge, they haven't had any issues either, but they'd have to answer that themselves. Even frogs can spread disease to fish. It's just a part of nature. It's certainly something to be aware of but I wouldn't worry too much about it.
As for Swimmer's Itch, rub briskly with a towel immediately after leaving the water & shower after being in water.
I'm personally looking forward to seeing my ducks return. They should be back any day now. I learned a lot by watching Mrs. Duck lay on her nest last year. Funny how she'd always leave at the same time every evening to meet up with Mr. Duck and go out to eat. He'd always fly her back to the pond, but he'd usually keep flying or land until she sat back on her eggs and then he'd leave.