My favorite training method is very similar to some suggestions above. Many of the people who have got Koi from me have reported back that it works great. If possible, feed the fish with the sun at your back. Let your shadow cast out over the water. Try to feed at approximately same time every day. As the sun sets earlier, I feed a little earlier. The fish will start to associate your shadow with the food.
If at first they still do not want to come over, go ahead and toss the food in and sit back and watch. It may take a little bit for them to get used to the practice. But they will get hungry enough.
As they get used to you more and more, then start slowly sliding your hand in to the water as they eat. Then start putting food in to your hand and down in to the water. As they get more and more at ease with you and you will be hand feeding them and petting them. If you have any darker Koi (Black, copper, Chagoi) they are the most friendly to start off with I have found. They love to eat and be friendly.
Soon enough they will come running across the pond to greet you!