This is normal operation since the server might be flooded with requests (I.E. UDP).
Check with your ISP to see if they have a designated NTP server. Many don't give NTP settings in their online help but do have a server set up. Often the DNS or SMTP server or a nearby router can be used. Some ISP's provide the server information by DHCP, and dhclient will write a new NTP config if this is the case.
