Well, ultimately it's always gonna be a connection issue of some manner, somewhere between your machine and our server. Whose "fault" it is won't always be clear of course.
And being able to download at high speed from some sites doesn't mean all is entirely well with your connection I'm afraid. This has pretty much always been true but the death of net neutrality just amplifies the problem. Some ISPs rejoice in strangling certain kinds of traffic from certain sources...
What you can do is to try a traceroute, which may help us form a picture of which networks are being iffy (we can also then check with our own provider to see if they are having issues with the connectivity to the networks involved). To do this on Windows, you can fire up a cmd box and type "tracert -w 500 racedepartment.com" (the -w 500 bit is optional and stops it waiting for a long time on every router which doesn't want to reply). If any of this is gobbledegook then just ask for help.
It's also well worth doing what I suggested to the OP which is to ask your ISP if they are aware of any reason why you should have speed issues to our server. (For the avoidance of doubt, I'm still seeing full wire speed to our server, which for me is around 70 Mb/s.)