Hmm, I don't know if this is what you meant with "it taking a few times" but evertime I try to connect, I get a "connection refused" messege...
EDIT: I went on a hunch and checked our firewall. Turned out that it was the one making all the problems. :p Now, I'm having no troubles at all.