or, why we should give up on service providers that don’t treat us well enough.
We co-locate
We (Haxx) have a server (technically speaking we have more than one but this is about our main one that hosts most of our public stuff). This server is ours. We bought it, installed it, configured it and then we handed it over to a company that “co-locates” it for us. It means they put our hardware in their big server room and we pay them for it and for the bandwidth our server consumes.
It also means that we have less control over it and we need to call the company to get access to our machine and so on. Ok, so we’ve used Crystone for this for a long time. They’ve been cheap enough and they haven’t complained when we’ve greatly overrun our bandwidth “allowance” for many months in a row.
A bad track record
We did have concerns a while ago (August 2009 and then again in March 2010) when they had power problems in their facility and we suffered from outages and server down-times. Crystone was then really bad at communicating with what happened, what they do and we started to look around for alternative providers since it started to get annoying and they didn’t seem to care for us properly. But we didn’t really get around to actually moving and time passed.
Maybe they had fixed their flaws and things were now fine?
A Saturday in May
Suddenly, on the early morning Saturday May 22nd 2010 our machine didn’t respond to network traffic anymore. We didn’t find out until we woke up and tried to use our services and after having tried a few things. we contacted Crystone to hear if the problem was theirs or if the problem was ours – we’ve had some troubles lately with the network interface card and we feared that perhaps the network might had stopped working due to this flaky hardware.
The customer service at Crystone immediately said that they were experiencing problems due to their move of the server park to the new facilities (they moved from Liljeholmen to Hammarby, both different locations within the general Stockholm area). They said they had network problems and that they were working on it. They did not give any estimation of when our machine would be back online.
They also said that they had mailed their customers about this move, and yeah we felt a bit bad about not having noticed such a mail so that we had been prepared.
The entire day passed. No network. Their web site mentioned problems due to this particular server move. We waited, we got no further info. We were unhappy.
Saturday become Sunday
How big problems can you have when the down-time for your customers exceeds 24 hours and you still haven’t fixed it nor told us what the problems actually are? The Sunday passed and they updated their web site a few times. The last update mentioned the time 16:03 and it said “most customers” are now back online and that if there’s any remaining problem we should contact their customer service. I spotted that message a couple of hours later, when our machine still wasn’t available. And what did customer service have to say to us about it? Nothing, they were closed. Our server remained dead and inaccessible.
Monday, now beyond 50 hours
In the wee hours of the Monday we passed 50 hours offline time and when the customer service “desk” opened in the morning and answered our phone call, they could get our machine back online. By rebooting it. No explanation from their part why our machine was like the only one that suffered this long.
A search in the mail logs also proved that Crystone never mailed us to tell that our server would move. Isn’t that odd? (not really, as we would find out later)
We won’t stand it
Already during the weekend we had decided we are fed up with this complete ignorance and crappy treatment. Down-times and problems happen, but the complete lack of information and care about us – their customers – is what made it clear we are not suitable to be their customers. We had to go elsewhere.
Crystone offered us a month fee worth of deduction on the hosting charges as a compensation for the troubles we had. That was nice of them, but really this service isn’t expensive so it’s not the cost of this that is burdensome. We just can’t stand having a service this unreliable and working with a company that is this uncommunicative.
This big server move was Crystone moving a lot of equipment over to the facility that is owned and run by Phonera, another ISP, and the one that we happened to have an offer from since before when we were looking for alternatives. Handy – we thought – perhaps we could just go there and carry our server over from one shelf to another and we’ll be fine. Phonera is slightly more expensive but hey, perhaps we’d get peace of mind!
“We don’t steal customers”
Phonera was first glad to accept us as customers, but surprised us greatly when they turned around and declined getting us as new customers, since they claimed they don’t want to “steal” customers from Crystone (that are now themselves customers of Phonera). Baffled, we simply sent off another request to Portlane instead and within minutes we had a decision made and a contract signed.
Later that afternoon, a Phonera guy got back to us and had changed position again and said that perhaps we could become customers anyway. They had figured out that none of them would gain by us going to a third company, but in any case it was now too late for them and we had already made up our minds about going Portlane.
“Sir, your server is not here”
On Tuesday 13:00, Björn (as co-admin of the server) had an appointment with Crystone to extract our server from their care to take it over to its new home. When he appeared in Hammarby at the new facility to get the server he was up for (another) surprise. It wasn’t there. Now Crystone could inform us that our server is still left in the old facility in Liljeholmen. It was never moved!
Glad our business with these guys would soon be over, Björn handed over our 1U of server to Portlane and within a short while it had found a new home, with a new IP address and a new caretaker.
We could once again take a deep breath of relief and carry on with whatever we were doing before again.
