Outrage at Outage for BlueHost, HostGator, HostMonster.
And lo! On the seventh day, was the Techdom of Provo built. They called it Endurance International Group. A perfect, pristine land pulsating with technology. Encased in stainless steel, with na’er a finger print in sight; temperature controlled, its infrastructure an engineer’s delight of colour coded connections.
And into this land, the Lord God of Tech on High placed humans; men who wore immaculately ironed chinos, and buttoned down collars; voted for Obama and drank only water and pulverised wheat grass. Many were those who worshipped the kingdom of Tech and dreamt of the day when they would be allowed to enter its hallowed polished stone corridors.
In the meantime the weekly collection plate was passed round and they gave up their small change; some gave more in the hope of inhabiting their own private VPN in paradise.
And it came to pass that a mighty storm visited the Techdom. And the faithful were vanquished. All 3.3 million of them. And their fury was palpable.
Techdom was not infallible after all. One of the chosen few, the perfect ones, discovered that a ‘vendor’ had supplied a faulty core switch to the very heart of Endurance-land. The Lord God of Tech was not happy with this imperfection in the centre of his paradise and had rent his servers, struck out the faithful into a land of silence and blackness. All 3.3 million of them.
It depends which side of Endurance International that you believe the Lord God of Tech sent his prophet down – some worshipped Bluehost; others claimed this was a Tech of shit straw and adhered to the creed of Hostgator; still others worshipped the server of Hostmonster, iPage, FatCow, Domain.com, iPower, and A Small Orange. So blindly did they follow their profit prophet that they failed to notice that there was but One True Data Centre in Provo, Utah, that all the prophets were beholden to – and he had just spat out his Core Switch.
In the hours of darkness and confusion that followed, the faithful examined their exported CSV files and vowed to worship another. (I was particularly enamoured with the chap who loftily announced to the world that it was going to take him 16 hours to migrate from Bluehost to Hostgator, but it would be worth it! I reckon he must have arrived at Hostgator around 4 hours after the core switch that Hostgator and Bluehost ‘share’ was repaired – and patted himself on the back. Such technological expertise obviously doesn’t leave any time for reading the financial papers…..).
Part of the fury of the faithful was the discovery that when they turned to that archaic instrument, the telephone, to demand that their personal web site be turned back on even if the other 3 million couldn’t be, they discovered that Techdom operated a VOIP telephone system – and that was screwed as well. As was the e-mail system. And the help line.
A lone creature in perfectly pressed chinos and button down shirt, sat on the company Twitter feed and made occasional reassuring utterances to the furious mob – until they exhausted their ‘daily Tweet limit’. That enraged the faithful further.
“I’ve got 120 web sites hosted with your company and I’m losing millions of pounds” cried one faithful.
I idly pondered the wisdom of having such a multi-million pound operation based on 120 cheap shared servers – but then what do I know? Others pointed out that WordPress had just sent their latest baby WordPress 3.6 out into the world and at just the moment that the Provo data centre went tits up, millions of WordPress users would have been upgrading (including Ms Raccoon, who did have her heart in her mouth!) – WordPress wasn’t the culprit.
Conspiracy theories abounded – hadn’t the US just issued a major terrorist attack warning? Was this Al-Quaeda’s revenge – disabling thousands of cheap viagra sites? Wrecking the on-line appointments system for the Penis-Extensions-R-Us private clinic? A tad subtle for Al-Quaeda I would have thought, but innovative if true.
Thousands of Bluehost customers paying $5.95 a month are demanding a refund of at least a month’s hosting to compensate them for their 12 hours in the wilderness or they will worship at another server. They think the company should have taken manpower off the front line looking for the problem in order to e-mail them personally and keep them informed….
I shan’t be.
I’ve been with Bluehost for years. We are old friends. I went to them as someone who barely knew how to switch on her own computer, never mind ‘upload file via ftp’. ‘What is ftp’ said I? ‘ ‘Er, um’, drawled the southern voice, ‘its er, um, something I’d better do for you ma’am’. And he did. Talked me through every step of the way. Found the Thesis file when I’d uploaded it and didn’t know where I’d put it. Patiently explained the perils of leaving my graphics open to hot linking. Unscrambled my config. file. Worked out how to make my little Raccoon clap hands. Those Bluehost technicians have taught me everything I know – which may be precious little; but always with southern courtesy, endless patience, and a ‘You have a nice day now Ma’am’ to send me on my way with a perfectly working web site once more.
So what if it didn’t work for 12 hours? It’s not the end of the world as we know it. It’s worked perfectly for the other 43,680 hours I have been plaguing them with stupid questions.
Does anyone know how much a core switch costs? I thought I might get them a spare one for Christmas.
August 4, 2013 at 07:56
-
Hi Anna, great to have you back – it did occur to me that Mr G might have
thrown you across the saddle of his snow white stallion and galloped off to
the sands of Arabee with you again !
How do I send you an e-mail, can’t see
an addy on the site ?
- August 4, 2013 at 12:03
- August 4, 2013 at 00:25
-
Cleverish, but I’d still bet on an overdose on the steroids
- August 3, 2013 at 20:21
-
I may be altogether wrong, but I suspect that the problems were nothing to
do with core switches. I reckon it was a piece of taut string between two
baked bean tins that had gone limp, and they didn’t spot it for hours.
On a more serious note, it does perhaps illustrate just how technologically
complex modern life is. Want to buy the weekly groceries with a credit card?
Nip into the bank for some cash? Tax the car? Visit the GP? All totally
dependent on computer systems working, which depend on servers, which depend
on the electricity supply and air conditioning. It all depends on a small army
of people maintaining it all and replacing obsolete bits at regular intervals.
Most people take it all completely for granted.
- August 3, 2013 at 18:19
-
All hail polymathic Renaissance Lady Raccoon! Couldn’t understand a word of
it but wildly content with my access to your Inn.
Alan Scott.
-
August 3, 2013 at 16:31
-
Unbelievable. 7 hours now without hosting and email service. You would have
thought there was a natural disaster. I can’t believe a hosting service
doesn’t have a backup plan or redundancy in place.
- August 3, 2013 at 15:41
-
I wonder where Twitter’s ‘core switch’ is, and if we can ask if we can have
a whip round to finance its disabling, without anyone thinking that merely
posing the question constitutes an electronic communication which is ‘grossly
offensive’ or of a ‘menacing character’?
For the enlightenment of those out there with an unduly sensitive nature,
or who are just plain dense, that was a joke.
-
August 3, 2013 at 15:02
-
Am I embarrassed.
I had noticed Anna’s site offline yesterday – it didn’t overly concern me,
these things happen occasionally – but I’ve just learned that my sites would
also have been down (same hosts). And that, I hadn’t noticed until I read this
post! I think I’ll lose my Nerd Badge for that.
- August 3, 2013 at 15:20
-
August 3, 2013 at 14:51
-
Er, is this article available in English ?
-
August 3, 2013 at 11:38
-
Noticed you were down, swore, thought “scheiss’ NSA Wichser, fucking
XKeyScore” and then went and put a bag of frozen peas (cheap ones from Tesco
not your proper Birds Eye) on my core switch…oops I mean, my facet spinal
junction.
Good to have you back.
- August 3, 2013 at 11:29
-
As drsolly says the core switch is in the $25,000 and $250,000, and
can be higher in some instances like the google data centers. In the case of
the data center running the hosting services I would put the value in the
upper third of the range – something they should be able to accommodate with
out too much trouble – just some techs having to physically change it.
From what I saw the site wasn’t totally out, just hard to get to at times –
a couple of page refreshes and it was up and running.
-
August 3, 2013 at 11:08
-
If you get that level of technical support for $5.95 per month, then you’ve
got a *terrific* bargain.
Right now, they’re dealing with what I call the “aftermath”. There will be
some problems around that were caused by the outage, and there will be the
usual crop of problems that weren’t caused by the outage, but people think
they might be. And there will be issues that have been around for a long time
and didn’t matter, but will suddenly get noticed and cause more support calls.
It’s a nightmare, and will continue for some days.
I’ve been in this kind of situation, although nowhere near as bad as
they’ve gone through, and are going through now. You might send them a
postcard to tell them that you appreciate them. Send it to the CEO, and
mention the name of guy that you got especially good support from.
-
August 3, 2013 at 11:01
-
120 web sites doesn’t mean 120 shared servers, they could easily be on one
server, but they’re probably scattered across a whole bunch. Or, if it really
is a multimillion business, they’d be on servers owned by the business but
hosted at the data center; they’d still be dark when the core switch went
out.
I have my main servers hosted at a datacenter. In the last couple of years,
I’ve had a few multi-hour outages, mostly down to power problems they’ve had.
You expect this, and if you need 100% uptime (or 99.999%), you have multiple
servers at multiple datacenters, and load sharing/balancing. If all of them go
dark, then it’s probably the Rapture.
A new core switch would be somewhere between $25,000 and $250,000,
depending on capacity. Could I have one for Christmas too, please?
- August 3, 2013 at 11:30
-
$250,000 eh? I think it might have to be a bit of a whip round
then…..
-
August 4, 2013 at 01:45
-
I know a good fund-raising site!
-
August 3, 2013 at 10:51
-
And there was me thinking you’d done a runner at the prospect of upcoming
visitors. Never mind, there’s still time.
- August 3, 2013 at 10:27
-
Anna – anyone who taught you how to upload that raccoon video is worth a
lot more than £4 a month …… great – yes, I did notice some problems with your
site last night but that’s life isn’t it …….
- August 3, 2013 at 10:14
-
It was never shut down completely. I was browsing in and out all day
yesterday and until about midnight last night [my time] and the site was
showing as unavailable quite often, but then a while later it was there again.
If there was a 12 hour continuous gap it must have been when I was asleep last
night…..
….. in which case it was “playing up” for about sixteen hours before that. I
made a huge Orange hobby-website back in 2007 or so and in 2009 Orange just
turned the whole thing off. My future remains bright but it certainly will
never be orange ever again.
- August 3, 2013 at 10:01
-
It’s always a pleasure to find good tech support. Stay with them, they’re
worth their weight in gold.
- August 3,
2013 at 11:07
-
Seconded!
- August 3, 2013 at 12:42
-
JuliaM, is the link on your name deliberately wrong?
(thylacosmilus.blogspot.con)
I mention only as I am already familiar with your excellent site, and
would always encourage others to click through to it.
- August 3, 2013 at 16:46
-
As an addendum I got excellent tech support from AVG for their *free*
version of their anti virus software on a Saturday many years ago and
have therefore been a firm supporter of their paid products ever
since.
-
August 3, 2013 at 09:24
-
Is this your way of apologising for your site being down for 12 hours?
Would you be upset if I told you I hadn’t noticed?
(But only because I ticked the box that says “Notify me of new posts by
email”.
)
- August 3, 2013 at 09:52
{ 25 comments }