NYIIX , the east costs largest public peering exchange experienced some performance troubles today from 05:30 EST thru 16:00 EST. As of writing this post things seem to be back to normal.
My company Atlantic Metro saw a large influx of support requests relating to network performance to many ISPs that we peer with us on this peering exchange. Our engineers we able to quickly drain traffic and pass it over alternate routes. It seems we were not the only network to do this, as we got several notifications from GOOGLE who we also peer with, saying they were draining traffic from all peers until the issues were resolved.
As you can see from the traffic graphs something was clearly going wrong.
The following is a list of peering exchange members that were affected by todays outage.
As some of you may know, @reneeclements and I recently signed up to be foster parents for Curly Tail Pug Rescue. Yesterday we got a call about an abused 5 month old fawn puppy that was going to be surrendered. They needed an immediate home for it, so we sprang into action.
The puppy is now in safe hands, and will be evaluated by our vet. There is a small potential that is has Femoral head ostectomy (FHO) and will need surgery.
As I mentioned in one of my previous posts, from the early days of outbrain we chose not to base our system serving on a cloud solution and establish our own serving farm. I guess it is not because we did not know that cloud exist or was afraid to try cloud. I think it was because we were somehow funded and as graduates of big internet companies we thought that if we want this company be as big we have to start our infrastructure the way the big ones start – with our Owned and Operated data center.
Datacenter That doesn’t mean you own the facility but when you start it you buy your own hardware and host it with some Colocation facility vendor. There are lots of such vendors with different sizes and locations. Most of them gives very much the standard service of power, network, security, power redundancy,etc… but I want to talk here about 2 aspects that are most important and makes our Colocation supplier exceptional – Serviceability and Professionalism.
It’s been almost 2 years now that we are working with Atlantic Metro Communications (AMC) as our Colocation vendor. We have been hosted somewhere else before and forced to move in a short notice. We have also tried Amazon AWS for short period and got disappointed but now after long enough with AMC and handling almost 20X growth in our traffic I can recommend these guys to you all.
The reasons is the 2 factors that made them so exceptional and a good fit for company like outbrain:
* Best customer service – Like outbrain, for the AMC staff, customer satisfaction is a core value. Like an integral part in the DNA. Yes, they do have a proper ticketing system and a proper NOC like any other vendor, but you can also submit a ticket or send a mail and get an immediate reply, in the middle of the night from Steve Klenert, their CEO, taking care of the problem or answering one of your questions. You can see that with all AMC staff which is always eager to help and share from their experience and knowledge, even when the case is not in their direct responsibility. For a CTO of a company located across the ocean from his Datacenter, this is super helpful.
* Being professional- knowledge in the technology zone is always something you can argue about. I believe no one is perfect and knows it all. However, there is something that makes a professional staff… mania for solving problems. When it comes to handling internet WAN and LAN networks, it is such a huge domain that nearly every problem is different from the other. Experts with good understanding in this domain and enthusiasm for solving each problem will eventually nail it and solve it. That’s one of the things I see in AMC staff, these are guys I can count on for solving problems and will not give me answers like “That’s vendor X problem, nothing we can do about it”.
outbrain is growing fast and so does our infrastructure I’m pretty sure AMC will keep being on our help in the foreseeable future.
One word for you, when you want to choose a colocation vendor, contact @klenert and see what he has to offer, you’ll not regret it.
