Thursday’s Network Outage

Many of you noticed that CDD was unavailable for much of the day on Thursday, April 9th. We wanted to take a moment to explain in more detail the events that caused this outage, and to alert you to a new resource, http://status.collaborativedrug.com.

Those of you who live in the San Francisco bay area might have heard about the incident that occurred early Thursday morning, in which someone cut fiber communication backbone cables in two locations, one near San Jose and the other in San Carlos, CA. The police are pursuing a criminal investigation for this act that disrupted cell or landline phone service in several areas, including disabling ATMs and even 911 service. AT&T has announced a $250,000 reward for information leading to the conviction of those responsible.

The act also completely cut Internet connectivity to the facility where CDD houses its servers. All of the facility’s customers went offline. CDD’s servers were up and running during the entire incident, and would have serviced your requests the same as always had the facility’s Internet connection not been severed. To the facility’s credit, its staff was able to procure another backbone connection early in the afternoon, restoring complete connectivity around 2pm (the repair on the cut lines was completed late Thursday evening). Prior to Thursday, we had been very happy with our experience at the facility and with its staff, with no prior power or network interruptions. CDD is following up with the vendor to understand why their redundant connections to the Internet did not protect them from this failure. Please be assured that we hold our hosting vendors to high standards.

CDD’s uptime has been outstanding for an organization of its size. That said, Thursday’s incident made it clear that one area we could do better is in keeping our customers informed in the event of an extremely rare occurrence like this. As a result, we are in the process of setting up http://status.collaborativedrug.com as a place customers can go for status updates in the event that CDD goes down. This website is hosted at a separate facility in Europe, so any calamity that can be imagined would not cause both CDD’s main servers and http://status.collaborativedrug.com to go offline at the same time. We also plan to include a link to an independent third-party measurement of CDD’s uptime, so that we can prove how outstanding our uptime usually is.