Status Updates

2014

February 14, 2014 1:40 PM PST – CDD’s servers went down for fifty minutes this afternoon due to a power glitch at the colocation facility (UPS at the colo failed during maintenance).

2013

No unscheduled downtime in 2013.

2012

No unscheduled downtime in 2012.

2011

No unscheduled downtime in 2011.

2010

February 20, 2010 10:30 AM PST – CDD is back up. Preliminary indications from UnitedLayer indicate that another PG&E failure occurred this morning. Because UnitedLayer is in the middle of upgrading its power infrastructure redundancy, it is still susceptible to these issues. CDD also will continue to troubleshoot to find out why this particular server does not come back up after power failure.

February 20, 2010 9:08 AM PST – During a PG&E brownout yesterday, our colocation provider discovered “maintenance issues within [their] battery cabinets.” These did not affect us yesterday, but this morning our servers experienced power failure, and one has not yet come back up. We are working to resolve the issue, and it should be fixed shortly.

February 14, 2010 4:00 PM PST – Due to power failure affecting one of one of our colocation provider’s border routers, CDD experienced sporadic short term network outages between 2:14 and 3:12pm today. Everything should be back to normal, and UnitedLayer assures us they will look into why the router’s redundant power did not fail over smoothly and correct the problem.

January 19, 2010 3:30 PM PST – CDD’s servers are back up.

January 19, 2010 2:55 PM PST – CDD’s servers are down due to another power failure at UnitedLayer, our colocation provider. They tell us it was due to lightning. Our servers will be back up shortly.

January 13, 2010 1:00 AM PST – CDD’s servers went down for an hour this evening due to a power glitch at the colocation facility (high winds caused a brief PG&E power failure, and then a UPS at the colo failed). The colo facility has assured us they are investing $125,000 in infrastructure improvements to fix this recurring issue with UPS failure. We at CDD also need to understand why our servers didn’t immediately come back online after power was restored. We expect to resolve that within a week’s time.

2009

October 13, 2009, 10:38 AM PST – CDD’s servers are back up. There was a power outage at our colocation provider this morning, which led to a subsequent router/network outage. We are still awaiting a full accounting from them and will post our findings here as soon as they are available.

October 13, 2009, 7:20 AM PST – All of CDD’s servers are unreachable over the network. We are investigating and will provide updates here.

September 10, 2009, 2:45 PM PDT – We had 20 minutes of downtime today from 2:25pm to 2:45pm PDT, due to a miscommunication with our server collocation vendor about a networking change.

August 3, 2009, 3:20 PM PDT – The site is back up.

August 3, 2009, 2:40 PM PDT – Someone seems to have hit CDD with a barrage of intensive queries, and the server is not responding. We have sent someone in to hard reboot the server if it doesn’t recover itself in the next 15 minutes.

April 9, 2009, 3:26 PM PDT – The site is back up.

April 9, 2009, 12:42 PM PDT – Multiple fiber optic cables were cut early this morning in San Carlos and south San Jose, which has taken the secure colocation facility that houses our servers offline. The cables are expected to be repaired this evening, though there is a chance we could be online before then. Although CDD is not accessible during this time, your data is safe and secure.