Yes, the site has been down.

dvsDave

Benevolent Dictator
Administrator
Senior Team
CB Mods
Fight Leukemia
Joined
Jan 31, 2003
Location
DC Metro Area
Alright, I am sure everybody is wondering why the site has been down a lot lately.

Two reasons:

1) Hard drive failed. This is a report from the hosts support site:

02/21/2004 1521 CST: The hard drive on server PEGASUS has failed. The hard drive is currently being replaced. Accounts will be restored from backup as soon as possible. More information will be posted here as it arrives. Please do not open a ticket on this issue. We are doing everything possible to get your site up as quickly as possible. We are very sorry for this unfortunate hardware failure.
02/21/2004 1759 CST: Restoring accounts from latest full backup.
02/22/2004 0545 CST: 100% of accounts have been restored.
2) Network at the data center was down. This is a report from the hosts support site:
02/28/04 1600 CST: We are currently experiencing problems with our upstream transport provider. We are working on the issue and hope it have it resolved ASAP. At the moment there is no ETA. Please bear with us as we work with the telco to resolve this problem.
02/29/04 1746 CST: Latest update we have is that CTSI is enroute with a GIG-E test kit, and replacement card(s) for their equipment. No further information is available at this time. When we have such information, we will make it available to you.
02/29/04 1807 CST: Circuit is up but not for good. It came back up by itself without anything getting repaired by CTSI--just like it did last night. It is being taken down for testing by CTSI shortly tonight. Hopefully they will finally get to the bottom of this and fix the stability and packet loss on their equipment once and for all.
02/29/04 1816 CST: CTSI is taking down the circuit completely for testing tonight, rather than to have this ongoing until a new fiber is installed. This is a nightmare for us as well, as we feel all of your pain and stress because of an incompetent telco cannot seem to quickly fix their equipment once and for all. As you can imagine this is not easy for us either.
 

zac850

Well-Known Member
Joined
Nov 21, 2003
Location
New York
These things happen, I'm sure we all understand that, even if we do loose time from CB :( :(

Thanks for the info Dave
 

dvsDave

Benevolent Dictator
Administrator
Senior Team
CB Mods
Fight Leukemia
Joined
Jan 31, 2003
Location
DC Metro Area
Update:

02/29/04 2318 CST: The problem was discovered to be a multi-level issue within the providers network. Tomorrow more details should be available.

I'll try to keep you updated as I get them!
 

dvsDave

Benevolent Dictator
Administrator
Senior Team
CB Mods
Fight Leukemia
Joined
Jan 31, 2003
Location
DC Metro Area
More updates:

03/01/04 1037 CST: CTSI is currently working on the circuit again. They tell us they are swapping out GIG-E cards on their equipment in one of their POPs.
03/01/04 1325 CST: CTSI techs are at all locations (Scranton POP, Philadelphia POP, & the Scranton Data Center) and they are currently still working on the issue. We have no ETA on repair, as they will not give us one.
03/01/04 1446 CST: CTSI is now testing the whole circuit end to end, rather than from hop to hop. That is all the further information they have given us.
 

The_Terg

Active Member
Joined
Dec 3, 2003
Location
Yonkers, NY, USA
Well, atleast they are good enough to give you such comprehensive updates.....

And man, something naughty must be if for them to be pulling gigabit cards all over the network....
 

dvsDave

Benevolent Dictator
Administrator
Senior Team
CB Mods
Fight Leukemia
Joined
Jan 31, 2003
Location
DC Metro Area
LAST UPDATE:

03/01/04 2230 CST: Beginning on Saturday Feb. 28th and continuing through Monday March 1st the Scranton Data Center experienced several extended periods of downtime. The outages appear to have been caused by multiple variables in several facilities. During the outages, extensive testing, maintenance, and replacement of gear in our Scranton facility, in our Philadelphia facility, and in our fiber providers facilities was performed. Possible problematic pieces of equipment and cabling were found in all facilities, including the apparent cause of the recent packet loss issues. (tests show 0% packet loss since restoring of service this evening) The basic summary is that our fiber provider had gear that needed to be swapped out due to errors, and one piece of network gear and 2 fiber cables that also showed errors. It is basically impossible to exactly pinpoint which piece caused the issue. Likely, any one of these by themselves would not have caused difficulty in tracking and diagnosing the issue (such as low-level packet loss) but, when combined, seemed to trigger this weekends outages. We apologize for the large inconvenience, and thank our customer base for their patience. The managers of the Scranton Data Center will be meeting to seriously evaluate and discuss the outage and their response to it, as well as to discuss possible solutions for avoiding further problems of this magnitude.
 

dvsDave

Benevolent Dictator
Administrator
Senior Team
CB Mods
Fight Leukemia
Joined
Jan 31, 2003
Location
DC Metro Area
old host.. the new host hasn't responded to my emails yet so we haven't made the switch yet...