Level 3 & Packet8 major outage
BREAKING!
A source is telling me that Level 3 Communications and Packet 8 are having a system-wide network outage for the last 30 minutes. Level 3 is a huge provider of VoIP termination and I believe Packet8 uses their network for termination. Other VoIP service providers (Vonage?) will also be affected if indeed Level 3 is having an outage. I'm trying to confirm this as I type this. I'm currently on hold (5min so far) for Packet8's technical support and sent my contacts at 8x8 an email. I'll update this post when I find out more. Update: Just talked with Packet8 technical support and indeed they are experiencing outages. They also confirmed they use Level 3, so it appears Level 3 is the cause. Contacting Level 3 next. Stay tuned! Update: 1:13pm
This from my source of the outage who I asked to keep me aprised of when service was restored. The outage seemed to have occured from approximately 11am EDT to 12:15pm EDT (1hr 15min long). He makes a good point about some VoIP customers being medical offices, which are obviously greatly affected by any sort of extended outage.
Yes... I am an IT Management and Communications consultant - a few of my customers are Packet8 customers (mostly medical offices, which poses many potentially dangerous situations when phones go out). OK - As of 12:15pm EST Level 3 has restored their servers and service has been restored. PS - great blog, thanks for putting this out there... Looks like a few companies need a crash course on Crisis Communications Management and Planning - "keep you publics informed"... (feel free to blog that too.) Update: 1:34pm
Just spoke with Packet8. The issue was due to a Packet8 database crash at Level3's data center. Level3's NOC itself did not experience an outage. Also, stuff was failed over and business customers experienced NO outage - only residential. I'll upload a recording of the call that explains everything in just a minute. Here's a mp3 recording of my conversation with Packet8 representatives. Highlights of the conversation:
Packet8 was able to failover their business customers immediately. They were not affected at all. They prioritize the failover of their business customers over their residential. The residential customers would failover after 15 minutes or if they rebooted their device. Medical offices should not be affected if they use the business plan unless they use the softphone application or if they violate the ToS (terms of service) by going with the cheaper residential plan. (which shouldn't be used for business) Tags: 8x8, Level 3, outage, Packet8, VoIP
Copyright VoIP & Gadgets Blog Comments on this Entry: (Craig on Jul 31, 2008 12:59 PM) Um. I have 6 phone lines here, and a fax on the East Coast, and all work fine. I didn't go down once. We were up and running all day long. (Carolyn Schuk on Jul 31, 2008 1:25 PM) I've seen some Packet8 problems over that last week - calls not connecting or just getting a busy signal. (Alex on Jul 31, 2008 1:53 PM) This is a very interesting piece of info, since the medical office in question is on a Virtual Office business plan and was out for nearly 75 minutes - no dial tone, dead-air on pickups - on all 12 extensions.The auto-attendant answered calls, but caller choices were only routed to voicemail - no live operators. Having tested it myself the entire time of the outage, I guarantee this.In addition, I find it interesting that my first 8 calls to tech support were all transferred to dead-air, eventually being disconnected after approximately 2 minutes of wait-time. (Craig on Jul 31, 2008 2:01 PM) There has been absolutely no problems with QoS from Packet8 Virtual Office over the past week, or past month, or ever for that matter.I can't speak of Residential (we are a business using several business lines and fax). (Mike on Jul 31, 2008 3:01 PM) RE a crash course on Crisis Communications Management and Planning - "keep you publics informed"it was great watching this unfold. I am currently putting together a report on crisis communications at my company. this was a valuable lesson in both what to and what not to do (Eric on Jul 31, 2008 3:32 PM) We have many Packet 8 phone lines. Didn't experienced any issue. (C on Jul 31, 2008 3:52 PM) For the most part this guy can make calls out but nothing is coming in.Location = Mass (Robert Basye on Jul 31, 2008 4:29 PM) I just contacted customer service has no idea when they will be fixed Location KY (Kevin on Jul 31, 2008 5:30 PM) We have the new 675x SIP phones. We are still down. Our phones have been out since about 10:00 CST. Our location is Dallas, TX.I was able to reach Packet8 Tech support immediately both times I called today (from a different phone). They were helpful, offering to call me when the service is restored. I think it would be helpful if they offered updates from their web site, however. (Alex on Aug 1, 2008 11:02 AM) I am utterly disappointed by Packet8's response to this situation. Having listened to the audio recording and reading all subsequent blogs and articles written on the matter, I cannot grasp the reasoning behind Martin's remarks.As the IT consultant for THE medical office in question, I sincerely resent the statement that, "if you were one of our business customers, you were not affected.” This office, in addition to others who commented on this blog, did lose connection for extended periods; ours lasting for a total of 75 minutes. For an office of approximately 20 employees in a critical medical capacity, this was unacceptable. Furthermore, to insinuate that my client "opted – as they shouldn’t – for a cheaper residential line" is an outrage and a blatant disregard or even an acknowledgment of our communications loss.
A source is telling me that Level 3 Communications and Packet 8 are having a system-wide network outage for the last 30 minutes. Level 3 is a huge provider of VoIP termination and I believe Packet8 uses their network for termination. Other VoIP service providers (Vonage?) will also be affected if indeed Level 3 is having an outage. I'm trying to confirm this as I type this. I'm currently on hold (5min so far) for Packet8's technical support and sent my contacts at 8x8 an email. I'll update this post when I find out more. Update: Just talked with Packet8 technical support and indeed they are experiencing outages. They also confirmed they use Level 3, so it appears Level 3 is the cause. Contacting Level 3 next. Stay tuned! Update: 1:13pm
This from my source of the outage who I asked to keep me aprised of when service was restored. The outage seemed to have occured from approximately 11am EDT to 12:15pm EDT (1hr 15min long). He makes a good point about some VoIP customers being medical offices, which are obviously greatly affected by any sort of extended outage.
Yes... I am an IT Management and Communications consultant - a few of my customers are Packet8 customers (mostly medical offices, which poses many potentially dangerous situations when phones go out). OK - As of 12:15pm EST Level 3 has restored their servers and service has been restored. PS - great blog, thanks for putting this out there... Looks like a few companies need a crash course on Crisis Communications Management and Planning - "keep you publics informed"... (feel free to blog that too.) Update: 1:34pm
Just spoke with Packet8. The issue was due to a Packet8 database crash at Level3's data center. Level3's NOC itself did not experience an outage. Also, stuff was failed over and business customers experienced NO outage - only residential. I'll upload a recording of the call that explains everything in just a minute. Here's a mp3 recording of my conversation with Packet8 representatives. Highlights of the conversation:
Packet8 was able to failover their business customers immediately. They were not affected at all. They prioritize the failover of their business customers over their residential. The residential customers would failover after 15 minutes or if they rebooted their device. Medical offices should not be affected if they use the business plan unless they use the softphone application or if they violate the ToS (terms of service) by going with the cheaper residential plan. (which shouldn't be used for business) Tags: 8x8, Level 3, outage, Packet8, VoIP
- Related Entries
- Packet8 MobileTalk on iPhone Adds VoIP Calling - Jul 11, 2008
- Packet8 675xi VoIP for the SMB - Jul 24, 2008
- ACN IRIS 3000 Videophone & The State of the Videophone - Jul 15, 2008
- 8x8's Packet8 Posts Steady Growth - Jan 30, 2008
- How much data does fring consume? - Aug 05, 2008
- Well there goes VoIP's 5 nines (99.999) of reliability! - Aug 04, 2008
- Using Asterisk to Scam Credit Cards - Aug 04, 2008
- The IP-PBX Energy Wars... - Aug 04, 2008
- Skype 4.0.0.155 Beta 1 for Windows - Jul 31, 2008
- 3CX Free SIP Softphone - Jul 31, 2008
Copyright VoIP & Gadgets Blog Comments on this Entry: (Craig on Jul 31, 2008 12:59 PM) Um. I have 6 phone lines here, and a fax on the East Coast, and all work fine. I didn't go down once. We were up and running all day long. (Carolyn Schuk on Jul 31, 2008 1:25 PM) I've seen some Packet8 problems over that last week - calls not connecting or just getting a busy signal. (Alex on Jul 31, 2008 1:53 PM) This is a very interesting piece of info, since the medical office in question is on a Virtual Office business plan and was out for nearly 75 minutes - no dial tone, dead-air on pickups - on all 12 extensions.The auto-attendant answered calls, but caller choices were only routed to voicemail - no live operators. Having tested it myself the entire time of the outage, I guarantee this.In addition, I find it interesting that my first 8 calls to tech support were all transferred to dead-air, eventually being disconnected after approximately 2 minutes of wait-time. (Craig on Jul 31, 2008 2:01 PM) There has been absolutely no problems with QoS from Packet8 Virtual Office over the past week, or past month, or ever for that matter.I can't speak of Residential (we are a business using several business lines and fax). (Mike on Jul 31, 2008 3:01 PM) RE a crash course on Crisis Communications Management and Planning - "keep you publics informed"it was great watching this unfold. I am currently putting together a report on crisis communications at my company. this was a valuable lesson in both what to and what not to do (Eric on Jul 31, 2008 3:32 PM) We have many Packet 8 phone lines. Didn't experienced any issue. (C on Jul 31, 2008 3:52 PM) For the most part this guy can make calls out but nothing is coming in.Location = Mass (Robert Basye on Jul 31, 2008 4:29 PM) I just contacted customer service has no idea when they will be fixed Location KY (Kevin on Jul 31, 2008 5:30 PM) We have the new 675x SIP phones. We are still down. Our phones have been out since about 10:00 CST. Our location is Dallas, TX.I was able to reach Packet8 Tech support immediately both times I called today (from a different phone). They were helpful, offering to call me when the service is restored. I think it would be helpful if they offered updates from their web site, however. (Alex on Aug 1, 2008 11:02 AM) I am utterly disappointed by Packet8's response to this situation. Having listened to the audio recording and reading all subsequent blogs and articles written on the matter, I cannot grasp the reasoning behind Martin's remarks.As the IT consultant for THE medical office in question, I sincerely resent the statement that, "if you were one of our business customers, you were not affected.” This office, in addition to others who commented on this blog, did lose connection for extended periods; ours lasting for a total of 75 minutes. For an office of approximately 20 employees in a critical medical capacity, this was unacceptable. Furthermore, to insinuate that my client "opted – as they shouldn’t – for a cheaper residential line" is an outrage and a blatant disregard or even an acknowledgment of our communications loss.
