• Information Resources
• Blog
• What does an Internet problem look like?

What does an Internet problem look like?

Update - Didn't realize comments were disabled.  Sorry about that.  Still new to this.  Questions, comments or criticisms, they are all equally welcome. In case you didn't notice, our campus Internet connection was having problems yesterday afternoon, starting at about 2:55pm until about 4:10pm. You can see that outage memorialized on this activity graph for our Internet link: The green area represents download activity (including web browsing and streaming) while the blue line represents upload activity.  Clearly, the outage occurred when campus Internet usage was close to its peak and we appreciate everyone's patience and understanding. Boring technical details below the break. The problems resulted from our Internet Service Provider performing some routine network maintenance when a software incompatibility between two core routers resulted in an infinite loop. This caused the routers to become too busy to pass network traffic. This initial problem was compounded when, in an attempt to break the loop, network engineers rebooted one of the routers only to find that it did not come back up with the proper configuration. Everything was sorted out by about 4:10pm when many of you probably saw that the Internet connection was working again. The network engineers, however, had the tough task of looking through the logs and configuration of the routers to figure out what, specifically, started all of the problems. That work took another two hours. At 7pm, our ISP sent an email out to all of their customers and let us know that they had figured out the source of the software incompatibilities and are working with their hardware vendor to get that addressed so this problem doesn't happen again. Now, I know we have a lot of bright students here at Oxy. Anyone want to take a stab at figuring out how to prevent infinite loops from happening in the first place? There's probably a Nobel Prize in it for you if you do.