PDA

View Full Version : Tech nightmare thread.



Phelyx
09-04-2007, 11:26 PM
The last week and 1/2 have been the worse work experience in my near 10 year career in IT. The scenario started after power outage around Chicago area hit my school pretty hard. The power didn't return to the school till Sunday early morning right before classes start on Monday. I came in on Sunday and made sure all my critical systems/servers were up and running and they were. Monday came as normal and some of the teachers started using the PCs while others waited for lessons to require them. Towards to late Monday network activity slowed to a crawl.

Almost like clockwork my network was able to send data for every 8-10 minutes then stop every 2 almost campus-wide. I didn't think much of it at the time figuring since the power being out 3 days and many computers didn't have their IP renewed I figured that the network was readjusting. I came in early Tuesday morning (5am) to find my servers were crashing due to this, could my Windows 2000 server finally be reaching near its end-time and can't handle all the DHCP requests. I quickly start digging out network material and proceed to do ping logs to try to diagnose the problem. I figured that since the problem is system-wide it must be because of the default gateway location since all data is trafficked inside the main router and also going through the firewall. I call the companies that handled the firewall, they noticed that there was no problem from outside internet to the device. They ping the machines inside my network and noticed the same problem, they thought I had a duplicate IP causing the crash, I call the routing company and they saw the same and figured that one of my prefix gateways was pointed to a wrong spot on the router causing replication. They fixed the issue which lead into.... Wednesday evening into Thursday.

I walk in Thursday to see the problem is not as bad 20-25 minutes but still continues to crash. Called the companies again they suggest it is one of my switches inside my network, yah I only have 70 of them, this should be easy. I talked to a previous tech person at this school she stated to me that the firewall is flakey and has been replaced before. I proceed to take a laptop one hop away from the firewall and logged pings. I see that even if isolate a connection away from further switches that the problem still results in the same matter, I start to assume that the firewall ethernet port might of been damaged from the power issues. I have my home machine and a firewall company ping my internal devices over the weekend. I see the same result and since the one company office doesn't open till 9 I call the routing company to bypass the firewall and go wall-less to prove it once and for all that it was the firewall..... it wasn't.

Today frustrated beyond believe I call the level 3 tech support at the state of Illinois, they had Cisco certification and could see in my switch, they were able to diagnose it to one wing of my building and to a primary switch of 24 additional circuits. By pulling RJ-45's and watching the results through a wireless laptop I was able to see if conditions improved or not and eliminate the issue to one cable, but it wasn't labeled so I don't know what it was for. I later get a tech call in saying that these rooms lost all network activity and they were saying no connection.

It is about 4pm and I go to see about recabling these rooms to which i saw a switch hooked up to the internet with one cable hooked in a loop (BOTH ENDS OF THE CAT5 IN THE SWITCH) plugged up with additional PC's connected. Apparently this one looped cable sent multiple mac-address messages causing all the switches on the network to get confused and restart at a regular interval.

Apparently the person who taught in this classroom previously quit, in the process of cleaning the room he secured the switch around the table with a cable and left it unplugged. The new teacher who went in the room did as instructed and plugged in the switch when he started teaching his curriculum on monday afternoon. My boss instructed all teachers to unplug their items before leaving their rooms before summer vacation and to secure all electronics. :mad2: :smash:

I was very close to quitting and never got so frustrated.

Dan18960
09-05-2007, 08:48 AM
Don't you love it when someone HELPS :mad:

I have a customer that ALWAYS wants to help while I am troubleshooting a problem at their office. It has lead to some weird problems (but hey - it's all billable time :D )