Incident Response Faster Than a Speeding Bullet

When it comes to customer service and support, a lot of companies talk the talk, but most don’t walk the walk like Dojo Networks®. Here’s just one example of our team providing fanatical resident support: 

Around 2:00 AM one Sunday morning, just hours from the beginning of finals week, our network operations center (NOC) noticed that a data closet on the top floor of a 17-story student housing building went offline due to power loss. 

As our team began investigating the outage, the 15th floor data room also went offline. Realizing this could be the start of a significant outage, we checked and ensured there were no weather events or other power issues in the area. Our next step was to reach out to the apartment building’s emergency maintenance to see if the building had a power issue or if they were just doing late-night maintenance. Unfortunately, we couldn’t reach anyone in the building. 

Finding the Source of the Outage 

Unable to determine if the issue was with our equipment or simply a building power issue, we immediately deployed a technician to investigate. We updated the phone system message to acknowledge the outage and let any callers know our incident response team was investigating on-site. 

While enroute to the building in question, a member of our team was finally able to contact the head of maintenance. Apparently, he was just being released from the hospital. It seems that the building had flooded and, while investigating, he slipped, fell, and had to be taken to the ER. 

Our on-site technician arrived at the site at approximately 5:30 AM to discover that the data rooms on the 15th and 17th floors were flooded and much of the equipment had been destroyed by the water. Electronics and water don’t get along very well, after all.  

A Quick Fix 

Each data closet serves two floors, so the offline data closets were affecting four floors total. Working hand-in-hand with our remote engineers, our field technicians fished out any equipment that was still operable. After seeing what needed to be replaced, the Dojo team came up with a plan to get things back online.  

Using some on-site spare equipment that we maintain, as well as some creative use of other equipment we had on-hand, our technicians were able to get all offline floors back online and operational by 2:00 PM. 

Remember that it’s just before the start of finals week, so having functional Internet is even more crucial for students than usual. Here at Dojo Networks®, we treat Internet like a utility; we ensure it’s as reliable as the lights coming on when you flip a switch.  

Even though we had fixed the issue and everyone in the apartment building had reliable WiFi, we knew the fix was more like a band aid. So, on Monday, we overnighted equipment to replace the band aid and to restore the data closets. On Wednesday, during an early morning maintenance window, we reconfigured the site to its original design. 

This story is just one example of how Dojo Networks® delivers fanatical resident support.  

A Surprise Ending 

For those interested in the story, we did eventually find out what caused the flooding. A sprinkler on the 17th floor of the apartment building was set off, and it took the staff more than 30 minutes get it shut off. There was enough water to flood all of the stacked #10 apartments all the way down to the 3rd floor.  

What caused the sprinkler to go off? It wasn’t an exhausted student trying to get out of a term paper or looking for an excuse to get out of finals, as you might expect. The very unlikely culprit was a very random stray bullet in a very safe area fired somewhere on the street penetrated an apartment window that wound up hitting the sprinkler head.