Significant Havoc In Technology does happen!

In my years managing network infrastructure, I’ve seen outages caused by everything from human error to nature’s wrath. Each incident was a puzzle, often with a solution as bizarre as the problem itself.

At WBS, we had a microwave link that kept dropping, leaving everyone stumped for weeks. Engineers scratched their heads, pointing fingers at equipment configs and signal interference. Then, one day, someone noticed a massive crane on the neighboring construction site. Its loaded bucket swung directly into the line of sight (LOS) of our link every time it rotated. A classic case of “moerse” machinery reminding us to check the physical layer first.

Another time, we had intermittent failures at a high site. The IP Core team blamed the radio crew, and the radio crew fired back. Frustrated, I sent one of my guys out to investigate—genchi genbutsu, go and see for yourself. What he found was a new, hulking generator installed by the property owner. Its exhaust pipe was dumping burnt diesel fumes straight into the air inlet of our equipment container. The inside was coated in a grimy layer of soot, frying our gear intermittently. We had to pull everything out, clean it, and relocate the air intake to stop the madness.

Then there was the MTN Gallo Manor data centre fiasco. The whole network went dark one day, and we scrambled to figure out why. We had redundancy with cabinets at Teraco, but our DNS guy had fibbed about having dual servers. In a pinch, we pointed everything to Google’s 8.8.8.8, and, as they say, Bob’s your uncle—back online. The root cause? ICASA, armed with a court order to remove an unlicensed radio, mistakenly yanked our fibre switch instead. The data centre manager was utterly useless, letting ICASA run rampant without question. He even tried to block us from accessing our own racks. Total chop. We didn’t stick around—moved everything to a new facility in Midrand.

Perhaps the wildest incident was at the Toyota factory in Durban. The network kept failing, and no one could pinpoint why. I went down to check it out and found the culprit: pigeons. Their acidic droppings had eaten through the roof of the network rack, seeping onto the switches below. The corrosion was wreaking havoc. Who knew pigeon poop could take down a network?

These outages taught me one thing: never underestimate the chaos of the real world. From cranes to exhaust fumes to rogue regulators and pigeon poop, the network game keeps you humble.

The Wild Days at MTN Gallo Manor Data Centre

The MTN Gallo Manor data centre was a beast of its own kind, a place where chaos and ingenuity danced a gritty tango. It was supposed to be a fortress of technology, a humming hub of connectivity, but more often than not, it felt like a house of cards teetering on the edge of collapse. Outages were as common as morning coffee, and the whole operation seemed to limp along on hope and duct tape.

The guy in charge—let’s just say competence wasn’t his strong suit. He’d strut around with a clipboard and a scowl, but when the chips were down, he was about as useful as a screen door on a submarine. The real trouble, though, was the infrastructure. Power continuity was a pipe dream. When Eskom’s grid inevitably flickered out, we’d all hold our breath, waiting for the backup generators to roar to life. Too often, they didn’t. Those big diesel beasts, meant to save the day, would sit silent, leaving the entire data centre plunged into darkness. Servers whined to a halt, and we’d be left scrambling in the gloom, cursing under our breath.

Then there were the water chillers. Poorly installed and barely maintained, they were a disaster waiting to happen. They’d overload circuits like clockwork, tripping distribution boards and sending us into another round of firefighting. It was as if the whole setup was conspiring against us, daring us to keep the lights on.

But we weren’t the type to roll over. When the data centre failed us, we took matters into our own hands. We became the data centre cowboys, riding in with our own solutions. Picture this: flatbed trucks rolling up with portable generators, their engines growling where the centre’s diesels wouldn’t. We’d snake power cables through emergency exits, rigging up our critical equipment to keep the show running. It was messy, it was unorthodox, but it worked. We’d huddle in the server room, the hum of our makeshift power supply drowning out the silence of the failed systems, and we’d grin like outlaws who’d just pulled off a heist.

Those days at Gallo Manor were a trial by fire. The place was a mess, but it taught us resilience, improvisation, and a certain stubborn pride. We kept the data flowing, not because the systems were sound, but because we refused to let them win. We were the cowboys of the data centre, and in that chaotic, flickering world, we carved out our own legend.

When Water Waged War on Our Networks

In the high-stakes world of telecommunications and IT, network outages are the stuff of nightmares—those invisible gremlins that can bring entire operations to a screeching halt. Over my career, I’ve battled my fair share of them, from rogue software bugs to power failures, but none were quite as dramatic (or soggy) as the times when Mother Nature, or rather faulty plumbing, decided to turn our workplaces into impromptu aquariums. These floods didn’t just soak the floors; they drowned our networks, forcing us to improvise, adapt, and ultimately rebuild. Two incidents stand out vividly in my memory, both involving burst water systems that turned the office space into watery wastelands.

The first deluge hit while I was at Datavia in Braamfontein, Johannesburg. It was a late-night grind; I was deep into coding my branch line management information program, tweaking algorithms under the hum of fluorescent lights. Around midnight, I heard it—a persistent rushing sound, like a heavy downpour. Assuming it was just another stormy evening in the city, I ignored it at first, focused on my screen. But the noise didn’t let up. Curiosity finally pulled me from my desk to the window, where I peered out expecting sheets of rain. Nothing. The streets were dry and there was no rain. Puzzled, I ventured out of our office area toward the lifts, only to be met by a surreal sight: water gushing from the ceiling like a broken fire sprinkler on steroids.

I hurried down the stairs, my shoes squelching on increasingly wet steps, and emerged into the foyer, which had transformed into a veritable swimming pool. Water lapped at the reception desk, papers floating like lily pads. Outside, the security team stood helplessly, watching the chaos. “A pipe burst upstairs,” one explained, phone in hand. “We’re calling a plumber.” I couldn’t believe it. “Why not just shut off the main water supply?” I asked, incredulous. They shrugged—nobody knew where the shutoff valve was. With the building flooding and no immediate fix in sight, I decided it was time to call it a night. I waded to my car, drove home, and collapsed into bed, leaving the mess behind.

The aftermath was a week-long ordeal. The water had seeped into everything: walls, floors, and crucially, our communications equipment. Servers shorted out, cables corroded, and the network went dark. We spent days drying out the space, replacing soaked hardware, and rerouting connections. It was a stark reminder of how vulnerable our digital lifelines are to the physical world—one burst pipe, and poof, connectivity vanished. But we emerged stronger, with better backups and a newfound appreciation for building maintenance manuals.

Not long after, fate decided to test my flood-resilience again at iBurst. I arrived at the office one crisp morning, coffee in hand, ready to dive into the day’s tasks. But as I pushed open the door, I was greeted by pandemonium: ceiling boards lay in sodden heaps on desks, papers plastered to surfaces like wet confetti, and puddles everywhere. It looked like a bomb had gone off—if the bomb was made of water. Turns out, the holding pool on the roof had ruptured overnight, sending a tsunami cascading through the building. The deluge had short-circuited the electrical systems, tripping the power and plunging us into darkness amid the damp.

We scrambled into action. The power was restored after a few tense hours of electricians poking at fuses and drying out panels, but the network outage lingered as we assessed the damage to routers and switches. For weeks, we worked under an open sky of exposed beams and wiring, no ceiling boards to shield us from the fluorescent glare or the occasional drip from lingering moisture. It was makeshift office life at its finest—desks relocated & fans blasting to dry things out. Yet, in that stripped-down environment, creativity flourished; we bonded over the absurdity, turning potential disaster into a story of resilience. The full repairs took time, but the experience hammered home a key lesson: in tech, as in life, adaptability is your best defense against the unexpected.

These watery woes were more than mere inconveniences; they were humbling encounters with chaos, underscoring how interconnected our digital and physical worlds truly are. In the end, they didn’t just disrupt networks, they rebuilt my perspective on preparedness, one soggy step at a time.

Ronald Bartels | LinkedIn | Instagram