IT Networks in Dangerous Environments

Greg: Today we're gonna be talking about IT networks and dangerous work environments.

Not many of us aspire to be miners, to live on cargo ships several months out of the year, or even work for gas companies. But, of course, those who do work for these jobs are the working backbone of our economy. When you think of these types of jobs, consider the technology that needs to be managed and secured to get the job done effectively. IT in these industries is, just like any other tech company or financial institution...is fundamental to the inner workings of each business and of course, it might even mean life or death.

Today, we're going discuss the nuances of IT in a few of these industries. I'm joined with Dan Kirkland, who is a Tier III Technical Support Engineer at Ipswitch. How is it going, Dan?

Dan: Hey, it's going great today.

Greg: Dan's work has actually got him exposed to how IT personnel have to get creative and actually solve network issues in these industries. In likeness, Dan, what are some of the industries you've had a chance to work within? How are they doing IT differently, say, than your typical office environment?

Dan: We usually see people monitoring networks here. And it's usually, you know, small to medium-size network infrastructure. People branch out and do servers and some other items. But some of the items I've seen over the years go between mining equipment, gas valves that deliver gas to municipalities, dredging ships that are located around the bays and the coasts, internationally, and even deep freeze coolers for meat-processing plants out West.

Greg: Well, I'm just trying to fathom, you know, how maintaining a network at one of these companies could be like. I mean, with all this equipment, everything you have to monitor and the remoteness of these networks, that kind of brings a whole new definition to the word "remote."

Dan: Oh yeah, and, you know, you can step into some of the network rooms and they're climate-controlled and they're really great. Net admins have their wiring all laid out beautifully and in color-coded and so forth. But in, say, for the mining industry, for example, you have these huge mining trucks that are, you know, carrying earth from wherever the dig site is back to a spot for them to process through and extract the materials.

And these trucks are usually, you know, two to three stories high. They're in rugged terrain. They're having tons and tons of dirt dumped in on 'em. And these trucks are outfitted with little laptops that are connected wirelessly back to a monitoring tool such as WhatsUp Gold and giving off statistics on where they are, how long it's taking them to get through the field back into the process place, and also the physical health of the truck itself. So it gets pretty involved. These trucks pass through areas where they hit dead spots and the connection has to be remade, so, definitely a lot more challenging, monitoring environment than your typical corporate server room.

Greg: Yeah, that seems pretty intense, to say the least. You know, when I think of a network monitor I'm usually thinking about if your WIFI access point is up or not. So, the equipment that they're monitoring, I mean, it's not just for, you know, the health of the equipment. I would imagine that they use these monitors for safety as well.

Dan: Oh, absolutely. So, you know, with the trucks, well, say, if the hydraulics are starting to give out on a three-story truck, we're not worried about dumping the load. We're worrying about the brakes being able to apply. And, of course, it took a company that went in and developed a way of translating the counters or the information from the controls on the hydraulics into a format that could be broadcast through a network, and then reported on by a software. And the software happens to be WhatsUp Gold in this situation.

So, it's a way of having almost a centralized view of your entire business, because they also can look at equipment list and determine what's in service, what's out of service. And that will determine what gets monitored for that day.

Greg: So, you were talking about these dead zones. So, how would you know the difference between, say, the trucks at a dead zone versus, maybe, it fell into a ditch?

Dan: Well, they have an understanding of where the dead zones are, right? So they have an expectation of about every five minutes there may be a drop in connection due to a large geological formation, you know, a large pile of rocks or something. So, they've written it to ignore a five-minute drop or a drop every five minutes. But if the device stops responding within five minutes, then that escalates the device or, in this case, the truck...

Greg: Gotcha.

Dan: ...or the large...they have a huge shovel operator which is...I was looking at one last night, I don't know, it's like half of somebody's yard, and each scoop is very enormous.

So, those devices get monitored to conveyor belts, all the equipment that actually processes the soil to do the extraction. All those have some way of communicating to the outside world, and we're able to go out and bring that into WhatsUp Gold, and have a kind of a single management point for the entire mining infrastructure.

Greg: Oh, and before we forget. For everyone who's listening out there who doesn't know what the heck WhatsUp Gold is, it's Ipswitch's network monitoring tool.

So, we were talking about the IT personnel on cargo ships the other day. This is another instance of, you know, IT needing to be managed in really remote locations. So, how do these IT personnel go about doing this?

Dan: Well, okay, so on the modern cargo ships, you have the crew quarters. And these people are usually working these jobs for extended periods of time, so they all have WIFI for personal use. And then they have the ship network that actually goes back to the existing controls and you have engine controls. You've got all the hydraulics. You even have...on ships you've got a built tank pumps, you've got multiple built tank pumps because ships always have water in them, right?

Related Article: What is Network Flow Monitoring?

Greg: Yes.

Dan: So, these built tanks are always pumping the water out. They monitor those pumps to make sure that they're running efficiently. And then you have a monitor that tells you if the cargo has even shifted. But usually, if it has, I think most sailors would already know by the way the boat feels, right?

Greg: Yeah.

Dan: But this is monitored and it's pulled back into the ship, to the bridge. Again, WhatsUp Gold is up on the bridge being monitored, but it's also uploaded through a satellite link back to a base where if something goes on on the ship that the crew may not be able to respond to, the remote site is going to have a handle on as well. And then protocols begin once they establish, you know, there's an issue.

Greg: Wow, I mean these network monitors; I didn't realize they play such a vital role. It's like they're actually keeping the ship afloat.

Dan: Oh yeah, and, you know, again, the monitoring tool brings everything central and it acts as a secondary to the manual controls that an electronic setter built into the ship, right? So this is a failover. And there's a point that let's say, something doesn't show up on the panel but it shows up on WhatsUp Gold, they'll begin an inquiry back through, and maybe they'll identify an issue before something catastrophic can occur.

Greg: Yeah, and they do this by setting up like a certain threshold I would assume?

Dan: Yes, so even if you have, you know, some kind of...say there was a latency issue, and you had a time out, so you track that down and go, "Oh, okay, the network's crying wolf a little bit," but then you might go back and inspect and find where the actual mechanical control is not making clear communication back. And you can resolve that issue.

I was reflecting on an incident back in 2008 that came to mind when we were discussing this, and it was with a federal agency. I can say because it's been in the news. It was with the FAA (Federal Aviation Administration). I was driving to work. It was before 9 a.m. when our phones would snap on. And the news was out that they had to cancel all these flights nationwide because the air traffic controller network had gone down.

Greg: So, everything is grounded at this point in time?

Dan: Yeah, everything was grounded at this point. I sit down at my desk, my coworker is across from me and 9 o'clock hits, the phone snaps to life, and lo and behold it's the FAA. So what had occurred is they were monitoring their core routers which were located in Kansas City out...that was the other part, is like there's gotta be more redundancy than just that. And I'm sure after this incident that became gospel for them.

But the core routers had stopped working, but they were still pingable. So, the FAA, when they set it up, all they had us doing was pinging these monitors instead of going the full distance with SNMP (Simple Network Management Protocol), which would actually check to see if the device was working, it was operational. So the routers were on, but they were no longer speaking to anybody. And so we spent an hour and a half with them reviewing SNMP and how to set it up and how to monitor the devices. And as far as I know, that has not caused another grounding incident since then.

Greg: Yeah, so I guess it seems that everyone sets up their modules, their failsafes after catastrophe strikes.

Dan: Oh absolutely, yeah, you know, "Necessity is the mother of invention" as they say. So, yeah, people don't know what to monitor until something really key in their environment drops. And then it's you know, "We've got to figure out how to get this set up." And a lot of times it can be something rather simple.

Other kind of interesting call I had was with a gas supplier for a municipality. And so they would monitor the controls on the gas valves. And the gas valves would determine pressure that would feed out into each, you know, various lines. You get hundreds of thousands of gas lines running all over the place with a lot of valves on it. So it's a lot of monitoring, to say the least.

And evidently, this one particular gas valve had not been brought online with the monitoring system, with WhatsUp Gold monitoring system. And it released strong amount of pressure that, thankfully, was going to just one industrial building. So it was discovered at the station where the gas valve was located, but it was discovered after the fact.

And they had to assemble a lot of first responders to go out there and evacuate the building and then figure out how to back off all the gas that had filled these lines because, you know... Yeah, so it could have been a really tragic situation. They did catch in time. But I actually worked with this customer and, you know, making sure that he was collecting all the information off of that particular valve. And that was a really interesting call. I was like "Wow, we were in some big stuff here."

Greg: Yeah, now was it just bad luck that the one valve that didn't have the network monitor had the issue? Or was it other valves could have had his use but the monitoring tool was able to find that?

Dan: Yeah, the other valves...they had them monitored. They actually hadn't had a release issue on any of those. Those were running optimal. This was a valve that had gotten replaced, and when it was replaced it was not added back into the monitoring system.

Greg: Wow, I mean, it's fascinating that network monitors are, you know, that important to all these industries, and that it really is, you know, a failsafe to a life or death situation.

Well, unfortunately, that's all the time we had today, but I'd like to thank you, Dan, for chatting with us today, about all these fascinating stories of dangerous work environments and how IT is maintaining them.

Dan: Well, I appreciate you giving me the opportunity to share some of the exciting and diverse ways that WhatsUp Gold appears in the networking monitoring environment.