Friday, January 8, 2021

Man Insists He Knows IT Better Than IT, Learns Otherwise


Anyone who has ever worked tech support knows that the job is brimming with testy customers that seem to have a relentless vendetta toward taking out all their angst about this wild life, on the very people that are just trying to help them figure out how to use their technology. This results in some seriously unfortunate interactions. People just need to play nice with one another. Check out another tech support "glory story" with this person who changed their phone password, couldn't charge their phone, and then assumed that their phone was dead. 

1.

Text - r/talesfromtechsupport u/airzonesama · 3y •I Am Not Good With Comp. I know IT better than IT Epic r/ALL So a few years back, I was working in a manufacturing company as IT manager. Like many industries, we had a number of machines with embedded computer systems. For the sake of convenience, we called these "production machines", because they produce stuff. By and large, these PC's are just normal desktop PC's that have a bunch of data acquisition cards in them connected to a PLC, or a sec

2.

Text - I've been in manufacturing environments before, and this concept is not new to me. Thanks to a number of poignant lessons in the past, I make it my business to understand these PC's inside and out. I like to keep them on a tight refresh cycle, or when it's not practical (in the case of archaic hardware or software), keep as many spares as possible. Also, regular backups are important - you just have to understand that unlike a normal PC, it can be difficult to do and plan it well in advan

3.

Text - Airzone: Me, the new IT manager. TooExpensive: The site's facilities manager. He's in charge of the maintenance of the site, including all of these production machines. He's super paranoid about people trying to take his job, so he guards all his responsibilities jealously and doesn't communicate anything lest they get the drop on his efforts. Oh, and he has a fixation about not spending company money - even to the point of shafting the lawn-mowing guy out of a few hours pay - hence the n

4.

Text - I'm new, but in my first few weeks l've already had a number of run-ins with TooExpensive. I'm a fairly relaxed guy, but I have no qualms about letting someone dig their own grave and fall into it - and in the case of TooExpensive, l'd be happy to lend him my shovel. My pet hate was when organising new network drops, I will always run a double when we needed a single. We're paying working-at-heights money already, and a double drop is material cost only. i.e. Adding $50 - $100 material on

5.

Text - So you could see that we didn't exactly gel together well. Which brings us to these production machines, and the PC's nested within. Every attempt for me to try and document, or even understand them was shut down by TooExpensive. Me: Hardware and software specifications? TooExpensive: That's my job, get lost. Me: Startup and shutdown procedures? TooExpensive: That's my job, get lost. Me: Backup? TooExpensive: That's my job, get lost. Me: Emergency contacts? TooExpensive: That's my job, ge

6.

Text - Move forward a few months and I'm helping one of the factory workers on their area's shared PC. It's located right next to one of these production machines. It's old. The machine itself was nearly an antique, but the controls system had been "recently" upgraded. It had co-ax network of 2 PC's - one NT4 primary domain controller, and a NT4 workstation, and a network PLC (also on co-ax). The machines were pentiums running the minimum specs for NT4 to run, with a control application whose ap

7.

Text - It was test&tag day and TooExpensive was running around a sparkie to do the testing. My earlier instruction to the sparkie was to not disconnect any computer equipment if it was not powered off. And so it came time to test this production machine's PC. The sparkie wasn't going to touch it while it was on. Luckily TooExpensive came prepared with his thoroughly documented shutdown procedure: yank the power cords. The test passed, new labels were applied to the power cord, he plugged it back

8.

Text - 10 minutes later, the operator looks at me and asks for help. I call TooExpensive's mobile, and it's off. I called VPO's mobile and suggest that he comes over immediately. 10 minutes later, the operator, VPO, and I are looking at this machine. It's fucked. There's the better part of a million dollars worth of product to be processed by this machine, and the nearest alternate machine is in Singapore, belonging to a different company. And if the processing isn't done within soon, the produc

9.

Text - So as I have said, I have used this software before and have a basic understanding. I know enough that the configuration is everything, and configuration is matched to the machine. But I also knew a guy who did some of the implementations. A call to him gave me a lead, and I followed the leads until about 4 calls later, I had the guy who implemented this particular machine. OldBoy had retired 10 years earlier, but VPO had persuaded him to come out of retirement for an eyewatering sum of m

10.

Text - To cut long story short, we had to rebuild the database. But not from scratch. OldBoy's MO was when setting up a machine, when he was done, he'd create and store a backup database on the machine. The only issue was that 20 years of machine updates needed to be worked out. It also just so happens that through sheer effort, I am able to compare a corrupted database file to a good one, and fool with it enough to get it to load in the configuration editor. It's still mangled, but we are able

11.

Text - After OldBoy and I started working on the problem, he showed up again. He ignored any advice about a backup (because obviously there wasn't any), and instead demanded regular status updates for him to report to VPO. The little shit had screwed up the machine, run off to hide, and now a solution was in progress, was trying to claim the credit. When it was all running again, OldBoy debriefed VPO on the solution. I then had my turn with VPO. VPO: So Airzone. Thanks for your help. Your effort

12.

Text - VPO: He claims he was busy contacting his emergency contacts. Me: I see. VPO: I don't believe a word of that shit. Unfortunately it's your word vs his. If I had the evidence, l'd fire him. Me: (opening the email TooExpensive had sent me about meddling on my phone) You mean this evidence? Half an hour later, I got the call to lock TooExpensive's account and disabled his access card. Edit: Wow, this story seems to have resonated with so many people here.. And thanks for the gold, kind stran

Submitted by:

Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)