What I Learned as a Young Systems Engineer

The Unwritten Realities of System Engineering

I was 23 years old when I was first bestowed the title of Systems Engineer. I was proud to have finally reached the ranks of what I considered the elite at that time: the innovators, improvers, and improvisers, the ones that System Administrators aspired to be. This post reflects on the vital, often unspoken lessons I learned in those foundational years—insights that no classroom or textbook prepared me for.

1. The Unofficial Rule of Responsibility

In my first few months, I fixed an issue with a specialized system. It was one of those situations where the organization simply hoped the system would continue to function, as the single resource managing it moved on to bigger and better things. Eventually, and inevitably, the system failed. I was left to figure out what was wrong. Fortunately, I had some familiarity with the application/operating system and was able to recover the system after several hours. Additionally, I was able to explain, with a reasonable level of confidence, what went wrong. I was triumphant and felt like I had proven my value to my new organization. However, that success came with consequences, and I quickly learned that if you are the person to resolve an issue with a cryptic system, you become the point person for that system.

Life Lesson: “Solving a problem often means inheriting responsibility for the system you fix.”

2. The Constant of Change

The IT field is ever-evolving. Technologies that were cutting-edge during my younger years quickly became outdated. 10base5 gave way to 100Base-TX, which gave way to 1000BASE-TX. Storage capacities continue to grow exponentially. AI is a fast-emerging industry, which I’m not quite sure society is truly ready for. This field can be a grueling exercise as you try to keep up with the latest technology, security, and best practices. There aren’t many fields that evolve as fast as technology. For example, you could take an auto mechanic from the mid-80s, place them in front of an ICE vehicle, and, with the exception of a few upgrades to fuel efficiency and computer diagnostics, they would get the hang of it in a relatively short amount of time. Take someone in IT from that same time period and put them to work today; they’d be practically lost. In fact, I would argue that IT forces other industries to transform. Every industry is touched by information technology; heck, there are industries that couldn’t exist without today’s capabilities. However, keep in mind that this rapid evolution arguably started in the 90s. That’s when personal computers started to become a household product, quickly followed by dial-up internet or AOL for many.

• Life Lesson: “Mastering anything in IT is impossible, having a strong understanding is quite possible“
• Life Lesson: “Rapid Change is inevitable in information technology. Embrace it or find another career“

3. The After-Hours Assumption

As a young man without a family, there were definitely some unfair assumptions placed on me, such as the value I placed on my time off, whether that be at the end of the day or the weekend. There were many seasoned individuals with families who assumed their time was more valuable simply because they had children. While I can sympathize with the loss of time with your family, as I am now a family man myself, it’s unfair to assume that being single means you can work more on the weekends and holidays because you won’t be missed as much. Balancing work-life boundaries became an essential skill, ensuring that my personal time was respected.

• Life Lesson: “No ones life is more or less important than anyone else on your team. Defend your time off.“
• Life Lesson: “Never make assumptions on a persons’ availability based on their age or family status”

4. The title does not necessarily represent talent

Gosh, I remember my first System Engineer title back in 2002 (yes, I’m old). I was nervous. Now in the big leagues, I thought to myself, I’m going to have to study and learn and absorb as much as I can from this team to succeed. While some of that was true, I was surprised to discover many of my peers lacked Command Line Interface (CLI) experience, a fundamental skill in computer diagnostics. I quickly realized that some folks were promoted not because of skill but because of who they knew. Additionally, I quickly discovered that there were a handful of “true” engineers that others looked to for leadership. Out of those few skilled individuals, you will find one that will be happy to shepherd you along.

• Life Lesson: “Titles are just titles until you see them in action“
• Life Lesson: “Identify the true stars and absorb as much as you can”

5. The Art of Technical Documentation

Early on, I was tasked with creating extensive technical documents. Many in IT will relate to the fact that documentation on infrastructure is generally lacking or simply outdated. I somehow found a niche in creating technical documentation. I became the go-to person when it came to creating technical documents, instruction sets, physical technology models—heck, people even asked me to format their resumes. Over my 25-plus years in IT, I’ve probably created well over 300 various documents, diagrams, workflows, and spreadsheets. The largest document was approximately 225 pages on an EHR system. However, of the many documents I’ve created over my career, I would assume maybe 20% of them were actually read. Gradually, I realized that these documents, despite their detail and complexity, were seldom read.

• Life Lesson: “Documenting is a very valuable skill in information technology. Don’t expect many to actually read the document though”

6. Navigating the Blame Game

The struggle is real. IT is comprised of individuals who take great pride in their domains. Accusing their technology of causing a failure or even contributing to an issue is tantamount to a declaration of war. I’ve found myself in numerous conversations with other departments, attempting to navigate through their denials just to have someone take a look at the problem. Often, the time spent requesting a resource exceeds the time it would take to actually fix the issue. These battles are frequent, but none have been more intense than the skirmishes between system and network engineers. Ultimately, it doesn’t matter which side is assigning blame.

• Life Lesson: “If you’re told the firewall isn’t the issue, ask them to check it anyway.”
• Life Lesson: “If you’re informed the server can’t be the issue, insist on checking it anyway.”
• Life Lesson: “When you hear the router isn’t the issue, request a check on it anyway.”
• Life Lesson: “If you hear DNS isn’t the issue, it’s usually DNS. :-)”

7. Personal Reflections and Growth

These lessons, learned in the trenches of system engineering, have profoundly shaped my understanding of the IT landscape. They underscored the importance of continuous learning, effective communication, and mastering the art of negotiation and diplomacy in a technical environment. Moreover, they reminded me to also have a little fun. Recently, my friend Scott pinged me to wish me a Happy Thanksgiving. He attached an image—it was our creation: The Patch Pig. It represented the culmination of late nights and long weekends spent together. Whenever we faced a zero-day exploit, Patch Pig was there. Our journey together lasted barely a year, but it was one of the best years in my early system engineering career.

8. And to wrap up

As I navigated my career in system engineering, I realized that the most valuable lessons stemmed from experiences, challenges, and interactions that no training manual or college course could ever cover. These insights not only honed my engineering skills but also equipped me for the unpredictable and dynamic world of IT.

To those embarking on this journey, bear in mind: the path of a system engineer is as much about learning from your environment and experiences as it is about mastering technical skills. Welcome the lessons that come your way, for they are the building blocks of a rewarding career.