"Often in IT people can be scared to break things, and bring down production. Sure you can learn from a textbooks, one of my preferred methods. However, nothing truly helps you learn than when you broke it, now you gotta fix it!"
The realization
Imagine this, you're doing your normal job. You get asked to remove a disk from a virtual machine in vmware. This disk was no longer needed. You are already in the cluster, you click on the virtual machine, you go into the settings, and you detach the disk. Only to realize later that you detached the disk of the wrong VM, and not only that. But you disconnected the disk of a running database server. TALK ABOUT PANIC! 😨
Well, thats what happened to me a few years back. I accidentally removed the wrong disk, from the wrong vm. This isn't the first time I had a similar thing happen to me. Another time, i disabled the uplink on a distribution switch. Had to run to the location, and reboot it quickly before too much time went by.
The Impact
You can sit there and look at situations like this as the most absolute failures. Yes, for me these had some minimal business impacts. Which now you're probably wondering how removing a database drive has minimal impacts? Thats because the server it needed to be used on didnt need to be available for a few days. Allowing time to resolve the issue, and restore from backups. The business impact it did have was man hours. DBAs and other team members all stepped in to help clean up my mess. That was embarrassing. Embarrassing, and enlightening.
The Process to Improvement
Moving forward from those events felt hard. In the moment I felt ignorant, not smart enough, and down right like a failure to those I work with. I feel like it can be my responsibilities to be at the top of my game a lot. Especially when it comes to things I feel passionate about. Especially networking, virtualization and automatons. However, as time moved forward I did too. I moved forward with new ideas. Ideas to check and verify that I am working on the right server, that I am modifying the right ports, that I am verifying things the right way in the right order.
I truly feel that the moments where I have broke things myself, or others have broke things. Then, I have assisted to help resolve it, those are the moments where I have learned some of the most. Stepping in during a network loop or a broadcast storm, helps you learn how to track mac addresses better. Unable to find the source of flooding packets, plug a laptop in, look at Wireshark, you may see something new.
All of this is to say, I have borked things, you will borke things. To learn, you must borke.