Learning From Our Mistakes
I wanted to retell a quick story about a close call I had on the job some time back. If you’ve been doing this long enough, you will have these over the years.
A lot of times people only talk about their highlights or their accomplishments. But it is a good exercise to reflect and learn from our mistakes.
What Happened
I have a running list of notes I keep on my work laptop. Think things like a glossary with definitions to different tools, short code snippets, and more that I can re-use regardless of the situation.
One day I was working on Github and I pushed this notes file unintentionally. This normally wouldn’t be an issue if there is no internal data within it, after all it’s just general notes. You can revert, and be on your way.
The problem was there was an internal webhook in the file. Webhooks are “always on” listeners that will deliver data when a specified action occurs. Think of a Typeform or Slack notification.
A couple days later I was notified of this problem, I double checked the Github push, and sure enough there was the webhook. Due to the way they are formatted, Slack webhooks contain a secret at the tail end.
We then start an incident and remediation begins.
Due to the way Github works (version control), simply removing this file is not good enough from a security and privacy standpoint. It’s not a bug, it’s a feature.
Running a command of
git rm file ; git commit -a Will only remove it from your directory structure and repo, but it will still be in git history.
As the secret will remain in git history. I ended up using the git filter-repo tool to address this.
For information directly from Github on this see Removing Sensitive Data From A Repository.
What this tool does is remove the specified file, any commits referencing the file, and rewrite its git history. Making it seem as if the mistake never happened.
In the end, I was able to resolve the incident and learn how to use a new tool while at it.
I would like to say this is the only incident I’ve caused but that it is not the case 😅. Perhaps that’s a post for another time.
I like to go by the adage of “I either win or learn”, and this was definitely a learning experience.
Although this should not have happened, we can only learn from our mistakes and move forward. I think we can all take some time and remember moments where we really learned through adverse experiences and came out better because of it.
“I either win or learn” - Nelson Mandela
Whether we learn from preparing for interviews, hands on CTF’s or our own reflection, there are always takeaways for us to come back better.
As the year is well underway, it is a good exercise to reflect and self-assess.
If you enjoyed reading this, please share with a friend!