Pushing Internal Data To Github: Learning From Our Mistakes
Learning From Our Mistakes: Win Or Learn
Let’s take a break from our Spotify Wrapped. It’s a good time to reflect on learnings and how to get better going forward.
I wanted to recall a quick story about a close call I had on the job some time back. If you’ve been doing this long enough, you will have these over the years.
A lot of times people only talk about their highlights or their accomplishments, think modern day social media.
But it is a good exercise to reflect and learn from our mistakes.
What Happened
Here’s what happened.
I have a running list of notes that I keep on my work laptop. Think things like a glossary with definitions to different tools, short code snippets, and more that I can
re-use in various situations.
One day, I was working on some changes through Github and I pushed this notes file unintentionally as part of a change.
This normally wouldn’t be an issue if there is no internal data within it, after all it’s just general notes. You can revert, and be on your way.
The problem was there was an internal webhook in the file. Webhooks are “always on” listeners that will deliver data when a specified action occurs. Think of a Typeform or Slack notification. These can be considered secrets when in version control.
After an internal tool picked this up, I was notified of this problem, I double checked the Github push, and sure enough there was the webhook. Due to the way they are formatted, Slack webhooks contain a secret at the tail end.
We then started an incident and remediation began.
Due to the way Github works, simply removing this file is not enough from a security and privacy standpoint. It’s not a bug, it’s a feature.
Running a command of
git rm file ; git commit -m "removing file" Will only remove it from your directory structure and repo, but it will still be in git history.
As the secret will remain in git history. I ended up using the git filter-repo tool to address this. It was also the first time I had used this tool.
For information directly from Github on this see Removing Sensitive Data From A Repository.
What this tool does is remove the specified file, any commits referencing the file, and rewrite its git history. Making it seem as if the mistake never happened.
In the end, I was able to resolve the incident and learn how to use a new tool while at it.
I would like to say this is the only incident I’ve caused or contributed to but that it is not the case 😅. Perhaps that’s a post for another time.
I like to go by the adage of “I either win or learn”, and this was definitely a learning experience.
Although this should not have happened, we can only learn from our mistakes and move forward. I think we can all take some time and remember moments where we really learned through adverse experiences and came out better because of it.
I like to go by the saying of “I either win or learn”, and this was definitely a learning experience.
This reminds me of a time at my previous career.
Some of you may know, before Cybersecurity I used to work in Electrical Utility (talk about overlap am I right? 😉)
One day, we were on a big job for PG&E. It was a 500 foot cable run, and when it was time to cut the cable, and I cut the cable a bit short. For context, in a worse case scenario this could mean having to re-pull the cable and costing the company tens of thousands of dollars. For a second, panic wanted to set in but I focused on remaining calm.
With my crew, I confirmed we had some slack (extra cable) on the other end and was able to make it work where we weren’t short on my end anymore. Careless mistakes like this can happen when we’re doing a job we’ve done a hundred times. I never forgot to make sure we had enough slack on either end of the cable run ever again.
“I either win or learn” - Nelson Mandela
What I Read This Week
This write-up about persistence mechanisms in Linux kernel module rootkits.
Banshee macOS malware source code leaked, available on github, thanks to VXunderground.
Nearly 40% of phishing for the year came from gTLDs such as .shop, .top, .xyz
With the U.S. Postal Service being the most impersonated brand of the year
Wrapping Up
Whether we learn from preparing for interviews, hands on CTF’s or our own reflection, there are always key takeaways for us to come back better.
As the year is coming to a close, it is a good exercise to reflect and self-assess.
If you enjoyed reading this, please share with a friend!