Why develop when you have to audit
From BSides Boulder 2024 , let’s layer on the business and people complexities on top of this deeply technical problem. Despite all the hardships we’ve reviewed, building software and systems in highly regulated environments can still be rewarding, fast-paced, and fun! This is an expanded set of slides and resources since shown live on 14 June 2024 (YouTube recording).
📚 We’re almost an hour in - who’s learned something today? I bet your teammates, managers, and auditors are in the same position! Doesn’t it feel great to learn something new?!
Now let’s do the most difficult thing - explore the business and people facets of this far-reaching technical problem. 🥴
Some assembly required
The whole idea of an enterprise wide “software factory” has continually gained traction over the past ten years or so. It’s a fantastic premise with one, not so obvious, implication:
✨ This system is in scope for both itself AND everything developed on it. ✨
The first one is pretty straightforward, even if it can be technically difficult - configure your central remote in accordance with whatever hardening standards necessary (eg, NIST 800-171). Once that’s done, document compliance on your endpoints, identity management system, datacenter or cloud provider, “people stuff” like information security training, and all the other seemingly random stuff that’s needed.
That second one is where I spent way more time, though. It’s being organized on all of the above, then working with teams to make sure that the controls they said need are in place on the platform. Part of that is documenting your own controls in a reusable way to be included in many other audit packages. Another difficulty is in preventing “colliding controls” between teams or trying to scope settings such to cover everything needed and nothing more.
In moving into a more consultative role, I learned it wasn’t just me that struggled here. Many of these central git repositories assume things that simply aren’t true of complicated industries. A software factory is a SaaS service provider with a bunch of (internal) customers. Here’s a couple tips that made my life easier:
- Keep the paperwork in order at all times. It’s always in demand.
- Self-service is your best friend. The worst part of this “internal SaaS” idea is that every “customer” can direct message you all the time. There’s only so many teammates who can answer questions.
- Carve out time for proactive improvements. This is the hardest, yet most impactful long term, to the success of the project. The team running this needs time to learn new things, to experiment with new features or additions, to proactively improve the platform. Neglect this at your own peril.
- Don’t stop shipping code! Never forget what your “customers” experience daily. The further you are from writing software, the harder it is to navigate the conflicts of technology and people writing software successfully.
This is a job
Defense, manufacturing, energy, financial services, and healthcare are all huge swaths of the global economy on their own. They purchase and write and maintain and deliver software products to exist. That needs a lot of people to accomplish - or, better said, factories need people to function.
These are jobs.
People work for money.
There are tons of people, maybe even you, who take part in this system solely in exchange for money.
And it shouldn’t suck to do it.
It’s more secure, reliable, and economical when we factor in making systems delightful to use. Great user experience is a value in itself as kindness to your fellow humans.
Auditors are human too
Running centralized infrastructure for software developments gets caught at the many intersections of strongly held opinions, institutional politics, and third-party audits. There are always way more opportunities to disagree than expected - it doesn’t mean you have to fight about it all the time. Coming across as collaborative first has always helped me further platform security and adoption.
Having done this a time or ten, I wish I learned this one thing sooner:
✨ You don't have to show up to every argument you're invited to. ✨
Some examples I’ve had to work through in an audit:
- An auditor that didn’t understand the protocol used to fetch code doesn’t alter it. definitely an education opportunity
- A senior leader wants to change how teams structure their code review approvals and testing procedures, in the middle of an audit. do not argue, advise of timeline impact
- An auditor who misunderstood their scope, wanting to place all development company-wide into a single repository. this is misery should it succeed, time to engage!
Or in other words, auditors are human too. 🤖
Impact like no other
“The tech companies hired like crazy,” Saker says. “Those giants hired some of my top engineers.”
However, job switching sometimes doesn’t always go as planned. Some of the engineers who moved over to Big Tech got placed in sales engineering roles, which pull engineers further away from programming and working directly with technologies. The longer the engineers are away from the keyboard, the more the “itch” starts to creep back in to code and build systems that “touch people’s lives,” Saker says.
Lake, S. (2023, Feb 14). “Booz Allen-one of the largest cybersecurity employers-is reacting cautiously to big tech layoffs.” Fortune Education. (link )
One of the very first systems I worked on at Booz Allen was a website that enabled servicemen and women to find out how and where to request an absentee ballot to vote in elections in their home districts. Getting this improved in time for the next election was both important and urgent. It was in use by thousands of people, enabling them to have their voice heard.
Systems I had a hand in building have ended up in dozens of places that it’s alright to feel warm and fuzzy about. While I can’t speak for industries like health or finance, I couldn’t imagine working without the possibility of outsized influence you can have in the public sector.
Working in regulated industries can have an opportunity to make an impact like no other similar job.
Parting thoughts
GitHub’s original byline, circa 2008, “git repository hosting - no longer a pain in the ass”
Git supports a broad spectrum of development patterns. The right controls for your project or company depends on unique factors. The distributed assumptions it makes on version control and distributed identities (email, patch based workflows, gpg keys) can make auditing code difficult. This means the centralization of identity and controls has consequences to make proving changes easier, but impact which developer workflows function well.
Git was designed for that first one. GitHub, GitLab, and similar products were designed for the second.
Hanging out at the intersection of technology, people, and business usually is pretty murky. It’s possible to build software reliably and securely in the most constrained of industries. We just need to know
gitinside and out to make sure we’re covered in a code audit. 🕵🏻♀️
Footnotes
🍻 This talk was inspired by a conversation with a bunch of lovely folks (who will remain unnamed) about the utter uselessness of git log in an audit after my talk last year on Threat Modeling the GitHub Actions Ecosystem (and a few beers). It doesn’t reliably tell you anything about what goes into any repo and I have all the cuts in software audits to prove it. That no-longer-repressed trauma became this eight part / one hour talk … 🙊
🎨 I made lots of AI generated art for this talk and the expanded write-up from it. Unless I specifically said otherwise, it always came up with a “developer” as a white dude. Every single time - even in the dozens of images I made and didn’t use. This guy must have written the ratios of appearances here …
💞 That said, you belong in technology. We’re a great group of people overall. Please don’t ever feel unwelcome.
💼 I write software and build technology infrastructure for a living. Sometimes I do these as a hobby. The pleasurable craft of building things and what I do to pay bills overlaps quite a bit for me at the moment. I’m grateful for this coincidence.
✍🏻 I regretfully didn’t have the opportunity to work directly with Haluk Saker in my time at Booz Allen. As one of those engineers who moved into sales engineering at a tech giant in that time period, that quote captured a lot of my experience too. While I’m grateful to have made that career shift in hindsight, it was so much rougher than I expected for so many reasons. Perhaps I’ll write about that one day.
⚙️ GitHub published a deep dive into git’s database internals in multiple parts. While I’ve never had the database itself turn up in scope for an audit, I have had to explain packed objects and history queues several times over in order to meet controls around code execution and containment. These are all well worth the time to read and understand.
- packed object store
- commit history queries
- file history queries
- distributed synchronization
- scalability
Perhaps this is truly the correct answer. 👇
Allow `docker push’ to push multiple/a subset of tags · Issue #267 · docker/cli - the issue that inspires me to this day