Stop putting application security alerts in ticket systems
💖 This post contains unresolved professional trauma and is fueled by a desire to spare others that deep pain. You have been warned. 💖
Please stop automatically putting code security alerts into your ticketing system (eg, JIRA, ServiceNow, ZenDesk).
It’s an awful idea, especially at an enterprise scale. It creates a ton of noise for your security team, deceptively doesn’t tell you what you think it does, and is counter-productive to fixing things or keeping things from being unsafe to begin with. To understand why I hate this pattern so very much, we need to take a look at why it’s not effective and how it murders your team’s desire to fix anything.
Alerts generated by a code scanning tool (static or dynamic analysis, fuzzing, software composition analysis, etc.) …
Do not correspond to security incidents
There is, at the time of alert discovery, no evidence that alert was exploited or exploitable.
Ticket systems as an incident response platform want to track a very different problem than a developer fixing code - it wants you to
- Name an incident response lead
- Determine the “blast radius”
- Narrow down the time of exposure (how long was this vulnerable?)
- Figure out what type of data is involved and which systems, etc.
- Gather evidence
- Implement immediate measures to stop this (if warranted)
- Plan long term remediation
- Document all 👆
- (more I’m probably forgetting)
None of this is able to be correlated reliably in a programmatic way from repository/organization code security data (I tried). These alerts can happen off of the deployed branch, each team has different ideas of “deployed” (releases/tags, branches, copying files to a server, burning a finished copy of the code to CD, etc.) so this quickly becomes a dreaded task of sorting through ever-larger mountains of garbage.
A terribly common example is the noisy “flaws” returned by static analysis tools where it might be dangerous but the tool doesn’t know, so it makes you decide - a thousand times. Commonly, it’s when you get info from a user, but the tool isn’t sure if you are using that data unsafely. Another use case that often falls in this category are “code smells” or other bugs or stylistic concerns. Could be a security thing, could be a buggy thing, could be a boring thing - who knows so here’s a hundred tickets to fill out. 😍
Do not correspond to ITSM events
There’s not an easy automated way to correlate RITM00000000 to appsec/alert/0001 to developer-name-here calculate things like mean time to remediation, find who is slacking off in remediation, or other simple security metrics in a tool built to make sure your password gets reset or you get a new laptop within a specified time frame.
Even if you license the entire developer base as agents in said ITSM1 ticketing system ( 💰💰💰 ) and they’re going to all check in on it in a timely fashion ( 🤣 ) to make sure each ticket is classified/assigned/etc correctly, the chance of this data in this system telling you what you think it does dwindles over time.
📚 Story time! I was on a team that tried to correlate time spent resolving tickets. Once the ticket was marked resolved, the agent would have to click one of five radio buttons (single select) about if it was less than half an hour, 1-2 hours, etc. This “it’s only a single click” would totally give us all insight into how the team was spending time. We found that some team members always clicked the smallest option and that’d correspond to maybe 10-15 hours of work a week despite being quite visibly busy in the office with customers. Others always clicked the largest, signaling they were working for an impossible 200+ hours each week. This was the simplest and fastest option and it yielded tons of garbage data.
Now this gets even more complicated for code - creating and assigning the alert, going into ITSM and estimating it and classifying it and getting the right approvals to figure out when/how to fix it, actually fixing it and tying that PR to the ITSM number, adding in the correct number of hours, creating a change request to deploy that fix, etc … good luck with that!
Do not correspond to operations management events
😑 Let’s get metaphysical - What is a fix?
No really - how do you fix this security flaw? How do you know that fix is complete?
Do you need a maintenance window to update a dependency? What if you don’t? Or rotate a secret? What would need a maintenance window outside of core business hours? What can be done as a hotfix without the customer noticing?
The very broad array of software (and security alerts within it) don’t have this information because it’s dependent on each app built on top of it and the business logic around it. The people empowered with the knowledge to fix this are the only ones who can give a definitive answer.
Encourage bad behavior
All of this extra cost and “friction” between fixing things and getting credit for it encourages bad behavior like only scanning after deployment. Shifting those processes further from the people who have the power to Build The Thing causes those requests to be missed (or ignored) because it’s too noisy.
Most importantly, it removes the freedom to screw up experiment in development, where it’ll never actually be vulnerable or hit production systems. If building a feature costs me 50 tickets I need to spend time documenting, I’m going to do everything I can to create the least amount of feedback and ignore as much as possible.
You just hate < insert ticket system >
It’s fashionable to hate on Jira or ServiceNow. I have totally participated in hating on both of these and a few more, having used them at work in various roles, but I don’t hate ITSM tools as such. I just want you to use things for what they’re designed for. 😊
- Create a ticket in Jira (or whatever) to track the effort needed to fix a thing (or things!), or budget the time to do that into a sprint. That’s completely fine.
- Not okay - creating 20 minutes of tedium for each and every potential “flaw”.
- Create a change management request to deploy a fix to one or many findings from your appsec tool if doing so fits into your business process (user interruption, change approval, etc.) Also totally acceptable - this is what the tool was meant to do!
- Not okay - spending hundreds or thousands of engineering hours to shoe-horn quality data into security tools, security alerts into operations tools, etc.
- Who hurt you?! - giving your security, operations, and development teammates hundreds of pages of “action items” in a PDF. 😱
What to do instead
Do not despair, lovers of pretty charts and broad views of trends.
You have a better path forward through your Security Information and Event Management (SIEM) tool - eg, Splunk, Sentinel, DataDog, CloudWatch, etc. All of this application security data, plus a ton of other audit log data from all of your other tools, is available to integrate or separate or time-box to your every need. You can make the burn-down charts, the PDF reports, value stream metrics from there. Most importantly, you can create alerts from that data that hook into other tools.
When I ask why anyone would want to send thousands of alerts into automatic tickets no one ever says “because I want to torture my team”. The reasoning is always about understanding the codebase as a whole, tracking trends in remediation, and having an inventory of vulnerabilities that can be audited. Making sense of tons of data about security is what a SIEM was literally 🌟 designed to do. 🌟 It doesn’t replace a ticket system. Instead, it allows teams to map fixes or change requests in the systems that work well for those tasks and other teams to look at big sets of data in systems that work well for those tasks.
Application security (also infrastructure security, operations, development, <lots more>) is hard. Using tools for what they’re meant to do makes everyone’s life better and safer. ♥️
Disclosure
I work at GitHub as a solutions engineer at the time of writing this. All opinions are my own.
Footnotes
ITSM = Information Technology Service Management program ↩