Enterprise chargeback - finding business value

There’s nothing quite like the feeling of a successful proof-of-concept becoming a production system, but we’re not there yet. It turns out money makes for the hardest problems on the path to production.

Let’s solve the problem we set out to solve, getting an understanding of software costs to recover them. We’ll find some unexpected value in the process too.

How many ways can a vendor charge for a product?

At least several times as many vendors and products you have. 😢

Between different purchases that have their own discounts, bundled SKUs of features, different license models that vendor tried, and more, you’ll have maybe 2-3x the number of ways to buy any one product.

As an example, a “central” JIRA Datacenter purchase from Atlassian was licensed per unique named user (seat). The purchase had a start and end date, included a few features that were on additional SKUs but not all of them, and a negotiated discount on the list price. However, each project that ran their own did this on their own, resulting in paying full price or bigger discounts or a separate combination of features … etc.

This led to a couple of confusing situations to implement.

• Which purchase order does any named user pull that cost data from? Is it $10 or $12 or $20 per seat per month for the same product?
• If plugins are paid for separately, are they also licensed for differently? If only half of the named users have it turned on, do you pay for half the total seat count or all of them?
• Is there a consumption or usage-based component to the cost? If so, is there any API or programmatic way to tell who’s using how much?

Every product has a handful or more of these nuances to figure out. Plan to spend time here.

Accounting has some thoughts

Since each vendor and program has its own terms of service and methods for billing usage, we have to ensure everyone is abiding by these. Most contracts “pass through” the actual cost, not the list cost. This means that if you buy program at $x/seat/year, you must pass through that exact amount.

However, you should also track the list prices of everything. If program is $x/seat/year and you pay less than that because of enterprise licensing or volume discounts, that delta can be incredibly valuable long term. The same is true of consumption products like cloud compute sold at a pre-committed discount.

Having both the actual cost and vendor’s list price allows you to

• Add data to competitive bids to prove “we save the government $x per year with our product set”
• Encourage teams that don’t want to adopt a central solution with savings to their project
• Justify your team’s work on this with overhead cost savings

We didn’t use that program so why are we paying for it?

There’s no way to foretell what the billing dispute will be. Once central IT adds themselves as an intermediary between paying for things and delivering them, there will be some dispute.

Having detailed reporting, auditable calculations, and executive buy-in will all be critical for the first couple disputes. Once there’s a few settled, you’ll have any shortcomings in information or process fixed.

Who gets access to this data?

The moment this data collection starts, suddenly everyone wants to see it and have access to it. Some intriguing questions came from these new potential users.

• Are my devs reviewing code or just writing it?
• What do my project managers do?
• Can I tell who’s about to leave the firm?
• Am I staffed with the right skills for the next awarded work?

It’s easy to get excited by all the ✨ fun uses ✨ for this data, but remember that there’s PII in here. Consider who should have a say in what access controls should look like.

Everyone has a shiny new toy to play with

Resist the urge to over-engineer this.

It’s a database that ingests data once a month, then runs a report. That’s it.

It doesn’t mean that there’s no room for cool things, only that the value of many tools designed for scale is limited here. It didn’t quell the temptation to play with new tech.

• Application telemetry and observability? Cool, but also, a monthly spike to peg CPU usage for a few minutes is probably not that exciting either.
• Serverless? My friend, this is a couple gigs of data for years of monthly runs. Today, a Raspberry Pi could run this workload with a big enough hard drive. Back then, any node in the vSphere cluster wouldn’t notice this VM.
• Because of the size of the data set and how infrequently it was used apart from reports, literally any search was just fine.
• Given the size of the workload, the idea of running this in any container orchestration platform is also extra complication.

It works for 5 services. Can you do the next 40?

Tech was never the scaling problem. People are the problem. Adding more systems to ingest and analyze doesn’t have to be completely bespoke.

There were scaling problems, just not novel ones that could be solved with the sort of technology that gets conference keynotes. Database concurrency became a problem when we’d run all program ingests simultaneously. Instead, staggering the process to bring in data and process it became a simple and economical path. It could run all raw data imports on the 1st of the month, process on the 2nd and 3rd, have a day or two to review it, and then send invoices out within the week.

Kludgey fixes varied by service. Some seemed to not consider how long it took to generate a response at all. Workday reports worked in a weird way. You’d hit the API, then poll that endpoint for minutes until it generated the report you asked for over the same endpoint. In contrast, GitHub Enterprise worked equally weird in the other direction, where you’d hit the API and several minutes later the report you need would be available at a different endpoint. Both valid, neither a problem, and both need documentation to be maintainable in your “chargeback” system.

Unexpected value

Of course there’s better pricing from buying in bulk, reduced overhead from having fewer purchase orders to administer, and deduplicating some tools.

There were much more interesting patterns in the data, though. A few interesting trends immediately stood out.

• Despite some partisanship on not wanting to move between the two, usage patterns between GitHub and GitLab were nearly the same on every measure. This pattern held on other tools that do very similar things.
• There’s a long tail of super users in each tool, but they were never the best people to ask about how anyone else uses that program.
• CI systems never translated “equally” if migrated. Each system encourages more use to automate more tasks, so 1000 minutes on Jenkins became 1500 on GitLab or GitHub, and so on.
• Specialist tools like Primavera P6, GIS tools, and analytics software typically got the best value from bulk buys.

Matching developer talent to opportunities across the company was unexpectedly impactful. One of the (not perfect) data sets we could pull in was around the rough proportion of programming languages any developer checked into GitHub/GitLab/etc. Matching this with the project where their time is charged allows teams a “heads up” to better match folks to other work if there’s a promotion on another team, or to avoid “the bench” when the project ends and find a better placement within the company.

Parting thoughts

Building this was a lesson in the children’s song There Was an Old Lady Who Swallowed a Fly . From a purely technical standpoint, this was among the simplest, most boring solutions I’d built. Due to all of the stakeholders involved, it was still one of the hardest programs to get into a production environment. A simple collection of a few Python scripts adding data into a database became a much broader enterprise tool, step by step, as we added yet another set of policies/tech/controls to solve a problem and find another.