Advent(ure) in Systems Seeing — Days 11 and 12 — Sketch a Situation — Justifying a Performance Team
Think of a situation you’d like to explore with a systems lens as we practice various systems approaches and views. It’s good (since these 15–20 minutes of daily journaling add up) if it’s something that matters to you to explore and understand, and begin to shape responses to. Write a few paragraphs describing the situation. We’ve been doing “warm-up” workouts in systems and seeing (observing, noticing, seeing more deeply), and now we’re shifting to applying system lenses and practices to a situation of particular interest to you.
(Note: I’m writing this blog post in the UK and my computer has decided that I should be using UK centred or English specialised spelling for some reason.)
The situation I will describe is one that I’ve encountered a few times. It’s around the establishment, funding and goals of a systems performance team. My early career was as a performance tuning specialist, I wrote a popular book on Sun Performance Tuning in the 1990s, and worked with many people who specialise in this area.
Performance tuning teams operate on two distinct metrics, speed and cost. They make computer systems respond more quickly, which has an indirect effect on business value, and make them run more efficiently, which saves money by doing more with less resources. For poorly tuned systems, work on either of these metrics tends to improve both metrics, but situations also occur where a tradeoff between cost and speed can be made. In addition, resilience and stability tend to improve when systems are well tuned, although those are not usually a direct goal of a performance tuning team.
Business justification to create and fund a performance tuning team occurs when there is an emergency issue that needs to be fixed, or more proactively, a significantly large spend that needs to be optimised.
The system that I aim to setup is to get the performance team to be recognised as a profit centre not a cost centre. Most organisations are under-invested in performance work, so adding headcount/spend generates a high return on investment. Convincing the CFO to make an exception and dedicate extra headcount to an obscure engineering team deep in the organisation is left as an exercise to the reader.
An anecdote, back in 2011 Netflix was scaling up streaming rapidly on AWS but hadn’t put any effort into performance optimization. Then there was a botched attempt to spin off the DVD business into a new brand called Quickster, and Netflix had a bad quarter, with customers cancelling their subscriptions. It looked like we would make a loss for the quarter, but it was going to be close. The cloud team looked at the AWS bill and spent a few weeks optimising and tidying up our spend, saving several million dollars over that quarter, and Netflix ended up just above break-even, by an amount less than the cloud team had saved. We also effectively flat-lined our AWS spend for a few quarters even though the streaming business traffic more than doubled. Soon after that we created a dedicated performance team to create tools, find problems that needed fixing, and work with whoever owned the code to get it fixed.
In the datacenter world, performance teams don’t save money immediately, because they just delay the next purchase cycle, but in a cloud environment, you can get savings on next month’s bill.
The workload and goals of a performance team should be organised into a backlog of tuning project ideas. Those ideas get fleshed out into projects and rated using a methodology like Reach, Impact, Confidence, Effort (RICE) as described in the recent book on fixing developer experience Frictionless, by Nicole Forsgren and Abi Noda. In the case of performance projects, the incremental financial savings in each quarter should be enough to justify the headcount needed to make the projects happen. If the CFO wants more savings sooner, then more performance engineers are needed, and more pressure needs to be put on other teams to work on prioritising delivering performance over features on a regular basis.
Finally, I’d like to point out that in my experience most developers are not taught systematic performance tuning methodologies, and that many people working in the field of performance tuning, including myself, have physics degrees. Physics experiments teach data collection, mathematical modelling, statistics, and the scientific method of hypothesis testing. Good performance tuning engineers are a “special breed” and may not be great developers, so the typical developer oriented interview and hiring processes can exclude them. It’s hard to interview for people that can solve problems you don’t understand. My favourite interview question centres around queueing theory, Little’s Law, and how many threads a server is going to use.
We have an alternative here, where you can either draw a rich picture again (but with a new focus), or draw an actor map. They capture overlapping details, but Rich Pictures include interactions, whereas Actor Maps gathers more notes on the perspectives, orientations, and so on, of the actors (and other stakeholders). So, either:
- draw a Rich Picture that tells a visual story of the situation (that you described on Day 11). Identify who is involved (actors, groups, organizations, systems) and draw interactions among them, labelling the interactions and also jotting notes on the picture to capture the roles, cares and concerns, and orientations of the people, groups or entities. (See Rich Pictures on Day 4 to refresh.)
or
- draw an Actor Map, where you briefly summarize the situation (described on Day 11) in the center of your map (in a cloud to make it distinct). (See diagram above.) Identify actors (individuals or groups) who are involved in and influence the situation, and add them around the (summary) situation cloud. (If there are many, use your judgment about who to focus on.) Alongside each actor (or group), cluster notes about the actor. These notes may include: brief description of the situation from their perspective; what the actor or group cares about (as it relates to the situation) and what they are concerned/worried about; how they orient to the situation; how they contribute to the situation; and what their expections are. Repeat, for each actor (or group) that you’ve identified to focus on. (You could do this with notes in a table, if you really want a break from pictures. However, seeing the different perspectives around the circle, allows us to notice shared interests and potential conflicts (of interest, orientation, etc.).)
The image at the top of this post is the result. My additional thoughts are around the worries and blockers, and how to make performance optimisation part of the culture of the organisation, rather than a one-off event. It’s up to the CTO and senior management to publicly celebrate successes and goal the product organisations with latency and efficiency goals as well as feature delivery goals. It’s up to the performance team, working with the product teams, to carefully test and gradually roll out change so they don’t break the product. It’s up to the cloud economics team to estimate possible gains, then validate those estimates against actual measured results, and to calibrate their own estimation models. Platform management needs to propose the concept of the performance team as a profit center to both technical and financial management, to secure the resources and establish the measurements of results that will create a self sustaining system, with positive funding feedback, until the available project backlog doesn’t justify additional headcount.
