Don't treat people like serverless functions.

When I were knee high to a grasshopper we didn't have all this new fangled cloud infrastructure and we certainly didn't have the concept of serverless computing. How can you compute without a computer?..

But before my time (and I'm not that old!) computers were people. People like Sally. Actual humans sitting in offices with bits of paper, pencils and tables of logarithms and trigonometric functions. Adding, subtracting, scribbling down results, checking and verifying. Whether a human being or a hunk of metal, you need a computer to compute.

Well, most of the time the thing you're computing is important enough that you can't afford for it not to be computed. It's a bad thing if you need to calculate wage-packets and can't do it, whether it's because Sally's off sick and can't compute today or the server has gone down because it's run out of disk space. You're going to have a riot on your hands come Friday afternoon when the pubs open...

Which brings us to the concept of redundancy. 

Rather than relying on Sally alone we need to ensure we have someone else around who can also compute when she's out sick. Equally - in the world of tin - we need backups in case our primary fails - disks, networks, servers, power, cooling, data-centers etc. 

This is an expensive habit.

Almost every system will be important enough to warrant some degree of redundancy. The more critical they become, the greater the degree of redundancy required and the more time architects spend worrying about the impact of component failures, how long it takes to recover, how much data could be lost, acceptable error rates and so on.

In the bad old days we would literally have a standby server for every primary. Boom! Double the costs right there.

Just imagine if we needed to employ Jane as well as Sally just to cope with the days Sally was sick? Of course we'd not do this. We need to ensure the function can be picked up by someone else but that person could be someone else in a team where the team has a degree of redundancy built in (or even someone who's main job is something else, Bob from accounting or Sallys manager for example). Perhaps we can work out that if we need seven computers we best hire ten to cover the days some are sick...

Having a level of redundancy in the organisation provides the flexibility to handle outages and failures. Besides, people can't work at 100% capacity. They will burn out, productivity will fall, they'll hate you for it and will leave as soon as a better opportunity turns up.

Anyway, back in the world of tin, along came virtualisation and we could host multiple virtual machines (VMs), on one physical host in much the same way one person could turn their hands to multiple tasks. This was great ('ish) as it reduced the number of physical servers significantly and saved on hardware, power, space, CO2 emissions and consequently dollars. We still need to support some degree of redundancy in case a host node went down or a VM failed but it's much better than before. 

How much better?

Well, most systems aren't Google or Netflix. I know, surprising huh?

Most systems don't need to support 1000's of transactions per second. Mostly it's less than 1 tps. Yup, one! And often it's a lot less than one... like a few hundred transactions a day is typical of many systems. Me and a my Casio fx-85GT Plus can handle that!

So we can stuff a lot of VMs onto a single physical host with perhaps 50 VMs running across 3 physical hosts in a cluster whilst still maintaining enough redundancy to ensure availability. Make that tin sweat!

Suffice to say, if we treated Sally like the tin, she would not be impressed.

VMs are still pretty hefty though. Each VM runs its own copy of the operating system and associated processes which makes them pretty big (GBs of RAM for the VM compared with perhaps a few MB for application processes). We've had multi-tasking operating systems for decades now and there's little reason we can't run multiple application processes on the same server. Other than it's a really bad idea.

Developers make lots of assumptions about the environment they're running - which libraries and versions are available, what file paths they can use etc. - and a lot of these things aren't compatible with each other. They're also really bad at security and act like peace loving drug infused hippies... "hey, why would anyone else read my files man?". Add to this that bugs happen (always will) resulting in unstable or runaway processes crashing or consuming all the resources available and it's a recipe for disaster.

Running multiple disparate application processes on the same server is a bad idea. Or was...

Now along comes containerization and provides a degree of isolation between processes within a server to prevent one hippie process treading on another hippies toes. And we know how exposed hippies toes are don't we?

This can give us an order of magnitude increase in processes on a host so we're now up to 500 containers across our 3 node cluster of computers. Nice.

Sally on the other hand is seriously pissed.

But we still have to manage a bunch of physical servers underpinning our applications. Whether VMs or containers, there's a bunch of power hungry, raging hot physical computers burning away in the background. And in the case of human computers, really angry overworked ones.

Then came serverless.

Forget the server. You pay-per-use - that few hundred transactions a day - leveraging services provided by cloud providers. Everything from databases to messaging to raw compute can be provided as a pay-per-use service without needing to worry about the server or redundancy.

Erm, well, except for the cloud service provider - who worries about it a lot - and your architect who still needs to worry about the non-functional as well as functional characteristics of all those services we end up consuming (especially the economics of serverless if that one tps turns into many thousands..). 

Of course serverless isn't really server-less and there's always a bit of tin somewhere. We're really talking about building applications out of services (like AWS Lambda or Google Cloud Functions). Those services are carefully managed by cloud providers who supply all the necessary redundancy to support the resiliency, availability and scalability you'd expect.

But what about Sally? Does she still have a job?

Sadly no. Sally has now been moved into the gig economy on a zero-hours contract and works on a pay-per-use basis. She doesn't get any guaranteed work, an hourly rate or sickness benefits. Please don't treat people like serverless functions.

Docs

There, I said it. A four letter swear word. Something worse than the F’ word if the horror on the boss’ face is anything to go by.

We don’t do “documentation” anymore and besides, the agile manifesto says it’s immoral to write a word of documentation. The code is the documentation. That you have to get inside the twisted maze of my mind, and work out what drug infused insanity I was trying to convey at the time and which may or may not result in what was intended regardless… that’s your problem. 

Such nonsense pervades software development today. Although…

It’s not the lack of documentation per se but the lack of demonstrable thought that bugs me. A critical explanation of why things are the way they are.

Why has solution design been so poorly treated?

The answer to that lies in what design is trying to achieve.

Solution design aims to address the needs of the system, communicates how these needs are met and enables testing of the solution when the cost of change is lowest.

This testing is achieved by static walkthroughs and peer review.

However, in an agile world where we breakdown features and stories into small chunks that can be delivered rapidly and iteratively, the hefty tombs of yesterdays solution design documents are equally shrunk to focus on the few features and stories in scope.

Taken far enough it ends in a combined whiteboard design and review session with a handful of developers. And beyond taking a photo and circulating to the team, what's the point in doing anything more?

Firstly there's the supporting teams who are going to care for your solution through its early teething days, rebellious teenage years, into maturity and all the way to the grave – death being one of the few certainties in life. Making these guys bump their way around a darkened room trying to figure out how things hang together is unfair if not sadistic.

These guys don't work in scrum teams or story by story. They work from incident to incident, from shit-storm to fan-splattering shit-storm. They need a concise and holistic view of the solution for which a few poorly framed holiday photos doesn't cut it. Honestly, these guys are too nice to you.

Then there's the not insignificant matter of consistency. 

You can argue the marginal benefits of any technology but you need to justify any change which runs counter to the inherent design patterns of a system as contributing significant value to overcome the increased complexity and costs it brings.

Do things consistently and you get paybacks in terms of proven reliability, easier maintainability, reduced cognitive load, faster delivery and so on.

Do things inconsistently and whilst you may get to play with the latest  tech you're being pretty selfish - if that's your motivation. And if it's not then see above re justifying the increased complexity and costs.

Which brings me to my point on solution design.

Solution design is about more than addressing the needs of a few stories on a sprint by sprint basis. It's about addressing the broad needs of the system and providing a vision for the longer term. Defining the patterns which will be used time and again and which (should) enable that payback through consistency.

This isn't an argument that technological progress isn't a good thing or that we should never have crawled out of the sea. Or that your resident architect has all the answers. It's an argument to think about the broad needs, key decisions, responsibilities, patterns, principles, policies and costs - immediate and long term - that provide the foundation on which any system is based.

At times it may seem futile and time consuming but it's cheap to change this stuff early on rather than late on when it's most expensive.

And the way we do that best is through reasoned discourse. Discourse best expressed through quality documentation - words, diagrams and matrices.

The Con of Agile (or why agile reductionism is hard…)

Agile is, to a large extent, a radical breakdown of function into small incremental features delivered in a prioritised manner with rapid feedback to inform on the next evolutionary step to deliver ever greater customer value.

Executed as such you may not; probably should not, get what you originally intended and even if you fail then at least you’ve failed fast and saved yourself an expensive pipedream.

Bliss.

But agility is not easy. It is not something a tool, method or consultant will magically fix for you – despite what they tell you. Radical breakdown of function is hard. Reductionism is hard.

Most systems we build today are inherently complex with dependencies spread far and wide through the enterprise. Through reductionism we attempt to decompose these complex systems into their simpler component parts – a technique that has a long and successful history in software development – only to then recompose them in many and varied ways to deliver the outcomes we desire.

In a typical large organisation it’s not uncommon for a system to require integration with dozens of others and for any particular new feature to impact several of these at a time – in fact it’s rarer for a feature to be self contained within one system. Integration is the norm.

And here our problems start… 

According to Conways law an organisation will typically design systems that mirror the organisations communication structure. i.e. we end up with various systems mirroring the group and team structure of the organisation integrated along the same communication channels as the organisation.

The net result is that we have specialist teams with a deep understanding not just of the technology but the values and ethos of each system and the organisation they reflect. These teams own the system and have veto rights – rightly so – over what functionality they do or do not support. 

So on one hand we have a prioritised backlog of customer focused features and on the other we have a collection of disparate teams available to deliver them hell bent on aligning to the internal organisation structure. Something the customer usually doesn’t (and shouldn’t) give a toss about.

An instinctive reaction is for teams to start to breakdown features into stories that make sense from the organisational structure perspective. This can create a reflection of the organisation in the user journey and leads to stories which are really tasks because that’s easiest for teams to consume. And if you’re simply working off a prioritised list of tasks you’re not agile. You may be able to re-prioritise but you’ve lost the connection with why you’re doing something and that critical feedback loop from the customer so you can’t respond to their changing needs.

Perhaps worse still, a disparate collection of tasks spread across a plethora of teams does not provide a clear vision and direction for the people working in those teams. Treated this way people can start to feel like slaves to the machine, churning out widgets hour after hour with no clear understanding of why. Motivation and quality suffers and we no longer have small agile teams focused on delivering customer value but a collection of teams each with their own perspective on what we’re trying to achieve. 

As we breakdown features and stories into smaller chunks, maintaining a focus on customer value and the overall vision is hard – particularly in a large and complex organisation.

There’s now a twofold need both for well-articulated, customer focused and concise feature and story definition, and for solution designs which clearly stitches together the various parts of the organisation; and associated systems, connected to deliver those features.

The features and stories say what we’re trying to do and why we’re trying to do it.

The solution design says how we’re going to do it.

And that understanding of why we’re doing something is the “Ah-ha!” moment that gives us meaning and purpose, the reason we get out of bed in the morning. 

Ultimately it’s easy to explain what we’re trying to do and how we plan to do it – they’re concrete things we can action and it’s natural for us poor humans to want to focus on solving problems. It’s much harder to articulate why we’re trying to do something, and we all too often forget that others aren’t aware of the vision in the first place.

So as we break down capabilities and features into ever smaller chunks we need to consciously focus on the “why” we’re building this chunk and less so the “what” and the “how”. It may seem counter intuitive but solving problems is easy, finding them is harder.

As a consequence our features and stories should be heavy on the explanation of why we’re trying to do something and lighter on what we’re actually going to do. The “how” we’re doing something should be reserved for the solution design and provides traceability from the “what” and the “why” to the organisational structure, providing the foundation for that task list everyone seems to desire.

As we breakdown stories we also tend to reduce the number of teams involved and if we go far enough that they only impact one team then great. It won’t though always be the case and we shouldn’t try to twist stories to meet the internal organisation structure and make it fit us rather than the customer.

In short, as we breakdown features and stories we need to focus on why we’re doing something and what we’re going to do to address that need. Don’t worry about how we’re doing it – leave that to the technical team of architects and engineers – give them the easy task of connecting the dots and solving the problem. 

The trap in agile reductionism is that we, as human beings, have a tendency to want to solve the problem and in doing so lose the connection to the why that gives us meaning and purpose. We end up focusing on the solution and thereby reduce the solution to a problem, not the problem itself. The two are not the same. One is optimised for the organisation, the other for the customer.

An Observation

Much has changed in the past few years, hell, much has changed in the past few weeks, but that’s another story... and I’ve found a little time on my hands in which to tidy things up.

The world of non-functionals has never been so important and yet remains irritatingly ignored by so many - in particular by product owners who seem to think NFRs are nothing more than a tech concern.

So if your fancy new product collapses when you get get too many users, is that ok?

It’s fair that the engineering team should be asking “how many users are we going to get?”,  or “how many failures can we tolerate?” but the only person who can really answer those questions is the product owner. 

The dumb answer to these sort of question is “lots!”, or “none!” because at that point you’ve given carte-blanche to the engineering team to over engineer... and that most likely means it’ll take a hell of a lot longer to deliver and/or cost a hell of a lot more to run.

The dumb answer is also “only a couple” and “hell, I don’t care.” because, well... you’ll have different problems, mostly about whether you should be in business at-all I suspect.

So the balance is somewhere in the middle. Take your OKRs and have the discussion with the engineering team about what the concerns may be and where the hot-spots are. Run through some “what-if...?” scenarios to understand the potential impact of under/over-estimating. In many cases there are strategies that can be taken to reactively expand/shrink resources which can at least buy you some time... if not more.

Regardless, from an engineering perspective we need to:

1. Understand expectations and what really matters - driven through an understanding of  product OKRs.

2. Derive NFRs, agree these with product owners as acceptance criteria, and architect and design accordingly.

3. Monitor and observe what’s actually going on - how else do you prove you’ve met your acceptance criteria?

For example, a system I worked on recently involves processing transactions running into the millions of dollars. In this case, losing or duplicating a transaction can be seriously bad for your health.

The consequence? A lot of discussion with product teams resulting in significant design effort to ensure the integrity of transactions as they pass through the system. Realtime monitoring as transactions flow, near-time balancing controls as belt-and-braces and financial accounting and reporting on top (to be honest the business only really care about accounting but that feedback loop is way too slow from an engineering perspective (daily)).

Duplicates transactions are avoided - at the cost of reduced availability - and dropped transactions are detected and alerted on within a few minutes and can be automatically replayed (if we ever grow the balls to turn that on).

This isn’t cheap and involves a lot of testing and verification - including some chaos testing to simulate duplicates (not loses (yet)) and I hope it’s been worth it.

It also involves significant investment in monitoring, tracing and alerting to ensure we have good visibility of what’s going on across the platform so we can spot problems quickly if they do happen (and they can, they just shouldn’t result in a financial loss for customers or ourselves).

This is build I hope never to have to call on and the sort of failures which scare me most involve the outage of entire clusters and complete site failures - this stuff needs specific test focus. Auto-scaling, rebalancing and node failures are such common occurrences now as to be BAU and should not be the reason you’re called out of bed at 2am - we test this stuff by simply breathing these days. 

Through discussion with product owners on OKRs we can start to uncover our NFRs and what degree of failure can and cannot be tolerated. We use this knowledge to architect and design solutions appropriate to the problem space, and we monitor and observe to ensure we’re within bounds. 

And if the boundaries move? Then we adapt, evolve, change accordingly. Through discussion we’ve hopefully had a chance to consider what could go wrong and should already have some escape routes planned... just in case.

Inter-microservice Integrity

A central issue in a microservices environment is how to maintain transactional integrity between services.

The scenario is fairly simple. Service A performs some operation which persists data and at the same time raises an event or notifies service B of this action.

There's a couple of failure scenarios that raise a problem.

Firstly, service B could be unavailable. Does service A rollback or unpick the transaction? What if it's already been committed in A? Do you notify the service consumer of a failure and trigger what could be a cascading failure across the entire service network? Or do you accept long term inconsistency between A & B?

Secondly, if service B is available but you don't commit in service A before raising the event then you've told B about something that's not committed... What happens if you then try to commit in A and find you can't? Do you now need to have compensating transactions to tell service B "oops, ignore that previous message!"?

I'll ignore the use of queues/topics between services as this really just becomes the service B failure point although there are topologies in which risks can be mitigated this way (e.g. through queues local to the origin service A).

There are several options to address this issue:

Event Sourcing - In this model the event is king. Service A persists events to an event store and downstream consumers can subscribe to these events. The event itself does not present the current state of an entity but the history of what has happened. To understand the current state of the entity you need to replay the event history. Service B then consumes from the event store. Beyond this a read view can be maintained by service A to provide an efficient presentation of current state. Note though that in this model there is a risk of read inconsistency since that view is updated separately from writes to the event store. If you can tolerate the additional complexity and eventual consistency then this can work well. It also means you can have a fast write store for the events, independent of what may be a more complex view of current state. Such event sourcing is often used in CQRS implementations.

Outbox (store-and-forward) - In this model, instead of publishing directly to service B, we store the event in the same datastore as used by service A. With many traditional databases this means we can cover the event inside the same transaction as used to persist whatever data we have in the main service. A separate thread can then publish events from the outbox to service B independently.

A variant of the outbox could be to implement fallback options or circuit breakers to retry messages or fallback to the outbox only when messages fail. This can improve the general throughput and responsiveness of services.

Balancing Controls - An alternative to the above solutions is to accept some degree of failure and implement policing strategies to verify that all expected events were received by the subscribing service. For example, how many orders did service B process in the past hour v how many orders where accepted by service A. In some cases it may be acceptable to introduce such delayed checks but these checks can be invasive. In addition, where gaps are identified a strategy is needed to replay events and fill in those gaps. You'll know there's a problem, but you may not be able to fix it automatically...

To make such balancing controls easier and more consistent across services a standardised inbox (e.g. on service B) can be used to store inbound messages or signatures thereof once processed and combine this with a similarly standardised outbox (e.g. on service A) so that checks can be made more easily and allow replay from the outbox for missed events. Such standardisation can place undesirable restrictions on the freedom of microservices to vary themselves and optimise for their specific needs but may be the least worst option.

As to which option is preferable depends on what constraints you can accept.

If you can tolerate inconsistent reads and eventual consistency and have high throughput requirements then event-sourcing may be preferable.

If on the other hand you absolutely must be able to read-your-writes and ensure consistency within a service then an outbox may be simpler and provides better consistency.

Alternatively if you can tolerate some level of failure and workaround corrections then perhaps balancing controls may suffice.

Focus

I suspect most of us working in IT today use agile methodologies such as Kanban, Scrum and Safe. We also strive to keep up to date with the latest developments in languages, libraries, patterns, architectures etc.

All of this is with the intent of improving the delivery speed, quality, efficiency, maintainability and the cost effectiveness of the systems we build - oh, and whilst improving our CVs at the same time.

Care though is needed to ensure we don’t get distracted by these tools from delivering the solutions they were employed for. This often isn’t through any specific fault with the methods and tools themselves but how we set about using them. 

There can often be a blinkered tendency to maniacally focus on the tools and methods themselves and in doing so fail to deliver most effectively on actual requirements and customer needs. If all you have is a hammer then everything looks like a nail.

Worse still, the job can become about servicing the tools used to do the work rather than the product of the work itself.

Furthermore these tools are often barely distinguishable from each other and for most use cases it doesn’t much matter if you use one or another; Java or .NET, AWS or GCP, Scrum or Kanban. Use what works for your team and be the master of tools, not a slave to them.

The product of your creativity is that which you build, value is derived from how your customers use the product.

Focus on the product and value it gives to your customers. Produce working solutions first and foremost - functionally and non-functionally - as efficiently as can be.

Over time we learn new methods and tools that improve delivery and optimise value for the customer but we need always be mindful that tools and methods are only a means to an end and not an end in themselves.

A Years Worth

A few Christmases ago I was messing about creating bubble maps - no doubt in some mince pie and port induced state of inebriation (quality of code is consequently as you'd expect).

I'd long forgotten about this until Kent Becks recent post on medium.com about trying to understand A Year's Worth of effort.

This looked familiar and so with a little manipulation here's a simple utility to convert priority ordered stories into a visual bubble map.