An Observation

Much has changed in the past few years, hell, much has changed in the past few weeks, but that’s another story... and I’ve found a little time on my hands in which to tidy things up.

The world of non-functionals has never been so important and yet remains irritatingly ignored by so many - in particular by product owners who seem to think NFRs are nothing more than a tech concern.

So if your fancy new product collapses when you get get too many users, is that ok?

It’s fair that the engineering team should be asking “how many users are we going to get?”,  or “how many failures can we tolerate?” but the only person who can really answer those questions is the product owner. 

The dumb answer to these sort of question is “lots!”, or “none!” because at that point you’ve given carte-blanche to the engineering team to over engineer... and that most likely means it’ll take a hell of a lot longer to deliver and/or cost a hell of a lot more to run.

The dumb answer is also “only a couple” and “hell, I don’t care.” because, well... you’ll have different problems, mostly about whether you should be in business at-all I suspect.

So the balance is somewhere in the middle. Take your OKRs and have the discussion with the engineering team about what the concerns may be and where the hot-spots are. Run through some “what-if...?” scenarios to understand the potential impact of under/over-estimating. In many cases there are strategies that can be taken to reactively expand/shrink resources which can at least buy you some time... if not more.

Regardless, from an engineering perspective we need to:

1. Understand expectations and what really matters - driven through an understanding of  product OKRs.

2. Derive NFRs, agree these with product owners as acceptance criteria, and architect and design accordingly.

3. Monitor and observe what’s actually going on - how else do you prove you’ve met your acceptance criteria?

For example, a system I worked on recently involves processing transactions running into the millions of dollars. In this case, losing or duplicating a transaction can be seriously bad for your health.

The consequence? A lot of discussion with product teams resulting in significant design effort to ensure the integrity of transactions as they pass through the system. Realtime monitoring as transactions flow, near-time balancing controls as belt-and-braces and financial accounting and reporting on top (to be honest the business only really care about accounting but that feedback loop is way too slow from an engineering perspective (daily)).

Duplicates transactions are avoided - at the cost of reduced availability - and dropped transactions are detected and alerted on within a few minutes and can be automatically replayed (if we ever grow the balls to turn that on).

This isn’t cheap and involves a lot of testing and verification - including some chaos testing to simulate duplicates (not loses (yet)) and I hope it’s been worth it.

It also involves significant investment in monitoring, tracing and alerting to ensure we have good visibility of what’s going on across the platform so we can spot problems quickly if they do happen (and they can, they just shouldn’t result in a financial loss for customers or ourselves).

This is build I hope never to have to call on and the sort of failures which scare me most involve the outage of entire clusters and complete site failures - this stuff needs specific test focus. Auto-scaling, rebalancing and node failures are such common occurrences now as to be BAU and should not be the reason you’re called out of bed at 2am - we test this stuff by simply breathing these days. 

Through discussion with product owners on OKRs we can start to uncover our NFRs and what degree of failure can and cannot be tolerated. We use this knowledge to architect and design solutions appropriate to the problem space, and we monitor and observe to ensure we’re within bounds. 

And if the boundaries move? Then we adapt, evolve, change accordingly. Through discussion we’ve hopefully had a chance to consider what could go wrong and should already have some escape routes planned... just in case.

Inter-microservice Integrity

A central issue in a microservices environment is how to maintain transactional integrity between services.

The scenario is fairly simple. Service A performs some operation which persists data and at the same time raises an event or notifies service B of this action.

There's a couple of failure scenarios that raise a problem.

Firstly, service B could be unavailable. Does service A rollback or unpick the transaction? What if it's already been committed in A? Do you notify the service consumer of a failure and trigger what could be a cascading failure across the entire service network? Or do you accept long term inconsistency between A & B?

Secondly, if service B is available but you don't commit in service A before raising the event then you've told B about something that's not committed... What happens if you then try to commit in A and find you can't? Do you now need to have compensating transactions to tell service B "oops, ignore that previous message!"?

I'll ignore the use of queues/topics between services as this really just becomes the service B failure point although there are topologies in which risks can be mitigated this way (e.g. through queues local to the origin service A).

There are several options to address this issue:

Event Sourcing - In this model the event is king. Service A persists events to an event store and downstream consumers can subscribe to these events. The event itself does not present the current state of an entity but the history of what has happened. To understand the current state of the entity you need to replay the event history. Service B then consumes from the event store. Beyond this a read view can be maintained by service A to provide an efficient presentation of current state. Note though that in this model there is a risk of read inconsistency since that view is updated separately from writes to the event store. If you can tolerate the additional complexity and eventual consistency then this can work well. It also means you can have a fast write store for the events, independent of what may be a more complex view of current state. Such event sourcing is often used in CQRS implementations.

Outbox (store-and-forward) - In this model, instead of publishing directly to service B, we store the event in the same datastore as used by service A. With many traditional databases this means we can cover the event inside the same transaction as used to persist whatever data we have in the main service. A separate thread can then publish events from the outbox to service B independently.

A variant of the outbox could be to implement fallback options or circuit breakers to retry messages or fallback to the outbox only when messages fail. This can improve the general throughput and responsiveness of services.

Balancing Controls - An alternative to the above solutions is to accept some degree of failure and implement policing strategies to verify that all expected events were received by the subscribing service. For example, how many orders did service B process in the past hour v how many orders where accepted by service A. In some cases it may be acceptable to introduce such delayed checks but these checks can be invasive. In addition, where gaps are identified a strategy is needed to replay events and fill in those gaps. You'll know there's a problem, but you may not be able to fix it automatically...

To make such balancing controls easier and more consistent across services a standardised inbox (e.g. on service B) can be used to store inbound messages or signatures thereof once processed and combine this with a similarly standardised outbox (e.g. on service A) so that checks can be made more easily and allow replay from the outbox for missed events. Such standardisation can place undesirable restrictions on the freedom of microservices to vary themselves and optimise for their specific needs but may be the least worst option.

As to which option is preferable depends on what constraints you can accept.

If you can tolerate inconsistent reads and eventual consistency and have high throughput requirements then event-sourcing may be preferable.

If on the other hand you absolutely must be able to read-your-writes and ensure consistency within a service then an outbox may be simpler and provides better consistency.

Alternatively if you can tolerate some level of failure and workaround corrections then perhaps balancing controls may suffice.

Focus

I suspect most of us working in IT today use agile methodologies such as Kanban, Scrum and Safe. We also strive to keep up to date with the latest developments in languages, libraries, patterns, architectures etc.

All of this is with the intent of improving the delivery speed, quality, efficiency, maintainability and the cost effectiveness of the systems we build - oh, and whilst improving our CVs at the same time.

Care though is needed to ensure we don’t get distracted by these tools from delivering the solutions they were employed for. This often isn’t through any specific fault with the methods and tools themselves but how we set about using them. 

There can often be a blinkered tendency to maniacally focus on the tools and methods themselves and in doing so fail to deliver most effectively on actual requirements and customer needs. If all you have is a hammer then everything looks like a nail.

Worse still, the job can become about servicing the tools used to do the work rather than the product of the work itself.

Furthermore these tools are often barely distinguishable from each other and for most use cases it doesn’t much matter if you use one or another; Java or .NET, AWS or GCP, Scrum or Kanban. Use what works for your team and be the master of tools, not a slave to them.

The product of your creativity is that which you build, value is derived from how your customers use the product.

Focus on the product and value it gives to your customers. Produce working solutions first and foremost - functionally and non-functionally - as efficiently as can be.

Over time we learn new methods and tools that improve delivery and optimise value for the customer but we need always be mindful that tools and methods are only a means to an end and not an end in themselves.

A Years Worth

A few Christmases ago I was messing about creating bubble maps - no doubt in some mince pie and port induced state of inebriation (quality of code is consequently as you'd expect).

I'd long forgotten about this until Kent Becks recent post on medium.com about trying to understand A Year's Worth of effort.

This looked familiar and so with a little manipulation here's a simple utility to convert priority ordered stories into a visual bubble map.

SAFe Shite!

SAFe (Scaled Agile Framework - for which I will not provide the link because they don't deserve it) is worthless shite!

It's just a bunch of practices developed elsewhere, rebranded by a cabal of consultants to cream money out of large organisations. The only additions it brings are there to placate management by helping mascarade traditional waterfall processes and hierarchical structures as "agile" - it does little to nothing new to actually change the org!

Save your money and go open source with your processes. SAFe is bollocks!

Spectre & Meltdown

Good high level explanation of how Spectre and Meltdown work... and WHY RASPBERRY PI ISN’T VULNERABLE TO SPECTRE OR MELTDOWN

SQL Server 2017 on Linux (mostly)...

Nice piece of work. Begs the questions when we'll see Windows for Linux though ;)