Voyaging dwarves riding phantom eagles

It's been said before... the only two difficult things in computing are naming things and cache invalidation... or naming things and something else, but always naming things.

When I started in IT we used to name servers after Star Trek ships, the Enterprise, Excelsior, Voyager etc. This I could cope with because I was naïve and a bit of a trekkie (or should that be a trekker?) so this seemed amusing enough.

Then we started naming software packages after authors (Kafka), languages after snakes (Python) and chemical reactions (Rust), and the origin of some Javascript libraries is a complete mystery (JHipster anyone?).

Then there are scrum team names based on Lord or the Rings characters (Hobbits, Elves etc.) which, erm, ok, and Marvel universe characters, Iron Man, Batman, whatever-man... (sorry guys, but the Marvel universe is boring).

And if you have project code names based on birds of prey then please don't be surprised if I can't tell the difference between a goshawk and a sparrowhawk.

I have very little connection with these names and they often bear no resemblance to what these things do, or if there is then any connection is so obscure as to be missed by the vast majority (sometimes being clever is stupid). 

Occasionally this is a good thing - obscurity and ambiguity are sometimes beneficial - but often such names serve only to confuse, restrict the flow of information and require mental vlookups to cross-reference code names to something more readily understood. We may as well invent our own language.

So "voyaging dwarves riding phantom eagles" would be work done by the accounting team on the development database to onboard a new client (or a scene from the Hobbit). Which do you think is clearer?

Beat me over the head with Rick and Morty memes long enough and perhaps I'll get it, but I'd rather spend my time riding my bike than watching cartoons (personal choices folks).

Names matter - and they are hard - so if you want to communicate clearly then use clear, simple and unambiguous names. Fair to say, I don't work in marketing...

The internet is down, all is well

 A few weeks ago I switched from Zen internet (stable enough; a touch more expensive than the big boys; excellent customer service) to Community Fibre (1Gb/s; fibre to the premise; symmetric; cheaper than Zen; only if you live in London).

The speed has been excellent. Overall just under 1Gb/s (around 960Mb/s) direct from the hub, up to 600Mb/s from a laptop in line-of-sight and easily 150Mb/s - up and down - everywhere else in the house.

Until today.

When it broke.

No idea why, an engineer will be sent to the house tomorrow, and in the meantime I'm operating off a mobile phone (works better than I expected).

I shall give Community Fibre the chance to fix this teething problem though that is not the reason for writing this post.

I also have a so called smart thermostat and since the wi-fi was down (I turned the now useless router off) the poor wee little smart hub had no way of talking to the thermostats and no way for us to tell it to switch the heating on or off from the app (of course there's an app...).

This does not make for a happy house when the wife wants to have a shower.

These systems have fall-backs for such cases though and pressing a button on the hub at least fires up the boiler, even if it doesn't talk to the thermostats - cue running around the house turning on the radiators and regaining enough brownie points to at least not spend the evening in the dog-house. Pressing the button again turns if off. Magic.

Ah-ha! I thought. But what happens if the wi-fi is working in the house even though the big bad internet is down.

To my surprise, it worked! The wife was not impressed but that's ok, I was (there's joy in small things). I can control the heating from the app despite the internet being unavailable.

It would be all too easy to design a system whereby the app and smart hub need internet access to initiate the handshake enabling the two to talk - a middle-man if you will. That this system (Drayton Wiser) did not rely on this and after a little thinking (spinning) connected and controlled my boiler - despite the lack of internet access - re-assured my faith that there's at least some competent engineers out there. The design team behind this has thought to cover a number of failure scenarios with some graceful degradation in capabilities to cover scenarios such as the internet being unavailable or my local wi-fi being down.

In all cases it should be possible to operate the system - turn the heating on and off. It may not be pretty, but it is viable and preferable to freezing (or spending the evening under the cold stare of a woman with dirty hair).

We all too often ignore failure scenarios and exception cases - they're hard, complicated and expensive. Instead we fall into a trap of assuring ourselves that "it'll never happen", that the real world is a perfect and clean environment which never fails, or that 99.9% uptime is good enough. In truth the real world is random, noisy and unreliable and one in which we have an insignificant amount of control. 99.9% uptime is essentially telling one in a thousand customers that you don't care about them. If you want to scale beyond more than a few hundred customers that 0.1% matters, at a million customers that can be a significant existential threat to your business.

The amount of hardware and software between the user and your application is staggering and any of it could and will fail in ways you have not considered1. Network and node failures happen. We need to accept this and introduce measures to defend ourselves. Measures such as bulk heads to allow for failures in some nodes, retry mechanisms for transient failures, queues to decouple components, caching for reliability as well as performance, circuit breakers, and my favourite of the day, a get-out-of-jail-free card for key functions for when the shit hits the fan (like a button to turn a boiler on or a hot-spot through a mobile phone to stay connected).

The, if everything else fails I can still get the job done by doing x magic! Trust me. You'll sleep better.

1 Originally I wrote "likely not considered", but no, you should assume you ain't considered half of it.   

We can't go on like this!

I'm sitting here in the sun - yes, it's sunny in south London - and for the past 30 minutes I've been trying to buy another of Martha Wells excellent Murderbot diaries on Kobo books. It's failed. Several times. Once it insisted on an update first, grrr. Once it just timed out. Once I gave up. And now... the ebook has decided to reboot itself. It nearly worked once but Monzo approval was needed and the Monzo app decided it was "Refreshing..."... and I guess it's hot so perhaps it needed a long cold shower because, well, that didn't work either, or rather when it did the Kobo had given up!

In the midst of this Microsoft logged me out of OneDrive for some random reason (I don't care why right now and I don't need to be interrupted to be told), and the god awful Apple AirPlay stuttered, stop-start, repeatedly. You can't listen to music like this.

All I wanted to do was sit in the sun, read some trashy sci-fi (Murderbot is excellent trash sci-fi - and I mean that in the best possible way), and listen to some Metallica in the background (blame Stranger Things).

To add insult to injury my mother called and her computer's no longer running Skype and a reboot won't bring it back. And you can forget about talking her through going to Launchpad and clicking on Skype. She's 78, once threatened to leave my father if he ever bought a new computer again (this was the '80s), and despises them (computers, not men). I have her using gmail and Skype and I've not got the energy to add any more complexity to her life or be the IT support bod any more than is strictly necessary. I shall fix Skype the next time I visit (which btw, what's the f**king point of video chat if you need to visit in person to get it working!).

We can't go on like this. This stuff needs to be much more reliable, much more tolerant of noise, much less invasive, so much easier to use and just plain better.

On the up side, it's 18:30, time for a whisky and a game of cards.

Picture yourself on a boat on a river...

PO: We need a bridge over the river right here?

Me: Why?

PO: Because the customer needs to get to the other side?

Me: Why can't they use the bridge half a mile up river?

PO: Because that'll take them on a half hour round trip! That's not very nice...

Me: Right... and er, how many people do we think will use this bridge?

PO: None. We're not actually expecting anyone to use it. Not for the first year anyway, but we need to have it for compliance reasons.

Me: Which are?

PO: Compliance reasons.

Me: and...

PO: Urgh,... Sally told Peter told Ravi told Mike told Sue told Priya told Vlad told Jean-claude told me that it was "important compliance reasons". And Sally is very busy and even more important...

Me: <sigh>... So it's not a big deal if that no-one has to walk for 30 minutes then.

PO: Well, someone might use it, and what if they're in a hurry?

Me: They should have planned ahead...

PO: You can't tell a customer to "plan ahead"!

Me: Yes, you can. You do it every time you go on a road trip, you just need to know to expect it. Anyway, how about we have a boat that can ferry people across?

PO: No, no, no! We've told you before. We don't like boats. They're expensive and can't carry many passengers.

Me: Not as expensive as bridges and you just told me we don't have any customers!

PO: Look Mr Grumpy, are you going to design a bridge or not?

Me: .... what's your budget?

PO: Haven't thought about that, but it needs to be ready a week on Tuesday... and who's that fool in the river?

Me: That'll be a customer who fell off the rope bridge we built up river...

Don't treat people like serverless functions.

When I were knee high to a grasshopper we didn't have all this new fangled cloud infrastructure and we certainly didn't have the concept of serverless computing. How can you compute without a computer?..

But before my time (and I'm not that old!) computers were people. People like Sally. Actual humans sitting in offices with bits of paper, pencils and tables of logarithms and trigonometric functions. Adding, subtracting, scribbling down results, checking and verifying. Whether a human being or a hunk of metal, you need a computer to compute.

Well, most of the time the thing you're computing is important enough that you can't afford for it not to be computed. It's a bad thing if you need to calculate wage-packets and can't do it, whether it's because Sally's off sick and can't compute today or the server has gone down because it's run out of disk space. You're going to have a riot on your hands come Friday afternoon when the pubs open...

Which brings us to the concept of redundancy. 

Rather than relying on Sally alone we need to ensure we have someone else around who can also compute when she's out sick. Equally - in the world of tin - we need backups in case our primary fails - disks, networks, servers, power, cooling, data-centers etc. 

This is an expensive habit.

Almost every system will be important enough to warrant some degree of redundancy. The more critical they become, the greater the degree of redundancy required and the more time architects spend worrying about the impact of component failures, how long it takes to recover, how much data could be lost, acceptable error rates and so on.

In the bad old days we would literally have a standby server for every primary. Boom! Double the costs right there.

Just imagine if we needed to employ Jane as well as Sally just to cope with the days Sally was sick? Of course we'd not do this. We need to ensure the function can be picked up by someone else but that person could be someone else in a team where the team has a degree of redundancy built in (or even someone who's main job is something else, Bob from accounting or Sallys manager for example). Perhaps we can work out that if we need seven computers we best hire ten to cover the days some are sick...

Having a level of redundancy in the organisation provides the flexibility to handle outages and failures. Besides, people can't work at 100% capacity. They will burn out, productivity will fall, they'll hate you for it and will leave as soon as a better opportunity turns up.

Anyway, back in the world of tin, along came virtualisation and we could host multiple virtual machines (VMs), on one physical host in much the same way one person could turn their hands to multiple tasks. This was great ('ish) as it reduced the number of physical servers significantly and saved on hardware, power, space, CO2 emissions and consequently dollars. We still need to support some degree of redundancy in case a host node went down or a VM failed but it's much better than before. 

How much better?

Well, most systems aren't Google or Netflix. I know, surprising huh?

Most systems don't need to support 1000's of transactions per second. Mostly it's less than 1 tps. Yup, one! And often it's a lot less than one... like a few hundred transactions a day is typical of many systems. Me and a my Casio fx-85GT Plus can handle that!

So we can stuff a lot of VMs onto a single physical host with perhaps 50 VMs running across 3 physical hosts in a cluster whilst still maintaining enough redundancy to ensure availability. Make that tin sweat!

Suffice to say, if we treated Sally like the tin, she would not be impressed.

VMs are still pretty hefty though. Each VM runs its own copy of the operating system and associated processes which makes them pretty big (GBs of RAM for the VM compared with perhaps a few MB for application processes). We've had multi-tasking operating systems for decades now and there's little reason we can't run multiple application processes on the same server. Other than it's a really bad idea.

Developers make lots of assumptions about the environment they're running - which libraries and versions are available, what file paths they can use etc. - and a lot of these things aren't compatible with each other. They're also really bad at security and act like peace loving drug infused hippies... "hey, why would anyone else read my files man?". Add to this that bugs happen (always will) resulting in unstable or runaway processes crashing or consuming all the resources available and it's a recipe for disaster.

Running multiple disparate application processes on the same server is a bad idea. Or was...

Now along comes containerization and provides a degree of isolation between processes within a server to prevent one hippie process treading on another hippies toes. And we know how exposed hippies toes are don't we?

This can give us an order of magnitude increase in processes on a host so we're now up to 500 containers across our 3 node cluster of computers. Nice.

Sally on the other hand is seriously pissed.

But we still have to manage a bunch of physical servers underpinning our applications. Whether VMs or containers, there's a bunch of power hungry, raging hot physical computers burning away in the background. And in the case of human computers, really angry overworked ones.

Then came serverless.

Forget the server. You pay-per-use - that few hundred transactions a day - leveraging services provided by cloud providers. Everything from databases to messaging to raw compute can be provided as a pay-per-use service without needing to worry about the server or redundancy.

Erm, well, except for the cloud service provider - who worries about it a lot - and your architect who still needs to worry about the non-functional as well as functional characteristics of all those services we end up consuming (especially the economics of serverless if that one tps turns into many thousands..). 

Of course serverless isn't really server-less and there's always a bit of tin somewhere. We're really talking about building applications out of services (like AWS Lambda or Google Cloud Functions). Those services are carefully managed by cloud providers who supply all the necessary redundancy to support the resiliency, availability and scalability you'd expect.

But what about Sally? Does she still have a job?

Sadly no. Sally has now been moved into the gig economy on a zero-hours contract and works on a pay-per-use basis. She doesn't get any guaranteed work, an hourly rate or sickness benefits. Please don't treat people like serverless functions.

Docs

There, I said it. A four letter swear word. Something worse than the F’ word if the horror on the boss’ face is anything to go by.

We don’t do “documentation” anymore and besides, the agile manifesto says it’s immoral to write a word of documentation. The code is the documentation. That you have to get inside the twisted maze of my mind, and work out what drug infused insanity I was trying to convey at the time and which may or may not result in what was intended regardless… that’s your problem. 

Such nonsense pervades software development today. Although…

It’s not the lack of documentation per se but the lack of demonstrable thought that bugs me. A critical explanation of why things are the way they are.

Why has solution design been so poorly treated?

The answer to that lies in what design is trying to achieve.

Solution design aims to address the needs of the system, communicates how these needs are met and enables testing of the solution when the cost of change is lowest.

This testing is achieved by static walkthroughs and peer review.

However, in an agile world where we breakdown features and stories into small chunks that can be delivered rapidly and iteratively, the hefty tombs of yesterdays solution design documents are equally shrunk to focus on the few features and stories in scope.

Taken far enough it ends in a combined whiteboard design and review session with a handful of developers. And beyond taking a photo and circulating to the team, what's the point in doing anything more?

Firstly there's the supporting teams who are going to care for your solution through its early teething days, rebellious teenage years, into maturity and all the way to the grave – death being one of the few certainties in life. Making these guys bump their way around a darkened room trying to figure out how things hang together is unfair if not sadistic.

These guys don't work in scrum teams or story by story. They work from incident to incident, from shit-storm to fan-splattering shit-storm. They need a concise and holistic view of the solution for which a few poorly framed holiday photos doesn't cut it. Honestly, these guys are too nice to you.

Then there's the not insignificant matter of consistency. 

You can argue the marginal benefits of any technology but you need to justify any change which runs counter to the inherent design patterns of a system as contributing significant value to overcome the increased complexity and costs it brings.

Do things consistently and you get paybacks in terms of proven reliability, easier maintainability, reduced cognitive load, faster delivery and so on.

Do things inconsistently and whilst you may get to play with the latest  tech you're being pretty selfish - if that's your motivation. And if it's not then see above re justifying the increased complexity and costs.

Which brings me to my point on solution design.

Solution design is about more than addressing the needs of a few stories on a sprint by sprint basis. It's about addressing the broad needs of the system and providing a vision for the longer term. Defining the patterns which will be used time and again and which (should) enable that payback through consistency.

This isn't an argument that technological progress isn't a good thing or that we should never have crawled out of the sea. Or that your resident architect has all the answers. It's an argument to think about the broad needs, key decisions, responsibilities, patterns, principles, policies and costs - immediate and long term - that provide the foundation on which any system is based.

At times it may seem futile and time consuming but it's cheap to change this stuff early on rather than late on when it's most expensive.

And the way we do that best is through reasoned discourse. Discourse best expressed through quality documentation - words, diagrams and matrices.

The Con of Agile (or why agile reductionism is hard…)

Agile is, to a large extent, a radical breakdown of function into small incremental features delivered in a prioritised manner with rapid feedback to inform on the next evolutionary step to deliver ever greater customer value.

Executed as such you may not; probably should not, get what you originally intended and even if you fail then at least you’ve failed fast and saved yourself an expensive pipedream.

Bliss.

But agility is not easy. It is not something a tool, method or consultant will magically fix for you – despite what they tell you. Radical breakdown of function is hard. Reductionism is hard.

Most systems we build today are inherently complex with dependencies spread far and wide through the enterprise. Through reductionism we attempt to decompose these complex systems into their simpler component parts – a technique that has a long and successful history in software development – only to then recompose them in many and varied ways to deliver the outcomes we desire.

In a typical large organisation it’s not uncommon for a system to require integration with dozens of others and for any particular new feature to impact several of these at a time – in fact it’s rarer for a feature to be self contained within one system. Integration is the norm.

And here our problems start… 

According to Conways law an organisation will typically design systems that mirror the organisations communication structure. i.e. we end up with various systems mirroring the group and team structure of the organisation integrated along the same communication channels as the organisation.

The net result is that we have specialist teams with a deep understanding not just of the technology but the values and ethos of each system and the organisation they reflect. These teams own the system and have veto rights – rightly so – over what functionality they do or do not support. 

So on one hand we have a prioritised backlog of customer focused features and on the other we have a collection of disparate teams available to deliver them hell bent on aligning to the internal organisation structure. Something the customer usually doesn’t (and shouldn’t) give a toss about.

An instinctive reaction is for teams to start to breakdown features into stories that make sense from the organisational structure perspective. This can create a reflection of the organisation in the user journey and leads to stories which are really tasks because that’s easiest for teams to consume. And if you’re simply working off a prioritised list of tasks you’re not agile. You may be able to re-prioritise but you’ve lost the connection with why you’re doing something and that critical feedback loop from the customer so you can’t respond to their changing needs.

Perhaps worse still, a disparate collection of tasks spread across a plethora of teams does not provide a clear vision and direction for the people working in those teams. Treated this way people can start to feel like slaves to the machine, churning out widgets hour after hour with no clear understanding of why. Motivation and quality suffers and we no longer have small agile teams focused on delivering customer value but a collection of teams each with their own perspective on what we’re trying to achieve. 

As we breakdown features and stories into smaller chunks, maintaining a focus on customer value and the overall vision is hard – particularly in a large and complex organisation.

There’s now a twofold need both for well-articulated, customer focused and concise feature and story definition, and for solution designs which clearly stitches together the various parts of the organisation; and associated systems, connected to deliver those features.

The features and stories say what we’re trying to do and why we’re trying to do it.

The solution design says how we’re going to do it.

And that understanding of why we’re doing something is the “Ah-ha!” moment that gives us meaning and purpose, the reason we get out of bed in the morning. 

Ultimately it’s easy to explain what we’re trying to do and how we plan to do it – they’re concrete things we can action and it’s natural for us poor humans to want to focus on solving problems. It’s much harder to articulate why we’re trying to do something, and we all too often forget that others aren’t aware of the vision in the first place.

So as we break down capabilities and features into ever smaller chunks we need to consciously focus on the “why” we’re building this chunk and less so the “what” and the “how”. It may seem counter intuitive but solving problems is easy, finding them is harder.

As a consequence our features and stories should be heavy on the explanation of why we’re trying to do something and lighter on what we’re actually going to do. The “how” we’re doing something should be reserved for the solution design and provides traceability from the “what” and the “why” to the organisational structure, providing the foundation for that task list everyone seems to desire.

As we breakdown stories we also tend to reduce the number of teams involved and if we go far enough that they only impact one team then great. It won’t though always be the case and we shouldn’t try to twist stories to meet the internal organisation structure and make it fit us rather than the customer.

In short, as we breakdown features and stories we need to focus on why we’re doing something and what we’re going to do to address that need. Don’t worry about how we’re doing it – leave that to the technical team of architects and engineers – give them the easy task of connecting the dots and solving the problem. 

The trap in agile reductionism is that we, as human beings, have a tendency to want to solve the problem and in doing so lose the connection to the why that gives us meaning and purpose. We end up focusing on the solution and thereby reduce the solution to a problem, not the problem itself. The two are not the same. One is optimised for the organisation, the other for the customer.

An Observation

Much has changed in the past few years, hell, much has changed in the past few weeks, but that’s another story... and I’ve found a little time on my hands in which to tidy things up.

The world of non-functionals has never been so important and yet remains irritatingly ignored by so many - in particular by product owners who seem to think NFRs are nothing more than a tech concern.

So if your fancy new product collapses when you get get too many users, is that ok?

It’s fair that the engineering team should be asking “how many users are we going to get?”,  or “how many failures can we tolerate?” but the only person who can really answer those questions is the product owner. 

The dumb answer to these sort of question is “lots!”, or “none!” because at that point you’ve given carte-blanche to the engineering team to over engineer... and that most likely means it’ll take a hell of a lot longer to deliver and/or cost a hell of a lot more to run.

The dumb answer is also “only a couple” and “hell, I don’t care.” because, well... you’ll have different problems, mostly about whether you should be in business at-all I suspect.

So the balance is somewhere in the middle. Take your OKRs and have the discussion with the engineering team about what the concerns may be and where the hot-spots are. Run through some “what-if...?” scenarios to understand the potential impact of under/over-estimating. In many cases there are strategies that can be taken to reactively expand/shrink resources which can at least buy you some time... if not more.

Regardless, from an engineering perspective we need to:

1. Understand expectations and what really matters - driven through an understanding of  product OKRs.

2. Derive NFRs, agree these with product owners as acceptance criteria, and architect and design accordingly.

3. Monitor and observe what’s actually going on - how else do you prove you’ve met your acceptance criteria?

For example, a system I worked on recently involves processing transactions running into the millions of dollars. In this case, losing or duplicating a transaction can be seriously bad for your health.

The consequence? A lot of discussion with product teams resulting in significant design effort to ensure the integrity of transactions as they pass through the system. Realtime monitoring as transactions flow, near-time balancing controls as belt-and-braces and financial accounting and reporting on top (to be honest the business only really care about accounting but that feedback loop is way too slow from an engineering perspective (daily)).

Duplicates transactions are avoided - at the cost of reduced availability - and dropped transactions are detected and alerted on within a few minutes and can be automatically replayed (if we ever grow the balls to turn that on).

This isn’t cheap and involves a lot of testing and verification - including some chaos testing to simulate duplicates (not loses (yet)) and I hope it’s been worth it.

It also involves significant investment in monitoring, tracing and alerting to ensure we have good visibility of what’s going on across the platform so we can spot problems quickly if they do happen (and they can, they just shouldn’t result in a financial loss for customers or ourselves).

This is build I hope never to have to call on and the sort of failures which scare me most involve the outage of entire clusters and complete site failures - this stuff needs specific test focus. Auto-scaling, rebalancing and node failures are such common occurrences now as to be BAU and should not be the reason you’re called out of bed at 2am - we test this stuff by simply breathing these days. 

Through discussion with product owners on OKRs we can start to uncover our NFRs and what degree of failure can and cannot be tolerated. We use this knowledge to architect and design solutions appropriate to the problem space, and we monitor and observe to ensure we’re within bounds. 

And if the boundaries move? Then we adapt, evolve, change accordingly. Through discussion we’ve hopefully had a chance to consider what could go wrong and should already have some escape routes planned... just in case.

Inter-microservice Integrity

A central issue in a microservices environment is how to maintain transactional integrity between services.

The scenario is fairly simple. Service A performs some operation which persists data and at the same time raises an event or notifies service B of this action.

There's a couple of failure scenarios that raise a problem.

Firstly, service B could be unavailable. Does service A rollback or unpick the transaction? What if it's already been committed in A? Do you notify the service consumer of a failure and trigger what could be a cascading failure across the entire service network? Or do you accept long term inconsistency between A & B?

Secondly, if service B is available but you don't commit in service A before raising the event then you've told B about something that's not committed... What happens if you then try to commit in A and find you can't? Do you now need to have compensating transactions to tell service B "oops, ignore that previous message!"?

I'll ignore the use of queues/topics between services as this really just becomes the service B failure point although there are topologies in which risks can be mitigated this way (e.g. through queues local to the origin service A).

There are several options to address this issue:

Event Sourcing - In this model the event is king. Service A persists events to an event store and downstream consumers can subscribe to these events. The event itself does not present the current state of an entity but the history of what has happened. To understand the current state of the entity you need to replay the event history. Service B then consumes from the event store. Beyond this a read view can be maintained by service A to provide an efficient presentation of current state. Note though that in this model there is a risk of read inconsistency since that view is updated separately from writes to the event store. If you can tolerate the additional complexity and eventual consistency then this can work well. It also means you can have a fast write store for the events, independent of what may be a more complex view of current state. Such event sourcing is often used in CQRS implementations.

Outbox (store-and-forward) - In this model, instead of publishing directly to service B, we store the event in the same datastore as used by service A. With many traditional databases this means we can cover the event inside the same transaction as used to persist whatever data we have in the main service. A separate thread can then publish events from the outbox to service B independently.

A variant of the outbox could be to implement fallback options or circuit breakers to retry messages or fallback to the outbox only when messages fail. This can improve the general throughput and responsiveness of services.

Balancing Controls - An alternative to the above solutions is to accept some degree of failure and implement policing strategies to verify that all expected events were received by the subscribing service. For example, how many orders did service B process in the past hour v how many orders where accepted by service A. In some cases it may be acceptable to introduce such delayed checks but these checks can be invasive. In addition, where gaps are identified a strategy is needed to replay events and fill in those gaps. You'll know there's a problem, but you may not be able to fix it automatically...

To make such balancing controls easier and more consistent across services a standardised inbox (e.g. on service B) can be used to store inbound messages or signatures thereof once processed and combine this with a similarly standardised outbox (e.g. on service A) so that checks can be made more easily and allow replay from the outbox for missed events. Such standardisation can place undesirable restrictions on the freedom of microservices to vary themselves and optimise for their specific needs but may be the least worst option.

As to which option is preferable depends on what constraints you can accept.

If you can tolerate inconsistent reads and eventual consistency and have high throughput requirements then event-sourcing may be preferable.

If on the other hand you absolutely must be able to read-your-writes and ensure consistency within a service then an outbox may be simpler and provides better consistency.

Alternatively if you can tolerate some level of failure and workaround corrections then perhaps balancing controls may suffice.

Focus

I suspect most of us working in IT today use agile methodologies such as Kanban, Scrum and Safe. We also strive to keep up to date with the latest developments in languages, libraries, patterns, architectures etc.

All of this is with the intent of improving the delivery speed, quality, efficiency, maintainability and the cost effectiveness of the systems we build - oh, and whilst improving our CVs at the same time.

Care though is needed to ensure we don’t get distracted by these tools from delivering the solutions they were employed for. This often isn’t through any specific fault with the methods and tools themselves but how we set about using them. 

There can often be a blinkered tendency to maniacally focus on the tools and methods themselves and in doing so fail to deliver most effectively on actual requirements and customer needs. If all you have is a hammer then everything looks like a nail.

Worse still, the job can become about servicing the tools used to do the work rather than the product of the work itself.

Furthermore these tools are often barely distinguishable from each other and for most use cases it doesn’t much matter if you use one or another; Java or .NET, AWS or GCP, Scrum or Kanban. Use what works for your team and be the master of tools, not a slave to them.

The product of your creativity is that which you build, value is derived from how your customers use the product.

Focus on the product and value it gives to your customers. Produce working solutions first and foremost - functionally and non-functionally - as efficiently as can be.

Over time we learn new methods and tools that improve delivery and optimise value for the customer but we need always be mindful that tools and methods are only a means to an end and not an end in themselves.

A Years Worth

A few Christmases ago I was messing about creating bubble maps - no doubt in some mince pie and port induced state of inebriation (quality of code is consequently as you'd expect).

I'd long forgotten about this until Kent Becks recent post on medium.com about trying to understand A Year's Worth of effort.

This looked familiar and so with a little manipulation here's a simple utility to convert priority ordered stories into a visual bubble map.

SAFe Shite!

SAFe (Scaled Agile Framework - for which I will not provide the link because they don't deserve it) is worthless shite!

It's just a bunch of practices developed elsewhere, rebranded by a cabal of consultants to cream money out of large organisations. The only additions it brings are there to placate management by helping mascarade traditional waterfall processes and hierarchical structures as "agile" - it does little to nothing new to actually change the org!

Save your money and go open source with your processes. SAFe is bollocks!

Spectre & Meltdown

Good high level explanation of how Spectre and Meltdown work... and WHY RASPBERRY PI ISN’T VULNERABLE TO SPECTRE OR MELTDOWN

SQL Server 2017 on Linux (mostly)...

Nice piece of work. Begs the questions when we'll see Windows for Linux though ;)

Equifarce!

I was/am interested into the whole Equifax hack and how this happened. To this end I posted a brief link yesterday to the Struts teams response. A simple case of failing to patch! Case closed..

But then I've been thinking that's not really very fair.

This was (apparently) caused by a defect that's been around for a long time. The developers had reacted very quickly when the problem was identified (within 24 hrs) but Equifax - by all accounts - had failed to patch for a further 6 months.

What did we expect? That they'd patch it the next day? No chance. Within a month? Maybe. But if the issue is embedded in some third party product then they're dependent upon a fix being provided and if it's in some in-house developed tool then they need to be able to rebuild the app and test it before they can deploy. Struts was/is extremely popular. It was the Spring of its day and is still deeply embedded in all sorts of corporate applications and off the shelf products. Fixing everything isn't going to happen overnight.

Companies like Equifax will also have hundreds, even thousands, of applications and each application will have dozens of dependencies any one of which could have suffered a similar issue. On top of this, most of these applications will be minor, non critical tools which have been around for many years and which frankly few will care about. Running a programme to track all of these dependencies, patch applications and test them all before rolling them into production would take an army of developers, testers, sys-ops and administrators working around the clock just to tread water. New features? Forget it. Zero-day? Shuffles shoes... Mind you, it'd be amusing to see how change management would handle this...

So we focus on the priority applications and the low hanging fruit of patching (OS etc.) and hope that's good enough? Humm... anything else we can do?

Well, we're getting better with build, test and deployment automation but we're a long way from perfection. So do some of that, it'll make dealing with change all the easier but its no silver bullet. And again, good luck with change management...

Ultimately though we have to assume we're not going to have perfect code (there's no such thing!)... that we're not able to patch against every vulnerability... and that zero day exploits are a real risk.

Other measures are required regardless of your patching strategy. Reverse proxies, security filters, firewalls, intrusion detection, n-tier architectures, heterogenous software stacks, encryption, pen-testing etc. Security is like layers of swiss-cheese - no single layer will ever be perfect, you just hope the holes don't line up when you stack them all together. Add to this some decent monitoring of traffic and an understanding of norms and patterns - at least something which you actually have people looking at continually rather than after the event - and you stand a chance of protecting yourself against such issues, or able to identify potential attacks before they become actual breaches.

Equifax may have failed to patch some Struts defect for six months but that's not the worst of it. That they were vulnerable to such a defect in the first place smells like.. well, like they didn't have enough swiss-cheese. That an employee tool was also accessible online and provided access to customer information with admin/admin credentials goes on to suggests a real lack of competency and recklessness at senior levels.

Adding insult to injury, to blame an open-source project (for the wrong defect!) which heroically responded and addressed the real issue within 24 hrs of it being identified six month earlier (!?) makes Equifax look like an irresponsible child. Always blaming someone else for their reckless actions.

They claim to be "an innovative global information solutions company". So innovative they're bleeding edge and giving their, no our!, data away. I'm just not sure who's the bigger criminal... the hackers or Equifarce!

Equifax Data Breach Due to Failure to Install Patches

"the Equifax data compromise was due to their failure to install the security updates provided in a timely manner."

Source: MEDIA ALERT: The Apache Software Foundation Confirms Equifax Data Breach Due to Failure to Install Patches Provided for Apache® Struts™ Exploit : The Apache Software Foundation Blog

As simple as that apparently. Keep up to date with patching.

DIY

I should probably have learnt this some time ago...

Quite often we find no-one is willing to do the {insert-task-here}.

I don't know why. Fear of getting it wrong. Fear of ridicule. Fear of crayons. Whatever. Heres a tip on how to get things moving when no-one seems willing...

DO IT YOURSELF!

It doesn't even matter if you do it badly. In fact it's often better to do it badly on purpose!

You'll be amazed (or maybe not) at the number of people that come out of the woodwork to provide their own "advice". All of sudden you'll have no end of input. Just be prepared to bite your tongue and take solace in the knowledge that you took one for the greater good.

Someones got to get the ball rolling...

It's an older code..

(let's assume we're talking about encryption keys here rather than pass codes though it really makes little difference... and note that your passwords are a slightly different concern)

Is it incompetence to use an old code? No.

For synchronous requests (e.g. like those over HTTPS) there's a handshake process you go through every few minutes to agree a new key. Client and server then continue to use this key until it expires then they agree a new one. If the underlying certificate changes you simply go through the handshake again.

For asynchronous requests things aren't as easy. You could encrypt and queue a request one minute and change the key the next but the message remains on the queue for another hour before it gets processed. In these cases you can either reject the message (usually unacceptable) or try the older key and accept that for a while longer.

Equally with persistent storage you could change the key every month but you can't go round decrypting and re-encrypting all historic content and accept the outage this causes every time. Well, not if you've billions of records and an availability SLA of greater than a few percent. So again, you've got to let the old codes work..

You could use full disk/database encryption but that's got other issues - like its next to useless once the disks are spinning... And besides, when you change the disk password you're not actually changing the key and re-encrypting the data, you're just changing the password used to obtain the key.

So it is ok to accept old codes. For a while at least.

An empire spread throughout the galaxy isn't going to be able to distribute new codes to every stormtrooper instantaneously. Even if they do have the dark-side on their side...

Fnd Collaboration

https://youtu.be/VQJ0MIwvLUc

Demonstration using Fnd for collaborative modelling between two browsers - in this case Safari and Firefox.

Fnd works best with Chrome and Firefox, mostly with Opera and Safari (as you can see here) but not with IE... Hey, it's alpha and I have little time to deal with the mindless irritations of IE..!

Try Fnd here...

Lost and Fnd

I could spend the next two years tweaking things as much as I have the past two but it's time to get some feedback on this so here we go...

Software modelling tools are expensive, bad at collaboration and more complicated than the vast majority of solutions demand. They set their sights on an unachievable utopia, demand constant maintenance to avoid being constantly out of date and dictate that we all follow overly prescriptive rules which may be philosophically correct but are practically irrelevant.

Diagramming tools are cheaper, better at collaboration and simpler but are all too fluid, lack structure and meaning and are far too open to interpretation. The best such tool I use is a whiteboard, a pen and room - a shame that such are so transient and unstructured.

A middle way is needed.

To this end I introduce Fnd (alpha).

Fnd lets you build catalogs, diagrams and matrices for solutions in a collaborative manner.

For now you can only sign-in with Google and when you do you'll first be asked to create an account. Once you do you'll see the home page which provides a list of current solutions.

Hit [+]  to create a new solution and provide any details you need.



Save this and you should now see a new solution defined on the homepage.



To edit the entry click on the main icon. To access the solution page, click on the link icon next to the solution name:

Clicking on the solution link takes you to the solution page which provides a list of the catalogs, diagrams and matrices associated with that solution.



From here you can create a structural diagram



and in the diagram view you can set the title and description (and tags (a little more on tagging will come later)).



The nav bar allows you to specify the stencil, colours, save, reload, screenshot, document, delete (item) and delete diagram.



To add something to the diagram select the catalog item and "new". This will show a popup allowing you to define that it and add it to the diagram.



Each catalog type has attributes specific to its needs. Choose that which suits best. For example, a component shows as:

When added to the diagram it appears with a stencil relevant to its type. This stencil can be changed by selecting the object in the diagram then the stencil type from the "Shapes" or "UML" drop-down. In the example below there are two components, one shown as a UML component stencil, the other as a database component.



Double click on the object to edit the settings and make sure you save the diagram - editing an object is immediate, changes to diagrams need to be saved.

Relationships can be added by clicking and dragging on the link icon (green +) from one object to the link icon on another.



From actor to component results in:



Double clicking on the link itself allows the attributes of the link to be defined. By default every link is a dependency but this can be changed as desired.

... and so on to build up diagrams.

Perhaps more importantly, if you grant another user access to your account (email-address > account > add user) then if you can both edit the same objects/diagrams at the same time and will see updates reflected in each others browser.

Matrices provide views of relationships in tabular and animated form. For example the above diagram appears as:

and

And catalog lists provide access to all objects of that type.


There's more to this with multiple diagrams providing different views, the ability to search and add objects from one solution to another, using tags to provide filtered views, generating documentation from diagrams and so on. I can't promise diagrams are always "pretty" or that models are "correct" but instead aim for a hopefully happy compromise somewhere between the two - enough to be understood!

A word of warning... Fnd is alphaware, I'm looking for feedback, it has bugs - some I know, some I do not. I use it daily in my work and it works for me (mostly). Let me know how you get on - the good, the bad and the ugly - and in turn I'll try to improve it and make it more stable and functional.

You can access Fnd at Feedback to admin [at] nfa.cloud.

p.s. Fnd is short for Foundation and simply a tip of my hat to one of my favourite authors...

UPDATE 2021-3-28: Work on FND has halted and it is no longer available.