Entropy

Entropy is a measure of disorder in a system. A number of years ago I was flicking through an old book on software engineering from the 1970's. Beyond being a right riveting read it expounded the view that software does not suffer from decay. That once set, software programs, would follow the same rules over and over and produce the same results time and again ad infinitum. In effect that software was free from decay.

I would like to challenge this view.

We design a system, spend many weeks and months considering every edge case, crafting the code so we've handled every possible issue nature can throw at us including those "this exception can never happen but just it case it does.." scenarios. We test it till we can test no more without actually going live and then release our latest most wondrous creation on the unsuspecting public. It works and for a fleeting moment all is well with the universe... from this moment on decay eats away at our precious creation like rats gnawing away on the discarded carcass of the sunday roast.

Change is pervasive and whilst it's seems reasonable enough that were we able to precisely reproduce the starting conditions the program would run time and again as it did the first time, this isn't correct for reasons of quantum mechanics and our inability to time travel (at least so far as we know today). However, I'll ignore the effects of quantum mechanics and time-travel for now and focus on the more practical reasons for change and how this causes decay and increasing entropy in computer systems.

Firstly there's the general use of the system. Most systems have some sort of data-store; if only for logging, and data is collected in increasing quantities and in a greater variety of combinations over time. This can lead to permutations which were never anticipated which leads to exposure of functional defects or increase volumes beyond the planned capacity of the system. The code may remain the same but when we look at a system and consider it as an atomic unit in its entirety, it is continuously changing. Subsequent behaviour becomes increasingly unpredictable.

Secondly there's the environment the system exists within - most of which is totally beyond any control. Patches for a whole stack of components are continually released from the hardware up. The first response from most first-line support organisations is "patch to latest level" (which is much easier said than done) but if you do manage to keep up with the game then these patches will affect how the system runs.

Conversely, if you don't patch then you leave yourself vulnerable to the defects that the patches were designed to resolve. The knowledge that the defect itself exists changes the environment in which the system runs because now the probability that someone will try to leverage the defect is significantly increased - which again increases the uncertainty over how the system will operate. You cannot win and the cost of doing nothing may be more than the cost of addressing the issue.

Then there's change that we inflict ourselves.

If you're lucky and the system has been a success then new functional requirements will arise - this is a good thing, perhaps one for later but a system which does not functionally evolve is a dead-end and essentially a failure - call it a "panda" if you wish. The business will invent new and better ways to get the best out of the system, new use cases which can be fulfilled become apparent and a flourish of activities follow. All of which change the original system.

There's also non-functional requirements change. Form needs a refresh every 18 months or so, security defects need to be fixed (really, they do!), performance and capacity improvements may be needed and the whole physical infrastructure needs to be refreshed periodically. The simple act of converting a physical server to virtual (aka P2V conversion) which strives to keep the existing system as close to current as possible; detritus and all, will typically provide more compute, RAM and disk than was ever considered possible. Normally this makes the old application run so much faster than before but occasionally that speed increase can have devastating effects on the function of the system within time sensitive applications. Legislative requirements, keeping compliant with latest browsers etc., all bring more change...

Don't get me wrong, change is a good thing normally and the last thing we want is a world devoid of change. The problem is that all this change increases the disorder (entropy) of the system. Take the simple case of a clean OS install. Day 1, the system is clean and well ordered. Examining the disk and logs shows a tidy registry and clean log and temporary directories. Day 2 brings a few patches, which adds registry entries, some logs, a few downloads etc. but it's still good enough. But by Day n you've a few hundred patches installed, several thousand log files and a raft of old downloads and temporary files lying around.

The constant state of flux means that IT systems are essentially subject to the same tendency for disorder to increase as stated in the second law of thermodynamics. Change unfortunately brings disorder and complexity. Disorder and complexity makes things harder to maintain and manage, increasing fragility and instability. Increased management effort results in increased costs.

Well, that's enough for today.. next week, what we can do about it...

Cloud Jobs

Cloud is the current buzz in the industry and various cloud service-providers are jockeying for position to be #1. Beyond the hype and bravado I've been wondering who is really taking the lead because from my point of view it feels like it's down to Amazon and Google.

So I searched a few job-sites to see which cloud service providers are seen as being requirements for positions and the results are below.



 

Lots of "Cloud" jobs and AWS (Amazon Web Services) occurs quite frequently with Azure (Microsoft) and Rackspace relatively hot (compared to OpenShift, Softlayer and Oracle Cloud). Google App Engine (GAE) gets a few hits whilst the general search for "Google" (which covers "Google Apps" and so much more) if included would bring the search results into a comparable position to AWS but this is too general to include as "Cloud" so I've excluded it here. Google Compute Engine got no hits.

So Cloud is big, Amazon are #1 (currently) and Azure is pretty popular; which shouldn't be much of a surprise from the enterprise perspective. That OpenStack has a presence compared to end service-providers such as SoftLayer (IBM) and OpenShift (RedHat) indicates that there's work in the open-source cloud space which is good to see (AFAIC) and some of this looks to be in building private clouds. But the lack of any hits for Softlayer, OpenShift or Oracle Cloud is a bit of a surprise. I'd have thought someone would be after skills in this stuff. Anyway, my somewhat unscientific reckoning as to where we are based on a very small and selective sample of data is:

1. The notion that "Amazon=Cloud" is hard to shift and the rest look to be rather slow to the party.


2. Microsoft Azure is the preferred option for many enterprises who have a historic investment in all things MS and .NET.


3. Google may be late to the IaaS party but since the net is the bloodline for Google I suspect that in the wider context of "cloud" they'll probably do ok (they've also got a hell of a lot of compute capacity lying around).


4. Open-source cloud has a comparatively strong position compared to where OSS usually is (i.e. as the lowest cost option when you get down to IT as a commodity).


5. There's a lot of demand for cloud which doesn't have any of these big cloud service providers as a requirement so the space for competition should be pretty hot despite this apparent Amazon/Microsoft duopoly.

Ok, it's hardly scientific and the scope of these service providers varies significantly so comparison is perhaps unfair. There's also the fact that some search results are of the form "... help us move from X to Y" which yields hits on both X as well as Y and though skills are required in both items it's really Y that should be preferred. It's also a very narrow selection of jobs in Britian today and says nothing about the rest of the world or what's already in use. Anyway, for this evening it's answered my question and I'll be reading up on my AWS, Azure and OpenStack to keep my skills current this weekend... :)

For the record, job sites searched were; jobserve.com, monster.co.uk, and totaljobs.com.

 

Curve Fitting

I'm working on a curve fitting algorithm to help match against various performance profiles (personal project). Quite pleased with the first attempt to model a sine wave...



 

 

Reuse

Reuse! My favourite subject. Now, are you sitting comfortably? Then I'll begin...

Once upon a time in a land far far away, the king of computer-land was worried. Very worried indeed. His silly prime minister had borrowed lots and lots of money to build lots of new computer systems and programs and now they couldn't pay the interest on the debt. Worse still, none of the systems worked together and the land was becoming a confusing mess and there were lots of traffic jams and angry people. No-one knew how it worked and everything kept breaking. It was very expensive and it didn't work. The villagers were not happy and were threatening to chop the heads off the king and queen because they were French.

Then one day, a strange young prince claiming to be from a neighbouring country arrived bringing promises to sort out all the mess and clean the country up. And what's more, he'd do it cheaply and the country would have more money and better computer systems. The king and queen were very happy and the villagers were pleased as well - although they still want to chop the heads of the king and queen, because they were French and it would be fun.

So they listened to the prince and liked his ideas and gave him all the money they had left. The prince was going to build even more computer systems but they would all be based on the same design so would be cheap and quick to build. This meant he could spend more money on the design so it would be very good as well as cheap to build.

Then the prince said that he could also make a large hotel and everyone could live under the same roof. This would save on roofs because there would only be one and would be cheaper to run because there would only be one electricity bill. The villagers liked this because they liked going on holiday. The king and queen liked this because they had decided to go on holiday and so the villagers could not chop off their heads even though they were French.

Then the prince started to design the computer systems. He decided to start with the post-box because everyone sent letters. So he spoke to Granny Smith and Mrs Chatterbox about what they needed. They liked his design. It was round and red and pretty - it looked a bit like the old post-boxes.

Then he spoke to the bookshop keeper who didn't like his design because it was too small for him to post a book. So the prince made it bigger, much bigger.

Then he spoke to the postman who didn't like it because it was too big and would give him too many parcels to carry but the prince decided to ignore the postman because he was clearly an idiot.

So two of the postboxes were built; one in case the other was full, and the villagers liked them a lot even though the postman did not.

Next the prince decided to build the hotel so asked the villagers how they would like their room to look; because there could only be one design. Some wanted it round, some square, some with a balcony, some with stairs... and everyone want an en-suite with bidet even if they did not know how to use it. So the prince designed a flexible framework consisting of transformable panels which could be positioned wherever the villager chose. No-one liked the tents and the bidet was missing. The villagers were very angry and started to build a guillotine because they were French.

Then some of the villagers started to place their tents at the entrance to the hotel so they could get out quickly. But this stopped other villagers from coming in so made them angry. Then another villager blocked the toilet and all the villagers were angry and the hotel staff decided to go on strike because they were French and they hadn't had a strike yet.

So the villagers decided to summon the king and queen to come back from holiday and sort out the mess. So they each sent a letter recorded delivery. But the postbox didn't understand what "recorded delivery" meant because it was just a big round red box and and postman didn't want to pick up all the letters anyway because there were too many to carry and they hadn't paid the postage. So the king and queen didn't return to sort out the mess and the villagers were apoplectic with rage.

So the villagers burnt all the tents and drowned the postman and the prince in the river. Then the king and queen returned from holiday to find the city on fire and lots of angry villagers carrying pitchforks and pointing to a guillotine. But the king and queen were fat and so couldn't run away. So the villagers decided to form a republic and elected the prime-minister to become the president. The president chopped off the heads of the king and queen and the villagers were happy so gave a gallic shrug; because they were French, and lived happily for the next week or so...

All of which begs the question... what's this got to do with reuse?

Well, two things.

1. Design reuse requires good design to be successful. And for the design to be good there must be lots of consistent requirements driving it. All too often reuse programs are based on the notion of "build it and they will come" where a solution is built for a hypothetical problem which it's believed many requirements face. Often the requirements don't align and a lot of money is spent designing a multi-functional beast which tries; and often fails, to do too much which increases complexity which increases cost. The additional effort needed to consider multiple requirements from disparate systems significantly increases design, build and maintenance costs.  To make this worse, true cases of reuse are often common problems in the wider industry and so industry standard solutions and design patterns may exist which have been thought out by smarter people than you or me. To tackle these in-house is tantamount to redesigning the wheel... generally badly.


2. Instance reuse sounds like a great idea - you can save on licenses, on servers and other resources - but this creates undesirable dependencies which are costly to resolve and act to slow delivery and reduce ease of maintenance. Furthermore, resource savings are often limited as you'll only save on a narrow portion of the overall requirements - you'll need more compute, more memory and more storage. Getting many parties to agree to changes is also time-consuming and consequently costly and makes management of the sum more of a headache than it need be.

Personally I believe if you're going to progress a reusable asset program you need to validate that there really exists multiple candidate usage scenarios (essentially  the cost of designing and building a reusable asset must be less than cost of designing and building n assets individually), that requirements are consistent and that you're not reinventing the wheel. If this is the case, then go for it. Alternatively you may find an asset harvesting program to review and harvest "good" assets may yield better results; technically as well as being more efficient and cost effective. Then there's the view that all reuse is opportunistic in so much as using something designed to be "re"used is really just "use", and not "reuse" - as I once noted, "wearing clean underpants is 'use', turning them inside out and back to front is 'reuse'".

In terms of instance reuse, in my view it's often not worth saving a few licenses given the headaches that results from increased dependencies between what should be independent components. The problem is complicated with hardware, rack space and power consumption so is often not clear and some compromise is needed. However, the silver bullet here is virtualisation where a hypervisor can allocate and share resources out dynamically allowing you to squeeze many virtual machines onto one physical machine. License agreements may allow licensing at the physical CPU level instead of virtual CPU which can then be over allocated so you can have many guest instances running on fewer host processors. This isn't always the case of course and the opposite may be cheaper so this needs careful review of licensing and other solution costs.

Reuse isn't always a good idea and the complexities needed in design and build and additional dependencies resulting may outweigh the costs of just doing it n times in the first place. Use of standards, design patterns and harvesting of good assets should be the first  front in trying to improve quality and reduce costs . Any justification for creating reusable assets should include a comparative estimates of the costs involved; including ongoing cost to operations.

Welcome!

Welcome to nonfunctionalarchitect.com.

This is a blog dedicated to all things relating to non-functional requirements (NFRs) and IT.

So what are non-functionals?

Most of the time we think of IT systems in functional terms - essentially the what it does for us. But this doesn't cover some of the most critical aspects of IT systems such as:

• Form - As an architect I like diagrams. Unfortunately I can't draw for toffee. Thankfully we have designers to fill this gap in my skills. But beyond the dark realm of sexy website design comes accessibility, cross-browser support and the dreaded and fluffy world of usability...


• Operability - Covering backup & recovery, monitoring, failover etc. How is the system to be maintained and supported? What are the core SLA's we need to support? Configuration management... and so on.


• Performance & Capacity - How fast does it need to be? How scalable? How many users? How much data? etc.


• Security - Always in the press; usually for the wrong reason, and something which is often missed, mostly though incompetence but occassionally through wilful malice.

These topics and my own psuedonymised experiences are presented on this blog.