Lost and Fnd

I could spend the next two years tweaking things as much as I have the past two but it’s time to get some feedback on this so here we go…

Software modelling tools are expensive, bad at collaboration and more complicated than the vast majority of solutions demand. They set their sights on an unachievable utopia, demand constant maintenance to avoid being constantly out of date and dictate that we all follow overly prescriptive rules which may be philosophically correct but are practically irrelevant.

Diagramming tools are cheaper, better at collaboration and simpler but are all too fluid, lack structure and meaning and are far too open to interpretation. The best such tool I use is a whiteboard, a pen and room – a shame that such are so transient and unstructured.

A middle way is needed.

To this end I introduce Fnd (alpha).

Fnd lets you build catalogs, diagrams and matrices for solutions in a collaborative manner.

For now you can only sign-in with Google and when you do you’ll first be asked to create an account. Once you do you’ll see the home page which provides a list of current solutions.

Hit [+]  to create a new solution and provide any details you need.

Save this and you should now see a new solution defined on the homepage.

To edit the entry click on the main icon. To access the solution page, click on the link icon next to the solution name:Clicking on the solution link takes you to the solution page which provides a list of the catalogs, diagrams and matrices associated with that solution.

From here you can create a structural diagram

and in the diagram view you can set the title and description (and tags (a little more on tagging will come later)).

The nav bar allows you to specify the stencil, colours, save, reload, screenshot, document, delete (item) and delete diagram.

To add something to the diagram select the catalog item and “new”. This will show a popup allowing you to define that it and add it to the diagram.

Each catalog type has attributes specific to its needs. Choose that which suits best. For example, a component shows as:

When added to the diagram it appears with a stencil relevant to its type. This stencil can be changed by selecting the object in the diagram then the stencil type from the “Shapes” or “UML” drop-down. In the example below there are two components, one shown as a UML component stencil, the other as a database component.

Double click on the object to edit the settings and make sure you save the diagram – editing an object is immediate, changes to diagrams need to be saved.

Relationships can be added by clicking and dragging on the link icon (green +) from one object to the link icon on another.

From actor to component results in:

Double clicking on the link itself allows the attributes of the link to be defined. By default every link is a dependency but this can be changed as desired.

… and so on to build up diagrams.

Perhaps more importantly, if you grant another user access to your account (email-address > account > add user) then if you can both edit the same objects/diagrams at the same time and will see updates reflected in each others browser.

Matrices provide views of relationships in tabular and animated form. For example the above diagram appears as:

and

And catalog lists provide access to all objects of that type.

There’s more to this with multiple diagrams providing different views, the ability to search and add objects from one solution to another, using tags to provide filtered views, generating documentation from diagrams and so on. I can’t promise diagrams are always “pretty” or that models are “correct” but instead aim for a hopefully happy compromise somewhere between the two – enough to be understood!

A word of warning… Fnd is alphaware, I’m looking for feedback, it has bugs – some I know, some I do not. I use it daily in my work and it works for me (mostly). Let me know how you get on – the good, the bad and the ugly – and in turn I’ll try to improve it and make it more stable and functional.

You can access Fnd at https://nfa.cloud/. Feedback to admin [at] nfa.cloud.

p.s. Fnd is short for Foundation and simply a tip of my hat to one of my favourite authors

!Attachments, Links

As I’ve said before, email is broken. Unfortunately, like smack, many seems hooked on it. For your methadone, start to wean yourself off by sending links and not attachments.

It’s more accessible, more secure, you can keep the content up-to-date and ensure everyone sees the current version. It’ll even stop your mail file from bloating…

Just use a wiki like Confluence and try not to attach docs to pages…

Blood-e-mail

We all hate e-mail. We all love e-mail…

E-mail is like writing  a letter. There was a time when sitting down to write a letter (with pen) was an almost pleasant task which you expected to take a good hour on a rainy day… including moments of displaced thought spent staring out of the window (at this points anyone under the age of 30 is probably wondering what the hell I’m on about!).

I still can (and do) waste a good hour or two writing an e-mail.

E-mail is not:

  1. A replacement for conversation. The best tech solutions we have for this are Google Hangouts, Facetime or Skype etc. or even; god forbid, the telephone (psst, don’t tell the kids they can talk into those things). Ping-pong emails are just an ineffective and tedious form of conversation – even worse when they’re to a cc list who mostly couldn’t care less about the topic. It’s morse-code compared to the telephone. If you can, get off your back-side, walk across the office and talk to them!
  2. A replacement for instant-messaging. IM deserves more credit than it typically receives and in many organisations is a fundamental necessity to improve communication. You should be ashamed of yourself if you use e-mail this way! IM is quicker, simpler and crucially doesn’t fill your day with a tonne of “in-box” items you’ll never get round to. And if you’re only experience of IM is Lync.. you need to get out more.
  3.  A replacement for group conversation. Get a room for god-sake! Co-location is the #1 solution for group communication. But if you can’t do that (and don’t give in, fight for this as it will revolutionise your working life) then many tools are available to ease comms over a distance. Many of these; Slack notably in my experience, can be truly engaging for group conversations.
  4. And the cherry on the cake… a replacement for documentation. If you think that sending an email with detailed information counts as “documentation” then you deserve to be taken outside, strapped to the stocks, de-trousered, painted in pigs-blood and have your children forced to throw a variety or spoiled food products at your sorry carcass till their tears run dry. Put it in a wiki or a teamroom or in a document on a file-system – I care not which. But stuffed in the crevice of some email chain where it’s neither obvious or available to those that need it only serves to deter the distribution of knowledge, increase confusion and encourage chaos and entropy to thrive. If your organisation works this way then your organisation is likely living off institutionalised knowledge which may walk out the door tomorrow.

We all hate e-mail. We all love e-mail… No, scratch that. E-mail is rubbish and should be relegated to the same historic status as letter writing. Occasionally nice to receive but quaint and you’d rather not spend your time writing them… It’s time to abandon e-mail!

Instrumentation Revolution

It’s long been good practice to include some sort of tracing in code to help with problems as and when they arise (and they will). And as maligned as simply dumping to stdout is, I would prefer to see this than no trace at all. However, numerous logging frameworks exist and there’s little excuse not to use one.

We have though gotten into the habit of disabling much of this valued output in order to preserve performance. This is understandable of course as heavily used components or loops can chew up an awful lot of time and I/O writing out “Validating record 1 of 64000000. Validating record 2 of 64000000…” and so on. Useful huh?

And we have various levels of output – debug, info, warning, fatal – and the ability to turn the output level up for specific classes or libraries. Cool.

But what we do in production is turn off anything below a warning and when something goes wrong we scramble about; often under change control, to try and get some more data out of the system. And most of the time… you need to add more debug statements to the code to get the data out that you want. Emergency code releases, aren’t they just great fun?

Let’s face it, it’s the 1970’s and we are; on the whole, British Leyland knocking up rust buckets which break down every few hundred miles for no reason at all.

British Leyland Princess

My parents had several of these and they were, without exception, shit.

One of the most significant leaps forward in the automotive industry over the past couple of decades has been the instrumentation of many parts of the car along with complex electronic management systems to monitor and fine tune performance. Now when you open the bonnet (hood) all you see is a large plastic box screaming “Do not open!”.

Screen Shot 2015-07-11 at 11.12.07

And if you look really carefully you might find a naff looking SMART socket when an engineer can plug his computer in to get more data out. The car can tell him which bit is broken and probably talk him through the procedure to fix it…

Meanwhile, back in the IT industry…

It’s high time we applied some of the lessons from the failed ’70s automotive industry to the computer systems we build (and I don’t mean the unionised industries). Instrument your code!

For every piece of code, for every component part of your system, you need to ask, “what should I monitor?”. It should go without saying that you need to log exceptions when they’re raised but you should also consider logging:

  • Time-spent (in milli or microseconds) for potentially slow operations (i.e. anything that goes over a network or has time-complexity risks).
  • Frequency of occurrence – Just log the event and let the monitoring tools do the work to calculate frequency.
  • Key events – Especially entry points into your application (web access logs are a good place to start), startup, shutdown etc. but also which code path requests went down.
  • Data – Recording specific parameters or configuration items etc. You do though need to be very careful here as to what you record to avoid having any personal or sensitive data in log files – no passwords or card numbers etc…
  • Environment utilisation – CPU, memory, disk, network – Necessary to know how badly you’re affecting the environment in which your code is homed.

If you can scale the application horizontally you can probably afford the few microseconds it’s going to take to log the required data safely enough.

Then, once logged you need to process and visualise this data. I would recommend decoupling your application from the monitoring infrastructure as much as possible by logging to local files or; if that’s not possible, stream it out asynchronously to somewhere (a queue, Amazon Kinesis etc.). By decoupling you keep the responsibilities clear and can vary either without necessarily impacting the other.

You then need some agent to monitor the logged output and upload it to some repository, a repository to store the data and some means of analysing this data as and when required.

Screen Shot 2015-07-11 at 11.39.04

Using tools like Kibana, ElasticSearch and LogStash – all from Elastic – you can easily monitor files and visualise the data in pretty much real-time. You can even do so in the development environment (I run ElasticSearch and Kibana on a Raspberry Pi 2 for example) to try to understand the behaviour of your code before you get anywhere near production.

So now when that production problem occurs you can see when the root event occurred and impact it has across numerous components without needing to go through change-control to get more data out whilst the users suffer yet another IT failure. Once you know where to look the problem is 9 times out of 10 fixed. Dashboards can be set up to show at a glance the behaviour of the entire system and you’ll soon find your eye gets used to the patterns and will pick up on changes quite easily if you’re watching the right things.

The final step is to automate the processing of this data, correlate it across components and act accordingly to optimise the solution and eventually self-heal. Feedback control.

Screen Shot 2015-07-11 at 12.18.38

 

With the costs of computing power falling and the costs of an outage rising you can’t afford not to know what’s going on. For now you may have to limit yourself to getting the data into your enterprise monitoring solution – something like Tivoli Monitoring – for operations to support. It’s a start…

Without the data we’re blind. It’s time we started to instrument our systems more thoroughly.

Communication Breakdown

G+ Polls are really very useful for a quick; if not terribly scientific, survey and I recently asked “How do you typically share solution designs?” on the IT Pros community given the variety I see from day to day.

Screen Shot 2015-05-18 at 21.20.17

That presentations are top and models are bottom is sadly unsurprising. I’m in two minds over documents and wikis as effective forms of communication – the former quickly gathers dust though at least provide a snapshot of what was intended at one point in time, the latter decays rapidly into a confusing contradiction of opinions in which the truth is a long lost fairy-tale.

But the really surprising thing for me was the number of votes for whiteboard + photos + email (and/or some online post). We all do it and it’s a excellent way to frame a discussion and share ideas. Does it really end there though? Personally I need to take the output of these sessions and work it into something more cohesive and focused which often yields insights which were not uncovered during the rabbit hole exercise that whiteboarding can become.

Truth be said, I still hunger for a good model and the liberating constraints of UML. Unfortunately it seems I need to both improve my PowerPoint and drawing skills instead.

Waterfall v Agile v Reckless

I was recently asked when I would use an agile instead of waterfall methodology to which I don’t think I gave a very good answer at the time. These things tend to dwell in the mind; in this case at 3am!, and though thoughts at such a hour aren’t necessarily too be trusted, here goes.

“Quality, time, cost – pick any two”, is an often quoted project management notion which I’ll not go in to. A similar triplet can be used to identify which methodology to choose – functionality, time and cost.

The idea goes like this – you can prioritise any two of functionality, time or cost and doing so will tell you which method to use:

Prioritise functionality & cost over time – use waterfall. You know what you want so you can plan it all out up-front, minimise technical-debt, reduce redundancy and keep costs down. It may though be that it takes longer than you really want before you see anything go live. Good for fixed-price work, anything where the scope is well defined and time is less of a concern and some package implementations (package customisation to be avoided (as a general rule!)).

Prioritise cost & time over functionality – use agile. You want something out there soon, at minimal initial cost (think MVP) and can limit the functionality that’s delivered within these time/cost constraints. You can define short delivery cycles, size these accordingly and constrain the functionality that gets delivered within them. Good for T&M projects and anything where the vision/scope or requirement priorities are less clear or subject to change – i.e. most projects.

Prioritise functionality & time over cost – use the reckless approach. This essentially says you don’t give a damn about the money but you want it all and you want it now! If you ever hear a customer claim “money is no object!” run away. Ok, there are a handful of cases; usually government and compliance projects, where this happens but these projects are rare and typically indicative of failings elsewhere in an organisation. In all other cases the customer has frankly lost the plot. Essentially the reckless approach is one where you start trying to deliver all the functionality ASAP without much concern for anything else. You will likely throw out a lot of testing, ignore non-functional requirements, use whatever tools are to hand; whether they’re right for the job or not, and ignore any corporate processes/procedures and standards on the way. Hell, it can work! But most often it leads to unforeseen costs late in projects, excessive rework and; if you do go live with it, a lot of tech-debt and long-term costs which needs to be paid back.

It’s just unfortunate that the reckless approach is used so often when it should be so rare. It’s also unfortunate that those that deliver reckless projects often get promoted (they delivered on-time) without the long term cost being fully understood – a corollary with the quarterly focus issues many large corporations suffer.

So agile should be most common, then waterfall, then reckless. I’d also suggest waterfall should be used sparingly as there’s a tendency for the project to disappear into dark-room for many months before resurfacing and being “not what I wanted” due to poor requirements definition and lack of a good feedback loop. Agile keeps the customer much more engaged on a day-to-day basis which provides that feedback loop and minimises such waste.

It should also be noted that neither agile or waterfall say anything about the work-products needed to deliver them. They don’t say “don’t do requirements definition”, “don’t do design” or “don’t test”, far from it, they just do it in different ways (user-stories v consolidated requirements definition for example). You also need to have an understanding of the long term vision, scope and dependencies within an agile project to avoid excessive tech-debt; or at least be conscious of the implications of short term decisions. From an architectural perspective you still need to identify the building blocks, do the design (logical and physical), establish non-functionals, identify key decisions and dependencies etc. But whilst you may go into all the detail up-front in waterfall you can spread this out in agile with a relatively shallow but broad scope solution outline up-front (essentially identifying the key building blocks and non-functionals) and go into depth as and when needed. To believe you can avoid such work is to make assumptions and to; consciously or otherwise, adopt the reckless approach…

LocalServe

One of the things I have found irritating in the past is the need to install and configure a web-server each time the urge takes me to try something out. I don’t run a local web-server permanently and being of a JFDI disposition the hurdle needed to get a server running is usually enough to stall what motivation I’ve managed to muster. Then I discovered that from Java 7 onwards it’s fairly simple to implement your own web-server in plain Java – no need for an application server.

LocalServe implements two types of handlers:

1. File-handler –  This serves any content (and sub-directories) in the directory in which localserve is run. Any file which is not found returns a 404 and  any request for the root of a folder (path ending in “/”) attempts to return the index.html file in the directory. Note that localserve does not provide any listing of directories.

If all you want to do is serve static content then the above is sufficient and LocalServe can be run using the command below in the directory you want to serve content from. This will run a webserver on port 8765 by default:

java -jar localserve.jar

The port number can also be changed by adding this to the end – e.g.:

java -jar localserve.jar 5678

2. SQL-handler – Often static content isn’t enough and you need to use a database. This handler provides access to a database that can be called from JavaScript (typically via an AJAX request). A configuration file can be specified on the command line when running localserve. This configuration file provides details of a SQLite database, and the SQL statements that are supported. Each SQL statement has a logical name, SQL statement (including bindings), supported methods (POST or GET) and optionally a redirect (where to send the user on success). Calls to paths starting “/sql/” are sent to the SQL handler and the path element after this is used to match against a logical name in the configuration file. If found the SQL statement is executed with any HTTP parameters matching the bind names being bound accordingly. Two special names “PAGE” and “PAGE_SIZE” are defined such that queries which may return many many rows can be restricted to returning only certain pages of a certain size. Results from SQL commands are returned in JSON format.

The configuration file can be specified on the command line when running localserve as below:

java -jar localserve.jar config.json
or to run on a port other than 8765:
java -jar localserve.jar config.json 5678

(note that the position of these two parameters doesn’t matter).

An example configuration file is shown below:

{ "wsm": {
    "connString": "jdbc:sqlite:/path/to/database/file.db",
    "driver": "org.sqlite.JDBC",
    "operations": [
        { "path": "listbob", 
          "statement": "select name, id from bob", 
          "methods": "GET,POST" },
        { "path": "querybob", 
          "statement": "select id, name from bob where id={ID} order by name desc", 
          "methods": "GET,POST" },
        { "path": "insertbob", 
          "statement": "insert into bob (id, name) values ({ID}, {N})", 
          "redirect": "/sql/querybob", 
          "methods": "GET,POST" }
    ]
}
}

The database here contains one very simple table (BOB) as:

CREATE TABLE BOB (ID VARCHAR(20), NAME VARCHAR(100));

The only database used is SQLite and the main JAR file contains all libraries for this to work. I have tried other databases (notably IBM DB2) which worked fine so long as the JAR libraries can be found.

An example response to something like http://localhost:8765/sql/listbob looks like:

{ "dataset": 
    {  "page": 0, 
       "pageSize": 20,  
       "record": [  
           { "name": "James Brown", "id": "1"} ,  
           { "name": "Simple Simon", "id": "2"} ,  
           { "name": "Ducky Duncan", "id": "3"}  
       ] 
    }
} 

The attribute names are derived from the query names and are usually lower case. HOWEVER, you may find that if you explicitly state a column name the attribute may come back in uppercase (e.g. “select id||’ – ‘||name LONG_NAME from bob” will result in an attribute with name “LONG_NAME”).

Once you have the database setup and working then it’s a relatively simple task to use JQuery to submit AJAX requests to the SQL handler to create/read/update/delete/list the database. A hastily knocked up example is below:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> 
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/> 
    <meta charset="utf-8"/>  
  
    <script src="jquery-2.0.3.min.js"></script>
    <script src="purl.js"></script>
    </head>
  <body>
      ...
  <script type="text/javascript">
      
      function query(searchId) {
        $.ajax({ url: "/sql/querybob",
            data: { "PAGE": 0, "PAGE_SIZE": 32000, "ID": searchId },
            success: function( data ) {
              var dataset = (JSON.parse(data)).dataset;
              var arr = new Array();
              for (var i=0;i<dataset.record.length;i++) {
                alert(dataset.record[i].id + " " + dataset.record[i].name);
          }
        });
      }
      query(1);
    </script>
  </body>
</html>

Anyway, it’s been useful when I just want to prototype something and this is definitely not intended for any production use. It’s simply a very simple webserver so that I can quickly get on with prototyping something.

If you’re interested then the source can be found on GitHub where the LocalServe JAR file can be downloaded. The code is what I call “prototype quality” which means it’s been made to work by beating it into some shape with a hammer – it is uncommented and not of production quality.

Java 7 and this JAR are all you should need to run LocalServe. As ever, no assurances, warranties, guarantees etc. are provided and; whether you lose a little data or the world goes into meltdown (and everything in-between), I’ll accept no responsibility for any damages caused…