2016/03/31
Hack a Mousetrap
My 10 year old son and I have been playing with Arduino recently... specifically to build a mouse trap alarm. And clearly we're not the only ones thinking about this... Some guys over at MS have tried to hack a mousetrap using every piece of technology they can get their hands on (I'm sure I saw the kitchen sink in there somewhere). Nice :)
2016/03/30
Security, Impact, Truth and Environments
I’ve been known to berate others for abusing environments - despite my personal habits - but I think its time for me to curtail my anger and reconsider exactly what distinguishes one environment from another and why.
We’re used to managing a plethora of environments - production, standby, prod-support, pre-production, performance test, UAT, system-test, development-integration, dev etc. - each of which has its own unique characteristics and purpose and each with a not insignificant cost.
With all those environments we can very easily have 5 or 6 times the infrastructure required to run production sitting mostly idle - and yet still needing to be maintained and patched and consuming kilowatts of power. All this for what can seem like no good reason bar to satisfy some decades old procedural dictate handed down by those upon high.
Unsurprisingly many organisations try to combine responsibilities into a smaller set of environments to save on $’s at the cost of increased risk. And recent trends in dev-ops, cloud and automation are helping to reduce the day-to-day need for all these environments even further. After all, if we can spin up a new server, install the codebase and introduce it into service in a matter of minutes then why not kill it just as quickly? If we can use cheaper t2.micro instances in dev and use m4.large only in prod then why shouldn’t we do so?
So we can shrink the number and size of environments so now we only have 2 or 3 times production and with auto-scaling this baseline capacity can actually be pretty low.
If we can get there...
… and the problem today is that whilst the technology exists, the legacy architectures, standards, procedures and practices adopted over many years by organisations simply don't allow these tools and techniques to be adopted at anywhere near the pace at which they are developing in the wild. That application written 10 years ago just doesn’t fit with the new cloud strategy the company is trying to develop. In short, revolution is fast (and bloody) and evolution is slow.
Our procedures and standards need to evolve at the same rate as technology and this just isn’t happening.
So I’ve been considering what all these environments are for and why they exist and think it comes down to three concerns; security, impact and truth.
- Security - What’s the security level of the data held? More often than not the production environment is the only one authorised to contain production data. That means it contains sensitive data or PII, has lots of access-control and auditing, firewalls everywhere, tripwires etc. There’s no way every developer is going to get access to this environment. Access is on a needs-to-know basis only... and we don’t need (and shouldn’t want) to know.
- Impact - Whats the impact to the business if the environment dies or runs slow? If dev goes down, no-one cares. Hell, if pre-prod goes down no-one bar prod-support really care.
- Truth - How true to version X does the environment have to be? Production clearly needs to be the correct release of the codebase across the board (MVT aside). If we have the wrong code with the wrong database then it matters. In the development environment?.. if a script fails then frankly it’s not the end of the world, and besides dev is usually going to be version X+n, unstable and flaky in any case.
So in terms of governance it’s those things that keep management awake at night. They want to know who’s got access to what, what they can do, on what boxes, with what assets and what the risk is to data exposure. When we want to push out the next release they want to know the impact if it screws up, that we've got a back-out plan for when it does and that we've tested it - the release, the install plan and the back-out. In short, they’re going to be a complete pain in the backside. For good reason.
But can we rethink our environments around these concerns and does this help? If we can demonstrate to management that we’ve met these needs then why shouldn’t they let us reduce, remove and recycle environments at will?
Production and stand-by will have to be secure and the truth. But the impact if stand-by goes down isn’t the same. There’s a risk if prod falls over but that’s not the same thing. So allowing data-analysts access to stand-by to run all sorts of wild and crazy queries may not be an issue unless prod falls flat on its face - a risk some will be willing to take to make more use of the tin and avoid environment spread. Better still, if the data in question isn’t sensitive or is just internal-use-only then why not mirror a copy into dev environments to provide a more realistic test data-set for developers?
And if the data is sensitive? Anonymise it and use that; or a decent sample of it, in dev and test environments. Doing so will improve the quality of code by increasing the likelihood developers will detect patterns and edge-cases sooner in the development cycle.
In terms of impact, If the impact to the business of an application outage is low then why insist on the full range of environments when frankly one or two will do? Many internal applications are only used 9 to 5 and have an RTO and RPO of in excess of 24 hrs. The business need to clearly understand what they’re agreeing to but ultimately it’s their $’s we’re spending and once they realise the cost they may be all too willing to take the risk. Having five different environments for every application for the sake of consistency alone isn’t justifiable.
And not all truths are equal. Some components don’t need the same rigour as others and may have lower impact to the business if they’re degraded to some degree. Allowing some components; especially expensive ones, to have fewer environments may complicate topologies and reduce the general comprehensiveness of the system but if we can justify it then so be it. We do though need to make sure this is very clearly understood by all involved else chaos can ensue - especially if some instances span environments (here be dragons).
Finally, if engineering teams paid more attention during development to performance and operability and could demonstrate this then the need for dedicated performance/pre-prod environments may also be reduced. We don't need an environment matching production to understand the performance profile of the application under load. We just need to consider the systems characteristics and test cases with a willingness (i.e. an acceptance of risk) to extrapolate. A truthful representation of production is usually not necessary.
Risk is everything here and if we think about how the applications concerns stack up against the security risk, the impact risk to the business and risk of things not being the truth, the whole truth and nothing but… then perhaps we can be smarter about how we structure our environments to help reduce the costs involved irrespective of adopting revolutionary technology.
We’re used to managing a plethora of environments - production, standby, prod-support, pre-production, performance test, UAT, system-test, development-integration, dev etc. - each of which has its own unique characteristics and purpose and each with a not insignificant cost.
With all those environments we can very easily have 5 or 6 times the infrastructure required to run production sitting mostly idle - and yet still needing to be maintained and patched and consuming kilowatts of power. All this for what can seem like no good reason bar to satisfy some decades old procedural dictate handed down by those upon high.
Unsurprisingly many organisations try to combine responsibilities into a smaller set of environments to save on $’s at the cost of increased risk. And recent trends in dev-ops, cloud and automation are helping to reduce the day-to-day need for all these environments even further. After all, if we can spin up a new server, install the codebase and introduce it into service in a matter of minutes then why not kill it just as quickly? If we can use cheaper t2.micro instances in dev and use m4.large only in prod then why shouldn’t we do so?
So we can shrink the number and size of environments so now we only have 2 or 3 times production and with auto-scaling this baseline capacity can actually be pretty low.
If we can get there...
… and the problem today is that whilst the technology exists, the legacy architectures, standards, procedures and practices adopted over many years by organisations simply don't allow these tools and techniques to be adopted at anywhere near the pace at which they are developing in the wild. That application written 10 years ago just doesn’t fit with the new cloud strategy the company is trying to develop. In short, revolution is fast (and bloody) and evolution is slow.
Our procedures and standards need to evolve at the same rate as technology and this just isn’t happening.
So I’ve been considering what all these environments are for and why they exist and think it comes down to three concerns; security, impact and truth.
- Security - What’s the security level of the data held? More often than not the production environment is the only one authorised to contain production data. That means it contains sensitive data or PII, has lots of access-control and auditing, firewalls everywhere, tripwires etc. There’s no way every developer is going to get access to this environment. Access is on a needs-to-know basis only... and we don’t need (and shouldn’t want) to know.
- Impact - Whats the impact to the business if the environment dies or runs slow? If dev goes down, no-one cares. Hell, if pre-prod goes down no-one bar prod-support really care.
- Truth - How true to version X does the environment have to be? Production clearly needs to be the correct release of the codebase across the board (MVT aside). If we have the wrong code with the wrong database then it matters. In the development environment?.. if a script fails then frankly it’s not the end of the world, and besides dev is usually going to be version X+n, unstable and flaky in any case.
So in terms of governance it’s those things that keep management awake at night. They want to know who’s got access to what, what they can do, on what boxes, with what assets and what the risk is to data exposure. When we want to push out the next release they want to know the impact if it screws up, that we've got a back-out plan for when it does and that we've tested it - the release, the install plan and the back-out. In short, they’re going to be a complete pain in the backside. For good reason.
But can we rethink our environments around these concerns and does this help? If we can demonstrate to management that we’ve met these needs then why shouldn’t they let us reduce, remove and recycle environments at will?
Production and stand-by will have to be secure and the truth. But the impact if stand-by goes down isn’t the same. There’s a risk if prod falls over but that’s not the same thing. So allowing data-analysts access to stand-by to run all sorts of wild and crazy queries may not be an issue unless prod falls flat on its face - a risk some will be willing to take to make more use of the tin and avoid environment spread. Better still, if the data in question isn’t sensitive or is just internal-use-only then why not mirror a copy into dev environments to provide a more realistic test data-set for developers?
And if the data is sensitive? Anonymise it and use that; or a decent sample of it, in dev and test environments. Doing so will improve the quality of code by increasing the likelihood developers will detect patterns and edge-cases sooner in the development cycle.
In terms of impact, If the impact to the business of an application outage is low then why insist on the full range of environments when frankly one or two will do? Many internal applications are only used 9 to 5 and have an RTO and RPO of in excess of 24 hrs. The business need to clearly understand what they’re agreeing to but ultimately it’s their $’s we’re spending and once they realise the cost they may be all too willing to take the risk. Having five different environments for every application for the sake of consistency alone isn’t justifiable.
And not all truths are equal. Some components don’t need the same rigour as others and may have lower impact to the business if they’re degraded to some degree. Allowing some components; especially expensive ones, to have fewer environments may complicate topologies and reduce the general comprehensiveness of the system but if we can justify it then so be it. We do though need to make sure this is very clearly understood by all involved else chaos can ensue - especially if some instances span environments (here be dragons).
Finally, if engineering teams paid more attention during development to performance and operability and could demonstrate this then the need for dedicated performance/pre-prod environments may also be reduced. We don't need an environment matching production to understand the performance profile of the application under load. We just need to consider the systems characteristics and test cases with a willingness (i.e. an acceptance of risk) to extrapolate. A truthful representation of production is usually not necessary.
Risk is everything here and if we think about how the applications concerns stack up against the security risk, the impact risk to the business and risk of things not being the truth, the whole truth and nothing but… then perhaps we can be smarter about how we structure our environments to help reduce the costs involved irrespective of adopting revolutionary technology.
2016/02/28
Al Gore: The case for optimism on climate change
Off topic perhaps but entertaining and inspiring...
[ted id=2441 x=640 y=360]
[ted id=2441 x=640 y=360]
2016/02/18
Blood-e-mail
We all hate e-mail. We all love e-mail...
E-mail is like writing a letter. There was a time when sitting down to write a letter (with pen) was an almost pleasant task which you expected to take a good hour on a rainy day... including moments of displaced thought spent staring out of the window (at this points anyone under the age of 30 is probably wondering what the hell I'm on about!).
I still can (and do) waste a good hour or two writing an e-mail.
E-mail is not:
We all hate e-mail. We all love e-mail... No, scratch that. E-mail is rubbish and should be relegated to the same historic status as letter writing. Occasionally nice to receive but quaint and you'd rather not spend your time writing them... It's time to abandon e-mail!
E-mail is like writing a letter. There was a time when sitting down to write a letter (with pen) was an almost pleasant task which you expected to take a good hour on a rainy day... including moments of displaced thought spent staring out of the window (at this points anyone under the age of 30 is probably wondering what the hell I'm on about!).
I still can (and do) waste a good hour or two writing an e-mail.
E-mail is not:
- A replacement for conversation. The best tech solutions we have for this are Google Hangouts, Facetime or Skype etc. or even; god forbid, the telephone (psst, don't tell the kids they can talk into those things). Ping-pong emails are just an ineffective and tedious form of conversation - even worse when they're to a cc list who mostly couldn't care less about the topic. It's morse-code compared to the telephone. If you can, get off your back-side, walk across the office and talk to them!
- A replacement for instant-messaging. IM deserves more credit than it typically receives and in many organisations is a fundamental necessity to improve communication. You should be ashamed of yourself if you use e-mail this way! IM is quicker, simpler and crucially doesn't fill your day with a tonne of "in-box" items you'll never get round to. And if you're only experience of IM is Lync.. you need to get out more.
- A replacement for group conversation. Get a room for god-sake! Co-location is the #1 solution for group communication. But if you can't do that (and don't give in, fight for this as it will revolutionise your working life) then many tools are available to ease comms over a distance. Many of these; Slack notably in my experience, can be truly engaging for group conversations.
- And the cherry on the cake... a replacement for documentation. If you think that sending an email with detailed information counts as "documentation" then you deserve to be taken outside, strapped to the stocks, de-trousered, painted in pigs-blood and have your children forced to throw a variety or spoiled food products at your sorry carcass till their tears run dry. Put it in a wiki or a teamroom or in a document on a file-system - I care not which. But stuffed in the crevice of some email chain where it's neither obvious or available to those that need it only serves to deter the distribution of knowledge, increase confusion and encourage chaos and entropy to thrive. If your organisation works this way then your organisation is likely living off institutionalised knowledge which may walk out the door tomorrow.
We all hate e-mail. We all love e-mail... No, scratch that. E-mail is rubbish and should be relegated to the same historic status as letter writing. Occasionally nice to receive but quaint and you'd rather not spend your time writing them... It's time to abandon e-mail!
2016/02/13
Traceability
We can have a small server...
...a big server (aka vertical scaling)...
.. a cluster of servers (aka horizontal scaling)...
.. or even a compute grid (horizontal scaling on steroids).
For resiliency we can have active-passive...
... or active-active...
... or replication in a cluster or grid...
...each with their own connectivity, load-balancing and routing concerns.
From a logical perspective we could have a simple client-server setup...
...a two tier architecture...
...an n-tier architecture...
...a service oriented (micro- or ESB) architecture...
...and so on.
And in each environment we can have different physical topologies depending on the environmental needs with logical nodes mapped to each environments servers...
With our functional components deployed on our logical infrastructure using a myriad of other deployment topologies..
... or ...
... and on and on and on...
And this functional perspective can be implemented using dozens of design patterns and a plethora of integration patterns.
With each component implemented using whichever products and packages we choose to be responsible for supporting one or more requirements and capabilities...
So the infrastructure we rely on, the products we select, the components we build or buy; the patterns we adopt and use... all exist for nothing but the underlying requirement.
We should therefore be able to trace from requirement through the design all the way to the tin on the floor.
And if we can do that we can answer lots of interesting questions such as "what happens if I turn this box off?", "what's impacted if I change this requirement?" or even "which requirements are driving costs?". Which in turn can help improve supportability, maintainability and availability and reduce costs. You may even find your product sponsor questioning if they really need this or that feature...
...a big server (aka vertical scaling)...
.. a cluster of servers (aka horizontal scaling)...
.. or even a compute grid (horizontal scaling on steroids).
For resiliency we can have active-passive...
... or active-active...
... or replication in a cluster or grid...
...each with their own connectivity, load-balancing and routing concerns.
From a logical perspective we could have a simple client-server setup...
...a two tier architecture...
...an n-tier architecture...
...a service oriented (micro- or ESB) architecture...
...and so on.
And in each environment we can have different physical topologies depending on the environmental needs with logical nodes mapped to each environments servers...
With our functional components deployed on our logical infrastructure using a myriad of other deployment topologies..
... or ...
... and on and on and on...
And this functional perspective can be implemented using dozens of design patterns and a plethora of integration patterns.
With each component implemented using whichever products and packages we choose to be responsible for supporting one or more requirements and capabilities...
So the infrastructure we rely on, the products we select, the components we build or buy; the patterns we adopt and use... all exist for nothing but the underlying requirement.
We should therefore be able to trace from requirement through the design all the way to the tin on the floor.
And if we can do that we can answer lots of interesting questions such as "what happens if I turn this box off?", "what's impacted if I change this requirement?" or even "which requirements are driving costs?". Which in turn can help improve supportability, maintainability and availability and reduce costs. You may even find your product sponsor questioning if they really need this or that feature...
2016/02/01
JBOSS Openshift Queue Deployment
Deploying to Openshift is, in theory, as simple as
To deal with this Openshift gives you the ability to define a set of action-hooks that get called during the various stages of the applications lifecycle.
These are just shell scripts and are pretty easy to define - just create a file in the
In the case of queue deployment we need a post_start script which uses the JBOSS CLI to create queues.
The script
Make sure the script is executable via
The use of various environment variables ensures the script will work regardless of the configuration the image fires-up with and the "echo" commands ensures some sort of output is dumped to stdout during push for confirmation.
This script references a cli script (cli/create-queues.cli) looking like this:
And hey presto! On deployment you'll see a couple of messages output showing:
And if you tail the logs (
Finally the JBOSS console will show your queues in all their glory...
git push. But if you've made any changes to the app-server environment you'll find it gets blitzed on deployment (this is actually a good thing since it'll force you to get into the habits of automating deployments).To deal with this Openshift gives you the ability to define a set of action-hooks that get called during the various stages of the applications lifecycle.
These are just shell scripts and are pretty easy to define - just create a file in the
.openshift/action_hooks directory in the project root matching the name of the hook you want. In the case of queue deployment we need a post_start script which uses the JBOSS CLI to create queues.
The script
.openshift/action_hooks/post_start looks like this:
echo "Starting JBOSS Queue Configuration..."
${OPENSHIFT_JBOSSAS_DIR}/bin/tools/jboss-cli.sh --connect controller=${OPENSHIFT_JBOSSAS_IP}:${OPENSHIFT_JBOSSAS_MANAGEMENT_NATIVE_PORT} --file=${OPENSHIFT_REPO_DIR}/cli/create-queues.cli
echo "JBOSS configuration complete!"
Make sure the script is executable via
chmod +x .openshift/action_hooks/post_start.The use of various environment variables ensures the script will work regardless of the configuration the image fires-up with and the "echo" commands ensures some sort of output is dumped to stdout during push for confirmation.
This script references a cli script (cli/create-queues.cli) looking like this:
jms-queue add --queue-address=queueA --entries=queue/QueueA
jms-queue add --queue-address=queueB --entries=queue/QueueB
And hey presto! On deployment you'll see a couple of messages output showing:
remote: Starting JBOSS Queue Configuration...
remote: JBOSS configuration complete!
And if you tail the logs (
rhc tail -a ) you should see confirmation of the deployment as below:
2016/02/01 16:57:57,641 INFO [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-8) trying to deploy queue jms.queue.queueA
2016/02/01 16:57:57,644 INFO [org.jboss.as.messaging] (MSC service thread 1-8) JBAS011601: Bound messaging object to jndi name java:/queue/QueueA
2016/02/01 16:57:57,738 INFO [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-8) trying to deploy queue jms.queue.queueB
2016/02/01 16:57:57,740 INFO [org.jboss.as.messaging] (MSC service thread 1-8) JBAS011601: Bound messaging object to jndi name java:/queue/QueueB
Finally the JBOSS console will show your queues in all their glory...
2016/01/30
Bearer v MAC
I've been struggling recently to get my head around OAuth2 access tokens - bearer and MAC tokens specifically...
Bearer tokens are essentially just static tokens valid for some predefined period before they need to be refreshed. They can essentially be passed around willy-nilly and will be accepted by a resource server so long as they can be validated. If a 3rd party manages to hijack one then they can use it to perform whatever the token is authorised to do just by submitting it in the correct manner. Consequently these tokens need to be looked after carefully. Shuffled over encrypted channels and protected by the client. They're arguably even less secure than session cookies since there's no "HTTP Only" option on an access token so preventing malicious access to tokens from dodgy code on clients is something the developer needs to manage. And given the number of clients around and quality of code out there we can pretty much assume a good chunk will be piss poor at this.
So Bearer tokens. Not so great.
MAC tokens aren't just static strings. A client secret and nonce is combined with some request data to essentially sign tokens. So if you hijack a request in flight you can't replay the token - it's valid only for the original request. This is good but really only protects against snooping over the wire which SSL/TLS does a pretty good job of managing without all the additional complexity. Beyond this a MAC token seems to make very little difference. The client needs to know the secret in the same way it would need a Bearer token. If someone manages to snatch this we're done for regardless and the false sense of security MAC tokens give isn't worth a damn.
The client application is often the weak point in OAuth since it's often an untrusted device - mobile phones and web-browsers (single page applications) etc. If the "client" is a downstream server (and this poor terminology by the way has caused way too much confusion and argument) then we've a reasonable chance to secure the server and data but ultimately we're still going to have a client ID and secret stuffed in memory just like we would have with a Bearer token. Ok, so we're adding some hoops to jump through but really it's no more secure.
So if we don't have transport level encryption (SSL/TLS) then MAC tokens offer some reasonable value over Bearer tokens. But if we do have transport encryption then MACs just add complexity and a false sense of security. Which I'd argue is a bad thing since increased complexity is likely to lead to increased defects... which in security is a very bad thing!
Besides, neither option allows me to identify the user or ensure the client calling the service is authorised to do so... Just that they appear to be in possession of a token saying the user has granted them authority (which may have been hijacked as per above).
p.s. One of my many failed new years resolutions was to post a new article every week of the year. This being the first four weeks in isn't a good start...
Bearer tokens are essentially just static tokens valid for some predefined period before they need to be refreshed. They can essentially be passed around willy-nilly and will be accepted by a resource server so long as they can be validated. If a 3rd party manages to hijack one then they can use it to perform whatever the token is authorised to do just by submitting it in the correct manner. Consequently these tokens need to be looked after carefully. Shuffled over encrypted channels and protected by the client. They're arguably even less secure than session cookies since there's no "HTTP Only" option on an access token so preventing malicious access to tokens from dodgy code on clients is something the developer needs to manage. And given the number of clients around and quality of code out there we can pretty much assume a good chunk will be piss poor at this.
So Bearer tokens. Not so great.
MAC tokens aren't just static strings. A client secret and nonce is combined with some request data to essentially sign tokens. So if you hijack a request in flight you can't replay the token - it's valid only for the original request. This is good but really only protects against snooping over the wire which SSL/TLS does a pretty good job of managing without all the additional complexity. Beyond this a MAC token seems to make very little difference. The client needs to know the secret in the same way it would need a Bearer token. If someone manages to snatch this we're done for regardless and the false sense of security MAC tokens give isn't worth a damn.
The client application is often the weak point in OAuth since it's often an untrusted device - mobile phones and web-browsers (single page applications) etc. If the "client" is a downstream server (and this poor terminology by the way has caused way too much confusion and argument) then we've a reasonable chance to secure the server and data but ultimately we're still going to have a client ID and secret stuffed in memory just like we would have with a Bearer token. Ok, so we're adding some hoops to jump through but really it's no more secure.
So if we don't have transport level encryption (SSL/TLS) then MAC tokens offer some reasonable value over Bearer tokens. But if we do have transport encryption then MACs just add complexity and a false sense of security. Which I'd argue is a bad thing since increased complexity is likely to lead to increased defects... which in security is a very bad thing!
Besides, neither option allows me to identify the user or ensure the client calling the service is authorised to do so... Just that they appear to be in possession of a token saying the user has granted them authority (which may have been hijacked as per above).
p.s. One of my many failed new years resolutions was to post a new article every week of the year. This being the first four weeks in isn't a good start...
Subscribe to:
Posts (Atom)
-
** Warning: Functional rant! ** I've spent a significant part of the past few years swearing at Lotus Notes. So much in fact that this p...
-
If you really wanted to know you'd have found it but for what it's worth, this site now runs on Redhats OpenShift platform . For a w...
-
PO: We need a bridge over the river right here? Me: Why? PO: Because the customer needs to get to the other side? Me: Why can't they use...
