Shit happens, data is stolen (or leaked) and your account details, passwords and bank-account are available online to any criminal who wants it (or at least is prepared to buy it).
But don't panic, the data was encrypted so you're ok. Sit back, relax in front of the fire and have another mince pie (or six).
We see this time and again in the press. Claims that the data was encrypted... they did everything they could... blah blah blah. Humm, I think we need more detail.
It's common practice across many large organisations today to encrypt data using full-disk encryption with tools such as BitLocker or Becrypt. This is good practice and should be encouraged but is only the first line of defence as this only really helps when the disk is spun down and the machine powered off. If the machine is running (or even sleeping) then all you need is the users password and you're in. And who today really wants to shutdown a laptop when you head home... and perhaps stop for a pint on the way?
In the data-center the practice is less common because the risk of disks being taken out of servers and smuggled out of the building is lower. On top of this the disks are almost always spinning so any user/administrator who has access to the server can get access to the data.
So, moving up a level, we can use database encryption tools such as Transparent Data Encryption to encrypt the database files on the server. Great! So now normal OS users can't access the data and need to go through the data access channel to get it. Problem is, lots of people have access to databases including DBAs who probably shouldn't be able to see the raw data itself but who generally can. On top of this, service accounts are often used for application components to connect and if these credentials are available to some wayward employee... your data could be pissing out an open window.
To protect against these attack vectors we need to use application level encryption. This isn't transparent and developers need to build in data encryption and decryption routines as close to exposed interfaces as practical. Now having access to the OS, files or database doesn't do enough to expose the data. An attacker also needs to get hold of the encryption keys which should be held on separate infrastructure such as an HSM. All of which costs time and money.
Nothings perfect and there's still the risk that a wayward developer siphons off data as it passes through the system or that some users have too broad access rights and can access data, keys and code. These can be mitigated against through secure development practices, change management and good access management... to a degree.
Furthermore, encrypting everything impacts functionality - searching encrypted data becomes practically impossible - or may not be as secure as you'd expect - a little statistical analysis on some fields can expose the underlying data without actually needing to decrypt it due to a lack of sufficient variance in the raw data. Some risks need to be accepted.
We can then start to talk about the sort of ciphers used, how they are used and whether these and the keys are sufficiently strong and protected.
So when we hear in the press that leaked data was encrypted, we need to know more about how it was encrypted before deciding whether we need to change our passwords before tucking into the next mince pie.
Merry Christmas!
2016/12/24
2016/12/23
2016/11/26
Scalable = Horizontal
There's two ways of scaling; vertical and horizontal, but there's only one which is really scalable.
Vertical scaling essentially means bigger nodes. If you've got 8GB RAM, go to 16GB. If you've 2 cores, go to 4.. and so on.
Horizontal scaling means adding more nodes. One node to two nodes, to three and so on.
As a rule, horizontal scaling is good. Theoretically there's no limit to the number of nodes you can have.
As a rule, vertical scaling is bad. You quickly run into constraints over the number of cores or RAM you can support. And for many of todays problems this just doesn't work. Solutions need to be both scalable at the internet scale and available as in 24x7. Relying on large single nodes in such situations is not ideal. (and those supercomputers with 250,000+ processors are really horizontal solutions as well).
The problem is, horizontal scaling isn't trivial. The culprits here are data and networking (plumbing really). State and caches need to be distributed and available to all. Databases need copies across nodes and need to be synchronised. Sharding usually becomes necessary (or you just end up with many very large nodes). And so on... Best bet is to avoid state as much as possible. But once you've cracked it you can build much larger solutions more efficiently (commodity hardware, virtualisation, containers etc.) and flex more rapidly than in the vertical world.
I could go on about how historically the big players love the vertical-scaling thing (think Oracle and IBM trying to sell you those huge servers and SQL databases solutions with the $$$ price-tags)... The world found NoSQL solutions which take a fundamentally different approach by accepting that consistency in many cases really isn't as important as we once thought - and many of these are open-source...
Whatever, there's only one way to scale... Horizontal!
Vertical scaling essentially means bigger nodes. If you've got 8GB RAM, go to 16GB. If you've 2 cores, go to 4.. and so on.
Horizontal scaling means adding more nodes. One node to two nodes, to three and so on.
As a rule, horizontal scaling is good. Theoretically there's no limit to the number of nodes you can have.
As a rule, vertical scaling is bad. You quickly run into constraints over the number of cores or RAM you can support. And for many of todays problems this just doesn't work. Solutions need to be both scalable at the internet scale and available as in 24x7. Relying on large single nodes in such situations is not ideal. (and those supercomputers with 250,000+ processors are really horizontal solutions as well).
The problem is, horizontal scaling isn't trivial. The culprits here are data and networking (plumbing really). State and caches need to be distributed and available to all. Databases need copies across nodes and need to be synchronised. Sharding usually becomes necessary (or you just end up with many very large nodes). And so on... Best bet is to avoid state as much as possible. But once you've cracked it you can build much larger solutions more efficiently (commodity hardware, virtualisation, containers etc.) and flex more rapidly than in the vertical world.
I could go on about how historically the big players love the vertical-scaling thing (think Oracle and IBM trying to sell you those huge servers and SQL databases solutions with the $$$ price-tags)... The world found NoSQL solutions which take a fundamentally different approach by accepting that consistency in many cases really isn't as important as we once thought - and many of these are open-source...
Whatever, there's only one way to scale... Horizontal!
2016/11/12
Instrumentation as a 1st Class Citizen
I wrote previously that we are moving into an era of instrumentation and things are indeed improving. Just not as fast as I'd like. There's a lot of good stuff out there to support instrumentation and monitoring including the likes of the ELK (ElasticSearch, Logstash, Kibana) and TIG (Telegraf, InfluxDB, Grafana) stacks as well as their more commercial offerings such as TICK (Telegraf, InfluxDB, Chronograf, Kapacitor), Splunk, DataDog, AppDynamics and others. The problem is, few still really treat instrumentation as a real concern... until it's too late.
Your customers love this stuff! Really, they do! There's nothing quite as sexy as an interactive graph showing how your application is performing as the load increases - transactions, visitors, response-times, server utilisation, queue-depths etc. When things are going well it gives everyone a warm fuzzy feeling that all is right with the universe. When things are going wrong it helps to quickly focus you in on where the problem is.
However, this stuff needs to be built into everything we do and not be an afterthought when the pressures on to ship-it and you can't afford the time and effort to retrofit it. By then it's too late.
As architects we need to put in the infrastructure and services needed to support instrumentation, monitoring and alerting. At a minimum this means putting in place standards for logging, data-retention polices, a data collection solution, repository for the data and some tooling to allow us to search that data and visualize what's going on. Better still we can add alerting when thresholds breach and use richer analytics to allow us to scale up and down to meet demand.
As developers we need to be considering what metrics we want to capture from the components we build as we're working on them. Am I interested in how long it's taking for this function call? Do I want to know how many messages a service is handling? How many threads are being spawned? What exceptions are being thrown? Where from? What the queue depths are?.. etc. Almost certainly... YES! And this means putting in place strategies for logging these things. Perhaps you can find the data in existing log files.. Perhaps you need to use better tooling for detailed monitoring... Perhaps you need to write some code yourself to track how things are going...
Doing this from the start will enable you to get a much better feel for how things are working before you launch - including a reasonable view of performance and infrastructure demands which will allow you to focus your efforts better later when you do get into sizing and performance testing. It'll mean you're not scrambling around look for log files to help you root-cause issues as your latest release goes into meltdown. And it'll mean your customer won't be chewing your ear off asking you what's going on every five minutes - they'll be able to see it for themselves...
So please, get it in front of your customer, your product owner, your sponsor, your architects, your developers, your testers and make instrumentation a 1st class citizen in the backlog.
Your customers love this stuff! Really, they do! There's nothing quite as sexy as an interactive graph showing how your application is performing as the load increases - transactions, visitors, response-times, server utilisation, queue-depths etc. When things are going well it gives everyone a warm fuzzy feeling that all is right with the universe. When things are going wrong it helps to quickly focus you in on where the problem is.
However, this stuff needs to be built into everything we do and not be an afterthought when the pressures on to ship-it and you can't afford the time and effort to retrofit it. By then it's too late.
As architects we need to put in the infrastructure and services needed to support instrumentation, monitoring and alerting. At a minimum this means putting in place standards for logging, data-retention polices, a data collection solution, repository for the data and some tooling to allow us to search that data and visualize what's going on. Better still we can add alerting when thresholds breach and use richer analytics to allow us to scale up and down to meet demand.
As developers we need to be considering what metrics we want to capture from the components we build as we're working on them. Am I interested in how long it's taking for this function call? Do I want to know how many messages a service is handling? How many threads are being spawned? What exceptions are being thrown? Where from? What the queue depths are?.. etc. Almost certainly... YES! And this means putting in place strategies for logging these things. Perhaps you can find the data in existing log files.. Perhaps you need to use better tooling for detailed monitoring... Perhaps you need to write some code yourself to track how things are going...
Doing this from the start will enable you to get a much better feel for how things are working before you launch - including a reasonable view of performance and infrastructure demands which will allow you to focus your efforts better later when you do get into sizing and performance testing. It'll mean you're not scrambling around look for log files to help you root-cause issues as your latest release goes into meltdown. And it'll mean your customer won't be chewing your ear off asking you what's going on every five minutes - they'll be able to see it for themselves...
So please, get it in front of your customer, your product owner, your sponsor, your architects, your developers, your testers and make instrumentation a 1st class citizen in the backlog.
2016/09/23
2016/09/10
MzE1NjE5NTI0ODYyfjIxMTAxNjI4NzMxNn5VMkZzZEdWa1gxOHorQVFYL2NQRFkwUVgyRm82anNJRGJHOW0vV1ZjdVEvVUR0cjZRZ0t3Y21WYWdCalo0dEZW
The title of this post is encrypted.
This page is also encrypted (via TLS (aka the new name for SSL)).
Anyone sniffing traffic on the wire must first decrypt the TLS traffic and then decrypt the content to work out what the message says.
But why bother with two layers of encryption?
Ok, so forgive the fact that this page is publicly accessible and TLS is decrypted before your eyes. It's possibly a poor example and in any case I'd like to talk about the server side of this traffic.
In many organisations, TLS is considered sufficient to provide security for data in-transit. The problem is TLS typically terminates on a load-balancer or on a web-server and is forwarded from there to another downstream server. Once this initial decryption takes place data often flows over the internal network of organisations in plain text. Many organisations consider this to be fine practice since the internal network is locked down with firewalls and intrusion detection devices etc. Some organisations even think it's good practice so that they can monitor internal traffic more easily.
However, there is obvious concern over insider-attacks with system-admins or disgruntled employees being in a good position to skim off the data easily (and clean-up any trace after themselves). Additionally requests are often logged (think access logs and other server logs) and these can record some of the data submitted. Such data-exhaust is often available in volume to internal employees.
It's possible to re-wrap traffic between each node to avoid network sniffing but this doesn't help data-exhaust and the constant un-wrap-re-wrap becomes increasingly expensive if not in CPU and IO then in effort to manage all the necessary certificates. Still, if you're concerned then do this or terminate TLS on the application-server.
But we can add another layer of encryption to programmatically protect sensitive data we're sending over the wire in addition to TLS. Application components will need to decrypt this for use and when this happens the data will be in plain text in memory but right now that's about as good as we can get.
The same applies for data at-rest - in fact this is arguably far worse. You can't rely on full database encryption or file-system encryption. Once the machine is up and running anyone with access to the database or server can easily have full access to the raw data in all its glory. These sort of practices only really protect against devices being lifted out of your data-centre - in which case you've got bigger problems...
The safest thing here is to encrypt the attributes you're concerned about before you store them and decrypt on retrieval. This sort of practice causes all sorts of problems in terms of searching but then should you really be searching passwords or credit card details? PII details; names, addresses etc, are the main issue here and careful thought about what really needs to be searched for; and some constructive data-modelling, may be needed to make this workable. Trivial it is not and compromises abound.
All this encryption creates headaches around certificate and key management but such is life and this is just another issue we need deal with. Be paranoid!
p.s. If you really want to know what the title says you can try the password over here.
This page is also encrypted (via TLS (aka the new name for SSL)).
Anyone sniffing traffic on the wire must first decrypt the TLS traffic and then decrypt the content to work out what the message says.
But why bother with two layers of encryption?
Ok, so forgive the fact that this page is publicly accessible and TLS is decrypted before your eyes. It's possibly a poor example and in any case I'd like to talk about the server side of this traffic.
In many organisations, TLS is considered sufficient to provide security for data in-transit. The problem is TLS typically terminates on a load-balancer or on a web-server and is forwarded from there to another downstream server. Once this initial decryption takes place data often flows over the internal network of organisations in plain text. Many organisations consider this to be fine practice since the internal network is locked down with firewalls and intrusion detection devices etc. Some organisations even think it's good practice so that they can monitor internal traffic more easily.
However, there is obvious concern over insider-attacks with system-admins or disgruntled employees being in a good position to skim off the data easily (and clean-up any trace after themselves). Additionally requests are often logged (think access logs and other server logs) and these can record some of the data submitted. Such data-exhaust is often available in volume to internal employees.
It's possible to re-wrap traffic between each node to avoid network sniffing but this doesn't help data-exhaust and the constant un-wrap-re-wrap becomes increasingly expensive if not in CPU and IO then in effort to manage all the necessary certificates. Still, if you're concerned then do this or terminate TLS on the application-server.
But we can add another layer of encryption to programmatically protect sensitive data we're sending over the wire in addition to TLS. Application components will need to decrypt this for use and when this happens the data will be in plain text in memory but right now that's about as good as we can get.
The same applies for data at-rest - in fact this is arguably far worse. You can't rely on full database encryption or file-system encryption. Once the machine is up and running anyone with access to the database or server can easily have full access to the raw data in all its glory. These sort of practices only really protect against devices being lifted out of your data-centre - in which case you've got bigger problems...
The safest thing here is to encrypt the attributes you're concerned about before you store them and decrypt on retrieval. This sort of practice causes all sorts of problems in terms of searching but then should you really be searching passwords or credit card details? PII details; names, addresses etc, are the main issue here and careful thought about what really needs to be searched for; and some constructive data-modelling, may be needed to make this workable. Trivial it is not and compromises abound.
All this encryption creates headaches around certificate and key management but such is life and this is just another issue we need deal with. Be paranoid!
p.s. If you really want to know what the title says you can try the password over here.
2016/09/02
Channel 4 in France
Slight obsession some would say, but I enjoy F1... not that much that I'm prepared to pay Sky whatever extortionate fee they're come up with today though so I tend to watch the highlights only on C4. Nice coverage btw guys - shame to lose you next year.
Anyway, I have a VPN (OpenVPN) running off a Synology DiskStation to allow me to tunnel through home when I'm abroad. Works a treat... normally. Channel 4 does not.
Initially I thought it was DNS leakage picking up that name resolution is from french servers. You can see this by visiting www.dnsleaktest.com and running the "standard test". Even though I'm reported as being in the UK, all my DNS servers are in France... Humm, I smell a fish...
Am I in the UK or France?
To work around this I setup a proxy server on the DiskStation and the same test now reports UK DNS servers as everything goes through the proxy.
Definitely looks like I'm in the UK... But still no luck on C4...
Finally, I set the timezone I was in to UK rather than France and this seemed to do the trick. Note that you need to change the timezone on the laptop, not the time itself or you'll have all sorts of trouble connecting securely to websites including C4.
In the end, the proxy doesn't seem necessary so they don't appear to be picking up on DNS resolution yet though it's the sort of thing that they could look at adding (that, and device geolocation using HTML5 geo API though for this there are numerous plugins for browsers to report fake locations).
Incidentally, BBC iPlayer works fine and does so without fiddling with timezone.
The net wasn't really designed to expose your physical location and IP to location lookups such as MaxMind are more of a workaround than truly identifying your location. Using TOR as a more elaborate tunnel makes you appear to be all over the place as your IP address jumps around and corporate proxies; especially for large organisations, can make you appear to be in all sorts of weird places. Makes you wonder.. All these attempts to limit your access based on an IP address to prop up digital rights management just doesn't work. It's all too easy to work-around.
p.s. Turns out that whilst France doesn't have free-to-air F1 coverage, most places have some form of satellite TV via CanalSat or TNT which includes the German RTL channel. It'll do nothing to improve my French but at least I get to watch the race on the big screen...
Anyway, I have a VPN (OpenVPN) running off a Synology DiskStation to allow me to tunnel through home when I'm abroad. Works a treat... normally. Channel 4 does not.
Initially I thought it was DNS leakage picking up that name resolution is from french servers. You can see this by visiting www.dnsleaktest.com and running the "standard test". Even though I'm reported as being in the UK, all my DNS servers are in France... Humm, I smell a fish...
Am I in the UK or France?
To work around this I setup a proxy server on the DiskStation and the same test now reports UK DNS servers as everything goes through the proxy.
Definitely looks like I'm in the UK... But still no luck on C4...
Finally, I set the timezone I was in to UK rather than France and this seemed to do the trick. Note that you need to change the timezone on the laptop, not the time itself or you'll have all sorts of trouble connecting securely to websites including C4.
In the end, the proxy doesn't seem necessary so they don't appear to be picking up on DNS resolution yet though it's the sort of thing that they could look at adding (that, and device geolocation using HTML5 geo API though for this there are numerous plugins for browsers to report fake locations).
Incidentally, BBC iPlayer works fine and does so without fiddling with timezone.
The net wasn't really designed to expose your physical location and IP to location lookups such as MaxMind are more of a workaround than truly identifying your location. Using TOR as a more elaborate tunnel makes you appear to be all over the place as your IP address jumps around and corporate proxies; especially for large organisations, can make you appear to be in all sorts of weird places. Makes you wonder.. All these attempts to limit your access based on an IP address to prop up digital rights management just doesn't work. It's all too easy to work-around.
p.s. Turns out that whilst France doesn't have free-to-air F1 coverage, most places have some form of satellite TV via CanalSat or TNT which includes the German RTL channel. It'll do nothing to improve my French but at least I get to watch the race on the big screen...
Subscribe to:
Posts (Atom)
-
** Warning: Functional rant! ** I've spent a significant part of the past few years swearing at Lotus Notes. So much in fact that this p...
-
If you really wanted to know you'd have found it but for what it's worth, this site now runs on Redhats OpenShift platform . For a w...
-
PO: We need a bridge over the river right here? Me: Why? PO: Because the customer needs to get to the other side? Me: Why can't they use...
