“You can’t manage what you can’t measure.”
Peter Drucker
This classic quote from management guru Peter Drucker equally applies to the measurement challenge we face in information security.
If we can’t (or don’t) measure, how can we identify if we’ve been successful with our security initiatives? For example, is a lack of security incidents an indication of success? If so, how can we demonstrate it? Likewise, without evidence of what is working (and where), how can we justify often significant security expenditure – or indeed, make effective security decisions?
In this article, we’ll take an introductory look at the use of metrics to measure cyber security effectiveness (sometimes also referred to as “security maturity”) with dashboards and benchmarking. We will also reference existing frameworks and models and provide links for you to explore in more depth.
But first, let’s start with a definition that some might find helpful:
Measurement vs Metrics – what’s the difference?
The US National Institute for Standards and Technology (NIST) defines measurement as something quantifiable and observable. Whereas a metric is defined as something supported by measurements that intend to facilitate decision making, improve performance and accountability.
Many experts agree that metrics are probably the most useful data points at our disposal. Those readers with experience in cyber security may like to dive deeper into the NIST Framework (2017) (although still in its draft form), which expands on the topic by identifying metrics (in section 4) to enable the measurement of cyber security effectiveness.
Similarly, the Center for Internet Security (CIS) Top 20 Critical Security Controls (CSC) library includes a measurement download that enables users to identify key information to help them track progress.
Here at Ascentor, we are fans of the NIST Framework and CIS Top 20 CSC – we have had excellent results working with them and would typically consider them for new projects where appropriate. For some of you, a lighter touch may be helpful, so we have also included a selection of alternative options below.
Report what you should, not what you can
In a similar vein to Drucker’s thoughts on the need for measurement, Phil Cracknell (CISO of Homeserve), put the case for metrics while speaking at the 2017 Cyber Security Summit and Expo. He said “businesses are waking up to the fact that they need metrics and risk indicators that our board, audit committees and non-executive directors are able to understand”.
He suggested that businesses should adopt a “report what you should, not what you can” approach and that metrics “can demonstrate effectiveness, measure exposure and agility, test organisation culture, pinpoint responsibilities and highlight levels of investment”.
Measurement: Two approaches and what to include
There are two common approaches to measurement – a dashboard of metrics, and benchmarking (where organisations compare their relative performance to others).
1) Dashboards
Many organisations have adopted a “dashboard” approach as a means of communicating with senior management. Meaningful metrics need to be quantified in terms of time, money and/or risk level, so suggestions for what a cyber security dashboard might include are:
- Financial losses due to security breaches
- Damage to reputation and trust
- The estimated cost of lost customers (see our article on the cost of the TalkTalk hack )
- Down-time due to business disruption
- Time taken to deactivate former employees access
- Frequency of attempts to access servers or applications by unauthorised users
- Types of vulnerabilities discovered
- Time taken to mitigate vulnerabilities
- Patch management – the regularity of patches and system updates and any errors detected (see our article on when patching goes wrong )
- Pass/fail results for employee information security training initiatives
For those following the NIST approach, the 2014 NIST Framework suggests that a dashboard could be aligned to their NIST Framework Core Functions. These consist of five concurrent and continuous functions: Identify, Protect, Detect, Respond and Recover.
Once your data has been gathered, what is it telling you? Look for data that might not be what you were expecting or if there are any inconsistencies. How does it compare to earlier dashboard measurements of the same data? And, what action will you take? Information informs action – or at least it should.
A2015 Tripwire magazine article asked a number of security industry experts for their tips on what they recommend a powerful dashboard must have . It is a worthwhile read and goes well beyond what can be covered in this blog article.
2) Benchmarking
Benchmarks are great for helping organisations compare their relative performance. The problem with using benchmarking to measure security effectiveness is the difficulty of identifying suitable metrics from comparable organisations, and even then, they might not have theirs right. Consistency of benchmarking metrics is a real challenge for information security.
Therefore, for information security, benchmarking by direct comparison with other organisations is probably futile in most cases. A viable alternative, which we support, is to use a form of maturity model as a benchmark.
For cyber security, we like the Cybersecurity Capability Maturity Model (C2M2) .C2M2 was developed by the US Department of Energy as a mechanism to improve maturity of cyber security for the US energy infrastructure – but it may be used by any organisation. It defines ten domains for analysis and four maturity indicator levels (MIL 0, 1, 2, 3). Self-assessment against the model is supported by an accompanying toolkit, which also generates a useful report that includes graphical results and a gap analysis.
Summary
With the plethora of frameworks, methodologies and guidelines, it is hard to know where to turn when it comes to metrics and measurement. While each approach can be comprehensive, no single one gives you everything you need. This brings on the added headache of a jigsaw approach where you may have to choose more than one approach and then workout how to fit things together.
For smaller businesses, a pragmatic set of cyber security measurements that dovetail with your corporate dashboard is a good place to start. You can get more sophisticated as your business grows and your cyber capability and knowledge mature.
Our work with major corporations and public organisations (where a more robust approach is needed), has led us to review and assess the best-known approaches and tools as well as some of the more obscure ones. From that experience, we favour the complementary NIST Framework, CIS Top 20 CSC and C2M2 trio.
