Online security awareness training is now the most popular form of security awareness training in the world. As we noted here, that’s good news when it comes to measuring the effectiveness of security awareness programs and training.
Offline, things aren’t so easy to track. However, online it’s possible to see who is doing what where and when. It’s little surprise, then, that measuring the effectiveness of online security awareness program has been chiselled onto the agendas of CISOs for some time.
And yet, measuring the effectiveness of security awareness training remains difficult. In this post, we’re going to go through how companies typically track the effectiveness of their security awareness program and where they may be going wrong, before suggesting an alternative approach.
How companies measure the effectiveness of security awareness training
As far back as 2012, Educause Center for Analysis and Research were looking into measuring the effectiveness of security awareness training. In 2013, the Center published the results of a survey that asked 95 universities how they measured the success of security awareness programs.
Reported methods of measuring the effectiveness of security programs included:
- Monitoring the number and type of security incidents they experienced (62%)
- Employee feedback (45%)
- Monitoring behavioral change (34%)
- Surveys (24%)
- Training attendance (24%)
- Alignment with university strategic goals (9%)
- Targeted assessments (9%)
- Pre- and post-testing to monitor performance (8%)
Given the survey took place in 2013, the reported efforts are certainly commendable – especially as it appears at least some research participants were measuring the impact of their security awareness campaigns in more than one way.
The right way measure the effectiveness of security awareness training
As revealing as the above survey is, it’s probably not revealing enough. Measuring the effectiveness of security training requires more than monitoring a selected set of metrics.
Step 1: Strategy
A coherent and well-planned cyber security strategy is a prerequisite for measuring the effectiveness of cyber security training. Without a strategy, any delivered security awareness program runs the risk of becoming a fleeting effort.
So before thinking about measurement, think about what you want to achieve and how you want to achieve it.
Let’s say you want to reduce the likelihood of people leaving computer screens unlocked when away from their desk by a 50% over the next six months. How might you do that? Posters? Prizes? Sticking watchful eyes to monitors to influence people’s subconscious?
Step 2: Questions
Similarly, before thinking about metrics, think about – in qualitative terms – what it’s really important to find out. We’ve touched on this before – when warning against becoming fixated on phishing susceptibility rates.
At CybSafe, we advocate keeping a close eye on three things: security awareness, security behaviors and security culture – the ABC of information security.
For us, that means CISOs need to focus on:
- What people know and understand about how to stay safe online (information security awareness)
- How people really behave when presented with attacks (behavior)
- What people think about – and how much they care about – cyber security (culture)
- And how confident people are in their abilities (culture)
So, back to locking screens. To improve the incidence of screen-locking, what is it that’s really important to find out?
On the awareness front, security leaders may wish to find out if people know about risks and best practices. When it comes to behaviors, you’ll presumably want to find out how likely it is for an unattended screen to be left unlocked. When it comes to culture, you’ll probably want to find out why people are behaving in the way they are.
Step 3: Metrics
Measuring awareness
The above questions help highlight useful metrics to monitor. Take what people know and understand about how to stay safe, for example.
People’s knowledge and comprehension of security can be monitored through online security awareness training performance. So long as you have access to a cyber awareness platform with analytical capabilities – such as CybSafe – it becomes easy to see how much people know about security best practices. Online quiz results can reveal whether or not people know of the risks of leaving unattended monitors unlocked. And that’s at an organizational level, a departmental level and an individual level.
Measuring behavior
Measuring behavior, meanwhile, is usually best achieved through simulated attacks such as phishing email campaigns etc.
Simulated attacks test the security behaviors of the people in your organisation. Monitoring how people respond to simulations gives you a metric of security behavior.
Simulated attacks may not be appropriate when attempting to measure the chances of an unattended computer screen being unlocked at any given time… but a coherent strategy makes finding alternatives easy. Spot checks, for example, should give you the information you need. You may consider combining checks with an incentive system, perhaps using yellow cards for unattended screens that have been left unlocked, and chocolate bars for screens that have been locked!
Indications of behavior change can also be measured in other ways, for example measuring triggers and motivations – two key components widely acknowledged as necessary for behavior change to occur.
Measuring culture
Measuring culture and human risk is perhaps the hardest of the three to do – but it’s not impossible. Anonymous surveys, for example, can give you an idea of why people take risky actions like leaving their screens unlocked. And they can be revealing.
Perhaps it’s an unwritten rule that it’s OK to leave screens unlocked when heading over to the printer. In theory, it’s low risk. But should the printer unexpectedly need more paper, risks increase.
Qualitative insights from surveys can help you change behaviours and reduce risks – but it’s important to note that finding an overall quantitative cultural metric is key. It’s only through quantitative metrics that improvements can really be pursued.
Through surveys, employee feedback and the intelligent analysis of sentiment and attitude, the CybSafe platform measures culture in both qualitative and quantitative terms.
Step 4: Timing
When measurements are taken is also particularly important.
To measure the effectiveness of security awareness training, measurements should ideally be taken at day zero – before security campaigns begin. Then, as campaigns unfold, measurements need to be recorded at regular intervals – ideally monthly, or even more frequently.
While this might be problematic when attempting to measure the impact of security awareness training manually, intelligent platforms like CybSafe naturally log awareness, behavior and culture metrics automatically, allowing CISOs to conduct analyses as necessary.
As well as fuelling continuous improvement, regular measurement helps identify incidents and where people may need further support, and administering further support can ultimately prevent data breaches.
What about the number of security incidents experienced?
When Educause asked people how they measured the effectiveness of their security awareness programmes in 2013, monitoring the “number and type of incidents” an organisation experienced proved to be the most popular approach. Clearly, it’s an important metric to track. But does it really give a clear idea of the effectiveness of security awareness training?
Unfortunately, as the metric fails to take into account the number or severity of attempted attacks, the statistic can be misleading. If the number and severity of attacks increases while training is delivered, the number of incidents experienced may well increase regardless of how effective security training is.
Measuring fuels machine learning and AI
At their heart, machine learning and AI are both fuelled by measurements.
Machines measure, then trial something, then measure again.
The difference between the two measurements is what allows machines to learn. It’s what makes AI possible – and it’s how our intelligent security awareness platform is able to evolve with the needs of your organisation.
CybSafe combines applied machine learning with cognitive computing technology to learn which actions to take to decrease your organisation’s human cyber risk. It’s still measurement – just automated. Without measuring the right things, the innovation wouldn’t be possible.
Why measuring the effectiveness of security awareness training is important
Historically, measuring the effectiveness of security awareness training has proven problematic. So problematic, in fact, that some of today’s top CISOs are unable to measure the effectiveness of their security awareness training as well as they might like.
Fortunately, the rise of intelligent cyber awareness platforms is making measuring simple. Those who wish to measure the effectiveness of security awareness campaigns are advised to give them proper consideration.
After all, monitoring the effectiveness of security awareness training is essential when it comes to truly reducing an organisation’s human cyber risk.
