Hello,
In my previous post we defined the importance of “security competence” as an important goal in the “Security awareness” campaign. Now that we have established this, let us define the metrics that will help you to measure the success of your efforts.

Metrics to measure success: For security awareness and security competence
Note: The metrics are based on the HIMIS (Human Impact Management for Information Security) methodology, the free copy of which can be downloaded here.
It is important to split the metrics into specific categories, viz. security awareness and security competence. This is to ensure that you do not get carried away with high “awareness”, whereas the “competence” could be low.

Security Awareness Metrics
Coverage
“Coverage” indicates the target workforce (employees, contractors, partners and other interested parties) that must be covered under the information security awareness program.
Format and visibility
“Format” indicates the different types of information security awareness content. “Visibility” indicates the channel through which the content is delivered. Channels are selected in order to put information security awareness content where maximum amount of visibility (eyeballs) can be gained.
Verbal: Trainer led classroom sessions, personal interactions
Paper: Posters, cards, quizzes or surveys
Electronic: Videos, Emails with messages, Animated games, Screensavers Quizzes or surveys
Frequency
“Frequency” indicates the gap between any two deliveries of information security awareness content. Frequency is critical because it influences “retention”.
Quality of content
These metrics are captured via qualitative analysis methods (survey and feedback) and the following measurement criterion can be used.
1. Impact visualization: Probably the most important factor. An example of impact visualization is visually depicting the damage (stealing a laptop, stealing valuable documents) that an intruder can cause by tailgating.
2. Business relevance:
The information security awareness program, specifically the content must capture the business requirements of information security.
Clarity and ease of understanding:
Style must not be sacrificed for substance. Emphasis must be given to conveying the message in a simple and clear manner first. Building style around the message should be done without diluting the message or making the content complicated.
Consideration of cultural factors:
It will be useful to consider cultural factors such as,
a.Language or terms used (usage of colloquial terms may be more effective),
b.Colour and design,
c.Characters represented
Retention measurement
“Retention measurement” indicates a method to measure how much the workforce has “understood and remembers” after the information security awareness delivery. Strategies that can be used are,
a.Personal interviews
b.Surveys
c.Quizzes
Security Competence Metrics
The following strategies can be used to measure security competence.
Observations: For example, observe for tailgating, observe how many meeting rooms still have sensitive information on the board after the meeting
Log review: For example, browsing and email patters can be observed through log reviews of corresponding systems
Data mining : For example, Mine through internet search engines to see how much sensitive information about the company is available online
Incident report review: For example, review of incident reports may show how many laptops were lost and a further investigation may reveal the cause as carelessness (poor behaviour) or not (may be the user was physically attacked).
Being practical, being creative, being reasonable
Too much metrics is also not a good idea. Use the ones that you will give a good degree of confidence so that you can trust your findings. With a list of success metrics and the strategy to measure them you are now ready to move forward.
Catch you with my next post.
Anup Narayanan