Richard Aldrich, Booz Allen Hamilton
Too often cyber security metrics reports tend to either be based on whatever data was most readily available, or include low-level data of a very technical nature. In such cases the response of the recipient is likely to be, “So what?” This presentation will address four key issues to consider to better anticipate and address that question.
First, know your audience. You can’t answer the “so what?” if you don’t understand to whom you’re presenting the metrics. Senior leaders need a different, usually much higher level set of metrics than the ones who are actually applying patches to systems. To develop meaningful metrics one must understand what is important to one’s audience. If your data is low level, but you’re delivering to a senior audience you need to roll it up. If your audience has budgetary authority you need to incorporate dollars into the metrics. If your audience has directive authority, you need to show what changes in policy need to be made to respond to the identified problems.
Second, know your data. Lots of data has significant limitations to it. Ensure you understand the limitations so you don’t overplay your hand. Also understand that cyber security’s value is over time, so generally point-in-time metrics are less meaningful than trending metrics.
Third, know the key types of metrics. Metrics can be broadly grouped into four types: time, risk, ROI, and effectiveness. Each may be important to a particular audience or at a particular time. The first answers the question, “Are we going to make it to the goal line in time?” The second answers the question, “Are we adequately responding to the threat, vulnerability and potential impact?” The third answers the question, “If I had more money to spend, where should I put my next dollars?” or “If I have to save money by cutting a program, where will I be hurt the least?” The last type answers the question, “Are our mitigation and remediation efforts successfully producing the desired result?”
Fourth, keep the message clear. Once you understand all of the above, make sure the presentation is clear. Don’t muddle it with distracting colors, overly complex charts, techno-babble or overly complex math. Make the data speak for itself with clear, intuitive visualizations that plainly make the point.