• Home
  • Resources
    • Find Resources by Topic Tags
    • Cybersecurity Policy Chart
    • CSIAC Reports
    • Webinars
    • Podcasts
    • Cybersecurity Digest
    • Standards & Reference Docs
    • Journals
    • Certifications
    • Acronym DB
    • Cybersecurity Related Websites
  • Services
    • Free Technical Inquiry
    • Core Analysis Task (CAT) Program
    • Subject Matter Expert (SME) Network
    • Training
    • Contact Us
  • Community
    • Upcoming Events
    • Cybersecurity
    • Modeling & Simulation
    • Knowledge Management
    • Software Engineering
  • About
    • About the CSIAC
    • The CSIAC Team
    • Subject Matter Expert (SME) Support
    • DTIC’s IAC Program
    • DTIC’s R&E Gateway
    • DTIC STI Program
    • FAQs
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
Login / Register

CSIAC

Cyber Security and Information Systems Information Analysis Center

  • Resources
    • Find Resources by Topic Tags
    • Cybersecurity Policy Chart
    • CSIAC Reports
    • Webinars
    • Podcasts
    • Cybersecurity Digest
    • Standards & Reference Docs
    • Journals
    • Certifications
    • Acronym DB
    • Cybersecurity Websites
  • Services
    • Free Technical Inquiry
    • Core Analysis Task (CAT) Program
    • Subject Matter Expert (SME) Network
    • Training
    • Contact
  • Community
    • Upcoming Events
    • Cybersecurity
    • Modeling & Simulation
    • Knowledge Management
    • Software Engineering
  • About
    • About the CSIAC
    • The CSIAC Team
    • Subject Matter Expert (SME) Support
    • DTIC’s IAC Program
    • DTIC’s R&E Gateway
    • DTIC STI Program
    • FAQs
  • Cybersecurity
  • Modeling & Simulation
  • Knowledge Management
  • Software Engineering
/ Journal Issues / Knowledge Management / Metrinome – Continuous Monitoring and Security Validation of Distributed Systems

Metrinome – Continuous Monitoring and Security Validation of Distributed Systems

Published in Journal of Cyber Security and Information Systems
Volume: 2 Number: 1 - Knowledge Management

Authors: Michael Atighetchi, Dr. Vatche Ishakian, Dr. Joseph Loyal, Dr. Partha Pal, Asher Sinclair and Robert Grant
Posted: 02/10/2016 | Leave a Comment

Distributed enterprise systems consist of a collection of interlinked services and components that exchange information to collectively implement functionality in support of (sometimes mission critical) workflows. Systematic experimental testing and continuous runtime monitoring of these large scale distributed systems, including event interpretation and aggregation, are key to ensuring that the system’s implementation functions as expected and that its security is not compromised.

To illustrate the need, consider an example Information Management System (IMS) that enables sharing of sensitive information between information publishing and consuming clients. Problems associated with configuration management can easily lead to situations in which the IMS allows unauthenticated clients to participate in information exchanges or allows unauthorized information to be disseminated to consumers. Furthermore, the loose coupling between subscribers and the IMS can lead to situations in which the IMS is unavailable and consumers believe that no new information is being published, causing significant misunderstandings across information sharing relationships. Finally, remnant vulnerabilities in the IMS can cause failures to happen at any time and cause significant damage to mission execution if not dealt with in a real-time manner. Unavailability of information sharing directly reduces situational awareness, loss of integrity can give adversaries control over mission execution, and loss of confidentiality can be detrimental to the reputation of actors and/or mission goals in general.

Monitoring and validation of IMS and client operations can aid in detection, diagnosis, and correction of situations like this. This is particularly important since 92% of reported vulnerabilities are located at the applications layer [1]. Despite the importance of experimental validation and continuous monitoring, and the increased support to adopt security assessment as part of the software development life cycle, current approaches suffer from a number of shortcomings that limit their application in continuous monitoring situations and their use in the validation of assurance claims.

First, current test practices favor unit tests over integrated tests for establishing correct functionality. Unit testing, e.g., performed via Junit [2], checks program functionality piece-by-piece but provides little to assess the overall information assurance claims of a system under test. Various tools exist for actively assessing the security of distributed systems, e.g., Nessus [3] and HP Fortify [4] to name a few, but their functionality is achieved by running specialized unit tests for security properties against either the code or the running system. In contrast, integrated end-to-end testing tools, such as YourKit [5] or Grinder [6], focus on performance and scalability. These tools enable operators to find bottlenecks or provision computing resources, but lack metrics associated with assessing security and correct functionality.

Second, integrated and end-to-end testing and experimentation is often postponed until software artifacts have matured significantly. This is because integrated testing and experimentation can be time consuming and effort intensive and the perception is that the cost of manually performing experiments early on frequently outweighs the benefits.

Finally, common testing and metrics frameworks add additional dependencies to existing systems, in the form of additional libraries that need to be loaded into the system under test and lines of code being added in support of instrumentation. This not only increases software complexity but more importantly can cause version dependency issues. It can also have unintended side effects on certification and accreditation as the software now has additional code that must be certified but that is not part of the core functionality, i.e., it is part of the continuous monitoring.

This article describes Metrinome, a metrics framework written in Java that is specifically designed to provide a platform for structured continuous security assessments throughout the software lifecycle. The novelty of Metrinome lies in its loose coupling with the system under test and its integration of end-to-end testing with continuous application-level remote monitoring. Specifically, Metrinome provides (1) runtime computation of a wide range of metrics from log messages generated by distributed components during system execution, (2) execution of assertions over the metrics to determine correct functionality while the system is operating, and (3) improved situational awareness via dashboard views and generation of experimentation reports. The outputs of Metrinome-based assessments can be used as input to Certification and Assessment (C&A) processes to precisely doc-ument the assertions that were previously checked to hold true in the system. Metrinome is available free of charge to government entities through AFRL.

II. Related Work

A. SNMP Dashboards

A number of management platforms exist that use the Simple Network Management Protocol (SNMP) for monitoring devices and nodes. Network Management Information System (NMIS) [7] operates at the networking level and enables monitoring, fault detection, and configuration management of large complex networks. Its main metrics deal with device reachability, availability, and performance. HP OpenView, IBM Tivoli, and Nagios provide similar functionality. Unlike these platforms, Metrinome specializes on monitoring at the application level and execution of fine-grained assertions.

B. Distributed Testing

Software Testing Automation Framework (STAF) [8] is an open source multi-platform, multi-language framework that enables a set of functionalities including logging, monitoring and process invocation for the main purpose of testing. STAF operates in a peer environment; a network of STAF-enabled machines is built by running STAF agents across a set of networked hosts. In contrast to STAF, the goal of Metrinome is more focused and hence no agents are required to be installed. Avoiding agents not only leads to reduced maintenance costs but also significantly reduces the attack surface across networked systems under test. Due to their complimentary nature, we have used Metrinome in conjunction with STAF for continuous testing and integration.

C. Application-level Metrics Frameworks

Several application-level metrics frameworks exist to monitor and measure the performance of applications. For example, Javasimon [9] exposes an API which can be placed into the code and allows inline computation of count metrics and measurement of durations. Metrics [10] is similar to Javasimon but allows data to be streamed to other reporting systems, e.g., Ganglia [11] and Graphite [12].

An important distinction between Metrinome and the above mentioned frameworks is Metrinome’s use of log messages to provide the same monitoring functionality. This makes Metrinome loosely coupled with the system being monitored and makes it applicable to any application that generates log messages, e.g., using Log4j or Logback.

D. Reporting/Graphing Backends

Ganglia, Graphite, and Splunk [13] are examples of highly popular platforms that offer the ability to search, analyze, and visualize data in real-time. Typically these frameworks consist of a processing backend that collects and stores the data. They also use statistical methods that provide new insight and intelligence about the data. Metrinome provides functionalities that intersect with the above mentioned applications, such as dashboard views and experimentation reports. One difference is that Metrinome focuses less on scalability but rather on ensuring correct execution of a system under test through the validation of assertions.

E. SIEM Platforms

Security Information and Event Management platforms (SIEMs), e.g. ArcSight [14], adopt many of the technologies described above, such as SNMP dashboards and reporting backends, to provide users with the ability to query, and analyze security threats generated by both hardware and software applications. Unlike Metrinome, these platforms require the deployment of agents on networked hosts to collect and report events.

III. Design and Architecture

Metrinome is designed to achieve specific objectives in portability and ease of use.

  • Portability – Metrinome can monitor a system inde-pendent of the implementation of the system.
  • Minimal coding overhead – Rather than adding new instrumentation libraries to monitored processes (caus-ing versioning conflicts and Java classpath pollution), Metrinome interfaces with existing logging and auditing frameworks, e.g., Logback [15].
  • Ease of use – To be of immediate use to experimenters and administrators, it should be easy to specify metrics and assertions that must hold over the metrics in a systematic way. In addition, results of metric computation need to be readily accessible by humans or other programs through a well-defined Application Programming Interface (API) and Graphical User Interface (GUI).

2014-01-30_1541

Figure 1: Metrinome High-Level Architecture

Figure 1 provides an overview of Metrinome high level architecture. Metrinome works with a set of monitored processes that have the ability to send log messages over TCP connections to the ingest API provided by the Metrics Server. Ingestion is performed via simple logging configuration changes on the monitored processes, e.g., by specifying the use of a SocketAppender in Logback to send certain log messages remotely to the Metrics Server over TCP connections in addition to or instead of sending those messages to the console or a local file.

Due to the fact that log messages issued by different processes may be similar, particularly if the processes are executing the same code base on different physical machines, the Metrics Server requires a descriptive unique process name associated with a specific logging instance as part of the log message. This requirement has already been built into most of the logging and auditing framework, enabling filtering of messages based on process names within Metrinome. The processing performed by Metrinome on received messages is defined using a XML-based Domain Specific Language (DSL), describing concepts such as sections, metrics, functions, and assertions. The Metrinome DSL allows administrators to specify processing logic in one file that can be dynamically loaded into the Metrics Server.

Finally, to ease access to information, Metrinome offers two interfaces: (1) a GUI, implemented in HTML and accessible through common web Browsers using HTTP(s), and (2) a RESTful [16] secure Web Services API for use by external programs.

Pages: Page 1 Page 2 Page 3

Previous Article:
« Search…Backwards
Next Article:
An Overview of the Schedule Compliance Risk... »

References

[1] Eoin Keary, Integration into the SDLC(Software Development Life Cycle), Retrieved Nov 06 2013, https://www.owasp.org/images/f/f6/Integration_into_the_SDLC.ppt

[2] JUnit Homepage, Retrieved Sep 06 2013, https://github.com/junit-team/junit/wiki/Getting-started

[3] Nessus Vulnerability Scanner, Retrieved Sep 06 2013, http://www.tenable.com/products/nessus

[4] HP Fortify My App, Retrieved Sep 06 2013,

[5] YourKit Profiler, Retrieved Sep 06 2013, http://www.yourkit.com/

[6] Grinder, Retrieved Sep 06 2013, http://grinder.sourceforge.net/

[7] Network Management Information System, Retrieved June 10 2013, http://www.sins.com.au/nmis/sample/

[8] Software Testing Automation Framework, Retrieved June 10 2013, http://staf.sourceforge.net/

[9] Java Simon - Simple Monitoring API , Retrieved June 10 2013, http://code.google.com/p/javasimon/

[10] Metrics, http://metrics.codahale.com, Retrieved June 10, 2013

[11] Ganglia Monitoring System, Retrieved June 10 2013, http://ganglia.sourceforge.net/

[12] Graphite - Scalable Realtime Graphing, Retrieved June 10 2013, http://graphite.wikidot.com/

[13] Splunk, http://www.splunk.com/ Retrieved June 10 2013.

[14] ArcSight, http://en.wikipedia.org/wiki/ArcSight

[15] Cody Burleson, “How to setup SLF4J and LOGBack in a web app – fast”, Apr 10 2013, https://wiki.base22.com/display/btg/How+to+setup+SLF4J+and+LOGBack+in+a+web+app+-+fast

[16] Fielding, Roy Thomas, “Architectural styles and the design of network-based software architectures”, Diss. University of California, 2000.

[17] Jenkins: An extendable open source continuous integration server, http://jenkins-ci.org/ Retrieved July 1 2013.

[18] Kelley Dempsey, Nirali hawla, Arnold Johnson, Ronald John-ston, Alicia Clay Jones, Angela Orebaugh, Matthew Scholl, and Kevin Stine, “Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations”, Retrieved June 25 2013 http://csrc.nist.gov/publications/nistpubs/800-137/SP800-137-Final.pdf

[19] Jason Miller, “Agencies struggle with continuous monitoring mandate”, Retrieved June 25 2013 http://www.federalnewsradio.com/513/2681377/Agencies-struggle-with-continuous-monitoring-mandate

[20] “R: Box Plot Statistics”, R manual, Retrieved June 3 2013, http://stat.ethz.ch/R-manual/R-devel/library/grDevices/html/boxplot.stats.html

[21] “Secure and QoS-Managed Information Exchange between Enterprise and Constrained Environments”, currently in submission to appear in Proceedings of ISORC 2014.

Authors

Michael Atighetchi
Michael Atighetchi
Mr. Michael Atighetchi is a Senior Scientist in the distributed computing group at BBN and technical lead on several DARPA- and USAF-sponsored research projects. Mr. Atighetchi has a Master of Science degree in Computer Science from UMASS Amherst and a Master of Science in Informatics from the University of Stuttgart/Germany. Mr. Atighetchi is a Senior Member of the IEEE, member of ACM, and has authored over 60 publications in peer-reviewed conferences and journals on topics including adaptive security, Red Team assessments, identity management, and Cross Domain Solutions. Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138
Dr. Vatche Ishakian
Dr. Vatche Ishakian
Dr. Vatche Ishakian is a Scientist in the distributed computing group at BBN working on USAF-sponsored research projects. Dr. Ishakian’s has a PhD degree in Computer Science from Boston University and is a member of the IEEE and ACM. His experience spans a broad set of disciplines across networking and distributed systems, including application-level scheduling and management, network economics, data placement, and network architecture. Dr. Ishakian has authored over 15 publications in peer-reviewed conferences and journals. Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138
Dr. Joseph Loyal
Dr. Joseph Loyal
Dr. Joseph Loyall is a principal scientist at Raytheon BBN Technologies. He has been the principal investigator for Defense Advanced Research Projects Agency and AFRL research and development projects in the areas of information management, distributed middleware, adaptive applications, and quality of service. He is the author of over 100 published papers. He is a Distinguished Member of the ACM and a Senior Member of the IEEE and of the AIAA. Dr. Loyall has a doctorate in computer science from the University of Illinois. Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138
Dr. Partha Pal
Dr. Partha Pal
Dr. Partha Pal is a Principal Scientist at BBN Technologies. His research interest is in the areas of adaptive cyber-defense, resiliency and survivability. As the Principle Investigator in a number of past and ongoing projects sponsored by various agencies, he has been leading the development, demonstration and evaluation of innovative cyber-defense mechanisms, strategies and survivability architectures, and using them to build survivable distributed information systems. He is a senior member of the IEEE and a member of the ACM. He has over 80 publications in peer reviewed conferences and journals, and holds a PhD in Computer Science from Rutgers University. Raytheon BBN Technologies, 10 Moulton St, Cambridge, MA 02138
Asher Sinclair
Asher Sinclair
Mr. Asher Sinclair is a Senior Program Manager at AFRL’s Information Directorate working in the Resilient Synchronized Systems Branch (RISB) at the Rome Research Site. His interests include research and development in enterprise systems management, service-oriented architectures, and Cyber security. He has contributed to more than 18 technical papers and conference proceeding publications. He holds a bachelor’s degree in Computer Information Systems from the State University of New York and a master’s degree in Information Management from Syracuse University. Air Force Research Laboratory, 525 Brooks Road, Rome, NY 13441, USA
Robert Grant
Robert Grant
Mr. Robert Grant works for the Air Force Research Laboratory Information Directorate in Rome New York. He has a B.A. in English from the University at Buffalo, a B.A. in Computer Science from Oswego State, and is currently working on his Masters in Computer Science at Syracuse University. Air Force Research Laboratory, 525 Brooks Road, Rome, NY 13441, USA

Reader Interactions

Leave a Comment Cancel

You must be logged in to post a comment.

sidebar

Blog Sidebar

Featured Content

Data Privacy Day - Jan 28

Data Privacy Day is January 28th

You can help create a global community that respects privacy, safeguards data, and enables trust. You can help teach others about privacy at home, at work, and in your community.

Learn How

Featured Subject Matter Expert (SME): Daksha Bhasker

A dynamic CSIAC SME, Senior Principal Cybersecurity Architect, Daksha Bhasker has 20 years of experience in the telecommunications services provider industry. She has worked in systems security design and architecture in production environments of carriers, often leading multidisciplinary teams for cybersecurity integration, from conception to delivery of complex technical solutions. As a CSIAC SME, Daksha's contributions include several published CSIAC Journal articles and a webinar presentation on the sophiscated architectures that phone carriers use to stop robocalls.

View SME's Contributed Content

The DoD Cybersecurity Policy Chart

The DoD Cybersecurity Policy Chart

This chart captures the tremendous breadth of applicable policies, some of which many cybersecurity professionals may not even be aware, in a helpful organizational scheme.

View the Policy Chart

CSIAC Report - Smart Cities, Smart Bases and Secure Cloud Architecture for Resiliency by Design

Integration of Smart City Technologies to create Smart Bases for DoD will require due diligence with respect to the security of the data produced by Internet of Things (IOT) and Industrial Internet of Things (IIOT). This will increase more so with the rollout of 5G and increased automation "at the edge". Commercially, data will be moving to the cloud first, and then stored for process improvement analysis by end-users. As such, implementation of Secure Cloud Architectures is a must. This report provides some use cases and a description of a risk based approach to cloud data security. Clear understanding, adaptation, and implementation of a secure cloud framework will provide the military the means to make progress in becoming a smart military.

Read the Report

CSIAC Journal - Data-Centric Environment: Rise of Internet-Based Modern Warfare “iWar”

CSIAC Journal Cover Volume 7 Number 4

This journal addresses a collection of modern security concerns that range from social media attacks and internet-connected devices to a hypothetical defense strategy for private sector entities.

Read the Journal

CSIAC Journal M&S Special Edition - M&S Applied Across Broad Spectrum Defense and Federal Endeavors

CSIAC Journal Cover Volume 7 Number 3

This Special Edition of the CSIAC Journal highlights a broad array of modeling and simulation contributions – whether in training, testing, experimentation, research, engineering, or other endeavors.

Read the Journal

CSIAC Journal - Resilient Industrial Control Systems (ICS) & Cyber Physical Systems (CPS)

CSIAC Journal Cover Volume 7 Number 2

This edition of the CSIAC Journal focuses on the topic of cybersecurity of Cyber-Physical Systems (CPS), particularly those that make up Critical Infrastructure (CI).

Read the Journal

Recent Video Podcasts

  • Privacy Impact Assessment: The Foundation for Managing Privacy Risk Series: The CSIAC Podcast
  • Agile Condor: Supercomputing at the Edge for Intelligent Analytics Series: CSIAC Webinars
  • Securing the Supply Chain: A Hybrid Approach to Effective SCRM Policies and Procedures Series: The CSIAC Podcast
  • DoD Vulnerability Disclosure Program (VDP) Series: CSIAC Webinars
  • 5 Best Practices for a Secure Infrastructure Series: The CSIAC Podcast
View all Podcasts

Upcoming Events

Wed 20

SANS Stay Sharp: Blue Team Operations 2021

January 18 - January 20
Organizer: SANS Institute
Wed 20

SANS Cyber Security Central: Jan 2021

January 18 - January 23
Organizer: SANS Institute
Wed 20

AI Champions, Online – Supply Chain

January 19 @ 14:00 - January 21 @ 15:30 EST
Thu 21

SANS Cyber Threat Intelligence Summit 2021

January 21 - January 22
Organizer: SANS Institute
Fri 22

SANS Cyber Threat Intelligence Solutions Track 2021

January 22 @ 09:00 - 17:00 EST
Organizer: SANS Institute
View all Events

Footer

CSIAC Products & Services

  • Free Technical Inquiry
  • Core Analysis Tasks (CATs)
  • Resources
  • Events Calendar
  • Frequently Asked Questions
  • Product Feedback Form

About CSIAC

The CSIAC is a DoD-sponsored Center of Excellence in the fields of Cybersecurity, Software Engineering, Modeling & Simulation, and Knowledge Management & Information Sharing.Learn More

Contact Us

Phone:800-214-7921
Email:info@csiac.org
Address:   266 Genesee St.
Utica, NY 13502
Send us a Message
US Department of Defense Logo USD(R&E) Logo DTIC Logo DoD IACs Logo

Copyright 2012-2021, Quanterion Solutions Incorporated

Sitemap | Privacy Policy | Terms of Use | Accessibility Information
Accessibility / Section 508 | FOIA | Link Disclaimer | No Fear Act | Policy Memoranda | Privacy, Security & Copyright | Recovery Act | USA.Gov

This website uses cookies to provide our services and to improve your experience. By using this site, you consent to the use of our cookies. To read more about the use of our site, please click "Read More". Otherwise, click "Dismiss" to hide this notice. Dismiss Read More
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.