The momentous NERC CIP v5 deadline of July 1 is now behind us.
Considerable work has been done by all NERC registered entities, but there is still considerable work ahead. Some entities are still working on implementing or automating required controls. On top of that effort, the time bound process requirements (e.g. review X every Y days) kicked in on July 1.
As entities have grappled with CIP v5, Tripwire has learned along the way. In this series, I will cover the aspect of CIP that has come up the most in the last year: how to meet the software monitoring requirements. It is a simple question on its face, but the more we peel back the layers, the more we see what a complicated topic it really is. This discussion may help some electric utilities with their ongoing efforts as well as provide ideas for companies not regulated by NERC about monitoring their software inventory as a security control.
In the first of this two-part series, we will walk through the background of this topic to illuminate the challenges facing entities. Part 2 takes what we have learned and presents practical and auditable approaches for success.
For readers who do not focus on NERC CIP, I still encourage you to keep reading. Being able to identify the possible actors (software) on your system and their level of patching has definite security value. Indeed, software inventory is the CIS #2 control. You have the luxury of implementing controls with a lighter weight process, no burden of audit, as well as all the lessons learned from our electric utility colleagues. Take the fire and leave the ashes!
A quick NERC CIP lingo primer for the general audience:
The North American Electric Reliability Council (NERC) has responsibility for setting various standards that apply to companies generating or transmitting electricity above certain levels. The standards are intended to help ensure reliability of the electric grid. Only one of the standards, Critical Infrastructure Protection (CIP), relates to cyber assets. The CIP standard is further broken down into sub-areas also referred to as standards. Standard 10 (CIP-010) contains requirements for configuration management around software, specifically those found in its first requirement.
Thus: NERC -> CIP -> CIP-010 -> Requirement 1 > Part 1.1.1 through 1.1.5.
Also, keep in mind that the CIP standards were written by lawyers, for lawyers, with engineers caught in the middle. The data generated for showing compliance could literally end up in civil court if evidence of compliance seems inadequate. The country is divided into geographic regions for audit, a reality which can lead to some region-by-region variance in how the requirements are interpreted.
So first, what is “software”, and what is asked for in the CIP requirements?
Within CIP-010 R1, there are sub-requirements listing what is needed regarding software monitoring. Additional information is provided in the Guidelines and Technical basis section at the end of the standard.
“Software” boils down to understanding these terms:
- OS and firmware version
- Intentionally installed software
- Custom software
- Security patches
Next, I will define these software categories briefly. If you know the NERC CIP-010 standard pretty well, feel free to skip ahead. The section on custom software is probably worth a look. though.
What is a baseline?
A “baseline” is a Tripwire Enterprise concept as well as a term in the CIP standard, but they do not mean the same thing!
In a Tripwire Enterprise context, “baseline” refers to a particular version (out of all observed versions of a file) that is considered the trusted or preferred version. Any new changes are compared to this baseline version, and a new version of a file can be identified as the new trusted version (aka, baseline) through a built-in workflow.
In a NERC CIP context, baseline refers to a standard of configuration, more like a platonic form or ideal configuration. This concept of baseline maps best to Tripwire Enterprise policy and test features, not the “change and promote” concept described above.
Most customers initially assume the “review change and promote” process will help with the CIP change management requirements. Both approaches are typically used by entities to one degree or another. The “change and promote” approach is much more granular and not as scalable, whereas the approach with policy tests allows for entities to manage compliance in a much more streamlined and scalable fashion.
OS and firmware version are the most straightforward to understand. “OS version” refers literally to the version of the operating system. “Firmware version” is usually read as pertaining to switches, routers, and other networked devices lacking a traditional OS. Proprietary systems that run a traditional OS (like Linux) but which cannot have a Tripwire Enterprise (TE) agent installed are usually treated like a network device, i.e. we connect by SSH or telnet instead of installing a TE agent. BIOS versions can be included in scope, but I have not seen any entities actually implement such monitoring for NERC CIP.
Intentionally installed software is software that is needed on the asset to support its bulk electric system critical function and that has been installed onto the operating system. This software is part of the declared (CIP) baseline, and it’s necessary for the role of the asset. Presumably, unintentionally installed software needs to be reviewed and then either approved or removed.
Custom software is the most interesting. Over the past year, I have seen entities across audit regions struggle with interpreting not only the language of the standard but also the Guidelines and Technical basis for this standard. WECC (the geographic audit region in the US West) made a public presentation in March wherein the advisement was to stick to the strict language of the standard (which does not mention scripts) and ignore the Guidelines and Technical basis (which does mention the scope “may include scripts”). WECC can only speak for their region of course, but it is a helpful data point in how that interpretation may have consideration in other regions.
Prior to this advisement from WECC, there was considerable angst among entities as they struggled with determining what qualified as a script and how to monitor systems for scripts in a scalable and repeatable manner. Some common example questions that arose included:
- Should we only monitor files with executable permissions on Linux?
- Do Excel Spreadsheets with macros count as scripts?
- Do we need to scan whole systems for executables, or can we meet the requirement with a partial scan?
The rabbit hole quickly became the Grand Canyon!
Defunct, at least in WECC:
The Technical Basis and Guidelines section of CIP-010-2 says “custom” software may include scripts which every entity interprets as “definitely include scripts”. But then what? I have seen an entity or two define the scope of custom software as any executable on the system. I have yet to see specific feedback from an audit on that point, so it is questionable in my mind whether the scope has to be so broad. The review process would certainly be quite burdensome and heavy on staff time to maintain! What I have seen more widely adopted is along the lines of “crawl, walk, run” – do something, perhaps modest, now and then adjust up or down later as audit feedback becomes available.
Finally: Patches. Patches refer to the hotfixes on Windows or analogous updates released for other traditional OS’s. Tracking the firmware version of network devices is the security equivalent of tracking patches from a CIP perspective. This facet of the CIP software scope is straightforward to define but trickier to report on, as we shall see. The fun part is that not all patches are security related, and they may not be distinguished as such by the OS vendor. “Here’s a bunch of patches, go have fun!”
Now that we know what needs monitoring, how do we do it?
You don’t want to be the best or the worst at CIP audit. If you are the best, then perhaps scarce resources were over-dedicated. If you are the worst, your organization will stick out in the mind of the auditor as they travel from site to site and there could be a higher risk of a finding.
In Part 2, I will describe approaches that have emerged over the years in collaboration with our NERC customers – NERC customers telling Tripwire what they need, Tripwire filling in with best practices, and sometimes even product enhancements arising from discussions of the art of what is possible.
Click here to read part 2!
 Please review comments at the 4 hour, 39-minute mark here from the San Diego CIP User Group meeting.