For a long time now, security evangelists have railed against the dangers of relying only on checkbox compliance. They warn that if you focus too much on the list of requirements, you’re bound to miss risks that may not actually be covered in rules and regulations.

That’s why organizations need to start evaluating effectiveness alongside these audits, in order to get a more holistic view into the systems they are assessing.

“Organizations are so focused on meeting the letter of the regulations and mandates that they lose sight of the risks that the individual controls in the mandates are intended to mitigate,” explained security consultant Brian Musthaler in a recent blog post.

It’s a theme revisited in a ComputerWorld article, which cited a survey showing that just 17% of organizations have what they consider a mature risk management program—i.e., one that goes beyond ticking off items on an audit list. The maturation to risk-based security, the article emphasizes, is “about a not so insignificant shift in objectives—from compliance to making systems more resilient to attack.”

The principle holds true not just when evaluating and shoring up in-house infrastructure. It also applies to how enterprises evaluate partners. As security organizations seek to find a sane way to measure the IT security stance of partners and vendors, the most common first step is to do it by following a requirements checklist or questionnaire, or by asking for an auditor’s attestation of compliance with some kind of standard. Assessment guidance from standards like the Statement on Standards for Attestation Engagements (SSAE) No. 16, ISO 27001, and FedRAMP all come to mind here.

Serving as a compendium of best practices, measuring against these standards can give good indicators of where to focus resources and are a good place to start your evaluation. The challenge is that while necessary, using these methods alone for assessing security risks is not sufficient. A company may be compliant with all the appropriate regulations and have excellent security policies but may be completely ineffective in the day-to-day implementation of these policies—rarely does a questionnaire ask how many compromised servers a provider is currently running on its network. Also, no matter how complete a checklist or audit is, its results are only a point in time reflection and can’t measure the dynamic nature of the risks it is meant to assess for the duration of the business partnership. Even if a penetration test or vulnerability scan is included as part of a vendor assessment, it cannot reveal issues that may appear the following week.

Complimenting an audit with a continuous evaluation of security effectiveness allows organizations to augment their view into the security risks of the extended enterprise. In addition to gaining visibility into the weaknesses of a network, a data-driven, evidence-based assessment can allow organizations to proactively mitigate new risks as they emerge and identify issues that a regulatory audit was not designed to catch.

By taking these steps, organizations can move towards a mature, risk-based security model and away from the more simple checkbox mentality.