A lot of people are always calling their PCI assessment an audit. However, certified public accountants (CPA) would tell them that there is a vast difference between the two.
An assessment is defined as:
“… to measure something or calculate a value for it. Although the process of producing an assessment may involve an audit by an independent professional, its purpose is to provide a measurement rather than to express an opinion about the fairness of statements or quality of performance.”
The key point of difference between an audit and an assessment is the “opinion”. While people would argue that a QSA is judging them PCI compliant, judging is not the same as offering an opinion. The reason is that a PCI assessment is done as of a point in time, not over a period of time. Yes there are some tests in the PCI assessment process such as with change management and vulnerability scanning that are tested over a period of time. However the bulk of testing for PCI compliance occurs at a given point in time, most often the time of the assessment. Such limited testing does not provide the basis for opining on any security program.
An audit is defined as:
“Audits provide third party assurance to various stakeholders that the subject matter is free from material misstatement.”
As an example, a financial audit comprises testing and sampling that is performed over the audit period, typically a period of one year. In addition, an auditor must conduct testing such that they can provide reasonable assurance that there are no material misstatements during the audit period.
The first important phrase is “reasonable assurance” and it is defined as:
“Acknowledgment that it is not possible to assert absolutely and certainly that an event will (or will not) occur.”
Going back to our financial audit example, what reasonable assurance points out is that it is impossible for a financial auditor to essentially redo all of the work performed by an organization’s accounting staff to prove that all of the transactions performed over the audit period were processed exactly as they should have been. As a result, an auditor creates tests of processes and controls and then generates sample sizes based on the risk and the number of transactions performed throughout the audit period such that it is likely the procedures will identify any errors or omissions. If the testing of those samples does not result in any errors or omissions being discovered, then the auditor believes that there is reasonable assurance that there are no material misstatements. If errors or omissions are found, then the auditor must increase their sample size to determine if the errors or omissions are systemic in nature (i.e., the process/controls are broken) or if they are true mistakes. The bottom line about reasonable assurance is that everyone (client, auditor, auditor’s certification body) agrees that if processes/controls are broken, the auditor’s procedures for detecting those breakdowns are sufficient to identify them.
And now we get to what we mean by “material”. Materiality is defined as:
“Information is material if its omission or misstatement could influence the economic decision of users taken on the basis of the financial statements. Materiality depends on the size of the item or error judged in the particular circumstances of its omission or misstatement. Thus, materiality provides a threshold or cut-off point rather than being a primary qualitative characteristic which information must have if it is to be useful.”
Materiality is a judgment call by the auditor based on an examination of risk and whether that risk could result in a misstatement of facts in the financial reports. Years ago we were working with a large client. We relied on their external financial auditor and their assessment of the point of sale (POS) systems user management and access controls audit for Sarbanes Oxley (SOX) to satisfy some of the PCI requirements 7 and 8 testing. However, two years in, the external financial auditor deemed that the controls surrounding the POS systems were no longer material to the financial audit and stopped their testing. As a result, we were left with having to assess the user management and access controls ourselves.
At this point, I am sure a lot of you are wondering other than getting you all to stop calling PCI assessments “audits”, what are you saying?
Business as usual (BAU) is going to change how PCI assessments are performed. Since organizations will have been required to embed controls and monitoring into their business processes, the PCI assessment will likely be changed into a true audit. The reason will be that BAU will require record keeping that will allow a QSA to test for exception conditions for PCI requirements and ensure that the exceptions were corrected and how quickly they were corrected.
While I know a lot of organizations will complain about this sort of process, this is how a proper information security program should work in the first place. Information security controls and monitoring should be embedded into all relevant processes in an organization. Business management and information security should be monitoring and measuring the controls and, when an out of compliance condition occurs, the appropriate actions are taken to either bring the controls back into compliance or the controls are updated/changed to reflect changing conditions.
In rare situations, an organization might find that a control is no longer required because changes have made the control obsolete. This is typically the case when an organization introduces new application software or new network architecture and the control environment wholly changes and controls end up as inadequate, monitoring for the wrong condition(s) or in the wrong place.
BAU is not a penalty; it is an approach to keep an organization on its security “game” by embedding controls and monitoring into the relevant business processes. By doing so an organization then has a mechanism in place to maintain its information security compliance as close to 100% as is humanly possible.
But that will be the rub. This approach will likely find a lot of organizations identifying that staying compliant is nearly impossible because of constant out of compliance situations that will be brought to light. The side benefit of BAU will be to demonstrate just how important security training for all personnel is and that security technology is not the biggest cause of security issues, it is human error. BAU statistics will provide the focus for security training of personnel to address shortcomings. In theory, that training should minimize the security issues from human mistakes and make an organization’s security posture all that much better.
Implementing BAU will take time. It is also not a silver bullet. Like its financial audit brethren, errors and omissions can still occur under BAU, but they are more likely to be caught and addressed before they can spin out of control.
