08 Mar PCAOB Change in Expectations Driving Increasing Scope for SOC Reports
ERP software provides organizations with tremendous benefits including vast configurable processes and standard reports used for reporting on data. ERP software comes in two flavors: those that organizations need to manage their own infrastructure (on-premise software providers) and those where the infrastructure is managed by the software vendor (SaaS software providers).
Software providers (SaaS and On Prem) are subject to audits where an independent auditor reviews and tests the controls defined by management and issues an opinion on their control environment. These audits are referred to as SOC (Service Organization Controls) audits.
Part of the SOC audit scope is testing of a population of reports provided by the software provider (what we refer to as “seeded” reports). Management routinely uses seeded reports in the execution of their controls.
Management is responsible for documenting how they concluded that the reports they are using are complete (C) and accurate (A). Auditors refer to this as C&A testing. This is true for seeded (as referred to as ‘default’ ‘standard’, ‘canned’, or ‘vendor-supplied’) reports as well as any custom reports or queries they develop.
External auditors are required by the PCAOB to evaluate and test management’s controls over such reports. For seeded reports, the review by external auditors has historically been limited to obtaining the SOC report. The SOC report provides general certainty over the software provider’s change control process. As reports are developed or changes are made to the reports, they must be subject to the software provider’s change management process.
The change management process should include controls such as:
- The developer of the code should be separate from the person (promoter) migrating the code to production
- The promoter of the code needs to move the code into a secure environment that the developer does not have access to – what we refer to as a User Acceptance Testing environment (UAT)
- A test script is prepared and executed by a person or group (tester) doing the testing
- Testing should verify the completeness and accuracy (C&A) of the report
- Documentation of the successful test should be kept as evidence associated with the change ticket in the software providers change management ticketing system
As part of the SOC audit, the audit firm would pick a sample of actual changes and ask the management of the software company to provide evidence that the changed report was properly subject to the change management process.
When the SOC auditor issues their opinion on the software company’s controls, it is based on the sampling it took during the audit. The SOC auditor does not test every change, only a sample of the changes. Historically, the SOC auditor does NOT provide the details in their SOC report of what reports were sampled.
When external auditors perform their audit, they look to management to identify a population of key reports that are being used to execute manual controls. Management, ideally, should be identifying this population by mapping it from the key controls in their Risk and Controls Matrix (commonly referred to by auditors as the RCM or RACM – pronounced “Rackum”).
External auditors should be testing the Completeness and Accuracy testing of these ‘in scope’ reports. Historically, any report that is provided by the software vendor, referred to as “seeded” reports, gets a pass by the auditor because it is assumed that the vendor tests new reports and changes to existing reports as part of their change management process. Even in these circumstances we believe it is prudent for management to test and document their own C&A testing because the software company does not disclose its methodology and management still needs to confirm the data is appropriate for their specific control.
Even though the SOC auditor does not test ALL reports, they test a sample of the reports. If the sample they pick has adequate change management controls, the software company’s change management system is deemed to be effective over all reports.
Transitioning now to what we are hearing from external auditors is a change in the expectations from the PCAOB. The risk being presented is that the sample the SOC auditor took did not include the specific reports that management is using. The expectation now is that management confirm with the software company or the SOC auditor whether the report(s) they are using to execute controls were specifically tested. This is a significant change in expectations and could cause a significant increase in scope for the SOC audit.
There is another risk that needs to be considered as well; the risk that the report is changed by the software provider. Most current SOC reports do not specify the reports that were samples and they also do not provide a list of reports that have been changed when they patch their applications. SaaS software providers typically patch their applications either semi-annually (2 times per year) or quarterly (4 times per year). Software providers of on-premise software provide patches, but not necessarily on a regular basis. Organizations often have the choice to defer applying patches to the software application for years such that the risk of changes to seeded reports is much less frequent than SaaS software providers whose patches are not optional and cannot be delayed.
Management should be re-performing their C&A testing whenever a report definition is changed. Therefore, it would be ideal for the software company to disclose the reports that have had a modification to the logic when they release each patch (often referred to as a ‘read me’ file). This would provide management a notice that they need to re-perform their C&A testing.
Following is a summary of the current risks that are often addressed by management and the suggested controls management needs to consider implementing:
|A report being used by management to execute a manual control has not been tested by the SOC auditor||
1. Ask the software company to disclose what reports are included in the sample used in the SOC audit
2. Ask the software company to increase its testing scope to include the specific reports being relied upon by management
|A report being used by management to execute a control has been changed by a patch and not disclosed to management||1. Ask the software company to update their internal procedures and include a complete list of reports that were changed as part of the release. As part of this attestation by management, they need to explicitly state that reports not mentioned in the report have NOT been changed.|
Following are our recommendations overall:
- Management needs to identify a population of key reports from the Risk and Controls Matrix. This includes controls that support manual controls (referred to as IT dependent controls) and IT general controls. This population MUST include all reports (seeded or custom) and queries that are used to execute any control.
- Given software providers do not identify the specific reports, management should baseline all reports – seeded or custom. Whenever a report is changed in any way, management should re-baseline their C&A testing.
- For seeded reports, given software providers typically do not disclose when they have modified seeded reports as part of their release notes, we recommend that management re-baseline and document the C&A testing for each report each time the software provider patches the applications.
- For custom reports and queries, management needs to put in place the following controls:
- All changes to reports and queries need to be subject to the change management control. This includes re-baselining the C&A testing prior to changes being made in production.
- Management needs to put in place a pro-active monitoring process to identify when any changes are made to custom reports or queries. This would include logging changes to the OS files, programs and files stored in the database, report definitions, changes to input controls such as a list of values, and any other place where logic / code in a report may be changed. This set of logs would constitute a change management population for changes to key reports. Management should regularly review, no less than monthly, this change management population to identify if any key reports were changed. If so, they should confirm these changes were subject to the change management process and that the C&A testing was re-baselined.
Summary and Conclusions
The PCAOB has asked some reasonable questions about ‘in scope’ reports being provided by software providers. Management needs to consider updates to their ITGCs to accommodate this change in expectations. Management should also request from their software providers to be more detailed in their SOC reports including specifying which reports are being tested and to include testing of specific reports as part of their SOC audit.