By Turksema, Rudi van der Knaap, Peter
All performance auditors share the same goal: to audit
the most relevant problems using the right people and
the most appropriate methods and techniques at the
right moment and to use the audit findings to make a
meaningful contribution to the quality of government
policy and operational management. The Netherlands
Court of Audit devotes a great deal of time to the
proper planning and selection of its audits. We work
in a very broad field but with finite capacity. The
Court of Audit's mandate covers all of the central
government and nongovernmental institutions that
receive public funds to carry out statutory tasks. In
all, we must audit the use of about 350 billion euros
with a staff of just 80 auditors who specialize in
performance audits. For staffing reasons alone, we
have to think very carefully about which performance
audits we will carry out. More importantly, if our
work is to be effective, we have to know where we can
make the greatest contribution.
In this article, we explain how the Court of Audit
tries to realize its goals. The first section
describes how we use the planning and selection tools
in practice - which tools we use, when we use them,
and how they are related to each other - and the next
section includes the results and the lessons learned
from our use of the planning and selection tools.
Planning and Selection Tools
At our SAI, audit planning and selection must help us
achieve the Court of Audit's mission: to "audit and
improve the regularity, efficiency, effectiveness, and
integrity with which the State and associated bodies
operate." Another task of the Court of Audit is to
"contribute to sound public administration through
cooperation and knowledge exchange at home and
abroad." For purposes of this article, it is also
relevant that we seek to be a transparent
organization.
To plan and select our performance audits, we use a
number of tools that are closely related to the Court
of Audit's overall strategy (see figure 1).
Figure 1: Netherlands Court of Audit Strategy and
Audit Planning and Selection Tools
Planning and selection begins with the Court of
Audit's strategy. Three domains-public services,
safety and security, and sustainable development-have
been prioritized to implement the strategy, and they
determine the most relevant objects for audit. We also
use three tools-monitoring, issue matrix, and
integrated risk analysis- to look beyond the strategy
at more topical developments in our field of
operations.
The first tool is monitoring, a standard activity
carried out by all Court of Audit organizational
units. In all events, we monitor the ministries'
policies and the performance of officials charged with
statutory tasks. Given our strategic goal of helping
to reduce social problems, we also increasingly
monitor social developments.
The issue matrix, the second tool, is an outcome of
monitoring and is intentionally designed to facilitate
discussion of the monitoring findings with the Court
of Audit's Board. Issues are summarized in memos that
answer the following four questions: (1) What is at
issue? (2) Is it undesirable? (3) Where does it occur?
(4) Who are the main players? The issue matrix
identifies relevant issues in our audit field that are
not necessarily covered by our strategy.
Figure 2 is an example of an issue matrix in the
transport, public works, and water management policy
field. The horizontal axis shows the likelihood of an
event's occurrence, and the vertical axis shows the
potential impact of an event. By ordering the issues
in such a matrix, we are better able to determine
their potential importance. To encourage further
discussion, we deliberately do not define the concepts
of "potential impact" and "likelihood of occurrence"
any further.
Figure 2. Example of an Issue Matrix forTransport,
Public Works, and Water Management Policy
According to this issue matrix, it is our opinion that
accessibility (motorway congestion) is almost certain
to occur and will probably have a high impact. In the
Netherlands, it is indeed a daily problem: overcrowded
rush-hour motorways harm both the environment and the
economy. Flooding, which may also have a high impact,
is thought to be less of an issue because it is less
likely to occur. The privatization of the national
airport, Schiphol, would have a low impact since it is
not expected to seriously affect the continuity of the
airport's operations. We have not identified issues
with a low likelihood of occurrence in this policy
field, though they may exist, Rather, the Court of
Audit is more interested in issues that are more
likely to go wrong.
The third tool is integrated risk analysis. In
contrast to monitoring and issue matrices, risk
analysis builds on the strategy and is more focused so
that it can be used directly for the activity program
and planning of regularity audits.
Integrated risk analysis (IPvA) is a systematic and
efficient means to generate, analyze, and record
information about the entire audit field to identify
and classify risks in the domains that are relevant to
the Court of Audit. The risk analyses are integrated
in that they
* combine all identified and prioritized risks so that
choices can be made for both the regularity audits and
the annual programming of performance audits and
* are conducive to detecting connections between
public administration operations and risks to public
administration performance.
The system used to calculate risk is very similar to
the one used for issue matrices. The risk is estimated
as a combination of the likelihood of an undesirable
event or situation occurring and its impact on one of
the risk domains: risk = {likelihood of risk x impact
of risk}. We use the model shown in figure 3.
Figure 3: Integrated Risk Analysis Matrix
The end product of integrated risk analysis is a
systematic overview of the main risks to the operation
and performance of public administration and
associated third parties. The analysis results serve
as input in our proposals for regularity audits
(selecting the auditee and approach) and the annual
programming of performance audits. Integrated risk
analysis also contributes to accumulating
field-specific know-how and exchanging that know-how
among organizational units.
Risks to public administration operations and
performance may be related to each other. For example,
IT problems (cause) at the Ministry of Finance may
lead to late payments of housing benefits (effect).
This might indirectly lead to another ministry's
failure to achieve a policy goal, such as reducing the
number of forced evictions. A simplified cause and
effect diagram for this example is given in figure 4.
Figure 4. Example of Potential Cause and Effect of
Risks to Public Administration Operations and
Performance
The tools we use to plan and select performance audits
ultimately generate input for programming. Ideally, by
comparing the outcomes of monitoring and issue
matrices (which have a broader and more topical
orientation) with the proposals arising from the
strategy (which have a more focused and longer term
orientation), we will program the right performance
audits at the right moment. The direct result of our
planning and selection activities is therefore an
optimal activity program, which in turn determines how
effective we are as an audit institution. However,
that is sometimes beyond our control.
Conclusions
This article has explained how the Court of Audit is
trying to perfect its planning and selection of
performance audits. We are pleased with the three
benefits we have identified in the planning and
selection process. First, the process produces a
systematic and lively internal and external debate.
Second - and this is a significant advantage of our
multiyear strategy - it enhances our external profile.
An auditee sometimes sighs, "Yes, if I read your
strategy and hear your arguments, it goes without
saying that you'll want to audit this." And although
we prefer not to hear people sigh, this often
increases their cooperation and willingness to take a
critical look at their own policy practices. Third, it
encourages rigorous, activity-based management, both
in the three domains recognized in our strategy and in
auditing the relationship between policy and its
implementation, the main focus of our performance
audit strategy.
People sometimes see an overlap in our preparations:
in addition to carrying out monitoring activities and
integrated risk analyses, we have identified special
domains and must prepare related proposals. We, too,
sometimes think that we have set our goals too high:
we must invest a considerable amount of time in the
planning and selection process, yet the capacity
available to carry out further audits is limited. We
also have to respond to current developments and
requests from Parliament. Finally, programming our
audits in response to social problems is a daunting
challenge. Sometimes, operational audit "magnets"
divert us from finding the causes of disappointing
results and lead us to check procedural and
organizational systems. Fortunately, we are getting
better at resisting these modern-day "Sirens."
References Netherlands Court of Audit (2003),
Performance and Operation of Public Administration:
Strategy 2004-2009 of the Netherlands Court of Audit,
The Hague, Netherlands Court of Audit.
de Vries, GJ. D. (2000), Beleidsdynamica als sociale
constructie. Een onderzoek naar doorwerking van
beleidsevaluatie en beleidsadvisering Delft, Eburon.
By Rudi Turksema and Peter van der Knaap, Netherlands
Court of Audit
CAclubindia
