Effectiveness Reviews

The Governing Boards of regulatory bodies will wish regularly to review the effectiveness of their organisation. When doing so, it is helpful to consider the following questions:

1. General Approach

As far as is compatible with the duties imposed by law, does the regulator leave the regulated free to live their lives as they wish and/or get on with their work, business or profession?

Do the regulated clearly understand what is expected of them?

  • Regulations may not be absolutely clear, and the law may not anticipate issues that may arise after the law is enacted.  But regulators should nevertheless strive to provide clear and accurate guidance whilst avoiding frequent repositioning of the goalposts.
  • Are regulated entities fully aware that they, not the regulator, are responsible for ensuring their compliance with regulations, and in particular for ensuring the safety of those who use their products and services?

Has the regulator done all it can to minimise the cost of compliance? Such costs include non-financial consequences such as the impact of regulation on the willingness of a business to innovate, or the willingness of volunteers to help a school, charity etc.

2. Over- or Under-Regulation?

Does the organisation achieve an appropriate balance between over- and under-regulation?

Under-regulation leaves too many loopholes that producers can avoid or evade

Over-regulation provides incentives to evade regulation, including

Note that:-

  • High levels of regulation can provide too much protection for large established firms relative to new entrants, and can foster corruption and collusion between producers and consumers to minimise the cost impact of e.g. building regulations.
  • The incentives almost always favour increases in the intensity and scope of regulation so as ...
    • to avoid obvious risks from under-regulation plus
    • to respond to understandable pressure from concerned lobby groups.
  • It may be necessary to involve the government/legislature in this discussion.

And are Inspectorates providing an effective stimulus to good management?  Or are they inadvertently reducing the ambition, innovation, responsibility and accountability of managers whose practices they have 'passed'?

3. Resource Allocation

Does the organisation divide its resources appropriately between its four key functions?

Enabling - licensing/permissions to carry out certain activities.
Policing - enforcing compliance with legislation/regulations.
Advising - providing guidance to help entities understand and comply with legislation/regulations.
Educating & Informing - helping the public understand and exercise their rights.

4. Contradictory Pressures

Has the organisation discussed and chosen appropriate responses to the contradictory pressures identified by Harvard's Professor Sparrow in his book The Regulatory Craft?:-

“Regulators, under unprecedented pressure, face a range of demands, often contradictory in nature:

  • Be less intrusive – but more effective;
  • be kinder and gentler – but don’t let the bastards get away with anything;
  • focus your efforts – but be consistent;
  • process things quicker – and be more careful next time;
  • deal with important issues – but do not stray outside your statutory authority;
  • be more responsive to the regulated community – but do not get captured by industry.”

5. Independence

Is the organisation satisfied with the extent of its independence from government?  The following indicators may be relevant:

  • Legal status and powers
  • Can Ministers issue directions and/or does the regulator seek Ministers’ informal agreement to controversial decisions?
  • Strength of character and experience of senior staff
  • Robustness of Board appointment process
  • Ministers ability to dismiss Board members (other than for criminality or insanity) 
  • Does the Board have fixed term (3-5 year) appointments, not renewable or renewable only once?

A more detailed discussion of this subject may be found here.

6. Accountability and Trustworthiness

Is the organisation sufficiently accountable for its decisions:

    • to Parliament
    • to the Media
    • to the Public
    • and to the Courts?

In particular, are its decision processes accessible and transparent? Does it accurately and clearly communicate the analysis that underlies its decisions? Is it candid about the compromises that may be needed?

Does the organisation appear to be trustworthy?  Does it deliver against expectation effectively, reliably, consistently, responsively?  And does it appear to operate with integrity, seeing others as equals; listening to and taking seriously their concerns, views and rights.?

Note that accountability involves explaining why the regulator has taken its key decisions. It does not mean 'obeying orders' - other than those contained in legislation and legal decisions.

A more detailed discussion of this subject may be found here.

7. Ethics

Has the organisation carried out, and implemented the recommendations of, a recent ethics health check as recommended by the Committee on Standards in Public Life (CSPL)? The health check should look at issues such as:

  • conflicts of interest
  • hospitality, gifts etc.
  • 'revolving door' and post-employment policies, and
  • codes of conduct - not only for staff but also for contractors, consultants, non-execs and secondees.

A helpful 2016 report by the CSPL is here.

8. Effective Regulation: Larger Organisations

Do senior staff understand the cultural and behavioural pressures that can cause catastrophic failure in the regulatory compliance of larger organisations?  These might include:

  • The Principal-Agent problem
  • Herd behaviour
  • Groupthink, and
  • The Prevention Paradox.

A detailed description of these pressures can be found here.

9. Effective Regulation - Individuals & Smaller Organisations

Has the organisation developed a suitable range of relationships with the individuals and smaller organisations within the various elements of the typical regulatory spectrum: 

More detailed advice about the regulation of the activities of individuals and smaller firms may be found here.

10.  Targets, Performance Management etc.

Does the organisation have performance management systems consistent with the NAO's good practice guide Performance Management by Regulators?  (And see the Notes below.)


It may not make much sense to try to quantify the effectiveness of your organisation, mainly because it is almost impossible to measure the deterrent effect, nor how this changes with numbers employed etc.  Most of the deterrent effect is probably achieved merely be establishing, and advertising the existence of, the regulator.  

Bruce Lyons commented as follows on reading the CMA's targets in 2016:

"Its performance management framework commits the CMA “to achieving direct financial benefit to consumers of at least ten times our cost to the taxpayer.”  Target setting and performance measurement are an important part of performance management.  However, the precise way that the government requires the CMA to justify its funding is dangerously distortionary.

The government requires the CMA to achieve a benefit to taxpayer cost ratio of at least £10 for each £1 of taxpayer cost (i.e. budget). This ratio is completely arbitrary and without empirical foundation.  I am actually quite confident that this ratio would be comfortably met if all the benefits could be credibly measured – but they cannot. My concern is that in order to tie tangible benefits to tangible case work, ‘benefit’ is essentially measured by expected direct price savings from the remedies on actual cases completed (averaged over three years).  This hugely underestimates benefits, in particular by ignoring deterrence.  Think how dangerous it would be if military leaders had to justify their budget by measuring the value of wars fought, ignoring wars deterred."


Martin Stanley

Spotted something wrong?
Please do drop me an email if you spot anything that is out-of-date, or any other errors, typos or faulty links.