Can’t get no satisfaction (statistics): reporting performance

One of the most promising developments in public services in recent years has been the increasing attention to customer satisfaction - whether those customers are students, patients, residents, or whoever. Often regulators require the publication of satisfaction statistics. When the results show improvement, public sector organisations do no need much encouragement.

I was therefore interested in the latest newsletter from my local NHS hospital trust. The headline was "Survey reveals patient satisfaction is on the up":

Significant improvements since 2007 included:

- The hospital room or ward was very clean - up 12%
- Always offered a choice of food - up 12%
- Doctors always washed or cleaned their hands between touching patients - up 10%...

But where did that take the Trust? What were the new percentages? (How many doctors had dirty hands?) How do the new percentages compare with last year's? What were the old percentages? What about the average rates for other comparable hospitals?

There were no charts illustrating any of this. Just words.

Perhaps the article was not intended to report performance - only tell of how the Trust was on "on the up". But accountability is about reporting performance.

School league tables: past performance is not necessarily a guide to future performance

Many of the opponents against choice in public services rely on weak arguments. “The middle class will benefit” – yet they already win by having the resources to choose through moving into catchment areas (or buying in the private sector); ”the poor don’t want choice” – yet surveys demonstrate otherwise; “what people want is a good local school/hospital/whatever” – yet choice (with competing providers) is a means to that end.

I was therefore interested to read an article in the latest bulletin of Bristol University’s Centre of Market and Public Organisation, Research in Public Policy. The authors of Are league tables any use for choosing schools?

George Leckie and Harvey Goldstein studied the statistical significance of value added scores and concluded:

... when taking account of this uncertainty, the comparison of schools becomes so imprecise that, at best, only a handful of schools can be separated from the average school or from one another with an acceptable degree of precision. This implies that publishing league tables to inform parental choice of school is a meaningless exercise, as parents are using a tool which is not fit for that purpose.

In particular, they noted the lag of over five years between the parents looking at league tables when choosing a school and the children sit their exams. Five years a long time in the life of a school.

Does this information problem blow a hole in the argument for empowering parents and other customers of public services? I would suggest not – there are other measures of performance other than exam league tables. (There may, of course, still be value in value added league tables if failing or coasting schools raise their game through being either “named and shamed” or spurred by fear of falling school rolls.) Nevertheless the research does pose more of a challenge than the arguments usually wheeled out against choice.

Board members, finance committees and financial monitoring – need to know basis?

I recently heard a partner of an accountancy firm (and provider of audit services to the housing sector) say that housing association board members did not need any quarterly financial reports – as long as the board had a finance committee doing the board’s financial monitoring. My immediate response was one of both disagreement and disbelief.

In case I had missed something. I thought I would consider Treading the Boards (pdf available) – the Housing Corporation’s self-assessment framework for board performance. (A guide to housing association governance published in 2001 but still relevant. It also informed the Corporation’s regulatory expectations).

Treading the Boards stated that the “key roles or functions” of a board include:

Monitor[ing] the association’s performance against agreed targets and milestones through regular critical appraisal of financial, operational and development information

It made it even clearer saying that the board should be able to demonstrate that it:

regularly and critically reviews management information on financial and operational performance against budget /targets, the previous year’s figures and external benchmarks

It interprets regularly as monthly or quarterly.

But can board members contract out their financial monitoring role to a finance committee?

Larger, more complex, associations may have established business activity sub-committees that can address the critical issues in more detail than would be possible for the main board.

I read that as giving finance committees a role of detailed scrutiny – not the rest of the board ducking out of this key function. (While I am a great believer in the value of committees in governance structures, I’ve seen governance failure arising from boards being oblivious to the concerns being raised about financial issues in committees.)

What about guides to good practice outside the world of social housing? The 2003 Review of the role and effectiveness of non-executive directors by a committee headed by the now late Sir David Higgs noted:

Non-executive directors should scrutinise the performance of management in meeting agreed goals and objectives, and monitor the reporting of performance.

That report considered this to be a “key element”. Subsequent events in social housing as well as the wider world have surely proven this and reminded everyone that we live in financially hazardous times.

Anyone suggesting that board members can pass the buck of financial monitoring to a finance committee should also pause and consider the fact that housing association board members will generally have the duties of a company director and/or charity trustee. These cannot be evaded in some kind of governance pass-the-parcel.

Better management reporting: How to make an Impact

I was disappointed yesterday. I saw that the Institute of Chartered Accountants is hosting a lecture by Jon Moon in Birmingham in November but then realised that I had a prior commitment.

Who is Jon Moon? He has written an excellent book on How to make an Impact – basically writing reports that are uncluttered so the message is clear. The report has certainly changed the way that I present information when I am working as an interim Finance Director.

I believe that so much management reporting in the public and third sectors is weak that I am sure that others could learn something.

People can attend the lecture even if they are not members of the ICAEW Finance and Management Faculty. I would also recommend the templates that Jon Moon gives away on his website.

Comparing performance: tenant satisfaction in council housing

The polling and research firm IPSOS-MORI have published an interesting analysis of tenant satisfaction data. The Housing Futures study (pdf available) aimed to offer “a more useful insight into tenant satisfaction levels than is possible from a simple league table approach”.

The study modelled the effect of largely external factors on tenant satisfaction – “the nature of place and the challenges it brings”. The model predicts tenant satisfaction on the basis of four factors:

1) Deprivation - the more deprived an area the lower the level of satisfaction.

2) Ethnic fractionalisation – “a measure not only of the proportion of people from ethnic minority communities, but the extent to which there are a wide range of different ethnic minority communities in an area” – with it being more challenging to meet the needs and expectations of more diverse populations

3) The proportion of elderly social tenants (aged 60+) - older tenants that are more likely to be positive about the services they receive.

4) The proportion of housing stock which is council owned – “As the percentage of total stock owned by the council increases, satisfaction levels decrease”

The study went on to identify where local authorities out- (or under-) performed the tenant satisfaction levels predicted by the model. Carrick, Ealing, Blyth Valley and Welwyn Hatfield were the joint top performers. (Three of the four have Arms Length Management Organisations.)

The fastest improving local authorities were identified as Ealing, Leeds, Redditch, Gloucester, Bury, Hounslow, Stevenage and Islington. (Again with ALMOs well represented.)

The study does usefully disentangle organisational performance from external factors. Perhaps similar studies should be performed and published for other public services. (If they have been, I don’t think they have been publicised enough)

My only criticism is the lack of any analysis of how relative performance maps across to different models – in particular, comparison of housing department performance with that of ALMOs. The study does contrast various types of local authority but ignores the ALMO debate.

Lies, damned lies and performance management

Last week’s Public Finance magazine carried an interesting article about performance management – yes, really. It included a disturbing statistic: over 70% of recipients in a recent survey admitted to fabricating performance data. (How many of the others didn’t like to admit it?)

We know about perverse incentives, erroneous errors and even issues with performance data to regulators and clients, but this is organisations telling themselves lies.

(Before I go on, I should stress that I am a sceptic when it comes to “targets” as reform and performance information as a panacea. But performance information is integral to good governance.)

What can be done? A mature attitude to performance information would be a start – seeing it as a means to an end rather than an end in itself; measuring what matters rather than what is easily measurable.

There is also value in assurance. Boards and managers should ensure that key information systems are reliable. How can they do their job if they are driving with a faulty dashboard?

Larger housing associations have to get certain operational performance information systems verified periodically. Other organisations could get internal auditors to check out performance information systems. Yet in most of the public sector and third sector performance information is just accepted at face value.

Housing repairs PIs, inspectors and targets

This week’s Inside Housing has a challenging article by John Seddon (the management guru who has popularised “lean thinking” in the UK). He writes on performance improvement in housing - and how, he suggests, targets and the Audit Commission inspection regime hinders it.

Sadly there is no copy of the article on Inside Housing website. However, there is an article making a similar case on John Seddon’s website, Systems Thinking. I have some sympathy with the case being made.

Inspectors (and politicians) often assume that everything can be fixed by targets (or higher targets if performance is not good enough). Sometimes “targets and terror” can work – but I would argue that such an approach cannot deliver sustainable improvement.

Housing associations and ALMOs should not give up all hope. They can do something. For a start, they should remember that the Housing Corporation has shifted the focus on repairs performance indicators from response times to resident satisfaction. It appears to be that few housing associations have re-thought their own repairs PIs and introduced a more holistic and sophisticated framework for repairs performance.

The Healthcare Commission learning from investigations – mergers, targets, governance and other common themes in public services

The Healthcare Commission watchdog has published an interesting analysis of lessons from 14 investigations into patient safety failures. Such analyses of failing organisations are always to be welcomed. In social housing there are the Learning from Problem Cases series of reviews into "supervision" cases (see the Housing Corporation website here and here). Sadly in other sectors, such as further education colleges, there are ad hoc reactions to failures rather than systematic reviews and efforts to promote learning.

The Learning from investigations report (pdf available) found common themes:

1) Leadership and management: Poor leadership was a problem in nearly all of the investigations carried out by the Commission.

2) Some boards had been focused on mergers or targets at the expense of their broader activities.

3) Lack of continuity in leadership was a problem in some trusts, where frequent changes in management were a factor in poor care. Bullying and harassment by managers was a factor in two cases investigated. The Commission found there was a fine line between promoting change vigorously and bullying.

4) Investigations often uncovered a breakdown in leadership and management, with a lack of clarity on responsibilities from board to ward. Poor teamwork, either between management and clinicians or between clinicians themselves was another common factor in failings.

5) Use of information: The Commission found that most of the trusts investigated did not have adequate systems in place to routinely inform the board of trends or potential problems.

6) Mergers and restructures: Seven of the trusts investigated had recently undergone mergers or significant organisational change.

7) Safeguarding vulnerable adults: Poor understanding of adult protection procedures and responsibilities was a serious problem in the two investigations into learning disability services and also a number of interventions in trusts.

8) Poor care on general wards: When its investigations looked at acute hospital care, the Commission noted that care on general wards fell well below the care provided on specialist wards. Older patients were most at risk as they were often most dependent on good nursing care.

It is note-worthy that the first five of these themes are found widely across public services.

Some facts about police pay and performance

The latest bulletin from the Reform think tank makes some interesting points about policy pay and performance which are worth considering in the heat and noise over the government’s reluctance to aware the review body’s recommended rise. For example it notes that over the last 12 years, police pay has risen at twice the rate of inflation and by more than the average of the public sector and the private sector pay increases.

It also puts falling crime rates in the context of how now 63 per cent of main family cars now have an alarm, compared to 23 per cent in 1992. So often performance indicators are seen in isolation rather than considering the external factors bearing on them.

Police, customers and targets

Its not news that target-driven policing doesn’t work. But it is good news that the government has admitted this.

We might even see the police service adopting an ethos of customer service. My local police station might even open at the stated time. When shut, the buzzer to the duty station may even work.

Perhaps not given uncertainty over the 101 pilots – where citizens could report non-emergency incidents without clogging up 999 lines and without being trapped in labyrinth-like never-ending call-waiting. These lines are showing customer satisfaction of 90%!

NHS targets - waiting for change

Apparently an investigation has been launched into waiting list figures at a Lancashire foundation trust. The case at Royal Preston Hospital will be an interesting one to watch.

Managing the public services by target is hazardous as I’ve noted here before. Hopefully the new regime will continue the moves away from targets. However, some experts, such as Professor Colin Talbot in Public Finance, are rather doubtful that a cull of targets will actually occur.

Maintaining trust: PIs, partnering and housing associations

According to the BBC, the Housing Corporation has launched an investigation after 5 Live Report alleged that it had uncovered evidence that a leading housing maintenance firm had falsified performance figures.

The case does raise some questions on how housing associations can be sure that performance information is reliable. I would certainly suggest that associations:

1. Periodically commission external validation of performance indicator systems whatever the future of social housing regulation is. (I declare an interest - I do PI system validation.)
2. Ensure that their audit committees (and, hence, their internal auditors) do consider the robustness of PI information - including PI collected by contractors and used to monitor partners

The alleged problems do make the case for associations (and other not-for-profits) creating a culture that stresses ethics both in within the organisation and in what is sometimes called the "extended enterprises" - not least suppliers and partners. Whistleblowing policies are a key part of this.

Good news on waiting lists - but lets not forget the flaws of targets in the NHS, housing, etc

It was welcome to read some good news about the NHS on Friday.

The NHS is on target to achieve the "historic goal" of eliminating long waits for treatment. By the end of 2008, no patient will have to wait more than 18 weeks from GP referral to hospital admission.

The chief executive of the King's Fund observed that the NHS had made "huge strides" forward. (Of course, it should do given the huge injection of resources.)

The Health Minister Andy Burnham was quoted by the Independent as saying: “I don't believe there should be further national targets for the NHS." (And certainly No 10 was thinking in terms of shifting the balance of public sector reform away from over-reliance on central targets.)

Let hope this is so. Targets have perhaps been a necessary evil – but they have harmful effects. They focus on what is measurable to the detriment of others concerns; they have unforeseen consequences (as with GP appointments); they encourage can encourage manipulation of data; they often centralise power in the hands of the target-setters rather than the customers etc.

One effect that I believe is often overlooked is a threshold effect. Where targets are set in terms of triggers or thresholds – as in waiting times less than 18 weeks – there is effort to get cases just over (or under) the threshold with cases failing the target getting relegated on To Do lists.

The classic case for me is responsive repairs waiting times for housing association. Generally emergency repairs are completed within 24 hours, urgent repairs one week and routine repairs one month. The average repairs times will be just inside the thresholds for the later categories. For the repairs that miss the target, the completion times will often be very long. If you are a housing association board member and don’t believe me – ask the managers at your housing association for the data.

Police, performance indicators and customer service

The claims from the Police Federation that Performance Indicators are distorting police work is hardly surprising.

The stress on PIs seen throughout public services is perhaps accentuated in the case of the police due to its peculiarities.

When I last visited my local police station (a minor motoring offence, I hasten to add), it wasn't open at the advertised hours and the bell/link to another local station that was (also) meant to be open just rang and rang. When I phoned to complain later, I discovered that the local constabulary had no mechanism for complaints unless it was about bent coppers. (Perhaps I was misinformed - if I was, thats not too impressive either.)

What other public service would consider normal or reasonable the absence of a feedback loop in the form of assessing and addressing complaints? Housing associations aren't only expected to have complaints processes - but they are meant to have one that does feed into performance improvement through organisational learning.

Surely there needs to be a focus on building a culture of customer service rather than just focussing (and perhaps manipulating) a set of PI "metrics".