I got myself in a debate on Twitter last night about this question. Someone made the following statement about people in Scotland

the majority wants Trident out.

I responded

Scot Soc Att Survey – 59% either in favour of nuclear weapons or no view

To ‘fess up straight away I was wrong about 59%, the true figure is 53%, but that’s still a majority. I gave a link to the correct data online (it’s set out in detail below) and the full source is the excellent Scottish Social Attitudes Survey.

What happened next is worthy of some examination because there is a view around the independence referendum that people don’t have enough information available to decide. This particular exercise in correcting one small misapprehension led to the following exchange

HIM: nice manipulation of the data. Kudos

ME: Since I gave rational answer to yr prev point I assume ‘nice manip’n of the data’ isn’t directed at me

HIM: no you attempted to manipulate data to substantiate your opinion.

HIM: it isn’t a factual error…Out of those that have an opinion, the majority want it out

ME: Have to agree to differ then because I think ‘neither in favour or against’ *is* an opinion

HIM: not when you’re claiming majority by manipulating stats. Majority of those of opinion want it out

HIM: is that correct? yes or no?

ME: I can’t explain further but I do have a reasonable understanding of statistics. Good night.

So without the constraints of 140 characters per message of Twitter who’s right, ‘him’ or ‘me’?

Here are the statistics I was referring to, courtesy of ScotCen Social Research:

Trident table

Click to enlarge

Source: Table in Scottish Social Attitudes Survey 2013

The first thing to say is the question asks whether Britain should have nuclear weapons, not Trident specifically. But since Trident missiles are the only nuclear weapons Britain possesses it’s a reasonable approximation. It should also be noted that the Scottish Social Attitudes Survey is a reputable, reliable and statistically valid source of opinion on the subject matter it covers. I know of no other up to date neutral source that addresses the same issue.

The nub of the difference between my interlocutor and me is whether people who answered ‘neither in favour or against’ should be included in the calculation of the percentage of people ‘against Trident.’ I say yes because to be neither for nor against is to express a view. Moreover, even a survey of this high quality is a relatively blunt instrument at catching the full subtlety of people’s opinions. So I could easily imagine a whole range of views underlying an opinion that someone is neither in favour nor against Britain having its own nuclear weapons. For example

  • You know, I couldn’t care less. I’ve got more important things to worry about
  • Well, I can see things for and against. It’s a fine balance
  • It’s not really relevant to defence these days but if the experts want it…
  • and so on.

In any event, the statement originally made was that ‘the majority wants Trident out’, not ‘the majority excluding “don’t knows” and those “neither in favour nor against” want Trident out’ – as the other person concerned amended his claim to when challenged. These are two quite different things.

To put it another way, if you lined up 100 Scots and said ‘Will everyone who is somewhat or strongly against Britain having nuclear weapons please step forward?’ 46 would. That’s a minority.

This sort of detail is important because it’s the only way to tease out the claims and counter-claims that accompany the independence referendum debate.

Incidentally, the question of Scotland being different from the rest of the UK features prominently in ‘Yes’ claims about the independence referendum. It is interesting to compare the results of the same question asked in the Scottish Social Attitudes Survey’s sister survey south of the border (the small percentages of ‘Don’t know’s have been excluded from this table).

Trident UK

As the authors of this comparison say

The differences in the level of support are not that large, and both parts of the UK could reasonably be described as being divided on the subject (the full report can be downloaded here).

To go back to the original issue, I maintain that there is not a proven majority of people in Scotland who ‘want Trident out.’ But I’m open to reasoned arguments that prove the opposite.

 


In my recent post on How to spot dodgy research about the public sector I mentioned publicity about potential procurement savings published by a company called Opera Solutions (OS) “to which I shall probably return”.  Well this is it – the return.

I’m uncharacteristically cross with myself because I’ve established a modest track record in commenting on stuff like this (see the footnote to the post mentioned above) but this is one I missed first time round.  Still, a reprise of what others have already said and a few additional comments are not out of order.

The first hit today on Google about this so-called “research” takes you to a press notice by the UK government department of communities and local government (DCLG) back on 17 June.  Headlined Shining a light on council spending could save up to £450 per household it’s worth quoting

…cutting edge analysis of council spending data by procurement experts Opera Solutions has revealed that greater transparency coupled with improved analysis is the key to unlocking massive savings by driving down costs…The report argues that Local Government, by adopting new processes and making better use of spending analysis, could replicate these kind of savings across a wide range of back office functions, with no impact on quality of service and reduce spending by up to £10 billion a year. 

And secretary of state Eric Pickles added approvingly

“Let there be no doubt whatsoever – today’s figures show that there is significant scope for councils to make taxpayers’ money work even harder. We’ve always said that sunlight is the best disinfectant, and the availability of financial data has helped identify numerous ways that councils can reduce expenditure while offering the same or better services to residents”

The nature of Opera Solutions’ cutting edge analysis was exposed by Ben Goldacre in The Guardian newspaper.  He said

Every now and then, the government will push a report that’s so assinine, and so thin, you have to check it’s not a spoof.

Goldacre went on to explain

The meat of it, the analysis, is presented in a single three-line table. Opera took the recently released local government spending data for three councils, and decided how much it reckoned could be saved by bulk purchasing.

[Opera Solutions] did its estimates on three areas: for energy bills (a £7m spend), and solicitors fees (£6m), it thought councils could save just 10%. The third category – mobile phone bills – were tiny in comparison (just £600,000) but here, and here alone, Opera reckons councils can save 20%, by getting people on better tariffs.

He then did a hatchet job on the whole, my word, sham.  I won’t steal his thunder but click through and read what he says.  It exemplifies precisely the ten infallible signs of dodgy research I posted recently.

Predictably, the Daily Mail had already waded in uncritically to condemn the alleged waste by “clueless councils”.  But others, like the PublicNet web site, acknowledged Opera Solutions’ work wasn’t all it seemed.  They admitted

Publicnet is among a number of media organisations that published this story in good faith.

And on the WhatDoTheyKnow web site you can find a number of items relating to Freedom of Information requests to the DCLG from one Edward Rudolf.  The DCLG seem to be finding excuses to not answer his questions but he has a keenly forensic approach to the subject and you can sense he’s not going to give up.  I don’t know him but more power to his elbow.

I’m not sure what I can add to all this but here are a few tit bits.

  • For a company seeking publicity about their work, Opera Solutions are remarkably coy about letting people see their “research”.  On their web site you’ll find you have to give your contact details to access what they optimistically call one of their White Papers.  Ironic in light of  Eric Pickles’ statement that “sunlight is the best disinfectant”.  Somehow Ben Goldacre managed to open up access to the report on the OS web site so despite their coyness you can find the whole six page magnum opus here (but don’t hold your breath).
  • Also of passing interest on the OS web site is their characterisation of this PR-dressed-up-as-research as “Prepared for the Government of the United Kingdom”.  Sounds impressive doesn’t it?  I’ve prepared many things for the Government of the United Kingdom.  They never asked for them and, unlike the OS effort, I doubt if they took any notice of them.  Let’s hope Edward Rudolf’s FOI requests tease out whether this work was commissioned or unsolicited.
  • A general lesson for all of us from this unfinished saga is a reminder of how major parts of the media publish PR guff as news without engaging their brains to analyse the latest press release that suits their prejudices or fills a last minute space.

As I’ve said before, the sadness of this sort of stuff is that mud sticks.  Once again people without full access to the facts will find yet another reason to believe the public sector in general and local government in particular needs a good shaking up.

PS – this doesn’t purport to tell the whole story on this subject.  There is a lot more out there in cyberspace including a critique by local government lawyers of the OS publication, as well as a small admission by them that their claimed savings for local government were “extrapolated” [and some]


Regular readers will know I’ve criticised dodgy research about the public sector on a number of occasions*.  Recent publicity about potential procurement savings published by a company called Opera Solutions, to which I shall probably return [I did], prompts me to share the 10 infallible signs of dodgy research, of which there seems to be an increasing amount.

  1. It’s carried out by an organisation hoping to sell something
  2. It says it’s based on a sample of public agencies/government departments/local authorities/NHS bodies/managers but it doesn’t tell you anything about the sample – how many, how they were chosen or their characteristics
  3. Where data is quoted the source is vague or unstated
  4. The organisation concerned wants you to contact them via a PR company
  5. As much money seems to have gone into the presentation and production of the report as the content
  6. The organisation has little or no track record of working with or for the public sector, or indeed has shown no previous interest in the sector
  7. They also have little or no track record of working in the UK
  8. You can find a press release about the research but the only way to get a copy of the research itself is to give your details through a web site
  9. A parliamentarian sympathetic to the interests of the organisation concerned almost immediately takes up its cause uncritically, quoting the conclusions as if they were proven facts
  10. If whatever the research promises (savings, improved service, greater efficiency) looks too good to be true it is.

Have you got any tell-tale signs of dodgy research about the public sector you’d like to share?  Let us know.

* See for example:


Bang on cue after the loaded and simple truths of think tank Policy Exchange’s report on public sector pay

public sector pay is higher and continued to grow faster than private sector pay during 2010

comes more objective and up to date evidence for what I’ve always claimed

the cycles [of pay in the two sectors] don’t move in the same direction at the same time

Not for the first time the evidence comes from the authoritative work of IDS (formerly Income Data Services) who earn their living charting trends in remuneration throughout the economy.  So they have to get it right and are not beholden to any partisan view of matters economic or political.

Their latest analysis of pay settlements across the UK economy states

A clear gap has emerged between the level of pay awards in the public and private sectors…The median pay award for the private sector in the three months to the end of April is 3 per cent, while in the public sector it is zero

This is in stark contrast to the Policy Exchange’s claims.  Let’s hope that if either of these organisation’s work has an influence on public policy it’s the more objective analysis of IDS.


…but it pours.

If there’s one thing that’s almost as bad as the faux survey carried out by PR people for companies seeking to generate publicity (see my post on Public sector workers unemployable – shock, horror), it’s the research carried out by think tanks seeking, er, to generate publicity.

This last week brings the latest example to knock the public sector, from the Policy Exchange (“David Cameron’s favourite think tank”) – Public and private sector terms, conditions and the issue of fairness.

This effort was widely reported in the media over the weekend.  Google News lists 142 articles so far.

The premise of the research is that

public sector pay is higher and continued to grow faster than private sector pay during 2010 and that significant reforms will need to be made to limit job losses in the public sector and to achieve equity and fairness in the labour market.

As with many of these things the conclusions and the inevitable recommendations (end national strike balloting in the public sector; replace the two-year pay freeze with a paybill freeze; reform public sector pensions) are reached after a cursory trot around a complex series of data sets.

Without spending a lot of time on the detail and seeking the views of those better qualified than me in the field of labour market statistics, I don’t have a snowball’s chance in hell of providing a reasoned critique of this work on the day (as I write) that this came to my attention thanks to the BBC Radio 4 Today programme.

In the meantime, of course, 142 – as we know thanks to Google – mainly UK media outlets have summarised and punted the findings in a mainly uncritical way, with the main countervailing viewpoint coming from the TUC who are just about as objective as the Policy Exchange, but in another direction.

And it’s out there in the public consciousness.  That vague feeling that yet again the public sector has been proven to be (choose your adjectives) bloated, inefficient, out of touch, unsustainable, etc. etc.

It’s a shame because reform, in all sorts of things, is needed in the public sector.

It’s just that a first examination of this particular work doesn’t provide the conclusive evidence that would lead to the particular proposals the Policy Exchange comes up with.

For example

  • Much of the analysis is based on a remuneration premium the authors claim public sector workers enjoy compared with the private sector.  Put simply this is a higher pay rate per hour for similar work
  • The main comparisons are made for 2009 and 2010 when the gap in pay rates between the two sectors is said to have widened for all levels of pay (except interestingly at the highest managerial level where the private sector has the edge)
  • Tucked away towards the end of the publication, and uncommented on for this purpose, is a table showing longer term trends in the public/private sector pay relationship between 1997 and 2010 demonstrating that between 1997 and 2003/04 the private sector enjoyed the pay premium.  This is followed by the authors’ own projections which show that if the current public sector pay freeze is maintained the current public sector premium will decrease and give way (or return) to a private sector premium by 2016/17, i.e. in five years time.  Which suggests there is some truth in my own conclusion that public and private sector pay relativities are both cyclical but at different times (see for example my earlier post on New year, hard times – public sector pay revisited)
  • In order to make sense (or not) of the analysis, you’ll need without much help from the authors to get to grips with some subtle statistical definitions like median gross, mean, median, hourly median, nominal change, and 10th, 25th, 90th etc. percentile
  • You’ll also have to take on trust that when they compare particular categories of worker in the two sectors, for example, housekeepers and related occupations, therapists, and security managers they are comparing apples with apples and not apples with pears
  • Interestingly, taking the comparability of these occupations at face value the evidence is not always as conclusive as the authors suggest.  For example, the median annual pay for sports and leisure attendants in 2010 was all of £162  more in the public sector, a whopping 1.6%, while the median pay of electronics engineers was actually a modest £3,500 or 8% higher in the private sector. Am I getting my percentages confused here or are they?
  • Some of the occupations exemplified do not allow easy comparison.  For example, how many policemen (sergeant or below) (sic) are employed in the private sector?

Was there ever a think tank with a political or ideological affiliation that found evidence to contradict its fundamental beliefs?  There was a reason that Private Eye many years ago dubbed the first ever Prime Ministerial think tank, “Ted Heath’s wank tank”.  Crude but telling.


Check out the comments on this post – many thoughtful contributions from the Local Government Improvement and Development web site are gathered together under the comment in my name

I’m getting fed up with dodgy research by private companies that draws unwarranted conclusions about the public sector from poor data.

An obscure column in the business pages of the Saturday Scotsman newspaper drew my attention to “research” by a company called uSwitchforBusiness.com which purported to show that

Poor perception of public sector workers puts small businesses off employing them

uSwitchforBusiness.com offers a “free, impartial energy brokerage service…focus[sing] on helping  business customers get the best gas and electricity contracts” receiving “a small commission payment when a customer chooses to switch or apply for a product through us”.  An interesting definition of impartiality but we’ll let that go.

Why they should waste their time hammering the poor old public sector workers being made redundant in their 1000s just now only they know.

They make their claims on the basis of

a uSwitchforbusiness.com poll, undertaken online with 240 key owners and financial decision makers of businesses with between 1 and 50 employees during January 2011.

The “findings” include

only 2% of SME employers would actively seek to recruit public sector workers

almost a quarter of small businesses (23%) would only consider a public sector worker if it was a role that they couldn’t otherwise fill – one in ten (12%) wouldn’t be prepared to employ a public sector worker at all

55% of SME owners believe that public sector workers have unrealistic expectations about pay, holidays and employment terms while just 11% consider public sector workers to be as productive as private sector employees

only 6% of private sector employers agree that public sector workers would fit in well in their company.

They have not published detailed results so we know nothing else about how the survey was carried out

  • how they chose the sample – whether it was random (and therefore some statistical reliability could be assigned to the results), what sectors of the economy it covered (hairdressers and florists might have a different view and level of understanding than accountants and management consultants), and whether the respondents were all uSwitchforBusiness.com clients
  • why it was of 240 businesses, an unusual number to survey and a very small sample for statistically reliable conclusions to be drawn from the results.  Latest government figures show that there are 1,187,275 enterprises employing 1-49 people.  240 is a sample of 0.019% of the total
  • what questions people were asked and the options they were given to answer them (their press release mentions some but it’s not clear if they are all the questions asked and options given)
  • why they’ve confined the survey to businesses employing 1-50 people.  They use the acronym “SME” in their press release, incidentally without explaining what it means.  The official definition of an SME (small and medium sized-enterprise) is 0-249 employees
  • the sizes of the businesses who responded.  A 1-person business might tend to have a different view from one employing 50 people
  • where in the UK the sample came from.  There could be significantly different views amongst small businesses in different regions
  • it’s not clear who was actually surveyed – the phrase “key owners and financial decision makers” has little meaning for a very small enterprise and the difference between a key owner and any other sort of  owner eludes me
  • the research credentials of those undertaking the survey are not known.  No independent research organisation is mentioned in the press release that forms the only source of information about the survey, although two members of staff of a public relations company are.

Some wider questions are also relevant

  • have any of the respondents actually worked in the public sector?
  • do they have any employees or close relatives who have?
  • when they completed the survey were they even considering what the public sector encompasses – not pen-pushing bureaucrats but teachers, firefighters, nurses, social workers, soldiers, customs’ officers, prison warders, librarians, judges…?

I’m not knocking small enterprises – I run one – just this survey, its opaqueness and its unwarranted conclusions.

As to what the results of the survey actually mean, consider just one conclusion

almost a quarter of small businesses (23%) would only consider a public sector worker if it was a role that they couldn’t otherwise fill – one in ten (12%) wouldn’t be prepared to employ a public sector worker at all.

 

To put it another way 67% (two-thirds) of the small businesses surveyed either would consider a public sector worker regardless of the role they are filling or have no view on the subject.  And the 33% who wouldn’t are only 80 out of 240, or if you prefer 80 of the total stock of 1,187,275 UK businesses of this size.  Incidentally, don’t you love that one in ten (12%)? Something wrong there I think.

Put like this the headline “Poor perception of public sector workers puts small businesses off employing them” isn’t quite so impressive.

Note also how in the presentation of their conclusions a sample of “240 key owners and financial decision makers” become “SME employers… small businesses…SME owners.. [and finally] private sector employers”.  These are not the same things.

In a related blog, the company claims

The government may need to rethink their employment strategies in light of the research.

I think not.

In short this is not serious research but a PR puff designed to draw attention to uSwitchforBusiness.com.  The shame of it, as with all endeavours of this sort, is that myths about public sector workers are perpetuated and extended and real people – redundant workers looking for a new job – can be seriously damaged.

Footnote – the uSwitchforBusiness.com survey bears an uncanny resemblance to a publication by another company in 2010, again PR puff dressed up as serious research


“The UK’s councils could do the same amount of work with 500,000 fewer staff if they matched the productivity of private firms, a report has claimed.”

This is the first sentence of an item on the BBC news web site today. Their You and Yours programme on Radio 4 also carried an interview with the report’s author, consultant Paul Weekes of a company called Knox d’Arcy.

This is a great topic worthy of serious examination.

The first thing to say is that it’s difficult because although the research has been widely publicised in the UK media this last week, none of the references actually provide a link to the report itself.  The BBC for one is usually meticulous about this.

An obvious place to find the report would be the Knox D’Arcy web site.  Unfortunately, it does not have a “News” section or make any reference to the report anywhere.

So my less than enthusiastic (as you will discover) look at this has to be based on media reports.  The first three thrown up by Google were the BBC, the Belfast Telegraph and the Conservative Home web site.  Others were listed but added little by way of more information.

The central claim in the report is that

junior staff in local authorities were, on average, productive only 32% of their time during working hours…compared with an average of 44% in the private sector – BBC web site

This conclusion seems to be based on 1,855 what the BBC call “workers’ surveys”.  The Belfast Telegraph more helpfully explains that the 1,855 surveys were of managers and supervisors, including 173 from local government officers.

There is no information available of how Knox D’Arcy defined a local government officer, which councils or even sorts of council they came from, or when the surveys were carried out.

There is no information about what sorts of organisations the other 1,682 managers and supervisors came from, or again when.  Knox D’Arcy claim an international clientele, almost exclusively private sector, so the comparators could be in the UK or elsewhere.  They could be manufacturers or service providers.

There is no definition of what Knox D’Arcy regard as “productive time”.

Although it may be the BBC’s choice of phrase (if Knox D’Arcy choose not to publish their full research how can we know?) we do not know what they mean by “junior staff”.

All this makes it extremely difficult to judge whether these claims are valid but two thoughts come to mind.

  • 173 “surveys” is a painfully low number to draw any conclusions from for a sector that employs 2,900,000 (Report on the Triennial Review of the Quarterly Public Sector Employment Survey, Office of National Statistics 2008).  On the arbitrary assumption that 1 in 10 might be a manager or supervisor that is a sample of 0.6%.  We do not know if they were chosen randomly, which would be the only sure way some statistical confidence could be assigned to the sample’s characteristics
  • the difference in productive use of time between private sector and local government is said to be 12% (44% – 32%). If 2,900,000 people could be made 12% more productive that would mean, crudely and all other things being equal, that their work could be done by 2,550,000 people, 350,000 fewer people not 500,000

Finally, we do not know if the report was commissioned by a client or whether Knox d’Arcy carried it out on their own behalf.

None of these points individually invalidates the report.  But taken together the best you could say about it is that pending further information, the jury has to be out on its conclusions.

The sadness, as so often, is that the media have taken up what is presumably a press release and reported it almost completely uncritically.

Footnote 21 August 2010 – this post refers to the apparent unavailability of the report on which the conclusions reported above are based.  Overnight a note has appeared on the Knox D’Arcy web site saying The research into public sector productivity will be available as a down load from this site when the report is released at the end of August.  I will return to the subject once it is available.