A new study by Semken and Hay subtitled ‘The Use and Abuse of Expertise in Estimating the Economic Costs of Brexit’ can only be described as utterly damning of the approach taken by the UK Treasury in its modelling of Brexit in 2016. The abstract is hard hitting enough, citing ‘methodological issues, unrealistic assumptions, and misrepresentations of established facts’. But the detailed findings of the paper are, arguably, even more damaging for the Treasury’s credibility.
The paper takes the sensible approach of trying to reproduce the Treasury’s estimates and then correcting these estimates using more robust assumptions and data. It needs to do this because, as the paper notes, the Treasury relied on ‘a series of potentially problematic (and in some cases, entirely arbitrary) premises’.
A similar approach was taken by Briefings for Brexit contributor Graham Gudgin and his co-authors in articles in 2017 and 2018. Those studies similarly found major flaws in the Treasury’s work that greatly exaggerated the supposed costs of Brexit. The new paper covers some of the same ground but also discovers some new issues.
One of the key flaws in the Treasury’s modelling which Gudgin et al. identified was that the Treasury estimated the costs of Brexit for the UK by estimating the average impact EU membership had on trade across all EU members. But the UK is not like other EU member states. It has a much lower share of trade with other EU countries than the average and this share has been behaving very differently to that of the average EU member state in recent decades (falling, rather than rising). Using an average EU effect thus introduced a major upward bias into the Treasury estimates of Brexit costs. Semken and Hay make the same point and find that removing this bias alone more than halves the estimated impact of Brexit on UK trade. Gudgin et al also showed that the Treasury had been aware of this important point years earlier but conveniently ignored it in their work on Brexit.
A range of other methodological issues are also uncovered. The sample used by the Treasury for trade equations includes a range of developing and developed countries, which introduces a further large bias – partly because the UK is not an average world economy, but an advanced one, and partly because of measurement errors and missing data for many non-OECD countries. The FDI equation leaves out time fixed effects, described as a ‘major problem’. And there are several others.
Startlingly, Semken and Hay do not shy away from suggesting some of the Treasury’s methodological choices may have been taken to massage their results. For the FDI equation, they suggest ‘time fixed effects may have been left out to aid statistical significance’ and that an ‘aggressive deletion of outliers’ may have been undertaken for the same purpose. Similarly, the use of logs is not ‘an innocent strategy’ as it requires zero or negative data points to be deleted which can also bias the results.
Correcting a number of these methodological flaws, Semken and Hay present a variety of new estimates for the effect of Brexit on UK trade and FDI. The contrast between their results and those that the Treasury produced is very stark. While the Treasury claimed EU membership more than doubled the UK’s trade with the EU, Semken and Hay’s modelling suggests the increase might only have been 12%. Gudgin et al had estimated an increase of 24% which like Semken and Hay was only a fraction of the Treasury estimate.
Moreover, when (correctly) considering the UK as different from other EU states and comparing the UK to other OECD countries, Semken and Hay find that the supposed link between UK trade and EU membership becomes statistically insignificant. Similarly, in their new estimates, the link between EU membership and FDI into the UK, which was only weakly statistically significant in the Treasury estimates, is now found to be statistically insignificant.
The Treasury used their estimates of losses in trade and FDI to calculate an overall impact on UK GDP. They did this by feeding the estimates into the NiGEM macro-economic model developed by NIESR. However, if EU membership is shown to have no statistically significant influence on UK trade or FDI into the UK, there are no credible figures for trade and GDP losses to feed into a macroeconomic model. This means the Treasury’s claims of massive losses in UK GDP due to Brexit no longer hold.
Semken and Hay also have criticisms of how the Treasury simulated GDP changes from Brexit using the NIGEM macroeconomic model. They point out for instance that the Treasury make no assumption for changes in migration due to Brexit. Gudgin et all also point this out but also make a more important point about productivity. The Treasury only feed two factors into the NiGEM macro-economic model. One is losses in trade, the other is an assumption about reductions in productivity due to lower trade and reduced FDI. Work by Gudgin et all and by Western demonstrate that there is no observed link between trade and productivity among richer countries. Omitting the productivity link would halve the Treasury estimate of the impact on GDP.
In their conclusion, Semken and Hay are at pains to say they still think Brexit is bad idea economically. This is no doubt still the mindset of many economists, but at least these two authors have had the intellectual honesty to carefully dissect the bad modelling the UK Treasury used to try to convince the public to vote ‘Remain’ in the UK referendum in 2016. They also make a clarion call for economic expertise not to be misused in the way it was in the run-up to the referendum. We can only agree – the politicisation of the Treasury on the Brexit issue was scandalous.
Nor is this just a bit of interesting, but ancient, history. Although the Treasury has grudgingly accepted now that the ‘gravity model’ approach it took in 2016 had drawbacks, it has never withdrawn its results. And it continued to produce skewed analyses, using other approaches, after 2016. In addition, many other international bodies and academics produced copycat analyses that used essentially the same approach as the Treasury, getting similar results by exploiting the same methodological swerves and introducing the same statistical biases. To our knowledge, these other studies have not been withdrawn or corrected either.
Indeed, some of the worst aspects of the Treasury modelling are still being used in public discourse in the UK and in analyses by bodies such as the OBR. The most notable of these is the claim that Brexit will slash UK productivity growth and lead to a 4% drop in long-term GDP. As we have repeatedly argued (most recently here and here) this trade-productivity link is not justified by the academic literature or empirical evidence yet continues to be touted as an established fact.
So while the new paper by Semken and Hay is very welcome, it could have gone further still. The flaws they reveal in Treasury work from 2016 are only a subset of the problems with ‘official’ estimates of the costs of Brexit. Apart from dubious claims about productivity, other issues include arbitrary assumptions about future immigration to the UK, ludicrous claims about City of London job losses, and attempts to dismiss or ignore any potential upsides from Brexit either in the trade sphere (via freer trade with third countries) or from regulatory reform. Most recently, as incoming data has failed to show collapsing UK GDP or trade, a new front has opened up with some economists torturing the data in an attempt to show these variables might have grown faster if the UK were still an EU member. It is to be hoped that Semken and Hay’s new paper represents the start of a new attitude in economics in which these skewed and politicised analyses will be challenged more broadly than has been the case over the last six years.
