It was Michael Gove who before the Brexit referendum said “people in this country have had enough of experts”. The highly educated Mr Gove was mining a rich seam of voters fed up with, and disregarding of, expert opinion. Brexiters have continued in this pejorative style. Only last week the foreign secretary, Boris Johnson, reportedly gave a terse and pungent imprecation to diplomats who raised the issue of companies doubting his wisdom about the UK leaving the EU without a trade deal. “Fuck business,” Britain’s top diplomat replied undiplomatically.

In fact both rabble-rousing Brexiters and experts have more in common than either would admit. Populists claim to have a special insight into the will of the people, able to dispense with debate and discussion. Hence Mr Johnson warning prime minister Theresa May against a “bog-roll Brexit” that was “soft, yielding and seemingly infinitely long”. Technocrats also argue it’s necessary to insulate policies from political challenge. They want more independent agencies to take over arms of the state. This unfortunately has captured thinking in the UK, where the last few decades have seen a steady growth in the number of agencies, commissions and regulators which draft legally binding rules. These bodies provide a way for politicians to look as if they are doing something while allowing them to duck tough decisions until they cannot. Just look at public sector pay, which could only apparently be raised via independent pay review bodies – until politicians under pressure decided they were unnecessary.

In an age of subdued economic growth, wage stagnation and a concentration of wealth among the very richest, it is not complete fantasy, as the former Bank of England official Paul Tucker, in his latest book Unelected Power, puts it, to see our democracy “flirting with a peculiar cocktail of hyper-depoliticised technocracy and hyper-politicised populism, each fuelling the other in attempts to maintain effective government and to re-establish majoritarian sensibility”. Mr Tucker’s hyper-depoliticised technocracy hovered into view last week when the head of Ofsted, Amanda Spielman, responded in Govian terms to claims her inspection system is biased against schools serving white working-class communities. She claimed poorer, white students lacked the drive of migrants. Ms Spielman is a public servant who has the right to judge schools and teachers but not social groups. Continuing to do so risks her legitimacy with audiences that she needs.

Unelected power is not new. Democracies have found ways of accommodating the military and the judiciary. Central banking has become part of that story. These power centres have realised that they cannot act in too overmighty a way, that circumstances and politics matter in a democracy and that people’s livelihoods cannot be sacrificed on the altar of intellectual purity. Over the years they have understood how important it is to respect voters and the deepest values that guide society. They did so through challenge. For an administrative state to truly work it will need councils to be reinvigorated, along with trade unions, cooperatives and civil society operating on the basis of more than single issues. Democratic societies must give people a voice as well as a vote. The public ought to be able to dissent on the grounds of their human, economic and social rights. If we forsake this approach then only two powers will remain: the state and the market. That is what the hyper-populists of politics and technocracy would want. Don’t let them have it.

• This article was amended on 25 June 2018 to correct “hyper-politicised technocracy” to “hyper-depoliticised”.