At least one in 200 people in Britain – one of the wealthiest nations that has ever existed – is homeless. Last year there was the biggest rise in child poverty for three decades. Benefit claimants have had their state support stopped – “sanctioned” – for attending a funeral, for missing a jobcentre appointment after being hit by a car, and – in the case of one 60-year-old army veteran – for selling poppies for the Royal Legion, thus allegedly showing he wasn’t trying hard enough to look for work. One woman I met had to pay the bedroom tax after her daughter died: she now had a spare room, you see. These are random examples of the institutionalised cruelty of austerity Britain. You cannot understand the political turmoil now afflicting Britain without understanding these injustices. That’s not to claim that all of those battered by austerity voted for Brexit - they certainly did not – but as one academic study found this year, its impact was decisive in winning the vote for leave.

Which brings me to the Tory remainers now cheered at anti-Brexit rallies and championed by “centrist” commentators. All of this – all of it – is on them. Take Jo Johnson, who last made headline news when he attempted to appoint serial troll (and personal friend of his elder brother) Toby Young to a new government quango. Johnson was the architect of the 2015 Tory manifesto that committed to a referendum, which he still defends on crude partisan lines, as “essential for party management” and neutralising Ukip’s electoral threat. We can “provide a brighter and better future for all of us and all of our children”, he declared to adulation at an anti-Brexit rally. Yet here is a politician who has championed and voted for George Osborne’s austerity programme, which has caused the worst squeeze in wages since the Napoleonic wars, and left children facing a worse lot in life than their parents. A brighter and better future has been stolen away precisely because of his own political decisions.

There’s Ken Clarke, frequently venerated as a no-nonsense, progressive, pro-EU Tory. It is somewhat amusing that this affable chap – who introduced the NHS internal market and has extensively profited from big tobacco – is held up as an anti-Brexit warrior, but has now committed to backing Theresa May’s worst‑of‑all‑worlds deal.

Then there’s Anna Soubry, perhaps the most revered Tory amongst the “centrist” faction of the anti-Brexit movement. I’ve met and interviewed her: she was likeable, charming even, as we chatted, a portrait of Margaret Thatcher hanging over us. This week Soubry tweeted about the case of a young woman born brain-damaged, who with her grandmother risks losing their home because of benefit cuts: “No ifs no buts no excuses this is a disgrace & NOT what I voted for when supporting much needed benefit reforms.” Except this is exactly what Soubry voted for, over and over and over again. Soubry has voted for every single attack on the welfare state over the past few years: the bedroom tax, universal credit, the benefits freeze and cuts to disability benefits. She shares direct responsibility for forcing some of the most vulnerable people in society into misery, the cost of forcing them to pay for a crisis caused by the financial vested interests who fund her party. And in her own leave-voting seat, many of her constituents driven into hardship and insecurity – some of whom were voting for the first time – marched to the polling stations in the 2016 referendum, cast their eyes over a ballot paper they interpreted as asking them to reject or accept the status quo, and voted accordingly.

This year, she launched a passionate and much-lauded attack on the Tory Brexiteers, jabbing her finger in the Commons as she issued her j’accuse: “Nobody voted to be poorer, and nobody voted leave on the basis that somebody with a gold-plated pension and inherited wealth would take their jobs away from them.” Here was a Tory accusing her own of class war. And yet does the same not apply to Soubry? Is she not an MP living in immense comfort, with a gold-plated pension, who has used her power to drive her own constituents into poverty and hardship?

When John McDonnell declared this week that he could not be friends with a Conservative minister because of the suffering he had seen in his constituency, including people who had attempted suicide, he provoked predictable outrage. The physicist Brian Cox even suggested it showed a belief in creating “a one-party state”. It is striking how alleged violations of personal etiquette – in this case, choosing who you spend precious time socialising with – is regarded as more heinous than using your power to impose suffering on strangers. You can be a good sport in polite circles if you do the latter, but not the former.

And that is why the anti-Brexit movement has to make a decision. There are those, such as the campaign Another Europe Is Possible, who see stopping Brexit as intertwined with removing the injustices that drove the referendum result. There are others who see pre‑2016 Britain as a relative nirvana, disrupted by an abhorrent administrative error which simply needs to be corrected. But it was Tory MPs who collectively plunged Britain into calamity. To hold up Tory politicians who remain unapologetically committed to socially ruinous policies as progressive heroes just because they opposed Brexit shows a failure to understand why Britain is in crisis in the first place. Toryism has already waged the sort of class war Soubry describes, with catastrophic consequences. And they have to be defeated – whether they voted remain or leave – if Britain is to have any hope at all.

• Owen Jones is a Guardian columnist