The former Australian prime minister Kevin Rudd says three of the world’s biggest mining multinationals have run sophisticated operations to kill off climate action in Australia and continue to wield day-to-day influence over government through a vast lobbying network and an “umbilical” relationship with the Murdoch media.

Australia’s climate policy paralysis has been ongoing for more than decade, from the Rudd Labor government’s doomed attempt to bring in an emissions trading scheme in 2008, to the Liberal prime minister Malcolm Turnbull’s failed attempt to introduce the national energy guarantee last year.

Successive attempts at serious climate action have faced fierce resistance, often led by industry lobbying, and have contributed to the downfall of three recent prime ministers: Rudd, his successor Julia Gillard and Turnbull.

The Guardian has collaborated with leading scientists and NGOs to expose, with exclusive data, investigations and analysis, the fossil fuel companies that are perpetuating the climate crisis – some of which have accelerated their extraction of coal, oil and gas even as the devastating impact on the planet and humanity was becoming clear.

The investigation has involved more than 20 Guardian journalists working across the world for the past six months.

The project focuses on what the companies have extracted from the ground, and the subsequent emissions they are responsible for, since 1965. The analysis, undertaken by Richard Heede at the Climate Accountability Institute, calculates how much carbon is emitted throughout the supply chain, from extraction to use by consumers. Heede said: "The fact that consumers combust the fuels to carbon dioxide, water, heat and pollutants does not absolve the fossil fuel companies from responsibility for knowingly perpetuating the carbon era and accelerating the climate crisis toward the existential threat it has now become."

One aim of the project is to move the focus of debate from individual responsibilities to power structures – so our reporters also examined the financial and lobbying structures that let fossil fuel firms keep growing, and discovered which elected politicians were voting for change. 

Another aim of the project is to press governments and corporations to close the gap between ambitious long-term promises and lacklustre short-term action. The UN says the coming decade is crucial if the world is to avoid the most catastrophic consequences of global heating. Reining in our dependence on fossil fuels and dramatically accelerating the transition to renewable energy has never been more urgent.

The mining sector’s power to influence Australian politics is infamous. Separate to climate policy, it led an intense campaign against the resource super profits tax – a 40% tax on mining profits – proposed by Rudd in 2010. The industry’s damaging $22m lobbying effort helped depose Rudd and gut the super profits tax, rendering it near useless.

Experts and environmentalists say the industry’s power is derived from political donations, gifts, advertising, paid lobbyists, frequent access and close political networks. Rudd said he felt the full brunt of that influence.

“Glencore, Rio [Tinto] and BHP ran sophisticated political operations against my government, both on climate change and the mining tax,” he told the Guardian. “They worked hard … to get rid of the resource super profit tax, against the interests of other mining companies and the national economy as a whole. They worked hard … in 2013 against the carbon price. They succeeded in both enterprises.”

Rudd attributes the day-to-day influence of the sector to two mechanisms. The first is what he describes as the vast lobbying network it uses to pressure political parties. The second is its close relationship with the Murdoch media, which owns most of the country’s print media. Rudd describes the relationship as “umbilical”.

“When did you last see the Murdoch media critical of any of these corporations?” Rudd said. “Rarely. If ever.”

BHP, Glencore and Rio Tinto, and the Murdoch-owned News Corp, all declined to comment.

The Melbourne University academic George Rennie, an expert on lobbying, sees the mining sector as one of the most powerful groups in Australia. Its power, he says, manifests in an ability to get senior politicians on the phone or to meet face-to-face with relative ease.

“Its ability to access decision-makers is famous, and the experience of the ‘mining tax’ lobbying campaign in 2010 has made both parties afraid to take on mining interests,” Rennie said. “The power of the resources sector comes from its profits – its ability to spend on donations and gifts, as well as its own political advertisements if it chooses.

“There is disproportionate power, so if you want to lobby government for something that the resources sector does not want, you’re very unlikely to get your way.”

The scant publicly available information on political access in Australia backs up Rennie’s view. A Guardian analysis of ministerial diary data compiled by the Grattan Institute thinktank shows the mining and energy sector enjoys frequent access to government leaders in two of Australia’s most important mining states: Queensland and New South Wales.

Mining and energy industry executives met the Queensland premier, deputy premier or treasurer 44 times in 2017-18 alone. That figure excludes renewables companies and energy retailers, and does not count meetings with lesser ministers, backbenchers or bureaucrats.

Access to the highest rungs of power was also significant, albeit less frequent, in NSW. There, the sector secured access to the NSW premier, deputy premier or treasurer 13 times in 2016-17.

The Grattan Institute data shows the mining sector is responsible for the highest proportion of major donations to political parties. Mining interests accounted for one in five donations above $60,000 in 2015-16 and 2016-17.

The Grattan Institute senior associate Kate Griffiths has researched lobbying, donations and political influence in Australia, helping to publish a report last year titled Who’s in the Room? She said the sector differed from others because of its high uptake of commercial lobbyists and its pursuit of face-to-face meetings with ministers.

“There’s also a fair bit of exchange of staff between mining and energy companies and political offices,” Griffiths told the Guardian. “Mining and energy companies have the resources to hire former politicians and political advisers so that can create a certain cosiness that makes both access and influence more likely.”

The revolving door between politics and the mining industry is a feature in both of Australia’s major parties. Brendan Pearson, a former head of Australia’s peak mining lobby group, the Minerals Council of Australia, recently joined the office of the Liberal prime minister, Scott Morrison, as a senior adviser.

On the other side, Gary Gray, a former Labor resources minister, joined the mining industry in his post-political career, including working for the mining services company Mineral Resources Limited.

Martin Ferguson, another former Labor resources minister, is now the chairman of the advisory board to the oil and gas peak industry body, the Australian Petroleum Production and Exploration Association.

Cameron Milner, a Labor strategist, lobbied for the Indian mining giant Adani for five years, before eventually parting ways so he could work on the campaign of the Queensland premier, Annastacia Palaszczuk, in 2017.

The industry also exerts influence in less obvious ways. An investigation by Guardian Australia this year revealed a covert, multimillion-dollar influence campaign being run by C|T Group, the lobbyist firm headed by Lynton Crosby, on behalf of Glencore.

The campaign, known as Project Caesar, included the creation of fake online grassroots groups that spread anti-renewable, pro-coal content to unsuspecting users. It aimed to shift the emotion and tone of the energy debate in Australia to favour coal by using personally relevant messaging, centred on themes of cost, reliability and family security.

Campaign teams also collected material that could be used to embarrass activist groups such as Greenpeace and 350.org.

Dom Rowe, Greenpeace Australia Pacific’s programme director, said Australia’s lack of credible climate policy could be directly attributed to the industry’s influence.

“Considering the vast network of influence and direct access that fossil fuel executives and lobbyists have to senior government ministers, it is little wonder that the Coalition has yet to take any meaningful action on reducing Australia’s greenhouse gas emissions, which have been rising for over four years,” Rowe said.

“It’s completely unacceptable that both major parties are abandoning their responsibility to protect Australians from the worsening impacts of the climate crisis, in favour of not biting the hand that feeds.”