It would be far-fetched to describe the US and Britain’s long-term relationship with Saudi Arabia as a love affair, although elements of romance, blind infatuation and lustful mutual gratification have never been entirely absent.

But what has become painfully clear from the furious row over the sycophantic official reaction in Washington and London to the death, this month, of King Abdullah is how much the relationship has changed, at least on the west’s side of the bed.

It is as if, after unquestioning decades of cohabitation, one partner woke up one morning with a jolt and blurted out: “I don’t have much in common with you any more, if I ever did. Truth be told, I don’t really like you, and what’s more, I don’t need you. In fact, I find you really annoying.”

As is common in such awkward situations, the outward trappings of the relationship appear undisturbed, for now at least, as shown by Barack Obama’s homage to Abdullah in Riyadh on Tuesday. But the ties that bind are shredding. In reality, the magic and meaning have fled.

In fact, the whole unreformed Saudi-west situation grows ever more embarrassing – and is thus ever less likely to endure intact.

Intent on offering his condolences and meeting Abdullah’s successor in person, Obama led an exceptionally high-powered delegation to Riyadh that included former secretaries of state, past presidential candidates and senior military commanders. Similarly subservient, Britain had already sent David Cameron and Prince Charles.

Yet when asked to justify this level of attention and, for example, the flying of flags at half-mast on government buildings, Downing Street was hard put to explain its stance. Saudi Arabia was an important ally and economic partner came the muttered reply from No 10, and others. To act otherwise would have been “aggressive” and impolite. A legion of critics vociferously disagreed.

This kneejerk diplomatic kowtowing, embedded in the thinking of a cold war, 1980s world that no longer exists, looks increasingly anachronistic and warrants close scrutiny. All the main policy planks underpinning the Saudi relationship are, more or less, under challenge.

Take oil, Saudi Arabia’s economic lifeline, and the main reason it has been so assiduously courted in the past. Saudi remains the world’s biggest oil producer and exporter, and the country with the largest proven reserves. It leads the mainly Middle Eastern, 12-member Opec cartel (Organisation of Petroleum Exporting Countries), and has unique influence over the global oil price.

But times have changed – as has the geopolitical power balance. Saudi Arabia’s refusal to cut oil production last year is a principal reason for the current, dramatic fall in prices.

Riyadh appeared to be trying to undermine shale-oil production in the US, which was reducing its export earnings. But the ploy did not work.

According to Citibank, shale oil and new Arctic oilfields may see US production doubling to 14.2m barrels per day (bpd) by 2020. That could leave the US free to become a net exporter, to the tune of 4.7m bpd of oil and LNG. The current US need to import 2m bpd, much of it from Saudi Arabia, may soon simply evaporate.

Increased supplies of oil from non-Middle East countries, such as Angola, coupled with the advent of alternative green-energy systems, improved conservation, and heightened awareness about the need to curb carbon emissions, are also reducing Saudi leverage.

In short, it’s possible the west will just not need the Saudis any more.

Away from commodity prices, increased western focus on human rights around the world, combined with the greater transparency forced on the kingdom by the globalisation and enhanced reach of conventional, digital and social media, has placed the Saudi record under unprecedented scrutiny.

Riyadh must endure unceasing bad publicity about cases such as that of the Saudi blogger Raif Badawi, sentenced to 1,000 lashes, and the recent street beheading of a Burmese woman. Public pressure means western governments are forced to take notice of these concerns in new ways. Lack of women’s rights is another hot topic, previously skated over, but no longer possible for US and British politicians to ignore.

The relationship with the west has survived several wars between Israelis and Arabs, in Afghanistan and in Iraq (twice); the chilling predominance of Saudi nationals in the 9/11 attacks and the rise of al-Qaida; serious bribery and corruption scandals and diplomatic rifts; recurring oil crises; deepening concern over Saudi funding for extremist religious teaching and its links to terrorism; escalating rows about egregious human rights abuses and the repression of women, and most recently, the Syrian calamity and the ascendancy of the black-shirted head-cutters of Islamic State.

But it has survived at what cost? For many in Britain and the US (which, post-1945, gradually assumed Britain’s geostrategic role in the Arabian peninsula, as elsewhere), the rationale binding western interests so closely to the Saudi state is no longer obvious, persuasive, welcome or easily justified.

Writing days before Abdullah’s death, the American author Stephen Kinzer warned that the basis of the west’s relationship with the Saudi regime was shifting in fundamental ways, while Saudi Arabia’s position in a region beset by insurrection and civil war was ever less secure.

“The most intriguing candidate for collapse is Saudi Arabia,” Kinzer wrote. “For more than half a century, Saudi leaders manipulated the United States by feeding our oil addiction, lavishing money on politicians, helping to finance American wars, and buying billions of dollars in weaponry from US companies. Now the sand is beginning to shift under their feet.

“After [King Salman, Abdullah’s successor, departs the scene], a power struggle within the royal family is likely. No one can say how intense or violent it might become, but the prospect of crisis comes at an especially bad time. The region is afire and oil prices are plummeting. It would be foolish to bet that Saudi Arabia will exist in its current form a generation from now.”

Memories of how the Saudis and Opec deliberately triggered an economic crisis in the west in retaliation for US aid to Israel during the 1973 Yom Kippur war still rankle. Manipulation of the oil price for political ends has been a common occurrence since. In 2008, as the world financial crisis hit, former US president George W Bush personally requested Abdullah to cut prices, and was flatly rebuffed.

The mood regarding human rights has also changed. In 1980, a British television documentary, Death of a Princess, based on the true story of Princess Misha’al and her lover, who were publicly executed for adultery, led the Saudis to expel Britain’s ambassador and impose sanctions, much to the London establishment’s discomfiture. Many countries bowed to intense Saudi pressure not to broadcast the film. Nowadays such bullying is not so easy.

Yet while the external environment has altered radically, inside Saudi Arabia itself, as campaigners testify, little, if anything, has changed. Intolerance of dissent, be it political, religious or ideological, remains almost total. Saudi jails are crowded with those whose only crime is to speak freely.

Curbs on women’s rights have not relaxed significantly, despite promises dating back to the 1990-91 Gulf war, when Riyadh was running scared of Saddam Hussein’s Iraq and cried out for western help. If anyone had believed King Salman would take a different tack, those hopes were quickly dispelled when, on Monday, the first public beheading took place under his reign.

The growing gulf between Saudi Arabia and its more sceptical western partners is nowhere more apparent than in the key area of security and defence cooperation, upon which the relationship was founded in 1915. The west has long viewed the Saudis as a pillar of stability in an unruly region. But Saudi policy since the 1980s has repeatedly given the lie to that over-hopeful analysis.

It was the Saudis, principally, who (encouraged by the US) funded the mujahideen in Afghanistan in their fight against Soviet occupation. But it was also the conservative Wahhabi Sunni Muslim establishment and their oil-rich billionaire supporters who went on to channel cash and arms to what morphed into the Taliban, who paid for the madrassa religious school system in Pakistan that produced new generations of extremists, and whose intolerant and anti-western views laid the ideological ground for the creation of al-Qaida, led by a Saudi citizen, Osama bin Laden.

It is the Saudis, according to regional and American reports, who helped create IS in Syria and Iraq, again by funnelling arms and cash. It was the unelected, despotic Saudi regime that, terrified by the implications of the Arab spring, opposed pro-democracy movements in Egypt and elsewhere, and energetically assisted in the brutal suppression of Shia Muslim reformers in Bahrain.

And it is the Saudis who now, in improbable alliance with Binyamin Netanyahu’s Israel, lobby most forcefully against any American nuclear deal, or broader western rapprochement, with Shia Iran, their sworn enemy.

Far from bolstering stability, Saudi policy actively works against western attempts to end the standoff with non-Arab Iran – still the natural regional partner for London and Washington that it was before the 1979 revolution. In Yemen, Iraq, Syria and across the Gulf, the Saudis’ age-old proxy war with Iran, formerly Persia, poisons hopes of peace.

They offer “intelligence-sharing” and token forces when the obvious reality, after Charlie Hebdo, is that the Islamist jihadist terrorism threatening Europe has now replaced Tehran-backed Bashar al-Assad as the west’s main security concern – and is the product, to a large degree, of the Saudis’ repeat mistakes.

To maintain its hold on western governments, the Saudi regime continues to hold out the prospect of lucrative arms purchases, such as the recent reconfirmed billion-dollar deal with BAE Systems to supply Eurofighter Typhoon jets. This despite the deeply unsavoury legacy of the Al-Yamamah bribery scandal, which revealed corruption on a scale previously unheard of in Britain.

To keep its grip, the regime uses its network of personal and official ties to Britain’s too pliant monarchy, to gullible congressional politicians, and to business and investment leaders overly impressed by its $1tn (£660bn) in cash reserves and its global investment portfolio.

But in the end it all comes down to values, not money or weapons or insider influence. In Britain and other western countries, as the past week’s events have shown, a sea-change is under way with which governmental authorities have yet to catch up. What was tolerable or ignorable 30 years ago is no longer so.

Happily, attitudes in British society, especially on individual rights, have shifted. Unhappily, in Saudi Arabia, they have not – not yet. But change there, too, is inescapable. The medieval game of thrones that is the absolutist Saudi system cannot endure. An unlikely 100-year-long affair is finally petering out.

Saudi Arabia: notes from history

A figure central to 20th-century history of the Arabian peninsula was TE Lawrence, better known as Lawrence of Arabia, a quixotic army officer whose championing of Arab nationalism contributed to early British support for what became Saudi Arabia.

Lawrence’s was colonialism with a human face. To a credulous public, his experience conjured a romantic world of proud and ruthless Bedouin horsemen silhouetted against a burning sky; of Saladin, scimitars, keffiyehs, and desert hawks. But colonialism it was, all the same, as demonstrated when Britain turned the Arabian peninsula into a protectorate exactly 100 years ago, in a 1915 treaty with Ibn Saud, the founder of the Saudi dynasty.

Ibn Saud became king and absolute monarch in 1932. Britain maintained its interest and back-room clout. It was among the very first states to recognise the new country of Saudi Arabia, in 1926. The Saudis opened their first embassy in London in 1930.