Harry Gordon Selfridge was out to win. The year was 1909 and the US retail magnate was opening his new store in London’s Oxford Street. His rival was the terracotta monolith of Harrods in Knightsbridge. The latter had just opened complete with an escalator, which so terrified customers that staff had to offer them glasses of brandy as they stumbled off it.

Selfridge’s answer was a facade like a Roman temple and an exhibition of the plane in which Louis Blériot had just crossed the Channel. In doing so, Selfridge also invented what is now called the retail experience, “shopping for pleasure not necessity”. He offered his largely female customers a library, a writing and “silent room”, a mini-golf course, a rooftop exercise terrace and a “girls-only gun club”. He also hosted the world’s first television show, by John Logie Baird in 1925. Oxford Street never looked back; it was now high street to the nation.

Today, however, it is in trouble. The great names have mostly gone. BHS is closed. John Lewis is retrenching. Debenhams went into administration in April 2019 and House of Fraser (which, as DH Evans, has had a presence on the street since 1879) is at the mercy of Mike Ashley’s Sports Direct. Selfridges survives in the ownership of the Weston family, but as an upmarket mall of other brands, which dust themselves in its glory. The street as a whole still boasts 300 shops, but it faces assaults from online shopping and crippling business rates. In my youth, I lived in a flat opposite Selfridges, and can still hardly imagine the downfall of my mighty hero.

A scholarly enterprise, the Survey of London has toiled in these parts since 1894. Mostly it records London’s parish buildings, but this year it has devoted volume 53 to a single thoroughfare – Oxford Street. It traces the story from Roman times to today. It has the aura of an obituary.

Oxford Street came slowly to glamour. It began as a muddy track linking London to Roman Watling Street, now Edgware Road. It was noted chiefly as the route along which convicts were taken from the Old Bailey to the gallows at Tyburn, later Marble Arch. Their fate was eased by the cart stopping at every pub along the way, where they and their friends got so drunk they hardly noticed being hanged.

By the 1780s, the gallows had gone. Oxford Street’s Georgian houses had begun to service the smart estates that had been established to the south and north, across Mayfair and Marylebone, those of Grosvenor, Portman, Hanover and Cavendish squares. The street’s eastern arm served Soho and what is now Fitzrovia.

By the early 1800s, there were 92 shops on Oxford Street, a third of them selling clothes and patronised almost exclusively by “unaccompanied” women. Known as the Ladies’ Mile, it was an early champion of women’s economic emancipation. Jane Austen, on her visits to London, was an early addict, once confessing to spending £5 in one go – roughly £500 today.

A German visitor in 1784, Sophie von La Roche, was equally enthralled. She wrote home to explain how she had walked “for half an hour past brilliantly illuminated windows, with pavements of people standing six deep”. She noted “the highest lord and humblest labourer … both receiving equally rapid and courteous attention.” There was nothing like it in Frankfurt.

As the shops grew in size, they found they had to accommodate their own staff, usually in crowded upper floors. One 19th-century department store, Marshall & Snelgrove, housed 250 shop girls who, after work, enlivened the pavements and pubs. Congestion became intense. Entertainment venues sprang up, with bands, concerts and travelling shows, one boasting the “fattest woman and smallest man on Earth”. When trams came to London in 1860, the shop owners made sure they would not run in Oxford Street, fearing that the loss of footfall would mean customers would stop crowding at their windows. A few years later, the tube was opposed for the same reason. When the Central line arrived in 1900, its four Oxford Street stations had to be located closer than those on any other network. The Central line became Oxford Street’s river of gold.

From the initial free-for-all emerged the “department store” partnerships that dominated the west end in the first half of the 20th century – and were still there in the 1960s. They were the emporiums of Marshall & Snelgrove, Bourne & Hollingsworth, Waring & Gillow, Swears & Wells, Lilley & Skinner and Peter Robinson. The upper-crust Debenham & Freebody turned its back on the street and opened a store 100m north in Wigmore Street. Its facade still stands like a stern Victorian aunt opposite Wigmore Hall. Curiously, all the big stores were on the smart north side of the street, allegedly to catch the sun. Those on the south side were smaller and cheaper – still being described in the 1880s as “rickety, tumble-down shanties”.

A few places were clothes-free. At first there was the Princess’s theatre, the Pantheon assembly room, a skating rink and somewhere Thomas de Quincy bought opium, not to mention prostitutes. There was Purdeys for guns, Bumpus for books, Morris & Co for wallpaper, Duveen for pictures. Shoes became an Oxford Street speciality, with 27 shoe shops at the peak, including True Form, Dolcis, Saxone and Bata. Originally, there was almost nowhere to eat, as ladies on “shopping expeditions” were assumed to prefer hotels. Fast food came from street vendors. That changed with the advent of the day tripper, bringing 10 Lyons corner houses by 1910.

Between the wars, Oxford Street began to suffer competition from Bond Street and then the rebuilt Regent Street, beginning what became a steady drift downmarket. Marks & Spencer, Woolworths and C&A took up the baton. The survey records them as supplying “ladies of the middle classes with goods of general utility for that exuberant pair, Mr and Mrs Everybody”. Harrods would have sniffed.

The crowds were awesome. One newspaper described “agitated spinsters holding hands while trying to cross the road”, while the writer Ivor Brown detected the “aroma of perspiration”. Wigmore Street behind was mercifully “disinfected by its many chemists”.

In the 1930s, Selfridge took to gambling and sold up, but his creation never lost its panache. It was Oxford Street’s talisman of quality and innovation. I remember the horror that greeted its first food offerings. The apex of fashion had become a grocer. It even boasted a bakery. In the event, food did not drive Selfridges downmarket; rather, Selfridges took food upmarket.

Like much of central London, Oxford Street in the 1960s came close to disaster. Its crowded pavements, dirt, noise and the shabbiness of its south-side shops offended the sensibility of clean-living modernist planners. One of them, Colin Buchanan, dismissed it as “the most uncivilised street in Europe”, which he was determined to correct. The street should be decked and pedestrians forced on to podiums while the traffic flowed freely beneath, he said.

London’s county council and Westminster’s city council produced one plan after another for taming Oxford Street. Most involved total rebuilding and decks. There were proposals to run minicars on air cushions down the centre, and for a fly-over at Oxford Circus. One fragment that did get built was a block designed by the LCC for the London College of Fashion, backing on to Cavendish Square. It still displays its podium, awaiting its deck.

The shops were strong enough to fight against this nonsense. But they could not stop the catastrophe that hit Oxford Street’s east end – its problem quarter at Tottenham Court Road. New Oxford Street had been pushed through the St Giles slum in 1847, and was linked by the Edwardians to Kingsway further east. It was instantly London’s most characterless neighbourhood.

This was followed, in 1963, by a secret deal between the LCC and the architect Richard Seifert to demolish most of old St Giles for a large traffic roundabout at the top of Charing Cross Road. Seifert agreed to build it, only if he was allowed to break height restrictions with an unprecedented London tower, Centre Point. The roundabout was never used and the tower was left unoccupied, although its owner, the property developer Harry Hyams, walked away with a £11m profit.

The scheme’s decks, concrete walls and dirty fountains formed an impenetrable barrier between Soho and Bloomsbury, a planning fiasco that stood for half a century. Westminster council is currently trying to rectify the Centre Point disaster. It is erecting two huge buildings at the top of Charing Cross Road and Tottenham Court Road, allied to a new Crossrail station and a £150m square. It is hard to believe it will be an advance on St Giles piazza.

Oxford Street has never had a true friend. It has never been pedestrianised, and is still clogged with half-empty buses, echoes of the horse-drawn ones that London Transport had the courage to reroute. As the street went downmarket in the 60s and 70s, it was written off. Its Christmas decorations are commercial tat, as are many of its tourist shops. In 1985, a Times editorial asked: “Why is Oxford Street such a visual mess?” Even the Survey of London’s Andrew Saint calls it “widely disliked, even disdained”.

Yet for all that, Saint admits “Oxford Street flourishes … indeed is a howling success”. It may not have seen off Harrods, but it has outlasted such historical west end rivals as Holborn (Gamages), Victoria (Army & Navy Stores), the Strand (Civil Service Supply Association), Kensington High Street (Derry & Toms) and Queensway (Whiteleys). Two hundred million shoppers still crowd its streets every year, despite its battered facades sheltering the same shops as any other high street; Next, Gap and H&M.

So how has Oxford Street bucked the national trend of our dying high streets and the giant shopping centres outside city centres? The street’s worldwide reputation, of course, means it is as much a tourist destination as a shopping street, while the range of shops in such close proximity makes it more like a linear mall than your average high street. Moreover, it learned its lesson in the 18th century, so it understands the retail experience and still has fashion shows, music, gyms, vegan cafes and calls to “detox the body and soul”.

Long blessed with the Central line, Oxford Street is also blessed by a government inexplicably eager to attract ever more people into central London. It will get two Crossrail stations, bringing it a forecasted 90 million more shoppers. Birmingham and Manchester can eat their hearts out.

Of equal importance is that Oxford Street has the right hinterland, one of the most attractive commercial communities in Europe. Studies by the architects Arup show that Marylebone, Soho and Fitzrovia contain the most intensive economic activity outside the City of London and Canary Wharf. It is not housed in towers, but in old workshops, warehouses, alleys and courtyards. Here lurk boutiques, clubs, galleries, studios, musicians and ethnic food outlets of the creative urban economy. Just as digital startups have flourished in scruffy Shoreditch, so in the streets off Oxford Street have developed two of the most advanced tech industries, Soho’s film post-production and Marylebone’s diagnostic medicine.

Oxford Street is thus “front of house” to the most marked feature of London’s success – that it blossoms amid traditional informality. For shopping, the new city spenders go to Covent Garden and Westbourne Grove, King’s Road and Marylebone High Street, Portobello Road and Borough Market. They hate hard edges. They seek reassurance, comfort and surprise – just what Centre Point will not give them.

For all its ugliness, Oxford Street’s longevity is astonishing. Generations of planners have tried to bludgeon it into respectability. It has defied them all. The street may have lost its looks, drifting from style and fashion to tackiness and bling. It is really just another high street. But it has survival in its genes. It will outlive us all.

• Survey of London Vol 53: Oxford Street, by Andrew Saint, will be published in April by the Paul Mellon Centre for Studies in British Art (£75). Simon Jenkins’s A Short History of London (Penguin, £9.99) appears in paperback in July

• This article was amended on 11 March 2020. Earlier versions implied that Mike Ashley and Sports Direct had control of Debenhams. To clarify, Ashley and Sports Direct held only a minority shareholding in Debenhams, which was relinquished when the company went into administration in April 2019.