Everlast sports socks at “mega value” prices jostle for space amid dozens of rails of Lonsdale tracksuit bottoms and Karrimor waterproofs. No, this is not a branch of Sports Direct but House of Fraser in Richmond, one of the wealthiest suburbs of London.

Just over six months after Mike Ashley’s retail group bought the upmarket department store chain out of administration for £90m, his apparent ambition to turn it into the “Harrods of the high street” is hard to detect.

As Ashley battles for control of Debenhams, he is trying to convince the struggling retailer’s board and shareholders he’s the only man with the “first-class leadership” skills to turn around its fortunes. But a look around his existing department store empire might raise questions over those ambitions.

In December, Sports Direct revealed House of Fraser was losing nearly £3m a week and some key suppliers, including Jigsaw and Edinburgh Woollen Mill group brands Jacques Vert, Jane Norman and Jaeger had pulled out.

The Guardian visited four House of Fraser stores this month, in Richmond, Wolverhampton, Birmingham and Sutton Coldfield, and found a grim state of affairs. Signs of decline included three entirely shuttered floors in Birmingham, where the restaurant and hairdressing concessions had also closed.

As much as 50% of clothing stock in some stores appeared to be Sports Direct brands, from Firetrap and SoulCal casual wear to No Fear, USA Pro and better-known labels such as Lonsdale and Karrimor.

In Wolverhampton, the cafe had just shut and the former beds concession was still empty, barring a lone shopper sat on a sofa next to a few shelves of Christmas decorations. In Sutton Coldfield, a section dedicated to the House of Fraser dress label Issa, once sported by the Duchess of Cambridge to announce her engagement, had closed down.

Far from attracting new designer labels in recent months, key brands have deserted the department store, including Chanel cosmetics, Clarins beauty salons, Weekend by Max Mara, LK Bennett and the Fragrance Shop, which operated in 27 stores. Others, including fashion brands Mint Velvet, Olsen and Marella, have significantly scaled back their presence.

Piles of Karrimor outdoor gear, discounted suitcases and bare rails of discount Sports Direct own-label hoodies and tracksuit bottoms filled large spaces where once stood House of Fraser’s discontinued clothing own labels Howick and Linea or the bed and sofa concessions And So To Bed and ScS.

“It’s not really the Harrods of the high street, it’s more of the Sports Direct of Richmond,” said the independent retail analyst Nick Bubb after a tour of his local store with the Guardian in south-west London.

On the ground floor, tables of trainers filled a space where the optician chain David Clulow once traded. An outdoorwear section filled with Sports Direct’s own-label Karrimor had been crammed into a space beside kitchenware on the top floor, where holes in the ceiling were covered in plastic.

“He’s just desperately trying to fill the gaps with his own stock and sell that through. It’s not a recipe for long-term success,” said Bubb. “Where is the depth of management talent to handle all this? I think he’ll regret ever buying House of Fraser. The mystery is why he wants to buy Debenhams as well.”

Of course, the damaged flooring, lighting and doors at several stores are not a particularly new feature for House of Fraser, which has suffered from years of under-investment. One supplier said brands exited before Ashley arrived in order to sell directly online, supported by a relatively small number of their own stores or concessions.

Once unwanted stock has sold, Ashley has promised suppliers he will bring in the same kind of upmarket brands, such as Gucci and Burberry, that are stocked in his successful Flannels chain.

In some stores, Ashley is thought to be planning entire Flannels floors as part of widespread changes. Sports Direct declined to comment on its plans for House of Fraser, but the company has already updated its tills and IT using Sports Direct systems and cut costs by shifting its warehousing and distribution to its main facility in Shirebrook, Derbyshire, and combining its head office with the Flannels HQ in London.

Industry chatter suggests Ashley wants only 40 of House of Fraser’s 59 stores and is still weighing up which ones. To date, he has acquired the landmark Frasers building in Glasgow for £95m, closed five stores and made deals on at least 37 others, though many of these are only one-year agreements.

“From his point of view, he’s clearing out the rubbish,” an insider said. “But as far as Harrods of the high street, there’s not much sign of that yet.”

There are questions over how patient suppliers will be during a revamp Ashley has admitted could take nearly two years.

But one supplier said top-end brands, such as Tommy Hilfiger, Hugo Boss and PS by Paul Smith, which are still in stores and want to continue trading in its sister chain Flannels, will help House of Fraser keep its head above water.

“For obvious reasons of brand integrity people are thinking about [pulling out]. But John Lewis only has 23 stores and is increasingly focusing on private label. The trouble is, he is the only show in town. If you exit House of Fraser where are you going to go?” said the supplier. “He’s a gambler and it’s a question of whether you want to gamble with him.”