One hundred years before Deirdre Kelly welcomed Channel 4's cameras into her living room and found unexpected fame – if not, yet, fortune – as mouthy, unemployed, single mother-of-two White Dee from Benefits Street, her house was home to a family called Ashforth.

The 1911 census records the occupation of Jesse Ashforth, 38, as "polisher-silversmith"; his son, also Jesse, 20, likewise. Ellen Ashforth, 41, was (one imagines) occupied with William, nine, Charles, seven, Naomi, four, and baby Ellen, "0".

Thirty-odd years later Ellen became the mother of Ralph Carpenter. "They had 13 children, my grandparents," says Carpenter, who was himself brought up 10 doors down the street. "Six survived. I don't think it was a very easy life."

Carpenter's memories of James Turner Street in Winson Green, Birmingham – now better known as Channel 4's infamous Benefits Street, so named because a majority of its residents (90%, some say) live on welfare – stretch to the mid-1940s. They are mostly, he says, in black and white.

"We lived in a black-and-white world," he says. "The cars, not that there were many, were black. There was the soot, from the factories and smokestacks. And I don't recall a single tree on James Turner Street when I was a boy. A few shops, front rooms really, and the fish-and-chips on Eva Road. But no trees."

What Carpenter does remember clearly, though, is watching all the men – including his father, Fred, a factory worker like practically everyone else – coming from work each evening. "They'd walk home up the street, all of them," he says. "All smoking: Woodbines, the great Brummie cigarette. They'd all have bags on their shoulders, knapsacks or haversacks, army surplus, or old gasmask bags, for their lunches. But they were all there, everyone, walking home. Because everyone worked then. Everybody had a job."

Whatever you think of this TV series – we seem to see it either as a cynical demonisation of the poor, or a laudable exposure of all that is wrong with our benefits system – what it has not done is explain how the James Turner Street of Carpenter's memory came to be the Benefits Street of 2014.

It is an edifying story, and it begins long before Jesse Ashforth. At the beginning of the 19th century, Winson Green was a small hamlet on Birmingham Heath, a vast stretch of largely uncultivated, open land west of town. The canal, cut in 1769, attracted a smattering of early industry to the area; in the 1850s the railway drew more. By then the Heath was also the site for three big new Birmingham institutions: the Borough Gaol, now HM Prison Birmingham; All Saints Asylum, housing 300 "pauper lunatics"; and – completing an unholy trinity – the Union workhouse, home to 1,100 unfortunates. Then came the houses.

For Birmingham, the City of a Thousand Trades, was booming, the small metalworking shops that had fuelled its earlier growth now joined by bigger mills and factories. And many of those were in Smethwick – next door to fast-urbanising Winson Green.

"The manufactures of Birmingham," marvelled Willey's History and Guide to Birmingham in 1868, "are almost infinite in their variety. Almost all articles of utility or ornament are manufactured in the town.

"From a pin to a steam engine, from pens to swords and guns, from 'cheap and nasty' wares to exquisite and elaborate gold and silver services ... All things are made in this hive of industry and give employment to its thousands of men, women and children."

Minutes from where James Turner Street would soon be rising from the ground, Matthew Boulton and James Watt's Soho Foundry, which built the world's oldest working steam engine, had put Smethwick on the map as early as 1796.

Now there were countless more manufacturers: Muntz Metal Works lined the hulls of all the Royal Navy's ships; rolling stock for the new railways came from the Birmingham Railway Carriage and Wagon Company; Guest, Keen and Nettlefolds made screws; Tangye Bros produced engines; and Evered made tubing. Phillips Cycles was in Bridge Street; British Pens nearby.

For the Great Exhibition of 1851 it was two Smethwick firms, Chance Brothers glassworks and Fox, Henderson and Co, that supplied the 25 acres of glazing that went into the Crystal Palace and the iron frame that held it together.

These factories needed workers, and the workers needed homes. Along with most of its neighbours in Winson Green, James Turner Street – originally Osborne Street – was built by speculative private developers during the 1870s and 1880s. Its first official mention occurs in 1877; council records show it was formally renamed in 1882. (A present-day debate about who the street was named after – a long-serving schoolmaster at King Edward's school, the owner of gun-barrel-maker Cooper & Turner, or a partner in a famous button-making firm, Hammond, Turner & Sons – seems to have been settled by Carl Chinn, chair of community history at the University of Birmingham, who is sure it was the latter.)

The street took some time to complete. Local amateur historian Bill Dargue points out that the 1890 Ordnance Survey map, doubtless surveyed a couple of years before publication, shows it only half-built. The 1891 census, on the other hand, shows a full street of residents.

The houses they moved into were, says Chinn, a big step up from the cramped and squalid early 19th-century working-class homes nearer the city centre. Building of new "back-to-backs", as they were known, was banned in 1876, and the terraced houses of Winson Green were of another order altogether.

"This was much improved housing," says Chinn. "Two, sometimes three bedrooms upstairs, a front and a back room downstairs, a separate kitchen. And much better built."

Chris Upton, reader in public history at Birmingham's Newman University, says the new districts tended to attract the more prosperous working-class: "People with steady jobs, good wages, moving out of the city centre, or to Birmingham from elsewhere. This was respectability: bay window, front garden, back yard, your own privy out the back, running water inside."

At a time when a skilled or semi-skilled working man earned maybe 30 shillings a week, the rent – and until the later 20th century, these houses were all rented – would be around 5s: less, as a proportion of income, than many people pay today.

"Food, though, was much more expensive in those days," Upton notes. "Thirteen, 14s a week to feed a family of four or five. Clothes and shoes were dearer. And school was 2d or 3d a week per child; paid up front, in the box, every Monday morning. There wasn't often much spare."

So who were those first residents of James Turner Street? They came, according to that 1891 census, from all over the country: a majority from Birmingham and Warwickshire, but also from Cambridgeshire, Cornwall, Devon, Herefordshire, Leicestershire, London, Norfolk, Staffordshire and Worcestershire.

But what stands out are their jobs. There are smiths, pressers, turners and stampers. Chain-makers, moulders, casters and brass workers. Solderers, burnishers, polishers, tank-makers, engine fitters, nut and bolt-makers, cycle-makers, core-makers, angle-iron smiths, axle turners, brass tap finishers, machine tool-makers, furnacemen, iron and steel wire drawers, rule-makers. Metal-bashers, almost to a man.

Not that there weren't other jobs: at No 19 was a gardener, Richard Webb, and at No 37 a jeweller, Thomas Sparkes; many others – silversmiths, gold chain-makers, silver swivel- makers – were also in the jewellery trade. The jewellery quarter, which still produces more than 40% of the UK's hand-made jewellery, was a short tram ride away.

Prison warders lived at 60, 63 and 119; railway servants at 25 and 49; there were labourers, bricklayers, a carpenter, a clerk or two. It is noticeable, too, how many wives worked: French polishers, lacquerers, brush-makers, steel pen slitters, burnishers, dress and shirt-makers, stud and cuff-makers, button and brooch-makers, ivory-button sorters. "There were huge numbers of factory jobs for women," says Chinn. "This was manufacturing, not heavy industry. They earned real wages – not as well-paid as men, but not pin money."

Past 12 or 13, most children were working, too: 15-year-old Clara Johnson was a chain-maker; Harry Woolley, aged 14, was a caster; Alice Baker, 13, a dress-maker; Roland Sparkes, 14, a stamper; Timothy Griffiths, 13, a boiler-maker.

Local employment supported local businesses: the Birmingham street directory for 1903 lists four shops, a  greengrocer and two "beer retailers" in James Turner Street alone (in 1932, there were still five shops, plus a "motor beading-maker"; in 1951, four. Two lasted into the 1970s).

Life was not necessarily easy. Many families had five, six, even seven children; many also took in lodgers, some with children of their own: No 15 housed labourer George Hirom, his wife Kate, their five children aged between nine and two, plus George Straight, lodger, with his two daughters of 12 and seven.

Poverty was never far away: in the harsh winter of 1895, the Birmingham Daily Post records the "heart-rending privations" of some of those helped by relief societies and charitable institutions: a brass caster with wife and two children, "on short time for a long period, then had his hand injured: £5 behind with rent and debts"; an unemployed harness stitcher's family "in danger of starvation".

Others on James Turner Street, records the paper, turned to crime: William Ashmore, 19, "a cripple", was jailed for a month in 1900 for the theft of half a hundredweight of nickel silver metal; Samuel Marshall, fitter, and his wife Jane, machinist, were charged with stealing 246lb of brass screw blanks; and, most dreadfully, Thomas Moreton, 36, brass caster, was indicted for the drunken murder of his wife, Elizabeth.

But to most, the city, now growing at a barely imaginable pace, was kind. Between 1891 and 1911, when the Ashforths appear in White Dee's house, Birmingham's population grew from 478,000 to 840,000. The job titles were changing – there was now, for example, an "electrical wireman" – but there were more and more of them.

The first world war, though it cost 13,000 Birmingham men their lives, only confirmed the city's status as the vital heart of British industrial production; at its close, prime minister David Lloyd George observed that "the country, the empire and the world owe to the skill, the ingenuity and the resource of Birmingham a deep debt of gratitude".

On winsongreentobrookfields.co.uk, a community history website run by amateur historian Ted Rudge, people who lived on and around James Turner Street in the 1920s and 30s recall corner shops that sold faggots and peas on Friday nights, as well as cow heels, pig's trotters and tripe.

Others remember playing in Black Patch Park at the end of the road, buying ½d of cake crumbs, and giving vegetable peelings and leftovers to the people who kept the pigs in Vittoria Street, in exchange for a piece of pork at Christmas. Unless the prison siren sounded for an escapee, another recalls, no one locked their front doors.

By now more advanced manufacturing, electrical engineering, cars and motorbikes were emerging as major industries. The second world war added jeeps, planes and munitions. And in the 30 postwar years – despite government planning measures actively restraining growth and transferring jobs to less thriving cities in the north – Birmingham's economy far outperformed that of any other British city outside London.

Between 1951 and 1961 only the capital created more jobs. In only one year between 1948 and 1966 did unemployment in Birmingham rise above 2% (most of the time it was below 1%). The city and its region grew wealthy; by 1961, household incomes in the West Midlands were 13% above the national average, higher even than London and the south-east.

John Cahill, now a Black Country bricklayer, grew up on James Turner Street in the 60s, leaving in 1974 at the age of 16 to join the navy. Cahill's father was a lathe turner, highly skilled, making parts for navigation systems, and his mother also worked in a factory. Even as late as the 70s, Cahill says: "Everyone was in work, women too. It was the same right along the street. Nobody worried about it. People lost a job today, they had a new one tomorrow. Good jobs; skilled jobs. Good incomes."

The area was, though, beginning to change. "By the mid-60s," says Chinn, "more and more prosperous working-class people, with good jobs, were starting to think they could do better than a terraced house in Winson Green. They began moving out, westwards, to newer, leafier suburbs."

Birmingham's housing stock was, indeed, decaying: Chinn cites a survey from 1971 that found 26,000 households in the city had no running hot water, and a further 28,000 no fixed bath. A second report, in 1979, concluded that 40,000 homes needed "substantial improvement" or demolition, with 26,000 more likely to be substandard within five years.

Many families, plenty in Winson Green, moved on, some in to council housing. The most decayed Victorian terraces came down, among them a whole chunk of James Turner Street, opposite Foundry Road primary school where Carpenter, Cahill and many of Rudge's contributors were once pupils.

As the more upwardly mobile moved out, immigrant families from the Caribbean and the Indian sub-continent were moving in, finding – mostly without difficulty – relatively low-skilled metal-bashing jobs, just like the street's very first residents. But not for long.

The implosion of Birmingham's industrial economy was as sudden as it was catastrophic. Two authoritative mid-80s studies, Crisis in the Industrial Heartland and The Midland Metropolis, chart the collapse with chilling clarity. In 1976, they record, Birmingham's GDP per capita was still the highest of any British city outside the south-east; five years later, it was the lowest in England.

Relative incomes, the highest in the country in 1970, were by 1983 the lowest. Birmingham lost 200,000 jobs, almost all in manufacturing, in the decade from 1971 to 1981; by 1982, its unemployment rate had skyrocketed to nearly 20%.

"It was dramatic," says Upton. "Birmingham went from a high-wage, low-unemployment city to a low-wage, high-unemployment city in a decade. UB40 sang about One in 10; it was more like one in eight."

The consequences, in places such as James Turner Street, were soon apparent. "It's a spiral of decline," says Chinn. "External forces, attacking the working class, because of their address. Manufacturing was the fabric of this city and it was ripped up. You can't divorce that fact from the social and economic problems of former manufacturing districts."

So the gradual transformation of James Turner Street into Benefits Street should properly be seen, Chinn argues, as a story of successive governments, both Labour and Conservative, failing abjectly to value manufacturing, and of a consequent, near-total collapse of economic opportunity.

For those on the sharp end of that collapse, Upton concludes, it boils down almost to "an accident of history. To living in the right or the wrong place, at the right or the wrong time."

Carpenter, for his part, has been watching Benefits Street with slightly shocked fascination. He remembers Black Patch Park, the road to the Soho Foundry, the men coming home from the factory, as if it were yesterday. But like many of his postwar generation, his opportunities lay elsewhere: he went to art college, travelled abroad, got out.

"Your horizons change," he says. "My father died young, in his 50s, though my mother stayed on, for a long time. But no friends, none of our family, live on James Turner Street now. Everyone's left. Moved on. That tells you something, I suppose."

Additional research by Phil Lewis