Almost a century on from the groundbreaking discovery of penicillin by Alexander Fleming, his scientific successors are racing to save modern medicine.

Infections that were once easy to cure with antibiotics are becoming untreatable, and a novel treatment for bacterial infection is the holy grail for teams of researchers around the world.

However, severe financial challenges have left the pipeline of new antibiotics thin and fragile – and treatments are unavailable in many of the places they are most needed. Big pharmaceutical companies have left the field in search of greater profits elsewhere, and talented researchers have opted for new jobs in more stable sectors.

The number of deaths caused by drug-resistant bacteria in 2019 was 1.27 million, and economic costs are on track to exceed $1tn (£765bn) by 2030. The death rate is highest in sub-Saharan Africa, where children under five are particularly affected.

“This is a problem which truly affects the whole world, rich and poor countries alike,” says Jeremy Knox, the head of infectious disease policy at Wellcome. “[But] the impact is definitely asymmetrical. People in low and middle-income countries are bearing a far greater burden.”

Global leaders will gather in New York this month to discuss antimicrobial resistance (AMR) at the UN general assembly. They will consider how to convince researchers and companies it is worth their while to create new replacement drugs, and how to improve access to tests and treatments.

The World Health Organization produces an annual list of drug-resistant pathogens that are of the highest concern. In June it warned there were too few antibacterials to fight them in development.

“We face a crisis of innovation,” says Damiano de Felice, the chief of external affairs at Combating Antibiotic-Resistance Bacteria Accelerator (CARB-X, a nonprofit organisation that aims to accelerate the development of such products. Only one new class of antibiotics has been discovered or patented since 1990, he says – a sharp decrease considering more than 25 were discovered between 1940 and 1979.

There are a lot of new, promising approaches in the early stages of development, he says, “but most of the product developers in this space are very vulnerable”.

Of 112 commercial institutions identified by the WHO as conducting preclinical research to develop new products against AMR, 97 had fewer than 50 employees.

Anand Anandjumar is the co-founder and chief executive of Bugworks, one of the small companies working in AMR research. “We are hardly 30 people,” he says, adding that the company – based in Bengaluru, India – “couldn’t be here” without the support of funders such as the Wellcome Trust, CARB-X and the Indian government.

In recent years, the few companies that have been successful at bringing new products to market have “done very poorly, financially”, says De Felice, with many going bankrupt.

That record deters commercial investors and contributes to a brain drain from the sector, with researchers who start out working on AMR moving to other fields after companies collapse or funding disappears.

“It’s really hard to make a lot of money from an antibiotic,” says Laura Piddock, scientific director at the Global Antibiotic Research and Development Partnership (GardP), which is working on new treatments.

Cheap drugs for chronic conditions such as diabetes or high blood pressure can still make companies big profits, because they are taken by a lot of people for a long period – often a lifetime. By contrast, antibiotics are used for a short period to deal with infections.

Piddock is optimistic that the scientific challenges of finding new chemical compounds to fight troublesome bacteria can be overcome, particularly with the advent of new tools such as artificial intelligence.

The bigger challenge is translating that research into new treatments, she says. “Whether you’re big pharma or a small nonprofit like GardP, it still costs millions.”

Access even to existing drugs remains an issue in countries of all income levels, she says, with many companies marketing their drugs in fewer than 10 countries “for financial reasons”. It means patients in hospital with sepsis may not have access to antibiotics “that you and I take for granted”.

An Access to Medicine Foundation report this year found that was unlikely to change. Looking at five big pharma drugs at a late stage of development for some of the “most severe drug-resistant pathogens”, researchers identified concrete commitments to register them for use in only five low and middle-income countries.

The issue needs incentives that push innovation, De Felice says, such as grants to support early-stage research from governments and the third sector.

It also needs incentives that pull drugs through to market and guarantee companies a return on their investment, even if the antibiotics are not used but held in reserve as a last resort for particularly severe infections.

Some of those programmes exist already. In the UK, drug companies can receive a fixed annual fee for new antibiotics regardless of how much they are used. The subscription model bases payments on how valuable the drugs are to the health system.

A similar approach is under consideration in the US – although some global health campaigners fear it is too domestic in focus and will drive up the price of new antibiotics across the world, making it harder than ever for people in developing countries to get hold of them.

Multiple countries will need to adopt similar incentives to sufficiently stimulate the market, Piddock says.

Some countries where it is harder to access drugs can also lack vaccines and even basic water and sanitation, which can make infection more likely.

“What we need is to ensure when we develop new innovations such as diagnostics and antibiotics that they are accessible and affordable in all countries, and to all populations within countries,” says Esmita Charani, an associate professor at the University of Cape Town and honorary reader in infectious diseases, AMR and global health at the University of Liverpool.

• CARB-X is funded in part by the Bill & Melinda Gates Foundation, which supports the Guardian’s global development journalism via theguardian.org. Read more about how the Guardian ensures its editorial independence here