The Australian Securities and Investments Commission has upended the way products including mortgages, insurance and investments are regulated, putting banks and other finance companies on notice that they will no longer be able to rely on the small print in contracts to rip off their customers.

In a move welcomed by consumer advocates, Asic said that rather than relying on ordinary Australians to make a rational choice based on the often voluminous disclosure documents that accompany financial products, it would now use new powers to make sure finance companies did the right thing.

Asic’s new posture comes amid an intense backlash from the finance sector towards regulation, reflected in an intense lobbying effort against Asic’s interpretation of responsible lending laws, and the announcement by the Morrison government of an inquiry by the competition regulator into the banks’ failure to pass on in full the Reserve Bank’s interest rate cuts.

It reflects the abandonment of disclosure-based regulatory principles introduced in 1997, following the Wallis review of the finance sector, that in recent years have been steadily whittled away amid a series of scandals including shoddy financial advice and insurance rip-offs.

The Asic deputy chair, Karen Chester, said the move was based on 10 years of research into the way consumers actually behave, collected from across four jurisdictions.

She said the research, conducted with the Dutch financial regulator, showed that “more often than not disclosure can be ineffective and can even backfire”.

“It’s a really healthy evidence base to keep fighting the good fight – we’re no longer in a world of anything goes as long as you disclose,” she told Guardian Australia.

She said the move also drew on commissioner Kenneth Hayne’s final report following his inquiry last year into the banks. More than 50 of Hayne’s recommendations related to making firms behave better, rather than providing consumers with more information.

“From Asic’s perspective, the limits of disclosure are not new, but we’re now in a world where the royal commission report proved more than a tilt away from disclosure,” she said.

And she rejected the idea that Asic was imposing too heavy a red tape burden on the financial sector.

Sign up to receive the top stories from Guardian Australia every morning

“We know that the rebalancing is needed because what people eloquently call a non-financial risk [not listening to your consumer] has not always been well monitored or well reported to the board,” she said.

“But we now know that can and has crystallised into a financial cost,” she said.

“Some are suggesting the remediation tally (including provisioning) is now approaching $10bn.”

Following six years of scandals in the finance sector, and continual criticism of it as a do-nothing regulator, the government has given Asic significant new powers and it has pledged to adopt a more aggressive approach to enforcement.

The regulator’s new weapons include the power to ban products, the product intervention power, which it has already used four times in the past three months, and a design and distribution obligation that comes into effect in April 2021.

This will require financial services companies to create products that are useful to a particular set of consumers and market it to them appropriately, rather than selling useless products to people as some have done with services such as junk insurance.

Chester said the new law encapsulated “just common good business sense”.

“I really think that’s got the potential to be a game changer for not only consumers but also the firms and their boards,” she said.

“Going forward they’ll now need to meet these obligations to say that the products you’re offering to your consumers, to your market, are fit for purpose.

“I think business is already on notice from us. It’s clear that when we see and substantiate significant consumer detriment we can and will act.”

The chief executive of Financial Counselling Australia, Fiona Guthrie, said Asic’s move was “fantastic”.

“Finally,” she told Guardian Australia. “It’s like the emperor with no clothes.”

She said consumers were forced to buy products such as insurance or a home mortgage but it was “impossible for any normal rational person who has a life to choose”.

“Nobody reads terms and conditions,” she said.

“Even when you try simplified disclosure, because you’re dealing with fundamentally complex products, people don’t make good choices.”

Susan Thorp, a professor at the University of Sydney who specialises in financial services, said some companies deliberately made their products complicated.

“The advantage for the firm is in creating a group of consumers who are confused so that the pricing becomes less transparent,” she said.

“Classic examples are things that have fees or charges that are in the background or people think they won’t incur them.”

Choice’s chief executive, Alan Kirkland, said that disclosure could actually result in a worse outcome for consumers.

“In areas like mortgage broking and financial advice, empirical research has found that disclosing that an adviser is conflicted actually increases trust in the adviser, right when a person should be more sceptical of the advice,” he said.

“What we need is better products and better conduct.”

He called on the government to resist industry’s lobbying for light-touch regulation following the Hayne royal commission.

“This report makes it clear that we need strong legislation that goes to the heart of the conduct revealed before the royal commission,” he said.

“Letting the industry get away with pushing responsibility onto consumers through more disclosure won’t cut it.”