Karren Brady has quit as chair of Sir Philip Green’s retail empire and faced immediate questions over what she knew of alleged sexual and racial harassment by the disgraced business tycoon.

Brady walked away after two years in the job and a fortnight after she vowed to remain in her post out of a “sense of duty” to look out for the firm’s staff, including her daughter, who works on Topshop’s social media accounts.

The Labour MP Jess Phillips called on Brady, 49, to “answer to what went on under her watch” following claims that Green’s company paid two staff about £1m each to settle sexual harassment and racial abuse claims.

The allegations include that the billionaire groped a female employee and told a black executive that his “problem” was that he was still “throwing spears in the jungle”. Two other female employees received hundreds of thousands of pounds each after alleging that Green had grabbed one woman by the face and put another in a headlock.

Brady, a Conservative peer, was chair of Taveta Investments, the company that ultimately owns the Green family empire of Topshop, Miss Selfridge and Dorothy Perkins outlets.

A star of The Apprentice and campaigner for “strong female leadership”, Brady quit following mounting pressure to speak out against Green. She refused to explain why she had resigned. Taveta also announced the resignation of the non-executive director Sharon Brown, an experienced retail executive.

Phillips, the MP for Birmingham Yardley, who sits on the women and equalities committee and backbench business committee, said: “Karren Brady will still have to answer to what went on under her watch and so she should, but it is Green who is ultimately responsible and, like always, it feels like powerful men can get away with anything.”

Vince Cable, the Liberal Democrat leader, said Brady “has decided she can’t defend the indefensible”.

Green, 66, who was described by a committee of MPs as the “the unacceptable face of capitalism” for his role in the collapse of BHS, has said he “categorically” denies any “unlawful sexual or racist behaviour”.

The Dragon’s Den star Duncan Bannatyne had described Brady as a “hypocrite” for continuing to work for Green despite using her high profile to attack “serial predators” such as the Hollywood producer Harvey Weinstein, the actor Kevin Spacey and the former minister Michael Fallon.

“I think it’s disgusting that she’s on the payroll of a billionaire who hasn’t got a great history with women,” Bannatyne said this month.

Brady had refused to quit, saying two weeks ago: “I hope you would agree that walking away is the easy thing to do; staying in the role and ensuring the company is robust going forward is much harder.”

Brady, who has built a reputation as one of the UK’s most prominent and successful female business leaders, has a track record of publicly shaming powerful men accused of sexual harassment.

She denounced Weinstein as “a serial predator with such colossal amounts of power that people think they have no choice but to put up with grotesque advances if they want a career in Hollywood”.

In her column for the Sun, she said: “What Michael Fallon [who resigned as defence secretary in 2017 over inappropriate flirtation], Harvey Weinstein and Kevin Spacey [who is accused of sexually harassing actors] are accused of abusing their power in an organisation to get someone to do something they don’t want to do, or tolerate something they don’t like.

“Lots of men are asking: ‘When is it appropriate to touch a colleague?’ If you are in doubt, how about … NEVER?”

In another column, she wrote: “It’s not just film. There is a Harvey Weinstein in every industry – the man at the top who abuses their power, who has no respect for women, who thinks women owe him something and it is a perfectly reasonable trade-off to exchange sex for work.”

Brady has not spoken about the allegations against Green. In her book Strong Woman, she said: “He is without doubt the most dynamic, the most energetic, the most relentless businessperson I have ever come across, a whirlwind of energy and ideas.”

Bannatyne said on Monday that Brady “should be answering questions about what she knew of the alleged non-disclosure agreements”.

Green appointed Brady as chair of Taveta in 2017 with a mandate to “beef up corporate oversight” and “sort out” corporate governance problems. She has been on the board of Taveta, which is controlled by Green’s wife, Tina, since 2010.

The company said: “Taveta would like to announce that Karren Brady and Sharon Brown (in their respective capacities as non-executive chairman and non-executive director) have resigned from its board. Taveta thanks them for their contribution and wishes them well for the future.”

Taveta, which owns the Arcadia Group, which in turn owns Topshop, Miss Selfridge and Green’s other retail brands, reported a 42% fall in annual operating profits to £124m in its latest accounts filed at Companies House. The group’s sales fell 5.6% to £1.9bn, but the company still paid £25m to Tina Green, who controls the business from tax-free Monaco.