Rishi Sunak has been urged to rethink the decision to keep £10m from the Conservative donor Frank Hester after police said they were investigating “racist comments which were allegedly made” by the businessman in 2019.

The investigation by West Yorkshire police was triggered by a complaint from Diane Abbott, after the Guardian reported that Hester told colleagues that looking at the MP made you “want to hate all black women” and said she “should be shot”.

Labour said Sunak should “do what he should have done a fortnight ago: pay back the money”, while a Conservative donor, Dr Maurizio Bragagni, said the party should not accept any more money while the investigation is taking place.

Abbott said she was “glad” that police were investigating the comments, which were made at a meeting at the offices of Hester’s company in Leeds, adding: “Women in public life should not have to live in fear, so I hope that this investigation is brought to a speedy conclusion.”

Hester became the Conservatives’ biggest ever donor when he gave £10m to the party last year, with another £5m reportedly under discussion.

Despite the calls for it to be handed back and reports the Tories are still in talks about the further £5m, party sources said the party planned to keep all of Hester’s £15m.

One senior Tory told the Guardian the party had not only received Hester’s most recent £5m donation, but had already spent it along with the rest of the original £10m, making it impossible to hand back. The person dismissed earlier reports that the donation had not yet been received and that the party was deliberating over whether or not to accept it.

“We got that long ago, and it’s gone,” the person said. The Conservatives have been spending heavily in recent weeks in the run-up to the local election campaign at which the party is expected to suffer heavy losses. A spokesperson for Hester declined to comment.

Anneliese Dodds, the Labour party chair, said: “Rishi Sunak should have returned the money immediately. The prime minister needs to now do what he should have done almost a fortnight ago: pay back the money, cut ties with Frank Hester, apologise to Diane Abbott and finally get to work rooting out the extreme views which appear to be tolerated in his party.”

Bragagni, who has given £300,000 to the Conservatives, said: “We should not accept any more money until the investigation will be completed, and a proper apology must be expressed, in any case.

“We did not know the comments at the time he made his donation, for sure we need to be more careful for the future … At the end it is Rishi’s call. He is the leader of the party.”

Another donor, Derek Calrow, who has given more than £670,000, told the Guardian the investigation was a “waste of police time”.

He said: “I don’t know Mr Hester but my understanding is that he’s made a full apology to Diane Abbott and frankly I think it’s a waste of police time. I think there are many other things the police could occupy their time without taking any view on this.

“The remarks I understood he made weren’t acceptable, he’s apologised for it, and as far as I’m concerned the matter should be closed.”

He added: “I think it’s a complete non-event and the Labour party are trying to make sure the Conservatives have this amount of money less.”

The Guardian reported last week that Hester said in a 2019 meeting at his IT healthcare firm that he did not hate all black women. But he said that seeing Abbott, who is Britain’s longest-serving black MP, on TV meant “you just want to hate all black women because she’s there”.

After the report last week, Abbott made a complaint to the Metropolitan police. The West Yorkshire force has taken on the inquiry into the remarks, as they were made at Hester’s company offices in Horsforth.

A police spokesperson said officers were “working to establish the facts and to ultimately ascertain whether a crime has been committed”.

West Yorkshire police said: “We recognise the strong reaction to these allegations and appreciate everyone who has contacted us since the article was published.

“As we continue our inquiry, we are keen to hear from anyone who could directly assist our investigation.”

The force appealed for any information to help with the inquiry, which is named Operation Brassminster.

After the publication of the remarks, a statement from Hester’s company, The Phoenix Partnership (TPP), said he “accepts that he was rude about Diane Abbott in a private meeting several years ago but his criticism had nothing to do with her gender nor colour of skin”. The statement said Hester abhorred racism, “not least because he experienced it as the child of Irish immigrants in the 1970s”.

The statement added: “He rang Diane Abbott twice today to try to apologise directly for the hurt he has caused her, and is deeply sorry for his remarks. He wishes to make it clear that he regards racism as a poison which has no place in public life.”

TPP, a healthcare technology firm, has been paid more than £400m by the NHS and other government bodies since 2016, primarily to look after 60m UK medical records. Hester has profited from £135m of contracts with the Department of Health and Social Care in less than four years.

Hester gave £5m to the Conservatives in May 2023 and announced this month a further £5m donation, which had been accepted by the party from his company in November last year.