The Barclays chief executive, Jes Staley, has been ordered to hand over more than £1.1m out of his own pocket as punishment for attempting to unmask a whistleblower.

Staley has been fined a total of £642,430 by the Financial Conduct Authority and the Prudential Regulation Authority, and Barclays has clawed back £500,000 of his bonus over the matter.

The bank will also have to report annually to the regulators detailing how it handles whistleblowing after they expressed concerns about its existing systems.

The whistleblowing involved letters sent to the board in 2016 raising concerns about the recruitment of Tim Main as head of the bank’s financial institutions group in New York.

Staley tried to hunt down the author of the letters using the bank’s internal security unit. He accused the letter-writer of harassment and trying to “maliciously smear” Main, a friend and former colleague when Staley was at JP Morgan.

The regulators said Staley failed to act with due skill, care and diligence, saying he should have identified that:

• He had a conflict of interest in relation to the letter and needed to maintain an appropriate distance from the investigation.
• There was a risk he would not be able to exercise impartial judgment in relation to how Barclays should respond.
• Once the complaint was in the hands of the compliance team, it was important they kept control of the investigation process.

The watchdogs have each fined Staley 10% of his £4.5m total pay package but the total was reduced from £917,800 after he agreed to settle at an early stage of the investigation. They said the fine was “appropriate and proportionate”, given Staley’s seniority and the impact of his actions.

Mark Steward, the FCA’s executive director, said: “Mr Staley breached the standard of care required and expected of a chief executive in a way that risked undermining confidence in Barclays’ whistleblowing procedures. Chief executives must act with a high degree of care and prudence at all times.

“Whistleblowers play a vital role in exposing poor practice and misconduct in the financial services sector. It is critical that individuals are able to speak up anonymously and without fear of retaliation if they want to raise concerns.”

Barclays said the regulators had not found Staley was unfit to continue as chief executive but it took the integrity of its controls seriously and had made “a very significant adjustment” to his 2016 compensation.

Staley said: “I have consistently acknowledged that my personal involvement in this matter was inappropriate and I have apologised for mistakes which I made. I accept the conclusions of the board, the FCA and PRA … and the sanctions which they have each applied.”

Last week Staley survived a bruising annual meeting, with one investor telling the Barclays chairman, John McFarlane: “The chief executive is irrevocably tarnished. I suggest you consider his continuation.

However, McFarlane told the meeting that 95% of shareholders backed Staley staying in his position. He repeated the board’s unanimous support for the chief executive and said: “I’d be quite interested to know how many people in this room have never made a mistake? Jes regrets his mistake. He does not deserve to resign.”

It is unusual for financial regulators to investigate and censure chief executives in the City and the FCA and PRA said this was the first case they had brought under the senior managers regime, which came into effect in March 2016.

However, the MP Nicky Morgan, chair of the Treasury committee, said she planned to question the FCA about the fine: “As the FCA’s decision notice states, a chief executive officer should set an example to the firm’s employees. Clearly Mr Staley has failed in this regard.

“In our next evidence session with the FCA, we’ll ask why it believes that the fines are appropriate and what it believes the implications are for the senior managers regime.”

Barclays had announced to the stock market in April last year that Staley and the bank were under investigation by the FCA, whose role includes monitoring bankers’ behaviour, and the PRA, which monitors the wider banking system.

New York’s Department of Financial Services has also been looking into Staley’s behaviour and on Friday, Barclays said it continued “to provide information to, and co-operate with, authorities in the US with respect to this matter”.