An influential investor group has written to more than 60 publicly listed companies with only a single female director, including Domino’s Pizza and JD Sports, raising concerns over a lack of gender diversity and warning of a backlash if progress is stalled.

The Investment Association (IA), which represents 250 firms with £7.7tn in assets under management, has joined up with the Hampton-Alexander review team to criticise FTSE 350 firms with “one and done” boardrooms that have a single female board member.

Domino’s Pizza Group; JD Sports Fashion; the pubs group Greene King, owner of the Wagamama and Garfunkel’s restaurant chains; and the gambling company 888 Holdings are among the 66 firms to receive the letter, having been singled out in the latest Hampton-Alexander review in November.

A further three companies with all-male boards – the property firm Daejan Holdings, Millennium & Copthorne Hotels and TR Property Investment Trust – have also received the letter.

Targets set by the government-commissioned Hampton-Alexander review aim for women to make up at least 33% of FTSE 350 boards and leadership teams by 2020. “Investors consider diversity to be a critical issue for business success,” the letter said.

Chris Cummings, the IA’s CEO, said: “Investors have been consistently clear that they want to see greater diversity in the boardroom so it is totally unacceptable that one in five of the UK’s biggest companies are falling so far short. Companies must do more than take the tokenistic step of appointing just one woman to their board and consider that job done.

“There is also compelling evidence that boards with greater gender balance outperform their less-diverse peers. These companies must up their game and explain clearly how they are planning to meet the Hampton-Alexander targets, or risk investor dissent at their AGM.”

The IA has already pledged that its Institutional Voter Information Service will give a “red top” or highest warning level to firms that fail to address a lack of female representation. It says those issues “will also be noted against the re-election of the chair of the nominations committee” ahead of each company’s annual general meeting.

The shareholder advisory firm Glass Lewis has historically taken aim at nomination committee heads and voted against their re-election if they continued to be led by all-male boards.

But Martin Garcia Mortell, Glass Lewis’s director of research for the UK and Europe, told the Guardian earlier this month that attention would be turned to boards with only one woman among senior ranks.

The IA and Hampton-Alexander team are now urging dozens of firms, including Acacia Mining, the online gambling group 888 Holdings and the Frankie and Benny’s owner, Restaurant Group, to draw up a clear roadmap for how they plan to address the gender imbalance.

The letter said: “We would like you to set out how you intend to make genuine progress to improving board diversity. We therefore ask you to set out what actions the board is taking to ensure that substantial progress is made to meet the Hampton-Alexander 2020 targets.”