Every “thought-leader” needs a catchy leading thought. Francis Fukuyama made his name and fortune with the definitive “one-liner” political meme The End of History?, which in the early 1990s seemed a smart way of describing the collapse of communism, and the “triumph” of the west. Since then, in the years in which history has clearly refused to end, Fukuyama, a senior fellow at Stanford University, has had various stabs at repeating that initial success. His new book, Identity, proposes the term “thymos” as the key to understanding our unnerving political moment.

“Thymos” (it does no harm, for credibility or book sales if the crucial thought-leading term is best understood by Ancient Greeks) comes from Plato’s Republic.It represents a kind of third way for a soul instinctively divided into two competing impulses – reason and appetite – by Socrates. If the former of those two made us human and the latter kept us animal, thymos fell somewhere in between. Most translations of The Republic suggest its sense for Plato as “passion”. For his purposes, Fukuyama takes it to mean “the seat of judgements of worth”, a kind of eternal status thermostat.

The importance of thymos, he believes, is not only that it has been seriously overlooked by other political theorists. Whereas classical economics tried to explain the world in terms of individuals acting to maximise their financial self-interest, behaviouralists, thinking fast and slow, have proved that our rational capacity is often undermined by more intuitive forces. Perhaps the most powerful of these, Fukuyama insists, is the desire for respect.

You might argue that this particular tension is the engine, say, of the 19th-century novel from Pride and Prejudice onwards but, in Prof Fukuyama’s telling, we have tended to lose sight of our fundamental human folly. From the 1930s onwards, broad Marxist class wars have fragmented into the competing interests of identity politics. Who we are, the kind of cultural or national group we identify with, is a potentially more insistent psychological force than our desire for more wealth or security.

“Thymos is the part of the soul that craves recognition or dignity,” Fukuyama suggests. When held in balance – “isothymia” – it gives rise to the demand to be acknowledged on an equal basis with other people. The Aretha Franklin battle cry of “R.E.S.P.E.C.T.”, which informed everything from feminism to the civil rights movement to gay liberation, could be heard as an anthem of isothymia. When that desire runs out of control, however, “megalothymia” – the need to be recognised as superior to others – is the result. The megalothymic individual or group is dissatisfied with the simple equalities and balancing forces of liberalism; it wants to take “big risks, engaging in monumental struggles, seeking large effects, because all of these lead to recognition of oneself as superior to others.” Sound familiar?

It is not entirely clear from Fukuyama’s thesis why thymos is a more useful way of explaining the irrational forces now shaping the political terrain than more familiar terms – ego, maybe, or just identity – but he takes it as a motivating factor that begins to explain both, say, the ruthless cunning of Vladimir Putin and the viral rage of the #MeToo movement. Trump and Brexit are a megalothymic backlash against the isothymic forces of multiculturalism and international cooperation.

In part, the latter phenomena also provide a retrospective negative answer to that question with which Fukuyama set out as a clairvoyant thinker in 1989. His famous essay The End of History?, which became the bestselling book, The End of History and the Last Man, was taken by many as an explanation of the ultimate triumph of liberalism over more extreme ideologies. Fascism had been defeated as an idea in the second world war; communism by the collapse of the USSR. There was only one system left in town – pragmatic liberal democracy.

Coming as it did at the historic moment when the Wall came down, Fukuyama’s loud caveats to this assertion (“it came with a question mark!”; “it was a deliberate provocation!”; “‘end’ meant ‘goal’ not ‘finale’”) were ignored. He has spent a good deal of the rest of his career – as the rise of undemocratic, illiberal China, the re-emergence of dictators and nationalism in former Soviet states undermined his case – determinedly adding nuance to that caricature. Thymos is, in this sense, his latest long explanatory footnote.

Last week, I talked to Fukuyama, now 65, on the phone from his home in California, about his latest analysis. He had just returned from 10 days explaining the pull of democracy to middle-ranking officials in Iraq. I began by asking him whether an overestimation of rational motives in the electorate was a universal condition or one particularly pertinent to our times?

He admitted it was certainly nothing new, but that our decade seemed to be supplying particularly startling examples of megalothymia in action. “You were told Brexit was clearly going to be very costly for the British economy, therefore it would be irrational to support Brexit,” he says. “But what has been proved is not only that a lot of people voting to leave the EU didn’t care about that, [but] they were actually willing to take a hit in terms of their prosperity. The issues were cultural and they were willing to pay a price, it seems, to have greater control of immigration. In general, the mistake a lot of elites have made is that you can have a politics led by economic rationality divorced from these feelings about national identity.”

But aren’t those populist movements also explained by a kind of desperate rationality? Since nothing else has changed personal economic circumstances for a decade, there comes a last resort when just upturning things looks logical?

“Globalisation has clearly left a lot of people behind. There is greater automation, greater inequality, but I think if you look in any detail at the voting patterns, both in the US with Trump and the UK with Brexit, a great many people who were not subject to the worst of those forces nonetheless voted [for Trump and Brexit]. And if you look at the number of foreign-born residents of Britain in the last decade, [the increase] is really quite phenomenal. I would be very surprised if you could ever have had that degree of social change without it producing some kind of backlash.”

He suggests the demands of political correctness, of R.E.S.P.E.C.T for difference of all kinds, has both generated and masked those latent feelings. In broad terms, does he believe that in insisting on acknowledging difference we have lost sight of the ties that bind us?

“You have leftwing and rightwing versions of identity politics. The leftwing version is longer-standing, where different social movements began to emphasise the ways they were different from mainstream culture and that they needed respect in various ways. And then there was a reaction on the right, from people who thought, ‘Well, what about us? Why don’t we qualify for special treatment as well?’ Politically, it is problematic in that it undermines a sense of citizenship. And now you get extremism on both sides.”

Fukuyama suggests his book would never have been written without the election of the 45th president of the United States. He was, he concedes, no better at predicting that sorry fact than anyone else, though nearly 30 years ago in The End of History and the Last Man he did, even then, single out Trump as the most egregious example of an “excess of freedom – of the arrogant display” of megalothymia that democracies must tolerate. I suggest that the portents of the near-decade of austerity after the financial crash, in addition to wars that produced mass migration, must have set off some alarm bells. The precedents were never great?

“I thought the financial crash would produce a more straightforward kind of leftwing populism,” he says. “This huge recession was created by plutocrats on Wall Street. But instead what you got was the Tea Party and rightwing activism. Part of the explanation of that is the way that these identity issues have evolved since the 1930s: people think about smaller inequalities rather than these bigger class battles.”

Does he see Trump’s scattershot illiberalism as a blip or a new reality?

“He is definitely not a blip,” Fukuyama says. “The most disturbing thing is the amount of support he gets despite all the damaging, racist, absurd things he does. There seems to be a third of American people who approve of what he is doing and another 10% or 15% who will hold their noses and tolerate him because of tax cuts.”

The other effect appears to be to drive the Democratic party further to the left, creating a situation in which the centre loses power. “Trump instinctively picks these racial themes in order to drive people on the left crazy and they get more and more extreme in their response. I think he sees an opportunity to divide people and make the Democrats less united as a competitor.”

When you witness these forces, I suggest, and see, for example, how the indefensible is defended by Brexiters with appeals to a fake “Blitz spirit”, it seems miraculous that they were held in check so long. The message of The End of History? seemed guardedly optimistic; his thesis appeared to chime with Martin Luther King’s assertion that “the arc of the moral universe is long, but it bends toward justice”. Is he less persuaded by that optimism now?

“I’m not sure I’m optimistic,” he says. “In some ways, I think it is surprising the backlash did not come sooner.”

Fukuyama himself is a third-generation immigrant to the United States. His paternal grandfather fled the Russo-Japanese war in 1905 and settled on the west coast, where he opened a corner shop. Fukuyama’s father was a minister in the Congregational church and received a doctorate in sociology from the University of Chicago. His mother, born in Kyoto, was the daughter of the founder of the economics department at Kyoto University. I ask him how much questions of assimilation and identity were live issues around his dinner table growing up. Does he think of himself as a Japanese-American?

“I never ever thought of myself like that at all,” he says, quickly.

Because?

“I didn’t grow up in any kind of Japanese community.”

He never learned to speak Japanese – there seemed no need. “I thought America was a really great country because it was possible to get ahead without anyone putting barriers in your way because of your ethnic background.” To this day, he says, he gets annoyed when he gets appeals from Stanford campus groups wanting him to go meetings with “some Asian American group”. He doesn’t feel that is him at all.

Did his parents feel the same way?

“My father could speak Japanese, but not well. He grew up in the Los Angeles school system in the 1930s and 1940s. My mother was born in Japan and came here after the war. She spoke English with an accent, my father didn’t. I remember my mother saying at various points that she thought Americans were racist and she experienced prejudice and so forth. But I remember my father saying that there had never been a moment in his life when he felt that his Japanese background held him back. I thought what that came down to is, ‘If you speak English like an American you will be accepted’. My mother couldn’t. If you look at Asian Americans in the United States they have done extremely well in comparison to whites on just about every measure.”

When he watched the Ku Klux Klan-style rally at Charlottesville last year, how did he feel about those long-held convictions?

“The issue of American national identity being based on a certain idea of citizenship was a battle that seemed to have been won by the civil rights movement,” he says. “All of a sudden, you see these people trying to get back to an ethnic understanding of what it means to be an American. That is very bad. Trump is giving aid and comfort to these people.”

Part of Fukuyama’s book proposes some solutions to this fractured politics. He emphasises the urgent need for politicians who can speak to the concerns of all citizens and not merely their base, who can provide a national vision that goes beyond narrow identity concerns. He advocates exploring policies such as national service to encourage young people to realise shared aims with people of all backgrounds. Can he really imagine that happening in the current political climate in the US or the UK?

He concedes that “it couldn’t happen except on a voluntary basis”. “Under Clinton,” he says, “there was a scheme where young people could volunteer to work in an inner-city public school or do other kinds of service. I think that could be expanded. But I think it would need a war or some big external event to make that service compulsory.”

Another urgent area of concern, he suggests, is the need for politicians to address the ways that the internet has acted as an accelerator for identity politics, with social media allowing individuals to listen to people in just their own narrow group and not have any sense of national or wider conversations.

“The idea was for it to be a tool of democracy, to give people access to information and therefore to power,” he says. “But I think all of the editors and the ‘gatekeepers’ and the fact-checkers that characterise the old media were actually extremely useful – in simply slowing down the spread of false information and guaranteeing a certain minimum level of quality. All of that is gone now. So basically anything you see on the internet seems as good as anything else you see on the internet.”

In the absence of such policies, how does he think politics will play out in the States? Does he have faith that the Mueller process will start reining in all that megalothymia?

“Check back after the midterm elections,” he says, with half a laugh. “In a democracy, the biggest check we have on power is an election. If the Republicans retain control of both houses, that will be taken as an affirmation of all the stuff they have been doing and that would be very bad. On the other hand, if the Democrats win the House of Representatives we will suddenly have this flood of information coming out from the investigation. This administration has been extremely destructive in every policy area – whether it is environment or health or housing – and I think they need to be held to account for that.”

Despite all appearances to the contrary, does he retain a conviction that the more benign forces of liberal democracy will prevail again – that western democracy can be brought back into balance, that the centre can be made to hold?

“It comes back to leadership,” he insists. “We wouldn’t be in this situation if there hadn’t been these political entrepreneurs who saw this gap and ran through it.”

That being said, says perhaps the best-known soothsayer in modern political life, “it is actually very hard to predict what happens next”.

Identity: Contemporary Identity Politics and the Struggle for Recognition is published by Profile Books (£16.99) on 4 October. To preorder a copy for £14.61 go to guardianbookshop.com or call 0330 333 6846• Identity: Contemporary Identity Politics and the Struggle for Recognition by Francis Fukuyama is published by Profile Books (£16.99). To pre-order a copy for £14.61 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99