David Cameron is planning to shake up his top team and has already met his inner circle to prepare the ground.
One Cabinet source suggested employment minister Chris Grayling, who is on the right of the Tory Party, would step up into Mr Duncan Smith’s job.
However it is far from clear that the shake-up in September would be welcomed by the Work and Pensions Secretary.
Mr Duncan Smith is pioneering a series of reforms, including a universal benefit which is due to be introduced next year.
It would also be unusual to appoint a non-lawyer as Justice Secretary.
Senior Tories point out that Mr Duncan Smith is passionate about the need to reform the criminal justice system.
For his part, Mr Clarke is seen to have undermined the Tories’ traditionally strong position on law and order, while failing to convince the party’s MPs of the need to reform short prison sentences and focus on rehabilitation and reoffending rates.
The 72-year-old is telling colleagues he wants to remain in the Cabinet, but may be offered the post of Leader of the House of Commons, a traditional last job for Cabinet grandees.