Mr Ellwood, who chairs the defence select committee, said “we need leadership” following reports of parties being held in Downing Street while Covid restrictions were in place.
It comes as hundreds of angry constituents contact their MPs.
The government has urged people to reserve judgement until senior civil servant Sue Gray’s inquiry is finished.
But Andrew Bridgen, Conservative MP for North West Leicestershire, said: “I don’t need to see what Sue Gray says to know that for me Boris Johnson has lost the moral authority to lead the country.
“If there’s another emergency where he has to call on the public to make sacrifices, he doesn’t have that authority. That makes his position in my book, as prime minister, completely untenable.”
MrBridgen is the fifth Conservative MP to publicly declare they have written to the chairman of the 1922 Committee – which organises Tory leadership contests – to say they have no confidence in the prime minister. Fifty-four Conservative MPs have to write a letter to trigger a vote.
Ahead of Boris Johnson’s apology on Wednesday, Mr Ellwood, MP for Bournemouth East and one-time defence minister, said Mr Johnson needed to “show some contrition” and get a grip of the situation – or he would be “out of office”.
Another former minister told the BBC: “Johnson is toast… if you were the chief whip looking at him you’d say he’s not fit to do any other jobs in government, you wouldn’t make him a junior minister, he doesn’t work hard enough.”
And a senior Tory MP said “there is a lot of scepticism around that there is anyone ready to take the reins. That buys Boris time. But he shouldn’t confuse that with another chance.”
Some Tory MPs said their inboxes have filled up after No 10 apologised to the Queen for two staff parties the night before the Duke of Edinburgh’s funeral.
Mr Johnson was not at either of those parties last April – but he faces questions over alleged Covid rule-breaking at No 10.
One senior backbench Conservative MP told the BBC they received more than 200 angry emails about the parties, with only five messages in support of the prime minister.
“Many colleagues now believe Boris won’t be leader at next general election… for many of us this feels terminal,” the MP said.
A Midlands Tory MP, who won his seat in a former Labour constituency in the 2019 election, said: “The inbox is bad, really bad.”
The gatherings which Downing Street has apologised to Buckingham Palace for took place on 16 April 2021.
The two parties last April involved about 30 people in total, and are reported to have converged at some point in the Downing Street garden, where they continued past midnight.
Staff were reportedly sent to a nearby shop with a suitcase, which was brought back “filled with bottles of wine”.
At the time, England was under “step two” restrictions that stipulated people could not socialise indoors, except with those from their household or support bubble. People could socialise outdoors in groups of up to six people or two households.
The behaviour of No 10 staff has been contrasted with pictures of the Queen sitting alone at the Duke of Edinburgh’s funeral because of Covid restrictions.
Boris Johnson’s spokesman said it was “deeply regrettable that this took place at a time of national mourning”.
Sue Gray is carrying out the investigation into alleged Covid rule breaking in Downing Street and government departments.
