'Worn out' Flick becomes first Germany coach to be fired
The 58-year-old became the first Germany head coach to be sacked since the position was created in 1926.
Sporting director Rudi Voeller, who coached the side to a 2002 World Cup final against Brazil, will take over on an interim basis.
German FA boss Bernd Neuendorf said the sacking was "unavoidable" after the team's recent run of "disappointing" results.
In a statement, Neuendorf said "with a view towards the European championships in our own country, we need a spirit of optimism and confidence", saying the decision was one of his "most difficult so far".
Voeller, who is reportedly unlikely to lead Germany to Euro 2024, said "Hansi has given everything to turn things around for the better after Qatar".
Germany suffered an embarrassing group stage exit at the World Cup in the Gulf state last year.
Voeller admitted "Hansi has worn himself out over the past few months.
"The Japan game showed us that we cannot make any more progress in this situation."
Out-of-work manager Julian Nagelsmann, who previously replaced Flick as Bayern Munich boss, has been touted as a possible replacement ahead of the tournament.
Former Eintracht Frankfurt boss Oliver Glasner, out of a job since May, has also been named as a possible successor.
The Germans were thoroughly outclassed by the Blue Samurai on home soil in the friendly in Wolfsburg, with only some acrobatic goalkeeping from Marc-Andre ter Stegen preventing the score from eclipsing 2001's 5-1 loss to England in Munich. 'Hansi out'
Germany have now lost four of their six matches since their early exit from the World Cup in December.
Euro 2024 kicks off in Germany in June 2024.
The home fans booed the German team at half-time, with some chanting "Hansi out" at the end of the match.
Germany have been historically reluctant to make drastic changes to the coaching position, with none of Flick's predecessors sacked.
Appointed in May 2021 after Joachim Loew's 15-year tenure ended, Flick was only Germany's 10th head coach since World War II.
Fresh off taking Bayern Munich to seven trophies in just 18 months in charge, Flick was victorious in his first eight matches as Germany boss.
Since then he won just four of 17.
Flick was bullish after Saturday's latest loss, telling reporters he "was convinced of what we are doing" and "wanted to continue" in the top job.
On Sunday, at a training session following the Japan defeat, Flick told fans "I'll keep fighting", but admitted to AFP subsidiary SID "many things are difficult to predict" in football.
Voeller said after the match emotions were running too high to make an immediate call on Flick's future.
"I suggest we collect ourselves and sleep on it," Voeller said, explaining "we are all a little in shock... a defeat like that hurts."
Taking over as interim boss in 2000, Voeller coached Germany to the final of the 2002 World Cup where they lost 2-0 to Brazil.
His first match as caretaker coach is on Tuesday, in a friendly in Dortmund against World Cup runners-up France.