He leaves the club just one day after they were thrashed 5-1 at his former club Frankfurt.
They are currently fourth in the Bundesliga, four points behind leaders Borussia Monchengladbach after 10 games.
Announcing the decision, CEO Karl-Heinz Rummenigge said: “The performance of our team in recent weeks and the results have shown us that there was need for action.
“Uli Hoeness (president), Hasan Salihamidzic (sporting director) and I had an open and serious conversation with Niko on this basis on Sunday with the consensual result that Niko is no longer coach of FC Bayern.
“We all regret this development. I would like to thank Niko Kovac on behalf of FC Bayern for his work, especially for winning the double last season.”
Salihamidzic added: “I now expect a positive development from our players and absolute motivation to achieve our goals for this season.”
Kovac admitted it was the correct time to leave the club he represented for two years as a player, saying: “I think this is the right decision for the club at the moment.
“The results, and also the way we last played, made me come to that decision.
“My brother Robert (assistant coach) and I thank Bayern for the last one and a half years. During this time, our team has won the championship, the DFB Cup and the Super Cup.