The decision comes after the team finished bottom of Group 3 in the qualifiers for next year's tournament in Angola.
"The national football coach has been relieved of his duties," said FERWAFA general secretary, Jules Kalisa, according to the BBC.
"His contract was expected to end in April 2010 but we have reached a mutual agreement. We wish him luck in his future engagements," added Kalisa.
Deputy coach Eric Nhsimiyimana has been appointed on an interim basis.
He will prepare the team for the forthcoming East and Central Africa Senior Challenge Cup championships in Kenya.
Last month the country's sports minister Joseph Habineza said Rwanda was close to signing former England and Liverpool star John Barnes as the new national coach.