Six state unions have written to Rugby Australia chairman Hamish McLennan, demanding he resign immediately and calling for an extraordinary general meeting of the board to oust him, if he refuses.
McLennan has come under increasing pressure, since Australia crashed out of the poolstage of the World Cup for the first time in October, under the leadership of coach Eddie Jones.
"We do not believe Mr McLennan has been acting in the best interests of our game," said the unions, who represent every Australian state and territory, except New South Wales and Victoria.
"We no longer have any trust or faith in his leadership, or the direction in which he is taking rugby in Australia."
The hiring of Jones and sacking of Dave Rennie in January was driven by McLennan, but backfired spectacularly, when the former England coach selected an inexperienced squad for the tournament.
The letter says McLennan exceeded the power of his position by "exerting an undue influence on the operations and executives of Rugby Australia".
It also criticised McLennan's much-trumpeted strategy of trying to poach playing talent from the wealthier and more popular rugby league code, which RA chief executive Phil Waugh has said will cease.
Jones resigned at the end of October, saying the structural changes to the game required to make the Wallabies competitive were unlikely to be realised in the short-term.
Waugh has launched a plan for structural reforms and announced that the New South Wales Rugby Union would be the first to hand over their high performance programme to centralised RA control.
Other states, most notably the powerful Queensland and Australian Capital Territory unions, have qualms about the wholesale takeover of their high performance units, which include their Super Rugby franchises.
Both unions were signatories to the letter and McLennan suggests that restructuring of the game is behind the attempt to oust him, which he vowed to fight.
"This will be the defining moment for the battle of rugby," he told the Sydney Morning Herald. "It's all about money and control, and we have been failing for years.
"We live in interesting times. This is about principles.
"They are actually not putting the game first, and it's about self-interest and parochialism."
Reuters