Several boxing fans have been calling for the heavyweights to fight each other since 2016 when all three men held versions of the world title.
Wilder recorded an emphatic first round knockout victory over Dominic Breazeale in Brooklyn at the weekend, and Fury believes the American is the only man looking to arrange a unification match.
“As a fight fan, I wish that Joshua and Wilder fought each other and I believe that Deontay Wilder is the man who is willing to fight,” Wilder told Behind The Gloves.
“And I’m British so I’ll admit it: I don’t think Joshua’s team want the fight, and that’s being brutally honest.
“Wilder wants to – he has fought the best in the division, he’s fought the avoided Luis Ortiz, he’s fought the avoided and elusive king of the ring, the dancing destroyer, the singing master, the colourful character that is me. Joshua hasn’t, facts are facts. Simple as.
“Until they fight and there is a victor, then I feel it’s Team AJ that don’t want the fight… Wilder has proven to me that he’s a fighting man and he’ll fight. Simple as. But I haven’t seen that from Joshua’s team.
“I want them two to fight each other. I want (AJ) to step up to the plate and prove he’s worthy of being a world champion.
“I’ve stepped up to the plate, I’ve been to Germany and dethroned (Wladimir) Klitschko, I’ve been America and beat Deontay Wilder, when has he stepped up? To beat an old Klitschko?… fight somebody who can fight back – Wilder or Dillian Whyte.
“Fight. You’re a fighting man, supposed to be a world champion, fight men who are dangerous.”
Joshua will defend his IBF, IBO, WBA and WBO titles against late replacement Andy Ruiz Jr at Madison Square Garden on June 1.