International Olympic Committee (IOC) vice president John Coates has warned the Kasumigaseki Country Club, the venue for golf at the Tokyo 2020 Olympic Games, that it must allow women to have full membership or organisers will be forced to find another venue.
Fierce criticism has been pointed towards the private club, which does not allow women to become full members and also limits the days they can play there.
It has led to calls for the venue in Saitama prefecture to be switched to an alternative location such as the Wakasu Golf Links.
The Tokyo 2020 Organising Committee have also written to the club's Board of Directors, urging them to make changes as it does not sit in line with the Olympic Charter.
Yuriko Koike, the Governor of Tokyo, has also criticised the rules.
Club members were due to take a vote earlier this month on whether they would change current regulations, but a Board meeting postponed a decision on the matter.
"We made quite clear that there has to be gender equality," Coates, who is the Tokyo 2020 Coordination Commission chairman, told Reuters.
"If they can't achieve the gender equality then we have to get another course, but the organisers are very confident that they will.
"They either will or they won't.
"There's plenty of time to move to another golf course if we have to."
Japanese national broadcaster NHK claimed last week that the club’s policy was not included in explanatory material when the International Golf Federation (IGF) approved its selection as host venue for the tournament in 2012.
According to NHK, a member of the Tokyo Metropolitan Government's Bid Committee warned the club that its policy could bring about problems for violating the Olympic Charter's rules on equality, and was told things would be fixed.
Assuming the issue was resolved, the Bid Committee member reportedly chose not to mention the policy in documents prepared in English.
A Kasumigaseki Country Club official confirmed to NHK that the club had been warned, but said it had replied: "If this becomes a problem, we will have a mutual discussion at that time".
It is believed some of the Board said they needed to allow women to become full members.
However, others felt more club members needed to be asked before making any alterations.
Board chairman Kiichi Kimura told reporters after the meeting that the matter has become frustrating.
A unanimous vote of the 15 Board members is required for changes to be made.