Tokyo 2020 organisers have claimed a report estimating a four-times increase on their initial bid budget is based on misunderstandings over what was originally included in their cost estimation.
They have also outlined their opposition to plans to move venues for four sports, including a new rowing and canoe sprint facility 400 kilometres outside the host city.
A Tokyo Metropolitan Government Task Force, commissioned by new city Governor Yuriko Koike, last week claimed the budget would balloon to ¥3 trillion (£22 billion/$30 billion/€26 billion) unless drastic changes are made.
It blamed rising costs on poor leadership, governance and cost control measures before proposing venues be moved in order to save money.
Rowing and canoe sprint could be shifted to Tome in Miyagi Prefecture due to problems renovating the Sea Forest International Regatta course on Tokyo Bay, it proposed, while existing swimming and volleyball venues could be used.
The sports world was largely taken by surprise by this report but is now deeply concerned at the proposed venue changes, as well as the negative impact on the reputation of the Olympic Games.
At present, all the Task Force recommendations are advisory, and no firm proposals have been made by Koike.
Japanese agency Kyodo News have reported that International Olympic Committee (IOC) Coordination Commission chairman John Coates has already participated in an emergency teleconference call with Tokyo 2020 chief executive Toshirō Mutō and the city's vice-governor, Takashi Yamamoto.
Coates reportedly expressed "grave concerns" over the plan to abandon venues already agreed upon.
Other officials, including International Rowing Federation President Jean-Christophe Rolland, have also been in Tokyo over the weekend for meetings.
Hidemasa Nakamura, director of the Tokyo 2020 Planning and Finance Bureau, told NHK News Watch that the 400 per cent rise from the bid time budget of ¥800 billion (£5.5 billion/$7.5 billion/€6.5 billion) is partly due to the inclusion of various aspects which were not contained within the initial budget.
This includes transport and security funding - which may account for 30 per cent of the latest projection - as well as the costs of designing or removing facilities, or facilitating for the five new sports added to the programme this year.
Nakamura also disputed the proposed venue changes due to the need to take an "athlete first" approach, as well as a "Tokyo citizens first" one.
Such a move would risk detracting from the atmosphere of the Games and the ability for them all to interact in one Athletes' Village, he claimed.
Other sports officials have blamed the budgetary changes on a misunderstanding over the differences between operating budget and wider infrastructural ones.
Writing on Facebook, former IOC marketing director Michael Payne claimed "this is getting silly & media totally failing to do any research - $30b nothing to do with Games but rebuilding Tokyo infrastructure."
The fallout is still a major concern for organsers and the IOC, however, after they insisted last year changes to Tokyo's bid introduced as part of the Agenda 2020 reform process had saved around $1 billion (£658 million/€919 million).
Only three venues - for shooting, modern pentathlon and one for football - were initially due to be held outside an eight kilometres radius in a bid that was billed as compact.
But venues for basketball, taekwondo and track cycling have already since been moved outside that area.