And a big hole is what we appear to have now - or at least a blank sheet of paper - following Japanese Prime Minister Shinzō Abe’s decision to “start over from zero” on the venue plan because of spiralling costs.

The decision, surprising in its abruptness, gives rise immediately to several trains of thought:

1. The appetite for prestige leisure/national vanity projects in countries democratic enough for their citizens to make their views clear remains small.

After the post-war surge, the Japanese population has been getting by in the face of sluggish economic growth for considerably longer than most of us in Western Europe.

At the same time, the younger generation has perhaps taken a leaf out of Westerners’ books by becoming more individualistic and assertive.

It is worth bearing in mind that real gross domestic product (GDP) growth in Japan in the run-up to its first Summer Olympics in 1964 was as follows: 1960 - 13 per cent; 1961 - 12 per cent; 1962 - 9 per cent; 1963 – 8.5 per cent; and 1964 - 11.5 per cent.

You could even argue that ordinary people, or the politicians who interpret their wishes, have, for the moment, usurped much of the power to determine where the Olympics are held and what sort of venues they are held in away from the International Olympic Committee (IOC).

2. The main Olympic Stadium remains a fiendishly difficult conundrum from a legacy perspective in almost any city.

Calls for a 60-80,000-capacity athletics venue - the anchor Olympic sport - are few and far between outside the Olympic and Paralympic Games.

At the same time, football clubs - the obvious long-term occupants in many countries - have grown too picky to want a running track around their pitch.

Engineers can square the circle, but of course this inflates the cost, whether you go back to square three or four, as in London, or install seats clever enough to move into different configurations depending on the event.

3. Of course, you can still do a lot with ¥180 billion (£928 million/$1.4 billion/€1.3 billion), apparently the new cost ceiling for the Tokyo 2020 venue.

And let’s hope they come up with something striking: every Olympics, I think, needs one stand-out, special Bird’s Nest-type project.

Not that these signature venues need be huge: the Kenzo Tange-designed Yoyogi National Gymnasium and the Nippon Budokan constitute, I would say, an outstanding architectural legacy from those Tokyo 1964 Games.

At the same time, with Rome, for example, talking about pressing six-decade-old venues into service for its 2024 Olympic bid and Tokyo itself using some of the sporting infrastructure - including both buildings named above - from 1964, you can hardly blame people and politicians for subjecting new-build projects to close scrutiny.

Especially as it is barely 15 years since Japan undertook its last concerted stint of sports infrastructure building, in preparation for the 2002 FIFA World Cup.

4. The most frustrating thing about this for Lausanne may be that the National Stadium cost issue has remained the prime focus of attention, notwithstanding a) the substantial cost savings that have been realised at other venues and b) the remarkable success of Tokyo 2020’s domestic sponsorship programme.

During the recent visit to Tokyo by the IOC’s Coordination Commission, chairman John Coates took justifiable pains to run through a list of figures underlining how little Tokyo 2020 was planning to spend on new venues compared to other recent Summer Olympic hosts.

Yet, insofar as the denizens of the streets of Peoria, Peterborough and Perugia think they know anything about Tokyo’s Olympic plans, it may well be along the lines of, "Oh, those extravagant Olympic so-and-sos, there they go again!"

That said, taking a longer-term view, Coates and his colleagues may well end up with a less costly, more popular Olympic project to publicise as a consequence of Abe’s initiative.

And, in spite of the false start and inconvenience caused to organisers of the 2019 Rugby World Cup, who were expecting until this morning to have use of the brand new National Stadium to showcase their tournament, Japan is not the sort of place where you instinctively worry that you might end up with a main stadium that isn’t ready in time.

5. This latest alteration to the blueprint does make me question even more the 2020 host-city contest, or rather the way in which it has apparently been deemed perfectly okay to implement root-and-branch change to the plan IOC members voted for, mainly in the name of Agenda 2020, without this being thought to have undermined the selection process in any way.

Yes, of course, Tokyo 2020’s ambitious initial plans would have been a tough PR sell at a time when so many sports fans around the world have been struggling to make ends meet.

Yes, Bid Book proposals always change – to an extent.

But, at this rate, one starts to wonder whether anything one actually encounters come 2020 will bear much of a resemblance to the original project.

Probably the judo: I think the world really would have to shift on its axis before the 2020 Olympic judo competition was moved out of the Nippon Budokan.

6. Further afield, I cannot believe global headlines about Japan’s $2 billion stadium, whose price tag nearly doubled from its original cost before the Prime Minister said enough was enough, are going to make it any easier for Boston’s 2024 Olympic and Paralympic bid to gain better traction.

7. This latest twist to the Stadium saga also does nothing to dissuade me from my hunch that one major legacy of Tokyo 2020 may be to accelerate the pace at which metropolitan Japan becomes more diverse and multicultural.

The tight labour market is taking much of the blame for rising costs; I think Tokyo 2020 President Yoshirō Mori said during the last IOC Coordination Commission visit that ordinary construction costs had risen 24 per cent because of an imbalance of supply and demand.

Given the drive to reconstruct areas damaged by the 2011 earthquake and tsunami, the claim does not seem implausible.

One means of addressing this - and I say this in awareness of the controversy still raging in Qatar - would be to increase supply by admitting more overseas building workers.

From the reaction when I tentatively raised the subject in Tokyo, I sense this is a delicate area the ramifications of which I comprehend scarcely at all.

But with Abe’s decision increasing the likelihood that contractors will be racing against deadline to ensure that stadium construction keeps pace with its revised schedule, the scope for market conditions to continue to work against Games organisers seems obvious.

An increase in the skilled labour supply in such circumstances might just help to relieve the pressure.