The uncertainty over Durban's ability to stage the Commonwealth Games in 2022 will bring back unhappy memories.
A South African city was due to host the British Empire Games (as they were called) in 1934 but was forced to hand them back, partly because of economic difficulties but also because a colour bar operated in the country.
When Canadian journalist and official Melville Marks Robinson, known to all as Bobby, initially proposed Games for the British Empire to be held in 1930, the South Africans were very enthusiastic.
That was hardly surprising for they were sports mad. The Springbok rugby union team first played the British Isles (later to become the Lions) in 1891. A South African, Sir Abe Bailey, had been a driving force behind the foundation of the Imperial Cricket Conference in 1909, and the more short-lived concept of a triangular test cricket tournament, held only once in 1912.
They had first sent competitors to the Olympics in 1908 and by the late 1920s, they were a fully fledged member of the Olympic Family. There was even an International Olympic Committee (IOC) member in South Africa.
At the time, many in Britain and other Commonwealth territories were deeply concerned with the question of amateurism at the Olympics. They felt amateur status was compromised by payment of money for "broken time" - compensation for work lost while training. A completely amateur Olympics for the Empire was an attractive proposition.
An unnamed South African authority even went so far as to suggest that "in the near future the dominion will have to decide between participation in the Olympic Games or in the Empire Games". However, the organisers of the inaugural Empire Games held in the Canadian city of Hamilton were quick to quash this notion. They even invited IOC President Henri de Baillet-Latour as a guest.
Most of the South African team gathered in London, before proceeding by sea to Canada on the liner Empress of France. They were joined by Bevil Rudd, who had won 400 metres gold for South Africa at the 1920 Antwerp Olympic Games. Rudd sailed with the team as athletic adviser. His wife acted as chaperone for the female competitors. All members of the team were white.
The Games proved a great success and South Africa's first gold medal was won by Oonagh Whitsitt in springboard diving. "Miss Whitsitt's success is a triumph for natural athletic ability," announced the Star newspaper in Johannesburg. "She has vindicated the shrewdness of the selectors. The South African success in this branch is higher than in swimming proper."
In the main stadium, Johannes "Snaar" Viljoen was voted by competing rivals as "the most versatile athlete of the whole meeting". Not only did he win gold in the high jump but he also took bronze in the long jump and set a Canadian all-comers record in the heat of the 120 yards hurdles, beating Lord Burghley who was described as "the English crack". His Lordship won the final but Viljoen was not disgraced by fourth. He also raced in the 220 yards and 440 yards hurdles, as well as throwing the hammer.
Harry Hart also doubled up and won both shot and discus titles, and the press back home showed their delight, describing him as "a veritable Apollo who long ago established himself as an outstanding all rounder".
A columnist writing under the name of "Olympic" suggested that "the success of the athletes should provide a fillip to the Cinderellas of South African sport".
Shortly before Games came to an end, officials gathered to established a British Sports Federation. At their first meeting it was unanimously agreed that the Empire Games should be perpetuated.
South African team manager Captain Norman Welsford was present and it became very clear that his country was keen to host the next event.
One official reasoned that "South Africa can never hope to stage an Olympiad with any degree of success because of its distance from the nerve centres of sport".
Before they left Hamilton it had been agreed that South Africa would host the next celebration in 1934, most probably in Johannesburg.
Welsford himself was delighted with the decision. Johannesburg was, he said, "a great city with great hospitality that would be an excellent centre for the athletic contests".
Except that not all members of the Commonwealth would be welcomed.
Racism in sport and life was still evident in many countries, including Great Britain and the United States, but particularly in South Africa. Back in 1913, the Natives Land Act restricted ownership of land among black Africans and marked the beginning of racial segregation, a road that would eventually lead to apartheid.
"The South African native found himself not actually a slave but a pariah in the land of his birth," noted activist Sol Plaatje, a co-founder of the group which eventually became the African National Congress (ANC).
In many sports, a strict colour bar operated and in 1931, the South African Athletics Association confirmed that they would continue to support it while staging the Empire Games. The Reuters news agency reported that the decision to enforce the colour bar had been taken "even at the risk of appearing discourteous".
In practical terms, the decision would have excluded almost the entire Indian team from the Games and competitors such British Guiana born Phil Edwards, who competed for Canada in the 1928 Olympics.
For London's press this was a "ticklish" issue but the great West Indian cricketer Learie Constantine (later to be made Lord Constantine) was in no mood for laughter. Constantine had just returned from a tour of Australia where the West Indies players had found their hosts to be very hospitable.
"While recognising that the South African authorities have felt themselves bound to defer to the prejudices of their public I feel they have missed a great opportunity of setting an example to their own people," he said.
"This is a serious blow to the racial understanding which has been slowly building up on the world's playing fields. It is also a serious blow at the prestige of Empire athletics which cannot fail to suffer from a sporting point of view."
Rudd's interpretation was something less than wholesale condemnation. "The colour problem in South Africa may occasion acute tension," he ventured.
"It is difficult for those who have not lived in South Africa to comprehend the dilemma in which the South African authorities are placed. I sincerely hope the decision will not be regarded as a piece of arrogance until the situation which gave rise to it has been sympathetically examined."
He pointed out that when the New Zealand All Blacks had toured South Africa, there had been no Maori team members "by mutual consent" as if this was something to be proud of.
Rudd did at least wonder "whether a dominion can accept an invitation to hold the Empire Games and then arrogate to itself the right to debar athletes within the Empire from competing?"
The response to the South African position was strong, particularly from Canada which had received so many plaudits for hosting the first Games.
In London, an unnamed Daily Telegraph columnist took up the argument that "the logically minded may ask why it is that South African representatives do not quibble at meeting coloured opponents in international competition in Europe and America and yet jib at doing the same thing in their own country".
South Africa's athletes had after all done just that in Chicago, when they joined forces to form a combined British Empire Team to take on the US shortly after the Empire Games in 1930.
The race question was compounded by financial difficulties. The Johannesburg press sounded a warning about the cost of staging the Games at a time when the Great Depression had hit the economy very hard across the world and particularly in South Africa. At this time, they had only just begun work on a pipe to provide fresh water for Pretoria and the provision of electric lights to other areas.
"It is a charming ideal but when we reflect on the miserable attendances at local athletics meetings, then the difficulties begin to loom very formidably," one report said. Some baulked at the cost which was estimated at £50,000 and warned that "clearly South Africa's entertainment would need to be on a scale so much less lavish than the Canadians as to lead to invidious and embarrassing comparisons". Calls were made for a "careful examination of ways and means".
In London it was noted that "now early enthusiasm was tempered with the sober consideration of the practical problems, there is a certain amount of well founded uneasiness".
When the British Empire Games Federation gathered in 1932, it was announced that Johannesburg would not after all be holding the Games. No mention was made in press reports of the colour bar, simply that South Africa and Australia "would be unable to assume the responsibility".
It was later confirmed that London would host the Games in 1934. The centrepiece was to be the White City Stadium which had been used for the 1908 Olympic Games. Swimming took place at Wembley. Cycling was held at Fallowfield in Manchester.
The Daily Telegraph correspondent "Fair Field" wrote optimistically that "it is safe to predict that these Empire Games will have all the glamour that attended the 1908 Olympic Games - and more popularity".
Competitors from India, Hong Kong and the West Indies took part. This would have been impossible had Johannesburg hosted the Games.
As with the Olympics, the Organising Committee produced a report after each Games. This was known as the official history and included a brief summary of previous Games. There were no references to South Africa's withdrawal as hosts.
They continued to send all white teams to participate in both Empire and Olympic Games but the election of the Afrikaner National Party Government in 1948 signalled the advent of hard line apartheid. In 1950 the Population Registration Act introduced eight different racial categories.
Independence came to many African nations during the 1950s and 1960s and opposition to South Africa's racial policies grew. They did not take part in any Olympic or Commonwealth Games after 1960 but the spectre of apartheid was still felt.
At the 1962 Games held in Perth, Ghana threatened to boycott any boxing matches controlled by a South African official. Continued British and New Zealand rugby tours sparked threats of boycott.
An agreement discouraging sporting contact was signed at Gleneagles in 1977, describing apartheid as "a dangerous sickness and unmitigated evil". In 1986, however, many nations stayed away from the Commonwealth Games in Edinburgh, furious that the British Prime Minister Margaret Thatcher had refused to apply economic sanctions to South Africa.
Distance runner Zola Budd and swimmer Annette Cowley were both chosen to represent England. Both were South African born which helped stoke the flames. Neither were ultimately permitted to take part but the damage had been done and for a time the very future of the Commonwealth Games seemed in doubt.
South Africa themselves did not compete again until apartheid had ended and made their first Commonwealth Games appearance as the Rainbow Nation in 1994.