But Robertson, who played a key Governmental role in helping London stage a successful Games, said it would be tough to host them again in Britain in the foreseeable future due to the huge international demand for the world's greatest sporting event.
"I would love to see the Games back here," Robertson told insidethegames.
"I have spent the last seven years of my life working on this project and I absolutely believe that the Olympics is not only the world's greatest sports event but that it is great for a country.
"You only need to look around to see that it has been great for our country in every single way.
"It has transformed our reputation as a sporting destination, it has given us enhanced economic opportunities, it has given us enhanced social opportunities and we have reminded the world of exactly what this country has to offer.
"So of course I would love to see it come back here as soon as possible.
"But it is just my knowledge of the way that international sport works that suggests it may not be quite that simple.
"Given that we have made such a success of the Games, quite a lot of other countries will be eyeing the Olympics up and wondering if it can do the same for them.
"And to be fair, the Olympics should not be confined to three or four destinations that can do it well.
"It needs to go out to new destinations and new markets and bring Olympic magic to places like Africa, the Indian subcontinent and the Far East.
"These are all the big emerging markets and the Games needs to go there.
"I know in my heart of hearts that the Games must go there before it comes back to London.
"So while I would love to see them back, I know, having done a lot of international sports politics over the last few years, that there are other parts of the world that, for the good of the Olympics, it needs to go to first."
The Sports Minister also admitted that he would like to see the Commonwealth Games staged in England at some point in the next 20 years but said that like the Olympics and Paralympics, it could be a difficult event to secure.
"The Commonwealth Games are obviously in Glasgow in 2014 and then it is the Gold Coast in 2018 so the next bidding possibility is 2022," he said.
"On that issue, we would first have to look at who is bidding for it.
"Again, I am a huge supporter of the Commonwealth Games and have been involved in several Commonwealth Sports Ministers meetings.
"So I would love to see the Commonwealth Games come here.
"But with the Commonwealth Games, the same sort of logic applies as it does with the Olympics in that it is an event that moves around the world.
"I think there will be quite a lot of countries that might look at a Commonwealth Games bid as a way of teeing up a future Olympic bid.
"So I would be very surprised if the competition in 2022 and 2026 wasn't very tough indeed."
Contact the writer of this story at [email protected]