The shortlist was announced on Wednesday in Quebec City by the 15-member executive board headed by Jacques Rogge, the IOC president.
Doha and Baku were rejected for a second time after failing to make the final list for the 2016 Games.
The remaining three cities will now embark on a lobbying period with the winner named on September 7, 2014, in Buenos Aires.
Japan's Tokyo hosted the Olympics in 1964, while neither Istanbul in Turkey nor Spain's Madrid have held the Games.
Madrid is bidding for a third consecutive time, Tokyo a second time in a row and Istanbul a fifth time overall.
Doha, which was proposing to hold the 2020 Olympics in October rather than the usual July and August schedule to avoid the searing summer heat, is already scheduled to host the 2022 World Cup football tournament.
Qatar possessed the financial muscle to carry off the 2020 bid, but reportedly could not convince the IOC to take a chance on holding the Summer Games in October.
It was not just the threat of high temperatures that frightened away the IOC but also the prospect of low television ratings if the Olympics was to go up against other major sporting events.
The IOC made it clear in the evaluation report that holding the Games in October was a non-starter.
"In July [and] August, people have more leisure time. There is therefore a risk that an October Games would become a
'weekend Olympics Games' and with a reduced demographic reach, broadcasters would have difficulties in attracting the same audience levels in terms of working people and youth."
If Doha's bid team were consoled by Qatar's successful bid to stage the 2022 FIFA World Cup, they were not showing it.
Baku aims to host the 2020 Olympics
"I think the two countries that were eliminated are emerging markets and I think it is a missed opportunity for the IOC," said Mayassa Al-Thani of the Doha bid.
"The traditional dates don't suit Doha and that is nothing we can change and we provided the dates we thought were most
"We believe we did everything in our capacity and in the end it was a decision of the executive board we have to respect despite our disappointment."
Flush with cash from natural gas reserves, Azerbaijan had invested more than a billion dollars in sports venues in the past 10 years and is likely to try its luck along with Doha luck again in 2024.
Rome pulled out of the running in February because of the country's efforts to head off a debt crisis.
London will host the 2012 Summer Games from July 27, while the Brazilian city of Rio de Janeiro will stage the 2016 Olympics.