Would it be more affordable if every country participating contributed an amount proportional to their GDP to help lessen the financial cost and burden for the host nation?
This would allow for smaller countries to host the Olympics and not have only a couple of cities rehosting it 2-3x when other cities have yet to even host once.
Just give poorer countries aid. Gifting them huge stadiums, etc. is just giving them additional burdens in the long run.