The problem here is entirely political. Cities have the power to zone. Thus, supply depends entirely on whether local community leaders accept more housing. This housing, almost invariably, goes to outsiders, who would dilute the community’s politics, forming alternative social networks and possibly caring about different political issues.
When asking why things are the way they are, it’s always useful to ask, cui bono? In most American coastal cities today, developers like to sell expensive houses, current homeowners like to see their property values rise, and those who benefit from subsidized housing don’t want to have to compete against more outsiders for a fixed number of subsidized or rent-controlled units. Everyone wins, except the people who aren’t here yet: the migrants, the new college grads, and anyone else who is an outsider today but hopes to be an insider tomorrow.
Levy’s post ties these threads together really well. Read the whole thing.