While everybody's been sweating over the housing and labor markets, the office market has been streaking toward a hard landing.

The Federal Reserve's new hawkish stance has pushed stocks lower and heightened concerns about the direction of the U.S. economy over the coming year. Anxiety has been centered on labor and housing crashes, but investors, city mayors and economic developers have kept up hopes for improvement in the office market. It's best to let go of that wishful thinking.

I can offer two reasons why office-market watchers aren't as pessimistic about the future of the sector as they should be — and not nearly pessimistic enough compared with all the negativity about housing and employment. First, they're still thinking about office activity as a pandemic-related behavioral change that's still in the process of normalizing. Reality: Office buildings are interest-rate and economically sensitive assets with deteriorating fundamentals despite still-booming employment growth.

Second is a lack of imagination. We know what it's like when unemployment hits 10%, home prices plunge and there are millions of foreclosed homes. But we — particularly we in the media whose professional lives are so wrapped up in cities and offices — struggle to imagine how that plays out in business districts.

The situation is already bleak and not only do we need to be able to see that, we need to start thinking about how to respond when things get worse. The stock price of Vornado Realty Trust, the Manhattan-centric office and retail REIT, is already below its March 2020 lows and, incredibly, threatening its March 2009 lows. Other office REITs, including Boston Properties, Cousins Properties and SL Green Realty are closing in on their March 2020 lows.

There should be no big surprise about why. Daily office traffic, while continuing to recover slowly, remains far below pre-pandemic levels. Vacancy rates are rising, particularly in older buildings, as leases come due. Higher interest rates are headwinds both for real estate values and for stock valuations.

And now the Fed is signaling a "higher-for-longer" interest-rate policy that will keep valuations suppressed and make refinancing debt more difficult. To the extent the Federal Reserve's 4.4% unemployment rate forecast for 2023 turns out to be accurate, it would mean a million more unemployed workers than we have today — and unemployed people don't need office space.

That doesn't even take into account belt-tightening that employers will do over the next year to cut costs as they're feeling squeezed by a slower-growing economy. Office space is an obvious lever to pull at a time when remote and hybrid work has shown to be effective — if not always ideal from employers' perspectives. Even big technology companies, which had been a bright spot for office landlords in the pandemic recovery, might decide to rethink their office footprints when they're scrutinizing their costs.

A bigger problem for the office sector is that it's not clear how well it would recover after any Fed-induced recession. If such a recession eventually led to lower interest rates, housing would probably bounce back quickly. The labor market continues to be the strongest part of the U.S. economy, so it might be the last to feel a recession and the first to be restored. But once a 1970's office building empties out, it's not clear that demand would come rushing back in an ensuing recovery. Maybe some office buildings could be converted into housing, but not all buildings are readily convertible.

Now consider that even empty or lightly-occupied office buildings cost money to operate: electricity, heating and cooling, cleaning, maintenance, security, taxes. And those costs have largely gone up with inflation. Offering tenants cheap rent might stoke demand but also might not provide a return for building owners. Cities could end up holding the bag on antiquated buildings that are slowly deteriorating and no longer generating tax revenue.

This was a plausible scenario for a lot of office buildings even before the Federal Reserve turned hawkish, but now it seems likely the problem will be bigger and happen sooner. There's still reason to think other parts of the U.S. economy will get through this period of supply and demand rebalancing in relatively good shape, but when it comes to office buildings and central business districts, we should be prepared for the worst.

Conor Sen is a Bloomberg Opinion columnist. He is founder of Peachtree Creek Investments and may have a stake in the areas he writes about.