Almost a decade after it all began, the Federal Reserve is finally talking about unwinding its grand experiment in monetary policy.

And when it happens, the knock-on effects in the bond market could pose a threat to the U.S. housing recovery.

Just how big is hard to quantify. But over the past month, a number of Fed officials have openly discussed the need for the central bank to reduce its bond holdings, which it amassed as part of its unprecedented quantitative easing during and after the financial crisis. The talk has prompted some on Wall Street to suggest the Fed will start its drawdown as soon as this year, which has refocused attention on its $1.75 trillion stash of mortgage-backed securities.

The Fed’s MBS holdings have long been a contentious issue, with some lawmakers criticizing the investments as beyond what’s needed to achieve the central bank’s mandate. Yet because the Fed is now the biggest source of demand for U.S. government-backed mortgage debt and owns a third of the market, any move is likely to boost costs for home buyers.

In the past year alone, the Fed bought $387 billion of mortgage bonds just to maintain its holdings. Getting out of the bond-buying business as the economy strengthens could help lift 30-year mortgage rates past 6 percent within three years, according to Moody’s Analytics Inc.

Unwinding QE “will be a massive and long-lasting hit” for the mortgage market, said Michael Cloherty, the head of U.S. interest-rate strategy at RBC Capital Markets. He expects the Fed to start paring its investments in the fourth quarter and ultimately dispose of all its MBS holdings.

Unlike Treasuries, the Fed rarely owned mortgage-backed securities before the financial crisis. Its purchases have been key in getting housing back on its feet.

Along with near-zero interest rates, the demand from the Fed reduced the cost of mortgage debt relative to Treasuries and encouraged banks to extend more loans to consumers. In a roughly two-year span that ended in 2014, the Fed increased its MBS holdings by about $1 trillion, which it has maintained by reinvesting its maturing debt.

Since then, 30-year bonds composed of Fannie Mae-backed mortgages have only been about a percentage point higher than the average yield for five- and 10-year Treasuries, data compiled by Bloomberg show. That’s less than the spread during the housing boom in 2005 and 2006.

Talk of the Fed pulling back from the market has bond dealers anticipating that spreads will widen. Goldman Sachs sees the gap increasing 0.1 percentage point this year, while strategists from JPMorgan Chase say that once the Fed actually starts to slow its MBS reinvestments, the spread would widen at least 0.2 to 0.25 percentage points.

“The biggest buyer is leaving the market, so there will be less demand for MBS,” said Marty Young, fixed-income analyst at Goldman Sachs.

The firm forecasts the Fed will start reducing its holdings in 2018. That’s in line with a majority of bond dealers in the New York Fed’s December survey. The Fed, for its part, has said it will keep reinvesting until its tightening cycle is “well underway,” according to language that has appeared in every policy statement since December 2015.