The announcement that U.S. Supreme Court Justice Anthony Kennedy is retiring has triggered joy among conservatives and anguish among liberals and moderates who follow legal issues — as well as concern among some Americans who fear that what’s next could put our nation in a fraught and ominous place.

While Kennedy is generally seen as a conservative, he is more of a liberal-libertarian on some high-profile issues. It was striking to see so many analysts agree that some of his most important decisions advanced progressive goals, including legalizing same-sex marriage, upholding abortion rights, forcing California to end mass overcrowding of state prisons, banning the death penalty for minors and upholding the right of detainees at military bases such as Guantanamo Bay, Cuba, to defend themselves and challenge their treatment before a judge.

This is why so many people are nervous about what will happen if President Donald Trump and the Republican Senate majority, emboldened by the recent string of 5-4 high court political victories, install a purer conservative as Kennedy’s replacement. It’s quite possible reproductive rights would disappear or be sharply limited in numerous states and that the criminal justice reform movement would face new barriers.

It’s also possible Kennedy’s departure would lead to efforts to roll back some gay rights he worked to establish and end the court majority’s limited support of affirmative action programs to help more minorities overcome obstacles they face in pursuing the American dream.

But while it is unrealistic to think that Trump will pick a Kennedy-style justice, there is hope that a hard-right court won’t go on a precedent-shredding crusade in coming years. That hope lies with Chief Justice John Roberts, who is likely to become the court’s swing vote and is acutely aware of the court’s image and the damage that’s possible to its reputation and to the nation if it is seen as just another wing of deeply ideological conservatism.

There’s evidence this is why Roberts was the fifth vote to save the Affordable Care Act in 2012 when President Barack Obama’s signature 2010 law came before the Supreme Court. Roberts’ institutional caution was evident again in another challenge to Obamacare’s legality, in 2015, when Roberts (and Kennedy) joined four liberal justices in preserving the law.

This is as it should be. Any chief justice who doesn’t appreciate the importance of preserving the idea that the Supreme Court is a judicial body, not a political one, risks damaging the institution. Yes, polls already show deep public skepticism that this is true. Yet not just Roberts but all justices must appreciate what’s at risk if this skepticism builds.