When musician/singer/producer Lionel Richie was first asked to be a judge on "American Idol," he said no. "I said, 'Guys, I'm busy. I'm an artist. I have to be on the road. I want to be out where my fans are.'"

The four-time Grammy winner, who also earned an Academy Award and sold over 100 million records, was where his fans were. That was some years ago, and on Sunday Richie returns as one of three judges on ABC's "American Idol," as it launches its 20th year.

"I got to a point in my life where you have all of this knowledge, you actually know a lot about what they [the contestants] are going through," says Richie. "So, when I was asked this time around, the answer was 'Yeah!'"

He and co-judges Katy Perry and Luke Bryan, he said, have suffered the same jitters that plague the hopefuls on the show.

"The difference is, now we know exactly how they feel. We've been there before," he said.

Getting there wasn't easy for any of the judges.

"The one thing for me was probably hearing 'no' so many times," said Perry, who rose to fame with such hits as "I Kissed a Girl" and "One of the Boys."

She knows what it is like to have three record deals and then be dropped, have two cars repossessed and sleep on couches. "It was all a part of the process," she recalled.

For country singer-songwriter Luke Bryan, best known for his winning numbers, "Drink a Beer," "Kick the Dust Up" and "Kiss Tomorrow Goodbye," it was not only arriving on top, but staying there.

"I think for me it was about each level of having to conquer," said Bryan. "You move to Nashville, then you have to conquer Nashville. Then you have to go get a record deal. And constantly — no matter what you do — you've always got to reinvent yourself and take it to the next level."

Bryan said it was stressful and challenging at every level. "That's what so many of these kids don't realize, that the stress never ends," he added.

Richie, who got his start with the six-man Commodores band, said it was different for him.

"I came from a group situation. So I could take the no's a lot better because there were five other guys that took the no's with me," he added.

But the one thing people easily overlook and don't understand, he said, is the time that needs to be served.

"There's always that moment where you say, 'I'm ready.' And we hear these kids on 'American Idol' say so many times, 'I'm ready. I'm 15 years old. I'm ready to go.' And we're saying, 'No, you're not,'" he said.

Hearing that phrase when he was younger was a bitter pill to swallow, Richie confessed. It can especially be tough when the performance goes very well and the only feedback given is: "You're five years away from being where you really want to be."

"And I just couldn't put that in my head. What do you mean five years from now? But we didn't have the experience," Richie recalled.

So as judges, he said, they constantly remind contestants that they are not ready yet. "We know what they're thinking, because at 15, of course you're ready. You sound perfect."

But there's more to it, he added.