PROVIDENCE, R.I. — A memoir by former U.S. Rep. Patrick Kennedy has created a rift with family members upset over his portrayal of family secrecy, substance abuse and mental illness, including that of his father, late Massachusetts Sen. Edward Kennedy.

Kennedy on Tuesday defended his book, "A Common Struggle: A Personal Journey Through the Past And Future of Mental Illness and Addiction," which was released Monday. He said he loves his family but feels it is important for him break the stigma and shame of mental illness and substance abuse by talking openly about what he and relatives have suffered — even if it does mean a change in the "family dynamic."

"If I can't talk about these things, who else is?" Kennedy said in an interview. "These aren't, frankly, big family secrets. They've been written about in tabloids."

In the memoir, the former congressman from Rhode Island discusses in detail his diagnoses with bipolar and anxiety disorders, and his own abuse of substances from alcohol to cocaine to prescription drugs. He also details his mother's alcoholism and his belief that his father was an alcoholic who struggled with post-traumatic stress disorder after the assassination of his brothers, President John F. Kennedy and Sen. Robert Kennedy.

Patrick Kennedy's mother, Joan, and brother, Ted. Jr., have distanced themselves from the book. His mother released her reaction through a friend, attorney Margo Nash.

"I had no knowledge that Patrick was writing a book, and did not assist him in the project in any way. I was not given a copy of the book and have still not seen it or read it," Nash quoted Joan Kennedy as telling her.

Stephen Fried, co-author of the memoir, said Tuesday that he had interviewed Joan Kennedy for an hour in March 2013, accompanied by Patrick's wife, Amy. Fried, a longtime journalist who is on the adjunct faculty at the Columbia University Graduate School of Journalism, said he had met Joan Kennedy several times before that, and that she was aware of what they were working on.

Ted Jr. said Sunday he was heartbroken that Patrick had written what he called "an inaccurate and unfair portrayal" of their family and said the narrative was "misleading and hurtful." His father's second wife, Vicki Kennedy, has not publicly commented on the book. Attempts to reach her for comment through multiple sources were not successful.

Patrick Kennedy said he understood why the dispute has arisen.

"This is something that's difficult for everyone to deal with, especially the torrent of media coverage around it," he said. He has spent much of the week giving TV interviews to outlets including CBS, MSNBC and FOX News.

Kennedy said these are wrongly seen as "personal issues" not as "medical issues," and if he wrote about his family's battles with cancer, no one would be upset.

"They're celebrated for being cancer survivors, but stigmatized for being survivors of mental illness and addiction," he said.

He sent Ted Jr. a copy of the book in August, and his brother called him after that to express his disapproval, Patrick Kennedy said. He last spoke to his mother Friday.

"I'm trying to take a deep breath," he said.

Patrick Kennedy has been sober for more than four years and has dedicated his work to advocating for better mental health care since leaving office in 2010. He said he wants his own children to grow up without the shame he had, and hopes his brother will someday understand that he is trying to help others who struggle with such illnesses.

"Like all families, things are going to be a little rough in the short run. We love each other," he said. "We've been through a lot more difficult things than this."

U.S. Rep. Joe Kennedy III, whose grandfather was Robert Kennedy, said Tuesday he had not seen the book yet but joked he was waiting for a signed copy from his cousin. He praised Patrick Kennedy's work as an advocate for mental illness.

Asked if it was difficult to see members of his family publicly at odds, the Massachusetts congressman replied, "It happens. Big family."