CHESTER, VT. - While most eyes warily watched the shoreline during Hurricane Irene's grinding ride up the East Coast, it was inland -- sometimes hundreds of miles inland -- where the most serious damage actually occurred. And the major culprit was not wind but water.

As blue skies and temperate breezes returned Monday, a clearer picture of the storm's devastation emerged, with the gravest consequences stemming from river flooding in Vermont and upstate New York.

In southern Vermont, normally picturesque towns and villages were digging out from thick mud and piles of debris that Sunday's floodwaters left behind. With roughly 250 roads and a number of bridges closed, many residents remained stranded in their neighborhoods; others could not get to grocery stores, hospitals or work. It was unclear how many people had been displaced, although the Red Cross said more than 300 had stayed in its shelters Sunday, and it expected the number to grow.

In upstate New York, houses were swept from their foundations and one woman drowned Sunday when an overflowing creek submerged the cottage where she was vacationing. Flash floods continued to be a concern into Monday afternoon. In New York's Catskill Mountains, where Gov. Andrew Cuomo led a helicopter tour of suffering towns, cars were submerged, crops ruined and roads washed out. In tiny, hard-hit Prattsville, what looked like a jumble of homes lay across a roadway, as if they had been tossed like Lego pieces.

"We were very lucky in the city, not quite as lucky on Long Island," Cuomo said. "But Catskills, mid-Hudson, this is a different story, and we paid a terrible price here."

In Vermont, officials recovered the body of a man who was tending the municipal water system in Rutland during the storm. They said his son, who was with him at the time, was also feared dead. A 21-year-old woman died after being swept into the Deerfield River in Wilmington, a small town west of Brattleboro. And a man was found dead in Ludlow.

Total of 40 deaths

As of Monday evening, Irene had caused at least 40 deaths in 11 states, according to the Associated Press.

"This is a really tough battle for us," Gov. Peter Shumlin of Vermont said after surveying the damage across the state from a helicopter. "What you see is farms destroyed, crops destroyed, businesses underwater, houses eroded or swept away and widespread devastation."

New York state and local officials had, by Monday afternoon, carried out 191 rescues across the state since the storm began, in many cases plucking people from cars or homes as water levels rose. State officials confirmed six people had died in connection with the storm: five drowned and one was electrocuted.

Gov. Chris Christie of New Jersey said his state was facing some of the worst inland flooding it had seen in years. Many small streams are now at flood level and some larger rivers -- including the Ramapo, Passaic and Delaware -- were peaking Monday or expected to peak over the next 24 hours, reaching record or near-record flood levels. Almost 200 New Jersey roads were either partially or fully closed.

Hundreds of miles to the south, in North Carolina, where Irene first made landfall, state-operated ferries began Monday to move personnel and supplies to Hatteras Island on the Outer Banks, where an estimated 2,500 residents remained cut off from the mainland by damage to the main highway.

And all along the East Coast, an estimated 5 million people remained without power Monday, with crews working frantically to restore service.

In Chester, Vt., a village of about 3,000 on the Williams River, Thelma Dezaine's 7-year-old son looked out their window as torrential rain fell Sunday and started screaming.

"He was saying, 'We've all got to get out of here; we're all going to die,'" said Dezaine, who was pulling sodden furniture and clothing from the first floor of the home she rents here. "We didn't have time to grab anything because the water rose too fast."

'Up to our windows'

Her neighbor, Mike Surething, who had 4 feet of water in his house, said, "As soon as the river crested that bank over there, within half an hour it was up to our windows."

Of his modest home, he said, "Everything in there is a loss."

Still, there were glimmers of good news. In Pennsylvania, the Delaware River largely remained in its banks, cresting several feet lower than feared. The forecast for flooding on the Mohawk River in New York also eased at Schenectady, N.Y., where officials had worried that high water might threaten the city's drinking water and sewage treatment plant.

The Associated Press contributed to this report.