Wisconsin Dells has a water park you may not have considered: The Wisconsin River

Our kayak bobbed on the water as my husband, Ed, tried to maneuver it into a tiny cave while I barked directions.

"To the right! To the right! Wait, now to the left!"

With a deft maneuver of his paddle, Ed sent us gliding into the cave's cool, dark interior. I reached out and touched the intricately carved sandstone walls, then asked Ed to hold us steady. Our outlet back into the Wisconsin River was just a few paddle strokes away, and I wanted to savor this experience. We were sitting inside one of Wisconsin Dells' famous rock formations, something millions of visitors probably don't even realize is possible.

After a minute, I signaled to Ed that we could continue, and soon our kayak popped out of the cave and back into the main channel of the river.

Wisconsin Dells is one of the Midwest's most popular vacation destinations, with more than 4 million annual visitors. Its popularity is largely due to its many water parks, which is why the town's booster dubbed it the Waterpark Capital of the World. The nation's first indoor water park opened here in 1989, and today this community of just 5,500 is home to more than 200 waterslides.

But Dells tourism wasn't always centered around water parks.

Tourists first began streaming to this town in south-central Wisconsin in the late 1800s, due to the alluring photography of a man named H.H. Bennett. One of the 19th century's top landscape photographers, he was captivated by the region's Cambrian sandstone formations, found in just three other spots around the globe.

Many of these early visitors experienced the area in an intimate fashion, just as Ed and I were doing. They hopped in rowboats or canoes and paddled past, and through, the rock formations. But it wasn't long before commercial steamboats debuted, luring passengers with comfy seats and the promise of no physical exertion. Commercial boat tours soon became the norm.

By the mid-20th century, ancillary attractions began opening for business, including the Ducks, which are World War II amphibious trucks transformed into popular tour vehicles. Then, a few decades later, the water park craze began. A century after Bennett's photographs of sandstone rock formations beguiled a nation, that fairy-tale landscape was relegated to the background.

After years of regular visits to the Dells, and perhaps one too many trips down a waterslide, Ed and I realized we'd never truly appreciated the area for what it is: a place filled with incredible beauty and unique natural artwork. To remedy this egregious oversight, we decided we needed to go back to the beginning. That is, to explore the Dells via personal watercraft.

After consulting with locals, we came up with three main options: Paddle south into Mirror Lake, tucked away from the bulk of the tourists in a forested state park; explore the Lower Dells of the Wisconsin River, below the Kilbourn Dam, where the river widens; or take a guided kayak tour of the Upper Dells, above the dam, where the winding river narrows. We opted to start with a voyage into Mirror Lake, the quietest area of the three.

We rented a double kayak from Dells Watersports, which offers kayaks, canoes, paddleboards and motorized watercraft at its headquarters on Dell Creek, a narrow waterway connecting Lake Delton and Mirror Lake. Hopping in, we began paddling southbound, away from the crowded water parks and larger-than-life attractions. The change in scenery was instant and dramatic.

Not too long into our voyage, we had to portage our kayak over Mirror Lake Dam, also known as Timme's Mill. Our reward for hefting it up a steep set of stairs to the relaunch site was discovering the remnants of the Grotto. The Grotto, also once dubbed the Hole-in-the-Wall Bar, is believed to have opened in 1966. Created inside a shallow roadside cave scooped into the base of a rocky outcrop, it had no running water or bathrooms. The village of Lake Delton reportedly closed it not too long after its debut, citing health hazards and neighbor complaints. Now vacant and graffiti-covered, it made an interesting, historic detour.

Ed and I spent the rest of the afternoon lazily paddling around placid Mirror Lake. Because the entire lake is restricted to slow/no-wake traffic, it was easy to gawk at the pine-covered bluffs towering above while waving to anglers, paddlers and picnickers.

Our foray into Mirror Lake was enjoyable. But it made us eager to explore the Upper or Lower Dells, where the bulk of the area's famous sandstone formations reside. So a few weeks later, we returned.

This time we rented a double kayak from Wis River Kayak Rentals in downtown Dells, adjacent to the Kilbourn Dam. Owner Jake Beard is in his third year of operation, and business is booming.

Beard said his customers "love the fact that there are no crowds, and that they can chill out."

Although we'd taken many tours on this part of the river over the years, we were surprised by how different things looked from our kayak. At least a dozen anglers were casting lines into the water as we floated past, largely obscured by tangled vegetation. Families picnicked on sandbars, splashed in the water and floated in tubes.

When we pulled up on a compact beach for a snack, I spied a clutch of tiny purple-pink flowers growing out of a jagged bluff. Then I noticed a carved inscription: A. Waldrich, 1917. The sight was as thrilling as an archaeological discovery.

Back on the water, we realized we'd missed spotting Hawk's Bill, one of the more prominent named rock formations in the Lower Dells. But my disappointment was assuaged when we glided past the Baby Grand Piano, a jumble of rocks that resemble a baby grand that has tumbled off the cliff above.

After exploring the caves, wishing we could find more of them, we were able to circle the Sugar Bowl, one of the Dells' free-standing rock formations, despite the strong current.

Our excursion ended at a tiny beach, where Beard arrived to shuttle us back to our car. As we chattered about our quest to uncover the "real" Dells, Beard, a fifth-generation area resident, suggested we hike out to Chapel Gorge.

We picked our way down to a cozy beach — one featured in an old Bennett photo — and found a small group of folks swimming, cooking out and playing Frisbee, all in front of those mystical sandstone formations.

"I wish we'd known about this spot when our kids were little," I said to Ed wistfully. We definitely would have had just as much fun here, and at no charge.

Before heading home, it was time for a bite. We opted for a stop at the Grateful Shed Truckyard, right on the Strip downtown. It opened this year and features a collection of food trucks clustered inside a massive building. Before its debut, the owners scoured old farms and junkyards, searching for items to use in their business. They found some old VW buses and one Greyhound bus, all of which they renovated into dining space.

After buying some treats, Ed and I climbed into a vintage VW bus to eat. The vehicle is probably 40 or 50 years old — a modern conveyance compared with the days when Bennett's photographs of Wisconsin Dells first enchanted the nation. Still, I think he'd appreciate the effort.

Melanie Radzicki McManus (melaniemcmanus.com) is a freelance writer focusing on fitness and travel. She lives near Madison, Wis.