When Stan Calow was growing up, frugality was a way of life: "You spend as little as you need to, and then save everything else." So, the 58-year-old engineer and U.S. Army veteran from Kansas City, Mo. always hated spending money.

It took his financial planner, Cindy Richey, to drill the point into him that it was actually OK to enjoy his savings once in a while.

After much prodding, the message finally got through. Calow and his wife just returned from a trip to France, touring the châteaux of the Loire Valley, just like they had always dreamed.

Says Calow: "I wanted to live life while I'm still young enough to enjoy it."

It's a tricky dilemma for many of us. As much as pundits tell us to scrimp, and save, and sacrifice for the future, when is it actually OK to spend a little on yourself and enjoy this life that passes all too quickly?

Indeed, according to a new survey, many of us are not enjoying it enough.

When Wells Fargo asked affluent Americans about what they regretted most about their finances, 15 percent said "not having enjoyed their money more."

It is an honest answer that you do not often encounter in financial surveys. After all, splurging on yourself is typically seen as selfish and gauche.

But as some planners point out, it's your money, and you should not be made to feel bad about enjoying it occasionally.

"People are so nervous about outliving their money, and sometimes they shoot too far in their saving," says Joe Nadreau, director of innovation and strategy for Wells Fargo Advisors.

Of course, leaving an inheritance is still an important consideration, according to 57 percent of affluent Americans in the Wells Fargo survey.

But just remember that once the will is read, you are six feet under, and no longer around to witness your family enjoy that wealth.

A bank of memories

So try thinking of the concept of inheritance a little differently: Instead of purely in terms of dollar bills, consider it as a set of memories, which you can create together as a family while you are still alive.

"We have recently noticed a sizable uptick in clients who are more interested in sharing their wealth in the form of experiential gifts," says John Fowler, a planner in Keller, Texas. "At the end of the day, our clients realize stuff is just stuff, but with a little effort, they can create a memory for their families that will last a lifetime."

Keep in mind that splurging on yourself doesn't mean you become miserly with others. It is not an either/or proposition. You can treat yourself once in a while, and also be generous with charitable causes that are meaningful to you.

"People call me all the time to get permission to enjoy their money, which I heartily give them," says Dave Ramsey, a popular radio host and author of "The Legacy Journey."

As for Kansas City's Stan Calow, he looks forward to traveling the world with his wife, and enjoying future grandchildren. This thought, in particular, came to mind when he was walking the streets of Paris recently:

"It's a shame to work so hard all the best years of your life, just so you can afford to survive in the worst years of your life."

Chris Taylor writes for Reuters.