With retirement planning, time is of the essence, but it's never too late to start saving. There are a lot of simple ways to build your retirement fund in your 40s and finish with a nice nest egg.

Determine your retirement number

Opinions vary on how much money is needed to retire comfortably — some experts argue that you should be shooting for eight times your ending salary, while others say you should be building toward $1 million. What's most important is that you determine the exact amount that you'll be saving for, so there's no guesswork; then you can start building your plan around that number.

Start increasing your savings

Generally, experts recommend that you save between 10 and 15 percent of your income for retirement, but that might not actually be enough, especially if you're getting a late start. See if you can start socking away 20 percent from your paycheck alone. If the discipline is hard to muster, set up an external savings account and have a portion of your paycheck automatically deposited there.

Set money aside for yourself first

To help others, we must first help ourselves, and that means putting retirement savings before other money responsibilities, like college tuition for your kids. Sound insensitive? It's not about denying your family financial help — it's merely about prioritization. Your kids have a lot of options to pay for their education, such as scholarships, loans and work-study. You don't have these resources for retirement.

Get rid of as much debt as you can

Tackle any outstanding debt you have and prioritize by starting with high-interest loans first, like credit cards. Seek out alternative solutions if your debt is too large, or it could compromise your retirement. See if you can consolidate your debt in a zero-interest card or low-interest personal loan.

Stop spending money on unnecessary things

Think about how much frivolous spending can add up over time. According to Financial Mentor, the value of a $5 latte bought when you're 40 can compound to over $1,000 by the time you're 80. Now imagine how much money you're missing out on by spending that extra $5 several times a week — potentially tens of thousands.

PAUL SISOLAK, GoBankingRates.com