Age discrimination at work could be costing the UK economy up to �31bn each year, according to the National Audit Office (NAO). A drop in the number of those working in later life means higher benefit payments and lower tax revenues.
Older jobseekers face "significant" problems and the government could do more to help them, though there had been some improvements, the NAO said.
The government plans to outlaw "ageism" in the workplace by the end of 2006. Barriers
According to the NAO, less than 70% of over-50s are in work, compared with more than 75% of those under 50.
The NAO report concluded that the UK economy will have to rely more and more on older workers as time goes by, as an ageing population means there will be fewer young workers entering the workforce.
Of the 2.7 million over-50s who are out of work, 1 million want to find a job but are facing barriers such as age discrimination, outdated skills, lack of confidence and health problems.
But the report found that progress was being made, with the gap between employment rates among the over-50s and those of the workforce as a whole narrowing by 1% in 2003.
The report pointed to the government's New Deal 50-Plus scheme having successfully found work for 120,000 people at a cost of �270m.
"Good progress has been made in increasing the employment rate for older people," said auditor general Sir John Bourn.
"There is scope for government agencies to focus their efforts on improving the employability of older people and for targeting disadvantaged groups."
"Success will bring benefits not only for the individual, but also for business, the taxpayer and the economy at large."
A spokesman for the Department of Work and Pensions told the BBC that measures were in place but more could be done to tackle the problem. 'Hurry up'
Help the Aged called on the government to hurry up and introduce legislation - pencilled in for 2006 - to tackle age discrimination at work.
"Age discrimination is still the major barrier in preventing people from enjoying extended working lives," said David Sinclair, social inclusion policy manager at the charity.
"Age discrimination in employment is bad for business, bad for the economy, and can result in older people living their later lives in poverty."
"The government must no longer delay issuing the draft regulations that will seek to bring age equality into the workplace by 2006."
In addition, the NAO report focuses on the numbers of over 50s claiming incapacity benefit and therefore not included in official unemployment figures.
Almost half of those over 50 and unemployed are receiving incapacity benefit, the NAO said, citing its report called Welfare to Work - Tackling Barriers to the Employment of Older People.
Cutting the numbers of people claiming incapacity benefit is widely rumoured to be a top priority of new Work and Pensions secretary Alan Johnson. BBC News