What’s the average annual household income in London? And what about the average property price? The answers – average household income was £38,688 in 2011 (the last year for which the ONS has statistics) and the average property price, even after years of  runaway inflation, is no higher than £514,000 – may surprise.

The so-called “mansion tax” would affect only about 2% of Londoners. Of London’s 3.4 million homes, even the tax’s critics estimate that only about 80,000 are worth £2m or more.

That the media, along with too many Labour politicians who should know better, obsess about the “backlash” against a mansion tax shows just how disconnected from real life they have become. Ed Miliband’s conference speech yesterday had its faults, but his confirmation that a Labour government would introduce a mansion tax was not one of them.

The tax could raise £1.2bn a year for the NHS, would have no effect on 98% of voters, and might help cool London’s overheated property market. Collecting it would be relatively simple: unlike cash, property cannot easily be concealed. So why are so many potential Labour mayoral candidates joining Boris Johnson in objecting to it?

The answer lies in the persistent myth that London is full of people on low, fixed incomes who will suddenly discover their homes are worth £2m and be clobbered with an annual levy they cannot pay.

Tessa Jowell has said she objects to the mansion tax because it could “be an issue for some people” in her constituency, and favours higher council tax bands instead. But higher council tax bands could be objected to on the same grounds that they would penalise mythical low-income mansion-owners.

Such timidity never wins elections, even where the mansion tax is “an issue”. For 16 years until May I was a Labour councillor representing a hyper-marginal ward in Blackheath, a leafy suburb much like Jowell’s Dulwich and West Norwood constituency, with some of south-east London’s highest property prices.  But even here only a tiny handful of homes are worth more than £2m: most “mansions” in London are divided into flats.

As a councillor I went doorknocking at least monthly – often weekly – and can recall fears of a mansion tax being raised precisely once, by someone whose home was not worth £2m and who was unlikely to vote Labour in any case. And judging by the huge car on his drive, he was no pauper. Assuming the threshold will periodically rise in line with property prices, teachers, IT managers, and doctors living in suburbs like Blackheath need not lose any sleep.

It’s certainly true that many people in places like Blackheath – my Mum and Dad among them – bought large houses cheaply in the 1970s and 1980s which are now worth five, ten or even twenty times more than they were bought for.  But these folk often downsize once their children fly the nest, either to “cash in” or avoid the hassle of running a household much bigger than they need. The idea that they would be involuntarily “forced” to do so by a mansion tax is absurd. In some parts of the country, older people can indeed find themselves trapped in big homes they can’t sell – but  these are in low-value areas where a Mansion Tax would affect no-one.

As for “a tax on London”, of course the majority of British homes worth over £2m are in the capital. So what? London already pays more tax than other regions, quite reasonably, as it is the wealthiest part of the UK. And this wealth is not spread evenly. While average property prices in one London borough – Kensington and Chelsea – may be nudging £2m, it’s a borough that is never going to be anything other than Tory, and the other 31 boroughs are well behind.

Between now and next May, Labour should keep on asking the Tories why they are forcing  people renting council homes to downsize – by the Bedroom Tax – but oppose a policy that might encourage the very richest to downsize as well. Opposition to the “mansion tax” shows how too many politicians are stuck in a 1990s time warp, where any suggestion of a tax on property prompts “Labour bombshell” posters, or resurrects the ghost of the Poll Tax. The gap between the wealthy and the poor is so great that the electorate has moved on. Ed Miliband should ignore the critics and stick to his guns.