I went to see Salman Rushdie in conversation with Lisa Appignanesi at the Southbank Centre last night. I have never been a particular Rushdie fan, so was pleasantly surprised by his wit, intelligence and affability. He was talking mostly about his new book the Enchantress of Florence, and made me want to read it.

I was struck mostly, though, by a comment he made right at the end, when he was asked by a member of the audience for his stand on the recent Martin Amis controversy. His answer was to say that it’s not helpful to make accusations of racism against one of Britain’s most serious literary figures. Amis has a right to find certain things hateful, he said, and has a right to hate those who perpetrated them. He then launched into an impassioned defence of Martin Amis’s right to freedom of speech, saying that freedom of speech is not a tea party — it begins with defending the right to remarks you find offensive. Being offended is not nice, but the only alternative is totalitarianism. (I am not using quote marks because I do not have a recording or transcript to refer to, but I’m confident this is very close to what he actually said).

Of course, especially since this was an event organised by PEN, the free speech argument went down a storm, and after a strong round of applause the evening was brought to a close. But I was left a little unsettled.

First of all, his characterisation of Amis’s remarks was not accurate. Of course Amis is justified to hate people who commit terrorist atrocities — almost everyone does, and if Amis had confined himself to hating Mohamed Atta or the 7/7 bombers there would have been no trouble at all. What made it racist was when he extended the hatred to the “Muslim community”, and advocated open discrimination and harassment of “people who look like they are from the Middle East.” That’s racism, no matter which way you slice it. To move from hatred of specific people to demonisation of a whole group is racism. It’s just like many people’s racist demonisation of African Americans as all being criminals or on welfare because of a few media-hyped examples. When you make that leap from criticising some people to generalising about a group, you’re going to get slammed for it, and rightly so.

The second thing that bothered me about Rushdie’s answer was that nobody, to my knowledge, has suggested that Martin Amis be locked up, or prevented from speaking or writing more books, because of his odious views. All that people have said is that his views are odious. So Rushdie’s argument against totalitarianism and in favour of Amis’s right to freedom of speech was completely irrelevant. It was the classic ‘straw man’ argument. Rushdie is clearly an intelligent man so I’m sure he was aware of this — I guess he just wanted to slide out of answering an awkward question.

The shame of it is, though, that if he wanted to make the free speech argument about Amis, there’s a much better, subtler way to do it. Rather than championing Amis’s right to advertise his ignorance, he could have made the argument that if you did silence people like Amis, you would take away not only his freedom but also, more importantly, the freedom of sensible people to point out his errors. If, hypothetically speaking, we did live in a society where Amis could be locked up for saying what he did, then of course he would keep quiet, but his views would not change. Indeed, he would probably keep sharing his views, but only in secret, with people he could trust. The ideas would spread underground, without being published and so without giving the Guardian opinion-page editors a chance to trash them. It’s not Amis who is the loser in this scenario, but the crowds of people who would write articles and blog posts refuting his ideas. It is our freedom of speech which is under threat — the freedom not just to have an idea, but to disagree with it.

To me, this is the most important part of freedom of speech. Of course people should be allowed to express offensive views. But then it is all the more important for people of good conscience to point out precisely why they are offensive, and why they shouldn’t be adopted. Rushdie seemed to suggest that because Amis is one of Britain’s most serious literary figures, he shouldn’t be accused of racism. I believe the opposite — precisely because he is a serious literary figure, his views will be taken seriously by many people, and so he should be loudly criticised when he talks nonsense. If Salman Rushdie had really wanted to strengthen freedom of speech last night, then instead of just talking about it, he should have taken his cojones in his hand and used his freedom of speech to call Martin Amis an ignorant, bigoted old fool.