It's not what you say, it's how you say it. Well actually it's both.
In Words Don't Mean What They Mean, I discuss an article on a new book explaing that a sentence has two purposes: to convey meaning and also to negotiate a relationship with the listener. That's why the way we word things often makes no literal sense, and literal wording often seems overly frank and rude. Interesting stuff.
Words Don't Mean What They Mean