It's not what you say, it's how you say it. Well actually it's both.
In Words Don't Mean What They Mean, I discuss an article on a new book explaing that a sentence has two purposes: to convey meaning and also to negotiate a relationship with the listener. That's why the way we word things often makes no literal sense, and literal wording often seems overly frank and rude. Interesting stuff.