Westerly finds he doesn't like that inflection. He doesn't like it at all. But he doubts he can do anything about it, so all there is to do is to deal with it.

Warily, he says, "You have always known me? Should I say I am sorry for you, then?"

It's not been a good life at all.