'Which is why you're not worth talking to. Real people don't run away from stuff, and just...I don't know, live in indulgence, the way you seem to. Go away.'

Whatever else is going on with him, this is true enough. Javert's got no time for indulgence, or people who think it's a valid way to spend life.