Harvey is still thrown off by the shorts. It's like the guy who wore a bugs-bunny tie to a civil case he was trying when he first became a lawyer.

He shakes it off, "....I think?" Because it is a fair question, "...Mel Fray."

He offers the note to Raph before putting his hands in his pockets, "...I think I'm the first person I've seen here in a suit and tie and I came in from a-" he pauses, "...Well, from a party."

In other words, he's shallow and tends to associate appearances with status. But honestly can you blame him?