Wilford did nothing wrong.

But fine. If a fight is going to happen - which apparently, it is! - then he'd rather it happen outside as well, just in case someone decided he started it. Which he didn't.

Which is why as soon as they're out the door, he swings his elbow out, if for no other reason than to get Chirrut to let go of him.