Should I worry that my boyfriend doesn't say this?

I'm sure you don't have anything to be worried about but it might be a good idea to bring to his attention. Ask him why he hasn't. Maybe he simply doesn't realize he is doing it.