Yes, constantly running an amplifier on the ragged edge isn't doing the speaker any favor either.
A speaker cannot necessarily differentiate between a clean signal and a dirty signal....however, it does make a major difference in its thermal capacity whether or not the signal is compressed when the amplifier is driven into clipping. Why? Because the signal shape changes creating a longer under-power duration with less transition time. So, the speaker has less time to dissipate heat. A speaker is less than 5 percent efficient to begin with.