Strongly depends on the speaker. Vienna Acoustics pays particular attention to the design of their grills for use on or off. In some case, they say the sound could benefit with them on and the enthusiast should experiment thusly. Isn't that part of the fun of this hobby?
90% of my listening is done with the grills on.