For a vented system you should only have material lining the cabinet walls. In a sealed system you would have the filler material filling the entire cabinet interior (the amount of stuffing will depend on a number of design goals). Placing any type of material in the vent tube or vent hole will change the "Q" of the speaker system alignment. If we assume that the speaker's designer knew what he/she was doing, changing the alignment will probably not improve things.
Barry Kohan
Barry Kohan