15" vs 18" subwoofer - which to buy?

If price and room space/aesthetics were not a big issue, is there any reason to get a 15" subwoofer over an 18" one? My main issue is not disturbing the neighbors too much. I'm confused about the difference between a 15" and 18" subwoofer. I want to get the best sound quality possible for low-to-medium volume use.

I've heard that an 18" subwoofer can be played at low decibels and provide good bass resolution and fullness of sound, while to achieve the same volume with a 15" subwoofer, the power has to be much higher. So for any given sound volume, the main difference is in distortion- and higher distortion sound may be more obtrusive. Is this true? it seems counterintuitive that an 18" subwoofer could be better for neighbors than a 15" subwoofer.

However, I compared a B&W800 (12") vs B&W850 (15"). I listened from the other side of the store wall (not as thick as my apt), and at minimum levels which I found satisfying for HT, I found the 12" more obvious and 'boomy' sounding while the 18" produced a more subtle (though actually more powerful), lower frequency vibration, like a very low-level rumbling background earthquake. The 15" subwoofer did seem more of a disturbance because of it's 'obvious' sound. However, I worry that an 18" won't even start producing quality sound until a certain volume that was much higher than a 15", and consequently potentially more disruptive for neighbors. Is this true? What is the relationship between subwoofer size, low sound volume, and subjective listening experience?

Unfortunately, it will be difficult to get an opportunity to hear both the DD-15 and DD-18 which I am considering, so I hope someone can help out here.