My Audio Note speakers with hemp drivers sound terrific.
The bass presentation and overall image presentation changes dramatically depending on the recording, as one would expect.
However, I don't believe the hemp driver can be separated to any appreciable degree from the overall design of the speaker.
What you are describing sounds more like either driver break-in issues, or speaker placement issues.
Ultimately, if you prefer something else, then go with what gets your smile on.