That's right. With SET amps, frequency extension drops like a stone when the impedance gets much below 6-8 ohms.
What you REALLY need if you keep your current amplification is a speaker made to work well with SETs such as the Zu Druid. They claim 101dB sensitivity (and even if they exaggerate, it's still at least 96 dB which is what you need) and a 12 ohm nominal impedance. The review linked below shows the impedance never dips below 8 ohms.
The list price is beyond your stated budget but you may be able to pick up some used ones. Be on the lookout for something like this. Here's a fairly recent review.
What you REALLY need if you keep your current amplification is a speaker made to work well with SETs such as the Zu Druid. They claim 101dB sensitivity (and even if they exaggerate, it's still at least 96 dB which is what you need) and a 12 ohm nominal impedance. The review linked below shows the impedance never dips below 8 ohms.
The list price is beyond your stated budget but you may be able to pick up some used ones. Be on the lookout for something like this. Here's a fairly recent review.