Are you likely damaged the amp (or speakers) when you connect 4 ohms speaker to 8 ohms tap of the amp ?
No.
Do you get different sound with connecting a 4 ohms speaker to either 4 or 8 ohms tap of the amp.
Yes.
What is the best connection to get the best performance.
Try both and use the one that sounds best to you.

