EG,
I can't formulate a precise explanation for your findings, but I suspect that if you put the ground lift switch to "normal" you may very well obtain results with only one set of cables connected that are similar to what you have obtained with both connected.
The ground lift switch isolates the ground sleeves of the rca connectors (and possibly also the ground wire on pin 1 of the xlr's) from the amp's chassis. I would only use that position if it is necessary to prevent ground loop hum or buzz. That isolation could conceivably account, in subtle ways related to the grounds, leakage paths, stray capacitances, etc. within the amp, for the sonic effects you observed when only one set of cables was connected.
Having both cable sets connected, while the ground switch is in the lift position, could conceivably alter the paths and impedances through which signal return currents flow, between the input circuits of the amp and the preamp. That in turn could conceivably alter or undo the sonic effects of having the cable grounds isolated from chassis.
That's all conjectural, but I definitely think it is worth reassessing your findings with the ground lift switch set to normal.
Re your question about phasing, I don't see any relevance, because neither the relative phase nor the absolute phase (polarity) of whatever inputs are selected (balanced or unbalanced) would be affected by the presence of the other set of cables.
Regards,
-- Al