It all depends on how those inputs are arranged inside the amp. IF they are connected in some way all the time (ground? or negative leg?) then the power from the one that is on can go back up the line some to the one that is off and possibley cause problems. Transistors don't like current coming in backwards. (simplistic concept but just the idea
So it can be risky (cause damage) to the other source output, even if that source is turned off.
So it can be risky (cause damage) to the other source output, even if that source is turned off.