When you play music, can you still hear this "buzz" through the music?
Make sure you use the most pristine recording possible for your test.
Don't mistake "take hiss" from the original recording for the hiss you're hearing with only the amp connected.
You may just be hearing "hiss" from your speakers due to bringing raw current to them from your amplifier.
This is common and not necessarily a problem. In fact, the more sensitive your speakers, the louder that "hiss" with no music playing.
You can "clean" that hiss with dedicated circuits and AC Conditioning, but it is unlikely you will eliminate it.
If that hiss disappears when you play music with no hiss in it, it isn't a problem.
Also, listen from your listening position, not with your ears so close to the speakers. If you can't hear the hiss from your listening position with only your amplifier connected, I wouldn't worry about the hiss, but I would still consider dedicated circuits and/or AC Conditioning.