The unbuffered really allows the source to be the drive to the amp, with only a volume control and perhaps a selector switch in the line. So IF your source can drive the amp like this, then it is the way to go.
IF your source has any difficulty driving the amp. that is it sounds lean, or lacks dynamics, THEN you will need to use the 'buffered output'. The buffer creates a proper impedence match between the source and amp, so the source is NO LONGER what the amp sees as the voltage source. The buffer is now the voltage source. The buffer adds a layer of electronics to the signal.
So if you are in need of using the buffer, well, use it.
Actually just LISTEN to the sound with, and without the buffer, and choose the one that sounds best. The one that sounds best IS the one to use.
Also, you can use the buffered output for a second device, a headphone amp or whatever if you find the non buffered out to the amp is good.
The reverse IS NOT TRUE. do not use the unbuffered out if the connection to the main amp is the buffered one.