Worth Upgrading Marantz 1504 AV Receiver with Power Amp?

i haven't heard your avr, tho it appears to be fairly modestly powered. although some mainstream avrs like nad have better amp sections than others, i personally haven't heard a mainstream avr which didn't benefit from external amping, and the named parasound would likely be a significant upgrade.

not sure re SQ of chromecast

but get the Parasound from one of the many dealers that allow free returns and do a listening test

Listing ID: lis7e6hh
Listing ID: lis7d377
Listing ID: lis7e1i7
Listing ID: lis7d550
Listing ID: lis7e86b

Here's some current listings here on AG. Any one of them will be a huge upgrade from what you now have. Some of them are a little over budget, but you may be able to negotiate for a lower price.

Yes, adding a power amp to your AVR will be a huge improvement and one that you will immediately hear.  The Parasound A23 will be a very good choice.

But do you need an AVR at all?  If you only listen to 2 channels, then I suggest getting rid of it altogether and buying a good 2 channel integrated amp with a built-in DAC instead.  Pick the right one and you won't need an external power amp either.  This way you'll still get a better power amp section and the benefit of a superior preamp section as well.

Do not underestimate the importance of a good stereo preamp.  The preamp sections in AVR's, even in "pure direct" mode, are a mere afterthought.