Then I would say it IS the pivots are way too tight.
The outer 'ring' screw has to be loosened first.
Hardware stores often have sets of 'special' bits for screw/hex/philips for sale. Some of these are the 'two tooth' for the outer screw.
You can also make your own.
Just find a proper size flat screwdriver, and file uot the middle of the blade til it fits the ring screw.
It is worth doing!
Once you get the outer loose adjust the inner until the arm is a bet loose. then tighten the outer a bit.
You will have to experiment until the inner stays in just the right place when the outer is tightened.
Probably what happened is someone set the inner and tightened the outer tight. Thus making the inner way too tight.
Again, the inner screw is the one that is setting the bearing, and the outer 'sets' (tightens) the inner. So watch out for the inner turning when you tighten the outer.
The inner can be set deliberatly loose so the tightening of the outer gets them just right!
Trial and error will get the job done.
Patience and you will have a fine TT.
(Do not worry about the bearings being damaged. Just fix the tightness and play it!