Yeah, antiskate setting is moving the arm as it falls.
Not a lot to do about it.
Though perhaps wiping the lift bar (if you have one) off with alcohol.
And flipping the drop lever more slowly, instead of fast.
The arm now is free floating down, if you hold the lift and release it slower, the arm will stay resting on the lift bar and not move sideways as much.