Sinatra was perceived as a serious musician and man, with his Mafia connections and all. Dean had the tipsy comic public image and often sang while in that character, which made people take his singing less seriously. Making movies with Jerry Lewis didn't help either.