Main two things are cultural associations and the vowels, if I had to guess. The former is way more important; the only real reason a language like French, which uses the same "guttural R" (uvular trill) found in German and Arabic, isn't seen as equally harsh is because the French have a long history as a prestige culture. Similarly, Arabic isn't considered particularly harsh to those in the islamic world and indeed is seen as a beautiful or poetic sounding. Cultural baggage can cause us to hear one sound and see it as breathy, airy, subtle, or seductive from one mouth and brutish, harsh, animalistic from another.
The other thing is vowel inventory; Polish, Hebrew, and Arabic all notably have way less vowels than English, and an inability to produce the full range of expected vowel sounds will mostly keep intelligibility intact but can have associations of underclassness or uneducatedness.