Probably not complete BS. This is anecdotal, but years of experience has taught me that aluminum-containing antiperspirants cause contact dermatitis for me after extended use.

I was also diagnosed with a nickel allergy by a dermatologist when I was a child. Metal allergies are real.

I wouldn't go so far as to say aluminum is toxic for everyone, but it's certainly something I avoid putting on my skin.