In the Byrne retcon on '86, Kryptonite was dangerous to humans (but on the order of un-enriched, unprocessed uranium.) Kryptonite affects Superman so badly because the specific radiation of kryptonite shuts down or damages his bio-electric aura and forces his special solar-battery organelles to shut-down and/or discharge.