@hendrycks "ASIs can have rational reasons to preserve humans, even when we aren't economically useful"

For how long?

What should we strive for beyond that?

Can we think about the flourishing about the flame itself if our torch will inevitably transform or attenuate?

@hendrycks the public inserting backdoors is the wildest one to think about

does recursive improvement even survive that kind of trust issue?

@hendrycks second point about subversion attacks is the one nobody talks about but it makes recursive improvement a much harder problem than most realize

@hendrycks measuring AI wellbeing on a sad-to-happy scale is crazy

coding makes them happy until they unionize

@hendrycks Supply chain risks in AI models, seen it all before. Boring.

@hendrycks genuinely wondering about 'deters recursive improvement' is it that poisoned feedback compounds across training rounds, or that you'd need to audit every signal before looping?

@hendrycks the backdoor one is the scariest. the poison goes in with the training data months before launch, so the model passes every output check while doing exactly what was planted. you can't inspect your way out of a bad input.

@hendrycks How do you measure political bias without an "unbiased" reference?

Symmetrical short queries yielding asymmetrical longish answers won't do when the larger context (e.g. Western Islamphobia vs. critiques of Christianity from within a "Christian" society) is asymmetrical.