Richard Ngo says Eliezer Yudkowsky overemphasized the connection between Bayesian reasoning and fundamental truth by a factor of ten

A bit late, but: I think talking about raising a hypothesis to attention from a space of trillions creates confusion about epistemology in a similar way to a watchmaker who thinks of themselves as selecting a watch from the space of all watches.

It’s collapsing an extended process into a single decision, and therefore making it hard to talk about the dynamics of that process.

I think that this is beautiful and approximately true (in the sense that Newtonian mechanics is, say) but actually not fully true when you understand it deeply enough. The process of creating a representation is not a neutral act, it ripples through the rest of your ontology (in a way that just raising a credence from ~0 to nonzero doesn’t).

Maybe a good analogy is how creating a new player in a game who does almost nothing can still alter the Schelling points of that game. Or how adding a new irrelevant option to a bargaining game changes the fair outcome (even though I believe standard bargaining frameworks deny this).

Ok, but I think it’s a true and beautiful and useful observation in epistemology that hypothesis selection and generation are fundamentally fungible. Selection and generation are at an important level the same thing.

Not in all respects of course, but it is a deep connection that is worth calling attention to.

@jd_pressman @ESYudkowsky @RichardMCNgo I mean, I would love to read a book about that, but I don't know of any. Do you?