Researcher Urges Focus On Low-Compute Optimization And Sampling Strategies

I dont believe LLMs are incapable of novelty, I think in the way many loosely thing of it, but there’s definitely a drive towards cliche

I’d go further and say this has implications for generating “novel” text, that follows more typical samples than samples at the mode

Loosely defining novel as proportional entropy, while still being coherent/on manifold.

More Novel strings leading toward novel ideas.