Google DeepMind's Susan Zhang says LLM-generated fiction quickly degrades, requiring a professional ghostwriter to rewrite 98% of drafts

some more info on her usage:

> my current theory/test is to have the first 10k words establish patterns in dialogue and prose, and then setting up blocks within a chapter that i then prompt the LLM to write

> so that as the process goes deeper into the book, i'm slowly handing off the writing to the LLM but with me still in the director chair

> but i'm sick and tired of seeing endless "it's not X, it's Y" or "X lands and I file it away" or "I want to do X, I do not do X" within 3 sentences of each other

@suchenzang i like elegant code but i was forced to work with other programmers early in my career so i learned to accept poorly written code as long as it works

@suchenzang seems like she has "taste". (this thing we are about to collectively lose in a few years)

@suchenzang Relevant: https://github.com/sam-paech/antislop-sampler

@suchenzang You need to treat writing as code and do it in claude code, it actually works well, You can plan ahead and create plotline, characters, tone etc. and you can ensure the agent adheres to them.

@suchenzang I've looked into this recently, and the current crop of writing-centric benchmarks are awful to boot. At least with coding you have a general sense of how good the model will be at your task.

With writing, its essentially a black hole. Claude > GPT > On Your Own

@vikhyatk too real. can't appreciate the good stuff without slogging through the bad.

@suchenzang IANAL, but i understood that contract lawyers especially love to use certain fixed phrasings that are well-established in existing contracts because they are more standard, often tested in law, and are less open to interpretation.

I guess in a sense this is a lot like code?

@suchenzang I don't think you get the difference here. The author has a hidden imagined version of how the whole story is supposed to unfold that is underspecified no matter how much they prompt. What matters is whether *OTHER* readers like the story, not the author.

@suchenzang Base models are better for creative writing. I think the optimum would be to use a frontier model for the overall structure and coherence, and improve the content with a base model.

@Biggiethelad1 @suchenzang I was writing fanfiction smut of preexisting characters, not a smut novel

I was reasonably happy with ClaudeSmut, but it wasn’t good enough to write a book to sell

@bdefig clearly you know ones with impeccable taste and poetically elegant and clever code quality

@PMinervini @suchenzang uhm. human or llm skills?

@suchenzang > coders are fine with repetitive elements and inefficiently written statements

we must know a very different set of engineers

@stjaco @suchenzang LLMs can be pretty decent at style transfer etc. so maybe human?

@suchenzang coders are fine with repetitive elements and inefficiently written statements - I’m not fine but I also need to pay rent

I love how smut writers have discovered some modern NLP/AI concepts from first principles lol!

RL for reasoning tasks are only easily trainable on verifiable domains like code and math. And with code, the only thing the reward function depends on is if the code runs, not on anything else. Not on the process.

We need to find better ways to evaluate tasks with intermediate steps and not directly/easily verifiable ground truths!

@suchenzang it’s (currently) mid at writing for/as a human and tbh, that’s good. it’s a waste of a mind. it’s great at writing about things that are *like* AI but aren’t. like, claude is often beyond human-level at hypertextual fiction, but is below-pulp at writing a standard narrative.

@suchenzang I mean, I don't think this is any indication that it is a bubble. Like, it likely is, but for different more macroeconomic reasons. It's just that it seems AI excels at anything that is closer to a commodity than art, and sucks for the inverse.

@w_bsun @suchenzang It seems that the frustration is also on the overuse of certain linguistic devices and patterns which i agree gets v annoying