Researchers release GENIE, an evaluation metric that measures the originality of LLM-generated stories across distinct narrative elements

Want to know *how* novel/creative your LLM response is and why? Use GENIE! 🧞

@jessyjli @gregd_nlp @ManyaWadhwa1 @Asher_Zheng00 You can explore the questions generated per prompt and contents of the population to see what makes responses novel. For example, the story about a cactus wanting to become a raindrop scores high on character originality. 🔗: https://alliteraryalligator.github.io/GENIE/

@jessyjli @gregd_nlp @ManyaWadhwa1 @Asher_Zheng00 How does GENIE compare to existing novelty metrics? We tested this using minimal pairs of creative writing responses differing by one feature (e.g. plot). Key findings: 🧱 Many holistic metrics are not paraphrase-robust 🎯 GENIE is sensitive to interventions & paraphrase-robust

@jessyjli @gregd_nlp @ManyaWadhwa1 @Asher_Zheng00 GENIE captures novelty w.r.t a user-defined reference population, along task-specific features (e.g. plot or setting). For a target response, GENIE: ⚙️ Automatically derives features 🌐 Uses a population of LLM-generated responses ❓Extracts features via prompt-specific questions

@jessyjli @gregd_nlp @ManyaWadhwa1 @Asher_Zheng00 Check out our paper for more! 📝: https://arxiv.org/abs/2606.12790 🐙: https://github.com/AlliteraryAlligator/GENIE 🔗: https://alliteraryalligator.github.io/GENIE/

This was in collaboration with: @ManyaWadhwa1 @Asher_Zheng00 @gregd_nlp and @jessyjli