• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI
    Reaction

    A claimed 95% ‘user model’ score calls the User Sim Index into question

    A user who shared code for the model argues that the index probably shouldn’t be used to evaluate user models.

    Tal LinzenTL
    Eric ZelikmanEZ
    noahdgoodmanNO
    7 Sources, ,

    TLDR

    A user says people often evaluate user models on the User Sim Index but probably shouldn’t. They shared code for a ‘user model’ they say scored 95%, with scores of 97% for communication, 97% for information, 92% for clarification and 95% for error reaction. They also shared a sampled conversation and believe their model beats any released model.

    Combined views

    7.7K

    7 Sources, first seen 1h ago

    Combined views

    7.7K

    7 Sources, first seen 1h ago

    107 likes
    1h ago
    first seen 1h ago
    107 likes
    10 comments
    42 saves
    3 reposts
    Featured Source
    10 comments
    42 saves
    3 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    7 Sources

    Eric Zelikman@ericzelikmanPeople often eval user models on the "User Sim Index" - we probably shouldn't. I attached code for a "user model" scoring 95% on its 4 behavioral dims: 97% comms, 97% info, 92% clarify, 95% error reaction. I think this beats any released model. See also a sampled conversation1h
    noahdgoodman@noahdgoodmani find this especially amusing because i'm teaching ELIZA in Minds and Machines (Stanford's intro cogci w/ @mcxfrank) tomorrow...1h
    Tal Linzen@tallinzen@ericzelikman @noahdgoodman @mcxfrank Lol58m

    7 Sources

    Eric Zelikman@ericzelikmanPeople often eval user models on the "User Sim Index" - we probably shouldn't. I attached code for a "user model" scoring 95% on its 4 behavioral dims: 97% comms, 97% info, 92% clarify, 95% error reaction. I think this beats any released model. See also a sampled conversation1h
    noahdgoodman@noahdgoodmani find this especially amusing because i'm teaching ELIZA in Minds and Machines (Stanford's intro cogci w/ @mcxfrank) tomorrow...1h
    Tal Linzen@tallinzen@ericzelikman @noahdgoodman @mcxfrank Lol58m