• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
  • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    A creator outlines a math verifier for AI evaluation and later training

    The creator’s outline moves from extracting and normalizing model answers to checking mathematical equivalence, grading answers and evaluating a model on MATH-500.

    SR
    1 Source, 17d ago,

    TLDR

    A creator describes round three of “Reasoning from scratch” as building a math verifier for two purposes: comparing a base model with future improvements, and supporting later reinforcement learning with verifiable rewards. The outline walks through answer checking and model evaluation, then covers prompt sensitivity, memorization and reproducibility.

    Combined views

    33.4K

    1 Source, first seen 17d ago

    Combined views

    33.4K

    1 Source, first seen 17d ago

    779 likes
    first seen 17d ago
    779 likes
    33 comments
    615 saves
    111 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    33 comments
    615 saves
    111 reposts
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    1 Source

    @rasbtReasoning from scratch round 3: This time, I cover generating a verifier for... a) ...evaluation (base model versus any future model improvement) b) ...the reinforcement learning with verifiable rewards (RLVR) training later on 00:00 Introduction 01:21 Four approaches to LLM evaluation 07:20 Verifiers and reinforcement learning with verifiable rewards 10:52 Notebook setup and dependencies 13:43 Section 3.1 Building a math verifier 18:57 Section 3.2 Loading a pre-trained model to generate text 24:34 Generating and displaying model answers 29:23 Section 3.3 Implementing a wrapper for easier text generation 34:00 Section 3.4 Extracting the final answer box 37:29 Handling answers without boxes 43:17 Section 3.5 Normalizing the extracted answer 46:56 Section 3.6 Verifying mathematical equivalence 53:32 Implementing the equality check 57:48 Section 3.7 Grading answers 59:20 Building and testing the answer grader 1:03:18 Section 3.8 Loading the evaluation dataset (MATH-500) 1:07:51 Section 3.9 Evaluating the model 1:08:34 Prompt templates for evaluation 1:10:47 Prompt sensitivity and memorization 1:13:55 A minimal evaluation example 1:15:32 Building the evaluation loop 1:20:27 Comparing CPU, MPS, and CUDA results 1:21:54 Reproducibility and floating-point math 1:23:37 Base model vs. reasoning model 1:25:30 Summary and next steps

    1 Source

    @rasbtReasoning from scratch round 3: This time, I cover generating a verifier for... a) ...evaluation (base model versus any future model improvement) b) ...the reinforcement learning with verifiable rewards (RLVR) training later on 00:00 Introduction 01:21 Four approaches to LLM evaluation 07:20 Verifiers and reinforcement learning with verifiable rewards 10:52 Notebook setup and dependencies 13:43 Section 3.1 Building a math verifier 18:57 Section 3.2 Loading a pre-trained model to generate text 24:34 Generating and displaying model answers 29:23 Section 3.3 Implementing a wrapper for easier text generation 34:00 Section 3.4 Extracting the final answer box 37:29 Handling answers without boxes 43:17 Section 3.5 Normalizing the extracted answer 46:56 Section 3.6 Verifying mathematical equivalence 53:32 Implementing the equality check 57:48 Section 3.7 Grading answers 59:20 Building and testing the answer grader 1:03:18 Section 3.8 Loading the evaluation dataset (MATH-500) 1:07:51 Section 3.9 Evaluating the model 1:08:34 Prompt templates for evaluation 1:10:47 Prompt sensitivity and memorization 1:13:55 A minimal evaluation example 1:15:32 Building the evaluation loop 1:20:27 Comparing CPU, MPS, and CUDA results 1:21:54 Reproducibility and floating-point math 1:23:37 Base model vs. reasoning model 1:25:30 Summary and next steps