• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Researchers Call Continual Learning Undefined ML Jargon

    Will Depue questions what progress on continual learning would actually demonstrate.

    LB
    NL
    WD
    19 Sources, 49d ago, first seen 49d ago

    TLDR

    Will Depue posted that he is tired of the continual learning discussion and asked what solving it would improve on any evaluation. Kevin Kwok replied that the beatings will continue until the learning is continual. Aidan McLaughlin agreed. Lucas Beyer wrote preach. Chris J. Maddison said it reflects the state of the whole field. Eric Mitchell answered line go up faster. Jason Phang replied that continual learning is whatever we want it to be. Tanishq Mathew Abraham quoted the post and said Will has bravely said something I have been thinking.

    Combined views

    158.3K

    19 Sources, first seen 49d ago

    Combined views

    158.3K

    19 Sources, first seen 49d ago

    1.4K likes
    1.4K likes
    153 comments
    416 saves
    40 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Featured Source
    153 comments
    416 saves
    40 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    19 Sources

    @willdepuei’m so tired of the continual learning thing. what the fuck is continual learning. what do you expect ‘solving it’ will do, like what eval is supposed to go up here. just devolved slop ML lingo now
    @kevinakwok@willdepue The beatings will continue until the learning is continual
    @zhansheng@willdepue Continual learning is whatever we want it to be ❤️
    @cjmaddison@willdepue state of the whole field
    @ericmitchellai@willdepue line go up faster
    @natolambert@willdepue Parameter updates all the time model get better and not expensive, you know just like a few Turing awards in that
    @aidan_mclau@willdepue agreed
    @herbiebradleyIDK I basically think multi-agent swarms with infinite scratchpads and really good ICL still gets you AI that will always perform worse at hard to verify long-horizon tasks than the theoretical AI which could robustly gradient update at test time. There's actually a deployment focused reason why repeated midtraining won't work so well: if you're a company buying these models, you want the model to grad update itself within your ZDR container, and won't allow the real datapoints the lab would need to be used for training. So in practice the repeated midtraining approach has its distribution constrained by the market.
    @teortaxesTexI think the real problem is not so much "continual learning" as combining distributed in-context learning at test time with served model updates in an economical manner, and particularly propagating feature *un*learning to the weights. This would be a big deal.
    @giffmana@willdepue Preach.

    19 Sources

    @willdepuei’m so tired of the continual learning thing. what the fuck is continual learning. what do you expect ‘solving it’ will do, like what eval is supposed to go up here. just devolved slop ML lingo now
    @kevinakwok@willdepue The beatings will continue until the learning is continual
    @zhansheng@willdepue Continual learning is whatever we want it to be ❤️
    @cjmaddison@willdepue state of the whole field
    @ericmitchellai@willdepue line go up faster
    @natolambert@willdepue Parameter updates all the time model get better and not expensive, you know just like a few Turing awards in that
    @aidan_mclau@willdepue agreed
    @herbiebradleyIDK I basically think multi-agent swarms with infinite scratchpads and really good ICL still gets you AI that will always perform worse at hard to verify long-horizon tasks than the theoretical AI which could robustly gradient update at test time. There's actually a deployment focused reason why repeated midtraining won't work so well: if you're a company buying these models, you want the model to grad update itself within your ZDR container, and won't allow the real datapoints the lab would need to be used for training. So in practice the repeated midtraining approach has its distribution constrained by the market.
    @teortaxesTexI think the real problem is not so much "continual learning" as combining distributed in-context learning at test time with served model updates in an economical manner, and particularly propagating feature *un*learning to the weights. This would be a big deal.
    @giffmana@willdepue Preach.