Reactions from ranked influencers
5 posts
Very cool work by @eliebakouch! The delta on code data is interesting. 2.1 was an end-to-end fresh pre-train with a new version of our code dataset. I'd be very curious how this will compare to our next (sized up) model.
new jspace results :) this time focusing on @poolsideai laguna XS.2 and XS.2.1 version bump and optimizers for laguna, the question i tried to answer is "XS.2 -> XS.2.1 is likely training the model on more/different code, so how different is the jlens between the two versions if i use code or wikitext for the prompt?" and the answer is that it's indeed VERY different. the model workspace is totally different between XS.2 and XS.2.1 if i fit it on code (i use some of the agentic traces training data from nemotron), and quite similar if i do it on wikitext EXCEPT the end layers that are similar for both (called the "motor" section in anthropic paper)! confounding factor is that i took longer prompts for code compared to wikitext, fable and kimi K3 said it was not very important tho (i might test it to be sure!) for the optimizer (based on checkpoints from the fantastic optimizer paper by @wen_kaiyue and marin folks), the question was a bit more open ended. we found that the CKA between different optimizers is much lower (less similar) than between the same optimizer at different sizes. we also found that some optimizers are much more similar than others and that the PR (effective number of directions a layer uses to talk to the logits) is quite different (see other plots in thread!) the overall structure is still relatively similar, feels like muon has fewer "outlier" layers, the ones that have lower CKA with other layers and create those lines in the heatmap
@eisokant oh didn't know, i think this make even more sense to see this then!
more plots: 1. adam mini and scion are super similar // cautious adam and soap, very different 2. bunch of plots about how similar different optimizers are and PR per layer per optimizer. also the top right experiment is interesting, the question is: "at 1.2B i use optimizer A, does the most similar model at 520M also use optimizer A?" which is mostly true except for SOAP
probably one of the last jlens experiments btw (sorry for the spam aha), it was very fun to do! will write more about it soon, but one of the side quest of this is that almost every experiment is based on open science artifacts (lot of open models, olmo/smollm checkpoints, marin delphi scaling laws and optimizer, laguna quantized checkpoints, etc..)
@eliebakouch @poolsideai ooh i've been meaning to do this - did you try a mixture? also did you see any difference (or did you try) chat formatting the data vs doing it free-prompted like they do in the oss jlens release?