Builder Argues Symbolic AI Could Enable Personal Open-Source Systems

other than SupGen miracles, and as insignificant as I am in the big scheme of things, I wonder what is the most impactful action I could take to move us from where we are, to a Fable-class model being open sourced

that should be priority of all of us not in the club

@goldstein_aa @fchollet yes. not sure if he agrees this would result in a massive pre-training cost reduction, he's more focused on actual capabilities. but ultimately we both seek the same thing

@VictorTaelin Do you mean something like what @fchollet calls "symbolic descent"?

@Yeahokay_imsure AI models are just libraries of functions / algorithms. the problem is that these algorithms are stored extremely inefficiently in the weights. symbolic AI would be when the model is an evolved but "normal" program, without the "matmul attention" indirection / emulation layer

@goldstein_aa @VictorTaelin @fchollet Derivatives exist for symbolic rewrite systems. See lambda the ultimate backpropagater. So “symbolic” is a bit of a misnomer here, imo, because the structure of baysian updates, back prop, derivatives of types, etc are “the same” given right perspective.

@fraserpricee first big milestone would be training something comparable to GPT-3 without GD. many open problems preventing that. known symbolic regression methods are too slow, higher-order de-unification is necessary yet basically unexplored, composition is still exponential, etc.

@vr4300 meaning all symbolic ai labs to merge but you guys don't want that

the field will fail for the same reason Haskell did, and that's poetic

@luan_wav I mean a full symbolic model. A .exe program that receives a prompt, and produces coherent English responses, creates apps, automates tasks. Just like LLMs do. But no matmul, no weights. Just... a program. It'd still be evolved, trained, a black box. Yet, no weights, no matmuls.

@canalCCore2 caio com todo respeito nunca que qualquer coisa feita em cima de qwen vai chegar no nivel do fable, pelo amor

@VictorTaelin Working on it... https://arxiv.org/abs/2603.18104

@VictorTaelin Regarding the first coment: just want to say that your poasting on X is working because, at least for someone like me with limited knowledge of FP, you've convinced me there is something really special here.

Which is why I'm making it the focus of my thesis xd

@goldstein_aa @fchollet yes, but that just an hypothesis on my crazy mind

yet, I cannot see why that wouldn't work

there is *nothing* theoretical preventing that, at all

@VictorTaelin @fchollet So, for example, you could implement an LLM with a "symbolic transformer" instead of a deep neural-network? You were discussing something like that a while back, don't recall if you used the name "symbolic transformer".

@VictorTaelin there is only one platform missing that prevents an OS Fable, and that is an open source RLHF data aggregation platform.

people donate to foundation, foundation hires expert ICs to produce 100M high quality training data samples across every field, and open sources them.

@VictorTaelin It's time to build

@o1lo01ol1o @VictorTaelin @fchollet Well that's rather fascinating, thanks for the paper reference. But obviously that approach didn't get any traction since it was published in 2008, right?

@VictorTaelin There has to be a mechanism to maintain a window of input > output, input > output that is the magic of "thinking". A gate layer is what would find the right neural pathways, which is still matrix multiplication, but wouldn't require a pass over every layer to work. 10+ yrs away.

@VictorTaelin given how big models can use superposition to store concepts and do general representation learning, I'm not sure how a full symbolic approach would be able to beat current LLMs, although they can surely be useful for explanability

@nikestyler1 I did, literally estimated I'd fucking finish it by next week