Bend Deploys AI Agents to Close Pre-Launch Performance Gaps
Reactions from ranked influencers
3 posts
yes this is how I prompt now. just let the thoughts flow in, don't care too much about perfection, let Fable figure it out
we're at the start of Bend's final pre-launch optimization campaign. remember Bend's runtime promises: "as high level as JS (closures, objects, etc.), almost as fast as C, almost as parallel as CUDA". this campaign is to ensure we honor this promise. you will act as an agent coordinator and manager. never do manual coding or research work yourself, always call Fable agents (mandatory - no other model allowed) to work for you. prompt them, wait for their responses, audit their work, merge what is good, repeat. you will use 3 agents, each one with one goal: - SEQ adversarial: this agent will be trying to find programs that perform MUCH better on C than on Bend3. it will be tackling only the T=1 case (i.e., raw C performance). its deliverables are files on ./devs/issues/perf_<short_description>.md. these issues must be concise. they must provide context and the smallest possible reproduction of some Bend3 file that, when translated 1-to-1 to competent C, results in C being significantly faster. note the other way around is not relevant; we're not claiming that Bend can reproduce any C idiom. we're claiming that, on the idioms that Bend can reproduce, compiling them to C will result in performance on par to what a competent C programmer would produce, if implementing the exact same algorithm (same functions, same pattern-matches) in C. an issue can only be opened if this precise invariant is broken. we tolerate files up to 2x slower; anything slower than that is considered near-C. issues cannot be duplicated. before opening an issue, the agent must have read all others and be confident that what they're reporting is truly new and not already covered by other perf_ issues. if they find something somewhat similar to an existing issue but that they feel like is still worth reporting, the best direction is to DELETE the old issue and open a new one that covers both cases. a good issue must contain, beyond context and minimal reproduction, the root cause AND a principled fix, either inline or pointing to a commit in a separate branch that can just be merged in (principled = small, elegant, causes no regressions elsewhere; bonus if it is net neutral in token count, i.e., a CHANGE of representations that doesn't cause the codebase to grow, improving the inner architecture for all programs and thus generalizing, instead of a PATCH/addition on top of a slow architecture; unless, of course, the root cause IS some small bug / issue that can be patched out), if the agent can find one. if the agent, after reasonable effort, could NOT find a root cause / fix, they can leave these fields as unknown, and move on to finding other classes of errors. - PAR adversarial: exactly like the agent above, except the goal of this agent is not to inspect raw machine performance, but only parallelism. i.e., it must reason about the task scheduler, the usage of threads, it must investigate contention, coalesced accesses, and every other concern that might impact the speedup obtained by running the parallel modes - both on C and Metal. in particular, it is known (although no issue for it yet) that irregular tasks (as in, a tree that is deep in some regions, shallow in others) will NOT parallelize well, because Bend's compiler assumes that `x y = f(a) f(b)` is an user hint not only that f(a) and f(b) can be parallelized, but that both take roughly the same amount of work. the entire system, tomes, wave bsp scheduler, etc., were built on top of this assumption. because of that, though, irregular trees under-utilize parallel resources. it IS desirable, IF possible, for this assumption to be lifted, and for both CPU and GPU to achieve near ideal speedup EVEN when the workload is irregular. as such, one line of work that this agent can tackle is trying to improve here. a successful delivery would be a change to the runtime that allows a much wider class of programs to achieve parallel speedups, while not causing regression in the classes of programs currently supported (hard - the bsp assumption is precisely what allows us to avoid contention), and while not creating multiple execution modes (FORBIDDEN) nor increasing the codebase size significantly (i.e., swapping an approach by another, NOT adding new approaches on top). that said, this may not be possible, and it is FINE to ship it as is. worth exploring, nice to have, not mandatory. another issue that requires attention is matmul and radix. currently, matmul and radix are underperforming on GPUs and CPUs, if compared to bend2 (~/t/dev/bend - the language bend3 supersedes, never shipped) and bend3-hs (~/t/dev/bend3-hs - an early prototype of this repo). even though bend3-ts uses the same approach as these, it fails to deliver a speedup on these (and perhaps other?) cases. finding out why and fixing is not optional, and is one of the issues this agent MUST solve. finally, currently, users must pick either CPU parallel mode, or GPU parallel mode. ideally, though, Bend should be able to choose which mode to pick, without user decisions involved. parallel CPU mode must still be supported fully (for these that do not have GPUs), but, when GPUs ARE available, the runtime must be able to decide where to run a parallel kernel. a clean shipped solution to that is a strong deliverable. overall, though, this agent should work the same as the first one, except focused on one main goal: improve parallelism, get bend closer to platonic nead ideal speedup in all modes, in as many classes of programs as possible, with as little user input as possible (other than writing x y = f(a) f(b) for calls they want to parallelism - that won't go). deliverables are the same: ./devs/issues, documenting findings, pointing to fixes if existing. small addition: another issue is that our current model is hardcoded in some senses; Cx8 is needed even in a 12 core machine because the way we distribute work is optimal for 2^N parallel tasks on WORK mode, so 12 threads would not fit well; similarly, we use fixed Mx16k on Metal to ensure there are 128x128 tomes. the optimal dimension might vary per GPU though, and, in some, the optimal could not be a perfect square, like 64x128, similarly to how some CPUs don't have exactly 2^N performance cores. our fixed tome count is a byproduct of this; it is a great heuristic that works surprisingly well in most hardware. yet, making this configuration flexible and automatically adjustable to fit the hardware we run on perfectly is also desirable, but not an extreme priority. count this as polish / refinement once major concerns are handled. - CUDA implementation: currently, Bend has a Metal runtime only. we want to support both Apple and NVIDIA hardware. as such, extending runtime.c to also work on CUDA devices is mandatory before launch, and one of the last hard things in our TODO list. that said, since metal is here, and since cuda support more features than metal in general, this is tractable. the main difficulty isn't implementing and making it work; is implementing while ensuring the codebase doesn't explode in size and remains long term healthy and maintainable. as such, the one and most important principle we should follow is: keep the CUDA code in the same file as the Metal code, AND keep EVERY CUDA code in a 1 to 1 correspondence to same metal code. by enforcing this invariant permanently, strictly and religiously, we avoid both runtimes drifting / doing different things. as such, even though the code itself doubles, the complexity never will. fixing a bug in one fix it in the other. making an architectural change in one applies the same architectural change to the other. and so on. so, more important than the code itself, is setting up this guideline to ensure the codebase directionally grows in the way it should, and this must be done BEFORE any kernel runs. that's this agent's main responsibility. once the architecture is fixed and the principles are laid out, written and set in stone, then the agent can start working towards the full implementation. it is impossible to run CUDA in our minis, so, for this agent, we've made available a machine with an RTX 4090, accessible via `ssh rtx`. the agent must use this machine as needed to develop, test and benchmark the CUDA runtime. later on, we should update the bench.ts script to allow running on the RTX, too, in the right environment. the deliverable of this agent is not issues; instead, it is a worktree branch with a complete, working, correct, audited and efficient CUDA runtime added to Bend. when an user compiles a Bend program in an NVIDIA-enabled environment, it must automatically pick the CUDA route and compile via CUDA, not Metal. cloning that repo into an NVIDIA enabled machine and running bench.ts must successfully run all Metal benchmarks except on the NVIDIA hardware, and it must display similar or better speedups. all tests must pass. once these all are true, the agent can conclude their work and mark it as completed. remember: do not touch, not let any agent touch, main on this directory. use separate worktree branches. NEVER run any Bend code in this machine. use macs mini for that, both via bench.ts, and directly, if needed. spawn the Fable agents and leave them working. report back on reelvant progress and milestones. if any major blocker occurs that demands my attention or decision, let me know. start working now.
go go fabulous (meanwhile I'm working manually on the formalization / paper)