AI Shifts From Prompt Engineering To Goal And Eval Engineering

*i reread my post and i'm stupid 🤦

"hillclimb it for *LOWER cost / *HIGHER accuracy" smh

But it needs a task to be measurable and have real ground truth for its outcome. For a lot of other creative tasks like writing or creating image media, a lot of times there are no objective measures.

If you just let the AI run and loop, sometimes it will just be dragged into a rabbit hole with a lot of tokens consumed, but the end outcome is far away from what you wanted.

@jerryjliu0 My problems with it so far stem more from the fact that you usually want pretty well defined completion or optimization criteria, which is hard to enforce. It's great for "I don't want to have to babysit this agent now" but provides no strong guarantees yet

@jerryjliu0 Does the end ( “goal”) justify the means? 😄

@jerryjliu0 feel like most teams are still leaning more into workflows since the /goal token usage is very high

have woken up to 5 hour sessions with all my usage gone lol

codex does a good job of pausing the goal if it really gets blocked though, which is nice

@jerryjliu0 Token efficiency is one good remaining reason

The pay-off of /goal is simply that it orchestrates the execute->verify->continue loop for you.

Where we come in is ensuring that the /goal converts non-deterministic execution into deterministic results. The better your /goal design, the more true that becomes.

LLMs can achieve this on their own when verification is straight-forward, but verification is not always straight-forward.

@jerryjliu0 how are you handling the error cases when it misses the mark?

@jerryjliu0 The hand-built workflow only ever covered the paths someone imagined. A goal lets me route around the part that broke instead of waiting for a human. Where it gets hard is the context nobody wrote down: the workflow encoded that implicitly, the goal just assumes I can infer it.

@jerryjliu0 goal + eval engineering is clean when ground truth exists. the messy half of prod is the non-repeatable tasks with no eval set, just an owner holding the bag

@jerryjliu0 the failure mode just moved. workflows had explicit nodes you could debug. goal-only buries orchestration inside the model, and teams dont have muscle for opaque traces yet

@jerryjliu0 Yea some call it loop engineering, it's perfect for ML training since it has a clear verifiable goal.

@jerryjliu0 And I thought I was operating on frontier learning how to design/create agentic workflow automations

Oh boy

@jerryjliu0 How well does it work for you? I worry it's gonna waste too much trying to search for the right context when I could've save AI time and token just saying where things are

@jerryjliu0 The workflow layer might slowly dissolve into objective plus feedback.

@jerryjliu0 this works fine for happy paths, but debugging a failure when you didn't actually define the steps is a nightmare.

@jerryjliu0 that makes sense! it really streamlines the process. have you tried breaking down complex tasks into smaller goals yet?

The catch is observability — when /goal infers the trajectory per-run, the regression surface shifts from step sequences to goal completion, which is great when goals are checkable. Harder when a run goes off-rails mid-task, because there's no saved plan to diff against next time.