Founder Says Code No Longer Serves As Core IP For AI Products

Your IP cannot live in brittle prompts/harness/code. We are still missing the layer that defines and verifies IP/specs/behavior.

𝗖𝗼𝗱𝗲 𝗶𝘀 𝗻𝗼 𝗹𝗼𝗻𝗴𝗲𝗿 𝘁𝗵𝗲 𝗜𝗣. 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝘄𝗵𝗲𝗿𝗲 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗯𝗿𝗲𝗮𝗸𝘀 𝗮𝘁 𝘁𝗵𝗲 𝘁𝗮𝗶𝗹-𝗲𝗻𝗱 𝗶𝘀.

Attending a lively panel on Agent Evals hosted by Notion reinforced a conviction I have held for a while.

The panel drove a great discussion that kept returning to a few main themes: 🧵

𝟰/ 𝗘𝘃𝗮𝗹𝘀 𝘂𝗻𝗹𝗼𝗰𝗸 𝗲𝘃𝗼𝗹𝘂𝘁𝗶𝗼𝗻: 𝗮 𝗳𝗮𝘀𝘁 𝗹𝗼𝗼𝗽 𝗳𝗼𝗿 𝗶𝘁𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗵𝗮𝗿𝗻𝗲𝘀𝘀 𝗮𝗻𝗱 𝗮 𝘀𝗹𝗼𝘄 𝗹𝗼𝗼𝗽 𝗳𝗼𝗿 𝗰𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝘁𝗵𝗲 𝘄𝗲𝗶𝗴𝗵𝘁𝘀.

The fast loop is harness iteration using prompts and tools, where changing a prompt changes the agent and allows you to ship fast. The slow loop is fine-tuning the weights. You reach for this only when intelligence plateaus and cost or latency become top concerns, though most teams never actually need it. Both loops are reproducible by anyone, so neither serves as the moat.

𝟭/ 𝗣𝘂𝗯𝗹𝗶𝗰 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 𝗵𝗮𝘃𝗲 𝗾𝘂𝗶𝗲𝘁𝗹𝘆 𝗹𝗼𝘀𝘁 𝘁𝗵𝗲𝗶𝗿 𝗺𝗲𝗮𝗻𝗶𝗻𝗴.

Public evals have eroded as a trust signal. Anyone can top a leaderboard in a month. Enterprise buyers no longer care about your benchmark number; they try the product on their own use case and decide from there.

𝟮/ 𝗧𝗵𝗲 𝗻𝗲𝘄 𝗳𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝘁𝗵𝗲 𝘁𝗿𝗮𝗰𝗲 → 𝗲𝘃𝗮𝗹 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲, 𝗽𝗼𝘄𝗲𝗿𝗲𝗱 𝗯𝘆 𝗵𝘂𝗺𝗮𝗻 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸.

LLM-as-a-judge worked... until agents went long-horizon. Tasks got too long, too subjective, too context-heavy to grade automatically. Domain expert feedback matters the most, but building evals manually will never scale.

As one panelist put it: “If you don’t solve that problem [going from a trace to an eval automatically], I think you’re screwed.”

𝟯/ 𝗧𝗮𝘀𝗸 𝘀𝘂𝗰𝗰𝗲𝘀𝘀 𝗶𝘀 𝗻𝗼𝘁 𝘀𝘂𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗮𝗻𝘆𝗺𝗼𝗿𝗲. Smarter models hide their regressions. Failure shows up as +5 turns to completion, not an outright error. Turn count and latency are wildly underrated signals.