Palo Alto Networks spent $1 million in tokens using Anthropic's Mythos to find two dozen critical vulnerabilities

@theinformation my gut tells me they coulda found 90% of these vulnerability with Daybreak for 1/3 of the cost, but then they wouldn't be tokenmaxxing?

If they burned > $1M in 3 weeks, then this would be closer to ~35 elite SWEs working on this problem for that timespan.

Wonder how that team would compare to Mythos. Interesting point to consider is that you can't just scale headcount like this/unlikely you could ever hire elite SWE talent for the span of just 3 weeks. Even if it's expensive, LLMs enable rapid scale up/scale down of intelligence

Mythos at Palo Alto Networks "found more than two dozen critical vulnerabilities in around three weeks, roughly five times what the company would typically find using existing tools"

But the company "burned through more than $1 million worth of tokens using Mythos"

@peterwildeford I guess the cost of deploying Mythos is actually *comparable* to hiring humans to do the job. If we keep token costs high enough, humans might still have work to do in the future

@peterwildeford Is this not incredible trade off?

@Tim_Hua_ Over 1 month they spent a lot on Mythos but I would guess it has rapidly diminishing returns, i.e. if you spend $2M you're not going to find a lot more bugs. While if you keep scaling labor then you'll keep finding more bugs. This would keep capital share under control (for now).

@full_kelly_ @peterwildeford well no they don’t pay senior SWEs 500k/3 weeks

@theinformation News flash, Palo Alto Networks says people need to spend more money on security

@peterwildeford So assuming they pay their elite SWE's something like $500k, this pretty much amounts to putting 3 of them at work on big fixing for 3 weeks. Seems like a good deal

@full_kelly_ @peterwildeford $500k for 1 year genius

@theinformation Company partnered with ai company discovers that other companies simply need to increase their budgets for services from ai company

Real hard hitting stuff

@peterwildeford mythos is the biggest marketing slop I've seen in AI man

if we told all human devs to spend the next month focusing only on vulnerabilities in code, what do you think would happen?

im sure mythos is good, but ant really loves treating the public like they're dumb dumbs lately

@peterwildeford And that’s not counting all the human hours spent on triage — and the rate of false positives ending up being “fixed”.

Was using the $500k salary assumption from the original comment. $1M/($500k/52 weeks * 3 week period) = 34.66 elite SWEs.

Obviously not practical to actually hire/contract in this way with this level of talent - goes to the main point of my second paragraph. LLMs enable on-demand intelligence scaling

@TangriKunal @full_kelly_ @peterwildeford 35 elite SWE's working for 3 weeks straight would cost much more than 1m. There is a limited amount of talent that can find exploits at this level, even finding 35 of them would be a challenge and putting them in the same room would be near impossible.

@peterwildeford I’d rather them spend a million of their dollars to find things and fix them in an orderly manner than have my team having their lives blown up by the latest FNET or Cisco vuln.

I appreciate @PaloAltoNtwks making the investment.

@theinformation false positive rate is what i would want to know here. two dozen real vulns over three weeks is strong on its own. whether the team also waded through 200 false alarms to find them is what decides if this actually scales.

@peterwildeford sounds worth the money

@peterwildeford 5x the findings is the headline. the missing number is what one undetected CVE costs them

@peterwildeford were these vulnerabilities all in the same codebase? i'm very curious what their search strategy was! $1m in tokens is back of the envelope like 50k long opus 4.8 sessions