Princeton professor Arvind Narayanan warns Anthropic's Claude Tag integration introduces severe enterprise vendor lock-in risks

VIEWS3.2KBOOKMARKS21LIKES26RETWEETS2REPLIES4

Takes on Claude Tag: 1. A *team-level* AI like Claude Tag inherently causes higher switching cost, because the memory is less accessible. It's only fully viewable by admins, and requires much IAM management.

2. Continual learning & agents taking more async actions across surfaces & apps will only increase the complexity of access management issues.

3. Currently, admins have responsibility for managing which knowledge team-wide or org-wide Tag agents have access to—a high burden of complexity which probably will lead to some embarrassing internal exposes of confidential information. This probably can be done by the model, but will it be 100% reliable?

4. With just MD file memory, we can deterministically manage access because it's explicit. But true continual learning in the enterprise seems like it would require agents to have the judgement to decide themselves what to expose. If agents could construct their own RL envs from a company's context, and post-train, then they would also need to be able to somehow limit their use of in-weight knowledge.

5. In theory, we could unlock better capability from enterprise AI by letting agents always access all information, but decide themselves how to abstract that to preserve confidentiality. But this seems to require much more reliable agents at information control.

6. The token cost could balloon as team-level async agents do more and more work. It's higher friction to control the team-wide model choice in the Slack interface, so you can't restrict eg Opus use to save $ so easily. Eventually we might get a good router, but labs are not incentivized to give you a good one! CISOs and CTOs of companies should stay wary of letting a single vendor's pricing dynamics eat up their spend, especially given the risk of government intervention in model access.

7. Just as companies have stores of tacit/process knowledge in humans, they will also accumulate this as the collective memory of their team/org-level agent swarms. They will not be perfect substitutes, because much tacit knowledge is about relationships or not easily expressible in text.

8. Satya's recent piece on an AI ecosystem was timely, and he clearly sees the potential here for continual learning & "as many models as there are firms".

9. Presumably all major AI labs will have some sort of "async org-level agent swarm" product, and some companies will have multiple running at once. Someone will sell the model wrapper version (including OS models & routing), which could be cheaper.

Arvind Narayanan@random_walker

The new Claude Tag feature seems extremely useful, but at the same time, a dangerous bargain for enterprises because of the pricing model and the risk of lock-in. The four big changes together mean that you interact with Claude as a coworker instead of a tool (the same Claude instance for everyone instead of each worker; soaks up tacit knowledge without your telling it; acts on its own; and does so asynchronously). All clearly very useful, but completely flips the interaction paradigm. https://www.anthropic.com/news/introducing-claude-tag

Let’s talk about lock-in. As far as I can tell, Claude maintains its own memories in this new way of working; the human team members can’t see and edit them. (System administrators presumably can, but they have other things to do!) Tacit knowledge thus goes from a weakness of AI agents to a major strength — it seems inevitable that as teams and orgs start to use Claude this way, it will become the main queryable repository of all their tacit knowledge, creating dependence and stickiness. Effectively, Claude is a coworker that you can’t fire without *every* team losing workflows and know-how.

By the way, it also seems to introduce a new and pervasive security risk, since Claude can be integrated into private channels as well, and can be given access to repositories and tools even if the users in that channel don’t have access to them. Anthropic has introduced an interesting but complicated access control model to handle all this: https://claude.com/blog/agent-identity-access-model But I’m not sure I trust people to understand and implement it correctly, nor the LLMs to be sufficiently robust against threats like prompt injection.

What about pricing? Claude is not like regular coworkers, because it bills for every token it produces. And it can do an unbounded amount of work, asynchronously and without being asked. In the current model, when AI is a tool, enterprises set per-user budgets, which creates accountability and keeps cost somewhat manageable. When everyone shares a Claude, it will be much harder to track and control spending. Of course you can set a token budget, but turning off Claude for the month for everybody when the budget is hit risks bringing work to a screeching halt.

When AI companies talk about the next stage of AI being a “drop-in replacement” for human workers, it should be understood not as a technical innovation but a business model innovation, enabling more value capture and rent extraction. AI companies are no longer competing for a share of enterprises’ IT budgets but rather a share of their entire labor spend, which is orders of magnitude bigger. Claude Tag is a big milestone in this evolution. This shift is very good for AI companies, but it is unclear if it is good for their customers.

4h3.2K2621

Matthew Schrager@MatthewSchrager

@AgustinLebron3 Going all in on a single provider seems too risky IMO. Lock-in, single point of failure, etc. Hard to imagine large enterprises making themselves permanently vulnerable to this.

I suspect we'll see a shift towards model agnostic providers like @FactoryAI/@cursor over time.

8h1602

Tryn McCaffery@trynmccaffery

@herbiebradley i didn’t reference this in my tweet but another issue with the release is certainly that Tag has to be a massive token-burner, which will not fly with orgs that aren’t willing to increase their AI spends massively

3h21