Naveen Rao clarified that model serving is not performed by the model provider and identified the separation as a key factor in data flows for AI deployments that reference Databricks
Discussion addressed request reconstruction for rollouts and multi-turn histories.
@NaveenGRao @MinqiJiang @unconvAI @databricks what
Naveen Rao@NaveenGRao@MinqiJiang @unconvAI @databricks The model serving is NOT done by the model provider. That's the key.
@MinqiJiang @unconvAI @databricks The model serving is NOT done by the model provider. That's the key.
Minqi Jiang@MinqiJiang@NaveenGRao @unconvAI @databricks So I suppose this assumes it would be hard to reconstruct requests belonging to the same rollouts. But it would be straightforward for many common cases like requests from the same multi-turn message history. Or does infra here include model serving?
@MinqiJiang @unconvAI @databricks No, Anthropic is available this way
Minqi Jiang@MinqiJiang@NaveenGRao @unconvAI @databricks Ah, so no Anthropic, but OSS models and maybe GPT on Azure.
@NaveenGRao @unconvAI @databricks Ah, so no Anthropic, but OSS models and maybe GPT on Azure.
Naveen Rao@NaveenGRao@MinqiJiang @unconvAI @databricks The model serving is NOT done by the model provider. That's the key.