Tried to squeeze the most important bits about the entire stack for cloud deployment of transformer inference, from application layer concerns to hardware, debugging, and o11y, into one talk. Had to operate at a very high tok/s!

https://www.youtube.com/watch?v=ZUdIsRZhWXI

The business of Inference is so interesting and extremely difficult! This is an awesome talk from @charles_irl

@modal @_gongy @_dcw02 @saatwiknagpal @DevenNavani @be4ncurd @emilyhanyf @hsubbaraj @racerfunction @luiscape @jonobelotti_IO @mma12261 @teenychairs @_dcw02 i apologize for screen-capping our hinge-free DMs and then sharing it with the whole world

@charles_irl @modal @_gongy @saatwiknagpal @DevenNavani @be4ncurd @emilyhanyf @hsubbaraj @racerfunction @luiscape @jonobelotti_IO @mma12261 @teenychairs machine god go brrr

@_dcw02 @modal @_gongy @saatwiknagpal @DevenNavani @be4ncurd @emilyhanyf @hsubbaraj @racerfunction @luiscape @jonobelotti_IO @mma12261 @teenychairs in the grimdark future there is only brrr