Vincent Weisser#743
our observability agent has been running since we launched hosted training - managing heterogeneous hardware from h100 clusters to nvl72 deployments wouldn't be possible without it. auto-detects infrastructure issues, references historical patterns from sqlite, and creates detailed github issues + slack alerts like this one
@jannik_stra Any plans for a blog on the implementation details?
@peter58587 yeah @johannes_hage keeps pushing me to set time aside to do a proper writeup on how we actually build hosted training and all of the automations behind it
Vincent Weisser#743our observability agent has been running since we launched hosted training - managing heterogeneous hardware from h100 clusters to nvl72 deployments wouldn't be possible without it. auto-detects infrastructure issues, references historical patterns from sqlite, and creates detailed github issues + slack alerts like this one