Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.
One of my research goal since then has been trying to add N models and get close to Nx speedup in training. There are few fundamental questions to ask: 1. Why does distillation loses help? 2. How do we communicate more information that is identifiable between the networks? 3. How can each network implement a way to match the output of another network, while having its own beliefs on each prediction?
Other fun story is that this was the first time common crawl landed in Brain and was like 1T tokens — I remember Noam cornering me in MK and saying we should train a 1T model on it. this is back in 2018 - I was very production pilled and thought Noam is a bit ood, and we should instead work on optimization methods.
One of my research goal since then has been trying to add N models and get close to Nx speedup in training. There are few fundamental questions to ask: 1. Why does distillation loses help? 2. How do we communicate more information that is identifiable between the networks? 3. How can each network implement a way to match the output of another network, while having its own beliefs on each prediction?
Other fun story is that this was the first time common crawl landed in Brain and was like 1T tokens — I remember Noam cornering me in MK and saying we should train a 1T model on it. this is back in 2018 - I was very production pilled and thought Noam is a bit ood, and we should instead work on optimization methods.
Guardrails removed spam, off-topic, unclear, or duplicate replies.
Ask a question below.
Published answers will appear here.