Christian Szegedy says transformers were intuitive and simple from the start as the architecture replaced multiple complicated attention mechanisms then in use
Pedro Domingos, Professor Emeritus at the University of Washington, linked to a thread on the paper.
@pmddomingos Maybe, but transformers were pretty intuitive and simple from day one.
At that time, there were a lot of complicated models with various attention-based mechanisms.
As the title also suggested, the transformer was a significant simplification over those overcomplicated models.
Pedro Domingos@pmddomingosDeep learning papers are confusing because deep learning researchers are confused.
#87Edward Grefenstette@EGREFEN
Pedro Domingos@pmddomingosIf the transformers paper was written by one of my students, I wouldn’t let him graduate until he did a better job.
#654Pedro Domingos@PMDDOMINGOSIf the transformers paper was written by one of my students, I wouldn’t let him graduate until he did a better job.
@ChrSzegedy Simplification, yes. Intuitive, no. More than one of the authors have told me they didn’t know what they were doing, and it shows.
Christian Szegedy@ChrSzegedy@pmddomingos Maybe, but transformers were pretty intuitive and simple from day one. At that time, there were a lot of complicated models with various attention-based mechanisms. As the title also suggested, the transformer was a significant simplification over those overcomplicated models.