One of their claims is that their model is very parameter efficient, but they don't actually do an apples-to-apples comparison to an equally small conventional NN. I ran a small experiment on MNIST, and an MLP is actually the same or better than coupled oscillators.
Introducing our first model, Un-0!
We trained an image generator powered by a backbone of coupled oscillators in place of a more traditional conventional neural network.
Users questioned claims that coupled oscillators boost parameter efficiency on MNIST, arguing instead that any gains stem from unrelated factors or act as a constraint.
No Digg Deeper questions have been answered for this story yet.
Here's an experiment that codex ran on a single Mac Studio on MNIST. Blue is MLP, red is Un-0. I also reproduced their result that freezing the oscillator weights makes Un-0 worse (green). What they didn't run before is *any* conventional NN with their exact setup.
@zhaisf I mean that's true, although you could in theory stack multiple layers of it
This is just a small scale result. However, it does suggest that the "parameter efficiency" isn't from the oscillators. It's from something else in the setup, e.g. EqM, the decoder architecture, etc. It would take a lot to convince me that coupled oscillators are actually good.
@unixpickle The way I understand it, the parameter efficiency doesn’t seem like a feature but rather a big constraint — the amount of learnable parameter is exactly d^2 + d and there isn’t an obvious way of scaling up parameter count given a fixed input dimension d
To be clear, their setup actually uses a bunch of conventional components already, in a somewhat well-engineered way. The only diff in my MLP experiment is replacing the coupled oscillator component of their architecture with an MLP. Everything else stays the same.
You might argue that coupled oscillators could be accelerated better at inference time. Not necessarily: if we find out that we need 10x more coupled oscillators than MLP flops for the same performance, then it might still be slower to use coupled oscillators at inference time.
To be clear, their setup actually uses a bunch of conventional components already, in a somewhat well-engineered way. The only diff in my MLP experiment is replacing the coupled oscillator component of their architecture with an MLP. Everything else stays the same.
@unixpickle Yes, you can grow the depth easily, but just not width