Simple ideas that work are the best!
You have less hyperparameters to benchmaxx a solution without multiple hyperparameters and when you are trying to scope a publishable method, you're less likely to find a simple method that works.
one thing i've learned after doing language modeling research for ten years is that if you want your contribution to catch on it has to be *very* simple. researchers are brought up through the paper writing process and it makes some think that you need to have novelty but novelty is basically the enemy of actual advancement. you need to strive to have as *little* novelty as possible. every bit of additional novelty: 1. makes your contribution harder to explain 2. makes it harder to understand 3. makes it harder to implement 4. makes it harder to experiment with and verify 5. makes your contribution literally worse
i really do think that more complex things in machine learning just end up working worse. and then when you consider points 1-4, it just all piles up and makes complexity a silly endeavor.
keep it simple. it seems wrong at first, i think some people think that simpler things are stupider, but when you gain experience you notice that the simplest things are usually the smartest, best, and hardest to design and build, but they're so worth it.