Users criticized the COLM review process for ignoring rebuttals on multimodal safety and noted that such flaws appear widespread across venues.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@mariyaivasileva It's not just COLM, it's pretty much everywhere.
COLM reviews are out for my solo paper on multimodal safety. I put a lot of care into the rebuttal. The paper is a survey-ish position piece analyzing and taxonomizing failure modes of multimodal safety evaluator systems across 100+ papers and benchmarks, but one reviewer asked for more than observational evidence. So I added a 4-page anonymous supplement with three new experiments on a procedurally generated safety dataset I designed, to turn the question into a more concrete empirical test: how easily can a VLM’s response flip from safe to unsafe when the same adversarial perturbation is presented through a different modality? Two of three reviewers never engaged with the rebuttal or even acknowledged it. Private comments to the AC asking for engagement went nowhere. The third reviewer — originally the most conservative and lowest-scoring — raised their score because of the due diligence. The final decision was essentially a meta-summary of the original reviews, with no mention of the rebuttal. Like others have mentioned, I’m unfortunately inclined not to submit to COLM again. I put in the work because I believe this problem matters and wanted the paper to see the light of day, but the way the review process went was beyond discouraging.
The incident highlights a broader lack of peer-review engagement.
Users criticized the COLM review process for ignoring rebuttals on multimodal safety and noted that such flaws appear widespread across venues.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.