Microsoft's Dimitris Papailiopoulos asks if a symbolic Python solver can crack the GSM8k math benchmark without LLMs
Omar Khattab questioned if LLMs would optimize the parameters.
@DimitrisPapail Is an LLM optimizer involved :D
Dimitris Papailiopoulos@DimitrisPapailSay I am trying to solve GSM8k but no LLMs allowed! Only with a symbolic style solver with perhaps a few trainable parameters, so it's effectively a python program. How high do you expect it to go?
@DimitrisPapail that’s a yes then! i think it can get almost arbitrarily good in that case, but will be brittle :-)
Dimitris Papailiopoulos@DimitrisPapail@lateinteraction no calls to llms allowed beyond the building of this magical solver :)
@DimitrisPapail but you can look into your training set no? i'm saying it will only generalize in distribution
Dimitris Papailiopoulos@DimitrisPapail@lateinteraction you can't look into your test set!
@DimitrisPapail haha i was speculating only! makes sense
Dimitris Papailiopoulos@DimitrisPapail@lateinteraction yes you can look into your training data set. but it's very hard (been trying)
@lateinteraction no calls to llms allowed beyond the building of this magical solver :)
Omar Khattab@lateinteraction@DimitrisPapail Is an LLM optimizer involved :D
@lateinteraction you can't look into your test set!
Omar Khattab@lateinteraction@DimitrisPapail that’s a yes then! i think it can get almost arbitrarily good in that case, but will be brittle :-)
@lateinteraction yes you can look into your training data set. but it's very hard (been trying)
Omar Khattab@lateinteraction@DimitrisPapail but you can look into your training set no? i'm saying it will only generalize in distribution
@qberthet to the train? absolutely
Quentin Berthet@qberthet@DimitrisPapail Is a lookup table allowed?
@DimitrisPapail Pi
Dimitris Papailiopoulos@DimitrisPapailSay I am trying to solve GSM8k but no LLMs allowed! Only with a symbolic style solver with perhaps a few trainable parameters, so it's effectively a python program. How high do you expect it to go?
@DimitrisPapail Is a lookup table allowed?
Dimitris Papailiopoulos@DimitrisPapailSay I am trying to solve GSM8k but no LLMs allowed! Only with a symbolic style solver with perhaps a few trainable parameters, so it's effectively a python program. How high do you expect it to go?