Agarwal says marginalization is only feasible under binary rewards.
Thinking of optimizing tool-call chains in terms of RL makes sense to me (to the extent RL makes sense). You take actions, they have costs, they could be destructive: need a good "policy".
but for reasoning tokens, it is super restrictive imo. why optimize the max and not the marginal, for example? why think as RL and not as latent variable learning problem?
@yoavgo As it turns out, the KL regularized return maximization objective is exactly the ELBO from variational inference. One is forced to REINFORCE because you can’t use the reparameterization trick, but other than that it’s a VAE where action / reasoning tokens are the latents.
Thinking of optimizing tool-call chains in terms of RL makes sense to me (to the extent RL makes sense). You take actions, they have costs, they could be destructive: need a good "policy".
but for reasoning tokens, it is super restrictive imo. why optimize the max and not the marginal, for example? why think as RL and not as latent variable learning problem?
@yoavgo Yeah, if you use a single-sample estimator of the expectation. That's also true when you sample one z ~ p(z | x) in VAE. Of course you could use multiple samples or do fancier things.
@TacoCohen but you still commit to a single latent path, no? because this is how agents in an environment must act
Agarwal says marginalization is only feasible under binary rewards.
Thinking of optimizing tool-call chains in terms of RL makes sense to me (to the extent RL makes sense). You take actions, they have costs, they could be destructive: need a good "policy".
but for reasoning tokens, it is super restrictive imo. why optimize the max and not the marginal, for example? why think as RL and not as latent variable learning problem?
@yoavgo As it turns out, the KL regularized return maximization objective is exactly the ELBO from variational inference. One is forced to REINFORCE because you can’t use the reparameterization trick, but other than that it’s a VAE where action / reasoning tokens are the latents.
Thinking of optimizing tool-call chains in terms of RL makes sense to me (to the extent RL makes sense). You take actions, they have costs, they could be destructive: need a good "policy".
but for reasoning tokens, it is super restrictive imo. why optimize the max and not the marginal, for example? why think as RL and not as latent variable learning problem?
@yoavgo Yeah, if you use a single-sample estimator of the expectation. That's also true when you sample one z ~ p(z | x) in VAE. Of course you could use multiple samples or do fancier things.
@TacoCohen but you still commit to a single latent path, no? because this is how agents in an environment must act
