Information-Time Proximal Policy Optimization
Published in arXiv preprint, 2026
InfoPPO reparameterizes temporal progression in LLM reinforcement learning using information density rather than token count. This yields a shared state-dependent structure for credit propagation and policy updates, combining non-trivial discounting with adaptive clipping. The approach preserves terminal supervision over long reasoning trajectories while constraining local policy changes. A theoretical analysis establishes a policy-improvement lower bound, and experiments with Qwen3 models show consistent improvements across five competition-style mathematical reasoning benchmarks.
Recommended citation: Zeng, Y., Cui, X., Song, Y., Liu, G., Xin, H., Zhang, K., Deng, C., Zhan, K., Ying, J., Zhao, J., Zhang, H., & Wang, J. (2026). Information-Time Proximal Policy Optimization. arXiv preprint arXiv:2609.24380.
Download Paper