Scaling RL for Creative Tasks

In March Cameron and I trained a model to paint with code. One of the hardest parts of that project was designing the reward. LLM judges couldn’t tell good paintings from slightly better ones, handing out the same score again and again, and RL can’t reach models when the rewards are a tie.

This work, with Jacky Kwok, Cameron Neal Wu and Simon Guo, a collaboration between Stanford University and Thinking Machines, replaces the judge with a verifier. Instead of one coarse score, it reads how confident the model is across a finer scale, many times over, across several criteria. The result is a model that paints, critiques its own work, and improves from that critique, reaching the same quality in about two thirds of the training.

[Link to Thinking Machines blog post]