Training a coding model to paint watercolours with TRL and OpenEnv
huggingface ·
The article describes an open engineering reproduction of Surya Narreddi’s project in which a coding model writes JavaScript with p5.brush to create watercolour paintings, following a video that had surpassed 1.5 million views. The implementation uses TRL and OpenEnv, with training on Hugging Face Jobs, the environment and scorer as Spaces, and published datasets, scripts, models, and other artifacts. Instead of optimizing for a verifiable answer, the training uses aesthetic rewards consisting of compilation and anti-cheating checks, a code-length term, a pairwise vision judge, and HPSv3, an image-preference model. The environment renders sketches in headless Chromium and restricts the model to ten of p5.brush’s 47 methods to preserve a watercolour style. Its reference pool contains 178 model-generated hibiscus paintings, rated by the author into “love” and “okay” tiers after several refinement rounds and generated from openly licensed iNaturalist photos. The author trained judge-led, HPS-led, and HPS-only reward mixes, reporting that the runs using the hand-rated pairwise judge learned according to their metrics and final paintings. The supplied article excerpt ends while explaining how the two reference tiers influence rewards, so its later results, costs, and proposed next steps are not included.
huggingface