Charting Public Policy for RL Systems

Choices, Risks, and Reward Reports: Charting Public Policy for RL Systems. Thomas Krendl Gilbert, Sarah Dean, Tom Zick, Nathan Lambert (CLTC, 2022).

With new machine learning advances comes new risks. This white paper addresses the risks associcated with Reinforcement Learning with an aime to empower policy makers with some richer understanding to ensure safety. The unique challenges facing RL are summed up well in the intro - “In ML, the primary risks have to do with outputs that a model generates. In RL, howeer, the risks come from the initial specification of the task.” Therefore, more attention must be applied to the initial design. “Measure twice cut once”, kind of. Or maybe it’s more like “careful what you wish for”.

The authors advocate for “Reward Reports for AI systems”, which is to say documents that enumerate the design choices, including types of feedback, performance optimization metrics, why specific components were used, and a description of how updates will be made based on system performance evaluation. Organizations charged with public safety should have visiblity into these reports, placing the algorithm designer closer in feel to the civil engineer than a traditional software engineer (i.e. “a technical expert whose credentials and demonstrated skills are trusted to oversee critical social infrastructure, and are worthy of certification but public authorities”).