Sahar Admoni

Large language models (LLMs) seem to offer an easy path to interpretability: just ask them to explain their answers. Yet the features driving an answer often differ from those emphasized in its explanation, meaning post-hoc rationales can misrepresent what actually shaped the model’s output. We quantify this gap by comparing the feature-importance distributions of answers and their explanations. Prior analyses reveal such discrepancies, but large-scale study has been limited by the high computational cost of attribution methods. To address this, we introduce the Post-hoc Self-Consistency Bank (PSCB), a large-scale benchmark linking model decisions with diverse explanations and attribution vectors across datasets, methods, and model families. Using PSCB, we find that Spearman rank correlation provides a more reliable signal of alignment than cosine similarity. Building on this insight, we apply Direct Preference Optimization (DPO) to attribution-based preference data, improving alignment without degrading task accuracy. Our results pave the way toward scalable and faithful alignment between LLM decisions and their natural language explanations.

Explaining reinforcement learning agents remains challenging, as policies emerge from complex reward structures and neural representations that are difficult for humans to interpret. Existing approaches rely on curated demonstrations that reveal local behaviors but offer limited insight into global strategy, leaving users to infer intent from raw observations. We propose \emphSySLLM (Synthesized Summary using Large Language Models), a framework that reformulates policy interpretation as a language-generation problem. Rather than relying on visual demonstrations, SySLLM translates spatiotemporal trajectories (an input modality outside the natural domain of LLMs) into structured text and prompts the model to produce coherent natural-language summaries that describe the agent’s goals, exploration style, and decision patterns. SySLLM scales to long-horizon and semantically rich environments without task-specific fine-tuning, leveraging the world knowledge and compositional reasoning of LLMs to capture latent behavioral structure across diverse agents. Expert evaluations show that SySLLM summaries align closely with human analyses of policy behavior, and in a large-scale user study, 75.5% of participants preferred these textual summaries over state-of-the-art demonstration-based explanations. Together, these results establish abstractive-textual summarization as a new, scalable paradigm for interpreting complex RL behaviors through language.