Preprint, 2026
Jason Chan, Zhixue Zhao, Robert Gaizauskas
We introduce a new task and automatic evaluation framework to test whether LLMs can reconcile apparent contradictions through explanations, finding that models generally struggle even with additional test-time compute.
ICLR Workshop on Logical Reasoning of Large Language Models, 2026
Jason Chan, Robert Gaizauskas, Zhixue Zhao
We argue that relying on logic as a harness in fact-checking systematically fails to handle misleading claims, and propose a paradigm flip by viewing LLMs as a human-centric complement to logic-based systems.
ICML, 2025
Jason Chan, Robert Gaizauskas, Zhixue Zhao
We create the RULEBREAKERS dataset on which to evaluate LLMs, finding that these models over-rigidly apply formal logic when reasoning with natural language, disregarding common sense and factual knowledge.