Twilight of the Gods. Fable and 10 more LLMs on a Code Reorganization Task. Comparison.
Other languages
Эта статья также доступна на русском: Гибель богов.
Materials & raw data
All 11 model proposals, the cross-reviews, the theses runs, and the ranking script are published here: Materials & reproduce this experiment.
This is a detailed write-up of one experiment. I took a god node from a real LangGraph agent and asked 5 American and 6 Chinese models first to propose how to untangle it, then to evaluate each other's proposals. After that, I tried three different ways to figure out which of them to trust on the matter.