You compare two chatbots by asking a strong model which of their two answers to each prompt is better, and rank them by pairs won.
Swapping the answer order changes the verdict on about a third of the pairs, and the judge leans toward whichever answer it read first.