Tuesday, November 22, 2016

Why the Winograd Schema Challenge is not an advance on the Turing test

© Huma Shah, 22 November, 2016

Charles Ortiz of Nuance Communication states: “The Winograd Schema Challenge provides us with a tool for concretely measuring research progress in commonsense reasoning, an essential element of our intelligent systems." (from here ).

So does the Turing test!


The Winograd Schema Challenge is not superior to the Turing test. This is because, the kinds of common-sense reasoning questions that Hector Levesque proposes are the kinds of ‘statements-followed-by-questions’ that judges already ask in English-language Turing test contests.

For example, in the 2008 Turing test experiment I designed (around the 18th Loebner Prize for AI) at Reading University, one of the judges (a psychologist), asked Eugene Goostman machine ‘My car is red. What color is my car?’. In 2008 Eugene Goostman replied correctly with ‘red’.

To me the Winograd Schema Challenge is a condescension and presumption that Turing test judges don’t have the intellectual faculty to ask smart questions to determine human from machine. The transcripts show that most judges do. Whether the judges are members of the public including teenagers, computer scientists or journalists, they ask all sorts of visceral questions during Turing test contests.

What a major Turing test challenge needs is not pittance prizes but a major few-million -sterling-pounds or 10million dollars award to address and build a truly Turing-conversational system.

Conversational commerce will drive question-answer systems to beyond common-sense reasoning. What is that anyway? Knowledge gained by experience :)

© Huma Shah, 22 November, 2016