ChaCha Beats Google and Yahoo in Mobile Voice Search Tests

Mobile analyst firmMSearchGroovehas just published the results of a series of tests which show that the mobile search serviceChaChabeat out two other voice-enabled search applications on the iPhone when it comes to search query accuracy.[Update, Ed:a commenter points out that the report was actually sponsored by ChaCha]To test this, the researchers usedGoogle's own mobile applicationandVlingo for iPhone, an app that lets you search both Google or Yahoo. Oddly, they ignoredYahoo's mobile app, which also has voice search built in.

The results of their study aren't entirely shocking: if you want to be understood, ask a human, not a computer.

The Mobile Search Tests

ChaCha's mobile search servicecan be accessed both by SMS and by calling a toll-free 1-800 number. Since these tests focused on voice search, the phone-in method was used. When using ChaCha, the service identified the queries accurately in 94.4% of the cases and delivered accurate search results 88.9% of the time. Vlingo, which the researchers used to test Yahoo search, only interpreted queries correctly in 72.2% of the cases and delivered accurate results 27.8% of the time. Google, surprisingly, fared worst of all. Their mobile application only understood spoken queries in 16.7% of tests and delivered accurate results 22.2% of the time.

To test the applications, the researchers conducted two rounds of tests using both keyword search and natural language queries where they asked questions using sentences. The queries represented a cross-section of typical mobile searches in categories like navigation, directions, local search, general information, social search, and long-tail search.

It's not all that surprising to find thatChaChaoutperformed the other voice-enabled applications - after all, they have real, live humans on the other end of the line to interpret the spoken questions. What is surprising, though, is how wide the gap is in between the human-powered search and the speech recognition apps,especiallywhen contrasting ChaCha with Google.