Eye Tracking Bing vs. Google
By Anna Johnson on June 15th, 2009How are search engine users looking at Microsoft’s new search engine, Bing, as compared with Google? Well, according to preliminary research from User Centric, Inc. Bing users tend to look at the sponsored (paid) listings more so than Google users.
User Centric used eye tracking technology to capture 21 participants’ eye movements as they completed two informational (e.g., “Learn about eating healthy”) and two transactional (e.g., “Book a last minute vacation”) search tasks in each engine.
The researchers found that users devoted about the same amount of visual attention on organic search results and top sponsored links in both both search engines.
Sponsored links on the right, however, attracted more attention on Bing than they did on Google.
On average, across all four tasks, 42 percent of participants looked at Bing’s sponsored links on the right, while only 25 percent of participants looked at Google’s right rail links.
21 participants is hardly indicative, and it’s too early to tell how people will ultimately tend to interact with Bing versus Google (or any other search engine).
Still, we can certainly add User Centric’s research to our evolving understanding of the impact Bing will – or will not have – as a search engine.
July 7th, 2009 at 3:57 pm
The sample size collected had sufficient power to detect a significant difference. The type of study performed is a common practice in virtually all forms of research practiced in all behavioral science graduate programs in the world (defending dissertations and the like).
As you will find, User Centric consultants frequently publish papers in peer reviewed journals and publications, as well as present data at professional associations domestic and international. We believe that we run our studies in a sound and rigorous manner that is in line with the scientific method.
That said, we acknowledge some may argue about the sample size. Yes, we could have run more participants, but that only increases power. Would 30 have been better? 40? 100? While sample size is really a question of power, a majority believe sample size comes down to “this smells large enough”. But, the problem is that this assumes that the speaker says that these results or scores can be generalized to the population. This is not a question of confidence intervals, but a question of difference. Bing and Google’s scores were significantly different. This is an inferential statistics question whether we can infer from the data that the difference is significant. If we were to run the study 20 times what is the likelihood that we could have found this difference by chance alone (p value of .05).
User Centric did not say that the scores were generalizable, but that the difference between Bing and Google were significant. We are able to make that statement and yes, sample size played a role in the statistical analysis performed.
Now, we certainly welcome discussions on new research questions or even if we think the result is due to the novelty of the Bing design and whether this will fade away. Valid points where we could run a study with experienced users and other permutations. Please forward topics for research questions to me directly.
The result is what it is. Let’s talk about the implications, habituation, usefulness of features, whether the options were copied from Google and improved or made worse.
July 7th, 2009 at 6:48 pm
Thanks for clarifying the implications of User Centric’s research, Brad. It’s much appreciated.