Two-Way Repeated-Measures in UX

In addition to the one-way Repeated-Measures, ‘Two-Way Repeated-Measures ANOVA’ can also be conducted in UX studies. To show a real example, here is how I did it for my MS thesis…

For the purpose of this study, I conducted a 2 x 3 Two-Way Repeated-Measures ANOVA in order to examine the effects of two levels of search engine,  Traditional Search Engine (TSE) and Social Search Engine (SSE) and three levels of task type, objective, combo, and subjective search queries. The measured variables (or Dependent variables) were time on task, task completion, satisfaction, and emotion for each search query experience.

Abstract
20 participants performed six tasks, each at three levels; objective, combination, and subjective task-levels. Participants used Traditional Search Engine (TSE) and Social Search Engine (SSE) in order to perform the tasks. The purpose of this study was to examine whether SSE improved efficiency, effectiveness, satisfaction, and emotional experiences of users during web information retrieval. The hypothesis was that SSE, as opposed to TSE, would enhance user’s search experience. The results suggested that, while it took longer to find specific subjective information, task completion, satisfaction, and positive emotion was significantly higher using SSE for subjective tasks. In other words, using SSE for subjective queries enhanced effectiveness, satisfaction, and positive emotions of the participants.

Methodology
There were three levels of tasks: objective, subjective, and combination tasks. At each task level two tasks were assigned. Each task were to be performed using both SSE and TSE. The types of tasks were different types of information retrieval tasks at these three different levels. In other words, each participant completed a total of six tasks, presented as six different search queries.

The data sets were coded by time on task, task completion, satisfaction rate, and emotional state. To illustrate how the coding would look like for one case, let us say that a participant, using Traditional Search Engine (Google), took 60 seconds to find the targeted web resource, ended up completing the task, was satisfied with the search results, and felt happy while searching.

If so, the code would look like this:  P1 – TSE 60 1 4 5 

P1 = Participant number one
TSE = Traditional Search Engine (Google)
60 = Time on Task; It took 60 second for the participant to find the targeted web resource
1 =  Task Completion; The participant did find the targeted web resource
4 = Satisfaction Rate; The participant indicated that he/she was satisfied after the search results and their search experience
5 = Emotional State; The participant indicated that he/she felt happy while searching

Having numerical data, it gets pretty straight forward to enter the data into SPSS and to run ANOVA tests. More specifically, it gets pretty accurate to run multivariate tests and paired-sample t-tests in order to examine any statistically significant results. Moreover, one can quite easily determine any Main Effect and/or Interactions that may exist. This, of course, assumes that the data is accurate.

Disclaimer: I am not a Statistician…. just a UX researcher with (college) background in SPSS. My only attempt here is to bring statistical tools closer to UX studies.