Monthly Archives: November 2011

November 13, 2011

Two-Way Repeated-Measures in UX

By Dr. Nil

In addition to the one-way Repeated-Measures, ‘Two-Way Repeated-Measures ANOVA’ can also be conducted in UX studies. To show a real example, here is how I did it for my MS thesis…

For the purpose of this study, I conducted a 2 x 3 Two-Way Repeated-Measures ANOVA in order to examine the effects of two levels of search engine,  Traditional Search Engine (TSE) and Social Search Engine (SSE) and three levels of task type, objective, combo, and subjective search queries. The measured variables (or Dependent variables) were time on task, task completion, satisfaction, and emotion for each search query experience.

Abstract
20 participants performed six tasks, each at three levels; objective, combination, and subjective task-levels. Participants used Traditional Search Engine (TSE) and Social Search Engine (SSE) in order to perform the tasks. The purpose of this study was to examine whether SSE improved efficiency, effectiveness, satisfaction, and emotional experiences of users during web information retrieval. The hypothesis was that SSE, as opposed to TSE, would enhance user’s search experience. The results suggested that, while it took longer to find specific subjective information, task completion, satisfaction, and positive emotion was significantly higher using SSE for subjective tasks. In other words, using SSE for subjective queries enhanced effectiveness, satisfaction, and positive emotions of the participants.

Methodology
There were three levels of tasks: objective, subjective, and combination tasks. At each task level two tasks were assigned. Each task were to be performed using both SSE and TSE. The types of tasks were different types of information retrieval tasks at these three different levels. In other words, each participant completed a total of six tasks, presented as six different search queries.

The data sets were coded by time on task, task completion, satisfaction rate, and emotional state. To illustrate how the coding would look like for one case, let us say that a participant, using Traditional Search Engine (Google), took 60 seconds to find the targeted web resource, ended up completing the task, was satisfied with the search results, and felt happy while searching.

If so, the code would look like this:  P1 – TSE 60 1 4 5 

P1 = Participant number one
TSE = Traditional Search Engine (Google)
60 = Time on Task; It took 60 second for the participant to find the targeted web resource
1 =  Task Completion; The participant did find the targeted web resource
4 = Satisfaction Rate; The participant indicated that he/she was satisfied after the search results and their search experience
5 = Emotional State; The participant indicated that he/she felt happy while searching

Having numerical data, it gets pretty straight forward to enter the data into SPSS and to run ANOVA tests. More specifically, it gets pretty accurate to run multivariate tests and paired-sample t-tests in order to examine any statistically significant results. Moreover, one can quite easily determine any Main Effect and/or Interactions that may exist. This, of course, assumes that the data is accurate.

Disclaimer: I am not a Statistician…. just a UX researcher with (college) background in SPSS. My only attempt here is to bring statistical tools closer to UX studies.

Advertisement

Leave a comment | posted in Uncategorized

November 5, 2011

Repeated-Measures Design in UX

By Dr. Nil

Reviewing my SPSS notes, I recalled the many aspects where this powerful tool comes in handy in UX studies. Not everyone ends up using it on a regular basis, simply because many of our studies end up being more qualitative and observational. In addition, sometimes it is just hard to wrap your head around it when time/budget pressure keep coming at you left and right. But if we knew simple quick ways of running our analysis this way, perhaps we’d find the results more assuring and useful.

So, here I thought of gathering all my notes and try to come up with simple examples that may help me remember all the useful statistical analysis applications. Let us start with Repeated-Measures Design and go from there. Hope you find this as useful as I did!

What is  Repeated-measures?
Repeated-measures, or Paired-Samples t-test, assesses if the mean differences between paired observations are significantly different from zero. If the difference is zero, then the before and after effects are the same, i.e. there is NO difference between the observations.

How can it be used in UX research?
There are many applications for using repeated-measures, of course, but for the sake of simplicity, I am proposing the following (very) simple example:

Say we have two different products displayed somewhere on our site and we want to measure Findability. We ask the participants to perform the same task of finding product X and then product Y on the site. Assuming all other variables are the same, we have the participants perform the task one after the other. Note: In order to counter balance, we need to change the order tasks, i.e. every other participant first finds product X and then product Y.

After the tasks, we ask the participants to rate their experience on a 7-scale Likert scale, ranging from the product being real easy to find to real hard to find. At the end of the study, we should have two numbers (from 1-7) for each participant; one for findability of product X and one for findability of product Y. This is the reason this test is called repeated-measure… it is because we measure/observe the same participant several times.

How to run the test in SPSS
If the menus in SPSS have not changed drastically, the following steps should do the trick:

  1. Analyze –> Compare Means –> Paired-Samples T-Test
  2. Click on the two variables that you have collected, product X and product Y, and then click the little ‘arrow’
  3. The two variables should appear on the Paired Variables box
  4. Hit OK
Now you should see two tables as a result of this analysis. To keep things simple, the most important part of this table is where it says ‘Sig.(2-tailed)’, which is known as the P-value. If this number is less than 0.05, you have a significant result, which means that the there is a difference between the two observations. In our case, let us say that our P-value is 0.02. In this case, we have a significant result, which means that there is a difference between the two. In other words, one of the products is way easier to find than the other. Now the question is: Which one?

For us to figure out which product was easier to find, we’d need to look at the Mean value. In our case, let us say that low numbers indicated that the participants found it easy to find the product. Therefore, the product with the lowest Mean would be the easier one to find.

And there you have it! A simple case where one can easily use SPSS to run valid statistical analysis.

Disclaimer: I am not a Statistician…. just a UX researcher with (college) background in SPSS. My only attempt here is to bring statistical tools closer to UX studies.

1 Comment | posted in Statistics