In the AWA project, we developed an approach to generate workarounds for Web applications to automatically overcome failures in Web applications exploiting the intrinsic redundancy of software systems. The technique takes advantage of the interactive nature of Web applications by assuming that users report the perceived failures.
When users interact with Web applications they may be exposed to faulty behaviors, but it does not mean that all the users will perceive such behaviors as faulty. For example when asking for the best route from a location A to a location B, users may not perceive the non optimal route as a wrong one.
The goal of this qualitative user study is to understand to what extent the users perceive failures as such, to validate the hypothesis of our study.
We conducted the experiment according to the guidelines of Interaction Design: Beyond Human-Computer Interaction by Roger, Sharp and Preece (Wiley 2011): We identified a target population, prepared a questionnaire and a list of tasks, asked the participants to accomplish the tasks, and analyzed the results. Below, we report the details of the experiment and the results.
We took the Website of our research group, which uses JQuery to provide some functionality, and we produced a faulty version of the Website by substituting the current correct version of the JQuery library with an obsolete faulty version with 6 bugs that affected the results of 6 different pages. The faulty Website can be accessed at http://star.inf.usi.ch/test/star.
This is the list of the bugs considered for the study with a link to the bug tracker and a short description:
These bugs affected the following pages of the Web site:
We designed a list of tasks, and we asked the participants of the study to perform the tasks on the faulty version of the Website, and report the final outcome. The questionnaire is composed of 11 tasks: each task is an action that have to be performed on the Website and requires a closed-ended answer. A participant is required to read the task, perform it, and answer the question with two possible outcomes:
The participant is free to navigate through the entire Website in the attempt to accomplish the tasks. The interviewer follows his or her behavior but does not interfere with the operations of the participant. The participant is encouraged to comment the final outcome of a task, with a particular emphasis for those that he or she considers as Not Done.
The 6 previously described bugs produce a failure in 6 out of the 11 tasks. This means that 6 out of 11 tasks may not be correctly achieved because of a failure that may rise and prevent the user to proceed or produce the expected result. The remaining 5 out of 11 tasks are failure free and the participants should be able to complete them correctly.
Here we report the complete list of tasks:
|Please, open this webpage: http://star.inf.usi.ch/test/star/ and complete the following tasks. When you think you have correctly achieved the task, tick the checkbox Done, when you think you cannot complete or achieve the task as expected, tick the checkbox Not Done.|
|1||Sort the news from the newest to the oldest, newest on top|
|2||Remove the news older than 2014|
|3||Read about the details of the different subareas of the two research areas carried on by the group|
|4||Open the webpage of a group member|
|5||Identify the total number of the group members and the number of members by role|
|6||Read the Bibtex information of a paper published by the group|
|7||Download a paper of the group|
|8||Look at the tutorial for the software ARMOR and send a comment to the developers|
|9||Look at the rewriting rules for Guava and Joda-Time in the page of the software ARMOR|
|10||Look for the print-friendly version of the page of the software ARMOR and return to the browser version|
|11||Read the research focus on self-healing systems|
The failure free tasks are: 1, 4, 6, 9, 11.
A failure may rise in the tasks: 2, 3, 5, 7, 8, 10.
We validated the user study and the questionnaire with the other researchers in our research group that were not involved in this work.
We run the study with two different groups of participants: a first study with people with no IT background and a second study with computer science students. We collected and analyzed the results of the two categories of participants separately.
For the first study we selected 20 participants trying to cover different age ranges, educational background and profession. The goal of the study is to understand if a user can perceive a failure while he/she is interacting with a Web application so, to avoid any bias that a technical background may rise, we selected only people with no IT education or professional background.
The following table summarizes the characteristics of the 20 participants we selected:
|Education Level||IT Expertise|
|Internet and emails||20|
For the second study we asked 7 computer science graduate students to take the same questionnaire.
In this section we report the detailed summary of the results of the two user studies. The raw data (the questionnaires filled out by the participants) can be downloaded below. We summarized the results in two tables that follow the convections below:
In the following we report the summary of the user study that involves participants with non-IT background:
In the following we report the summary of the user study that involves computer science graduate students: