Evaluating User Experiences
Apply evaluation methods and criteria to assess user experience of IT services and solutions, and to assess design options relating to them. Evaluation methods might include, for example, surveys, interviews, focus groups, walkthroughs, productivity studies, and usability tests. Evaluation criteria might include, for example, system performance, task success rate, time on task, error rate, and satisfaction ratings.
Improvement Planning
Practices-Outcomes-Metrics (POM)
Representative POMs are described for Evaluating User Experiences at each level of maturity.
- 1Initial
- Practice
- No user-acceptance testing.
- Outcome
- Usability of developed systems is not considered.
- Metric
- % of projects focused on usability as a criteria for success.
- Practice
- Focus is on functional system testing pre-launch.
- Outcome
- Developed systems mainly tested from a functional perspective.
- Metric
- % of evaluation testing pre versus post system acceptance.
- Practice
- User evaluation not sought.
- Outcome
- Feedback is not sought once design has move into its production phase.
- Metric
- % of user involved in evaluation as a proportion of the total user population.
- 2Basic
- Practice
- User evaluation sought from small cross section of users.
- Outcome
- Informal approaches developing around evaluating user experience.
- Metric
- Num and variation in evaluation approaches across projects versus supported evaluation approaches.
- Practice
- User evaluation limited to sub-section of users.
- Outcome
- User experience may not be representative of overall user experience.
- Metric
- % of total population of users included in evaluation process.
- 3Intermediate
- Practice
- Basic evaluation techniques used.
- Outcome
- Based around asking users, asking experts, and controlled user testing.
- Metric
- Variation in methods used to evaluation prototype performance by project.
- Practice
- Focus is on summative evaluation.
- Outcome
- Evaluation focused on assessing the success of the finished product.
- Metric
- % of evaluation assessments carried out throughout design process by project.
- 4Advanced
- Practice
- Multiple evaluation methods used.
- Outcome
- Observation in situ, user focus groups, expert advise, user testing and modelling user task performance.
- Metric
- Average number of evaluation methods in common use across design processes.
- Practice
- Both summative and formative evaluation carried out
- Outcome
- Evaluation carried out during design phase to ensure design continues to meet end-user need.
- Metric
- Num of projects using both summative and formative evaluation.
- 5Optimized
- Practice
- Evaluation expanded to include heuristic and predictive evaluation.
- Outcome
- Use of know user behaviour and predicted response to help understand actual user response.
- Metric
- % change in user satisfaction based on evaluation techniques.