IVI Framework Viewer

Evaluating User Experiences

B2

Apply evaluation methods and criteria to assess user experience of IT services and solutions, and to assess design options relating to them. Evaluation methods might include, for example, surveys, interviews, focus groups, walkthroughs, productivity studies, and usability tests. Evaluation criteria might include, for example, system performance, task success rate, time on task, error rate, and satisfaction ratings.

Improvement Planning

Practices-Outcomes-Metrics (POM)

Representative POMs are described for Evaluating User Experiences at each level of maturity.

1Initial
  • Practice
    No user-acceptance testing.
    Outcome
    Usability of developed systems is not considered.
    Metric
    % of projects focused on usability as a criteria for success.
  • Practice
    Focus is on functional system testing pre-launch.
    Outcome
    Developed systems mainly tested from a functional perspective.
    Metric
    % of evaluation testing pre versus post system acceptance.
  • Practice
    User evaluation not sought.
    Outcome
    Feedback is not sought once design has move into its production phase.
    Metric
    % of user involved in evaluation as a proportion of the total user population.
2Basic
  • Practice
    User evaluation sought from small cross section of users.
    Outcome
    Informal approaches developing around evaluating user experience.
    Metric
    Num and variation in evaluation approaches across projects versus supported evaluation approaches.
  • Practice
    User evaluation limited to sub-section of users.
    Outcome
    User experience may not be representative of overall user experience.
    Metric
    % of total population of users included in evaluation process.
3Intermediate
  • Practice
    Basic evaluation techniques used.
    Outcome
    Based around asking users, asking experts, and controlled user testing.
    Metric
    Variation in methods used to evaluation prototype performance by project.
  • Practice
    Focus is on summative evaluation.
    Outcome
    Evaluation focused on assessing the success of the finished product.
    Metric
    % of evaluation assessments carried out throughout design process by project.
4Advanced
  • Practice
    Multiple evaluation methods used.
    Outcome
    Observation in situ, user focus groups, expert advise, user testing and modelling user task performance.
    Metric
    Average number of evaluation methods in common use across design processes.
  • Practice
    Both summative and formative evaluation carried out
    Outcome
    Evaluation carried out during design phase to ensure design continues to meet end-user need.
    Metric
    Num of projects using both summative and formative evaluation.
5Optimized
  • Practice
    Evaluation expanded to include heuristic and predictive evaluation.
    Outcome
    Use of know user behaviour and predicted response to help understand actual user response.
    Metric
    % change in user satisfaction based on evaluation techniques.