Stakeholders' Perspectives

“A distinguishing feature of evaluation is the universal focus on stakeholder perspectives, ......,and evaluation are judged on if and how stakeholder perspectives are included” (Matheson, 2008).

So how to engage stakeholders and include the stakeholder perspectives during the evaluation process? (You can share your own experiences/stories or talk about the approaches you have heard of/read)

Page 194 of Mathison (2007), "A distinguishing feature of evaluation is the universal focus on stakeholder perspectives, a feature not shared by social science research. And evaluations are judged on if and how stakeholder perspectives are included. While evaluation models are based on different epistemological foundations, all evaluation models attend to stakeholder perspectives-the assessments, values, and meanings of the evaluand's stakeholders are essential elements in any evaluation." .... "When research participants are referred to as stakeholders, it is often a reference to whom the data are collected from rather than a consideration of the stakeholder's vested interests. "

How to engage stakeholders and include the stakeholder perspectives during the evaluation process?

Factors: Internal vs. External

How to engage with stakeholders depends on how you know the stakeholders. Evaluators, as discussed in Fitzpatrick, J., Sanders, J. & Worthen, B. (2011), can be internal or external evaluators. If you work within a company and are evaluating your own company, you are more likely to have a vested interests and therefor more likely to have bias, unless it's (ideally) for a client or other department (which would make you an external evaluator).

Foundational Elements

Evaluators also come into a project or evaluation with some knowledge of a proposal, business requirements, vision document, or other criteria. Often, or in the best case scenario, an evaluator would be part of writing, strategizing, and reviewing these documents. These documents are the foundation or backbone of what is known about stakeholders and the criteria for evaluation.

In practice - Factors

There are a lot of ways to engage stakeholders and include stakeholder perspectives in the evaluation process. Most of these will likely depend on a number of factors like:

• The type of business (B2B or Business to Business or B2C Business to consumer)
• The industry (advertising agency, government, retail, IT, education, policy/HR)
• Availability of quality assurance (QA) testers
• Background of the evaluator (social science, technology, design, level of training, government, policy, science, health care, specialized expertise and willingness to learn more about the project/program)
• Ability to learn more about the program/put the evaluator in the shoes of the stakeholder
• Whether or not the stakeholders are minors
• Type of data being evaluated
• Laws and legislature - and whether or not those laws/legislature change during the evaluation (this happens A LOT in health care)
• Privacy policy
• Stakeholders with financial/political/bias interests opinions
• Budget

Methods

I've used a number of methods to engage client stakeholders and users for user experience testing, design, and improvement of software systems. Some of those methods include:

• Meetings - for financial stakeholders
• Brainstorming sessions  - for financial stakeholders
• Surveys - for user stakeholders you can contact
• Behavioral flows  - for user stakeholders you might not be able to contact easily
• Analytics - for user stakeholders you might not be able to contact easily
• Data tracking - for user stakeholders you might not be able to contact easily
• Eye tracking software
• Heat mapping
• Focus groups
• Giving users a list of tasks and watch them complete the tasks, then ask some users to talk about their struggles and video the process
• A/B testings
• Prototyping
• Wireframes
• Demonstrations
• Feedback forms
• Training tools and walk-throughs with tracking software (scripted use of product)
• Natural use of a product (paid shoppers)
• Tracking search forms
• Personas - This is a great way to always keep stakeholders who are not present or can not represent themselves or a large few groups of people easily.
• Ethnographic field studies
• Usability benchmarking
• Clickstream analysis
• Email surveys
• Card sorting
• Diary studies
• Interviews
• Participatory design
• Measure performance
• First click testing
• Heuristic evaluation/expert review
• Use Cases
• Design workgroup

References 

1. Mathison, Sandra (2007). What is the difference between evaluation and research? And why do we care? In N.L. Smith & P. Brandon (Eds.). Fundamental issues in evaluation. New York: Guilford Publishers. https://books.google.com/books?id=l6oEU80PWwsC&pg=PA194&lpg=PA194&dq=A+distinguishing+feature+of+evaluation+is+the+universal+focus+on+stakeholder+perspective&source=bl&ots=7lmxhjF5vO&sig=5McBFL0N_DyEBAP9jc1T6qyXeU8&hl=en&sa=X&ved=0ahUKEwjbmp2S9dfKAhXDXhoKHSeDDgAQ6AEIHTAA#v=onepage&q=A%20distinguishing%20feature%20of%20evaluation%20is%20the%20universal%20focus%20on%20stakeholder%20perspective&f=false 
2. Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.), Chapter 1. New York: Pearson.