Most technology decisions start with a product. This guide starts with students. Work through the five steps in order; if a step has no answer, stop and get one before going further.
- Name the mechanismWhich theory, which lever?
TechnologyWrite one sentence: which students, which need, and which theory explains why the tool should help.
- Define the responseWho acts, and how fast?
TechnologyName the role that responds to what the tool surfaces and the time they have to respond.
- Plan the evidenceWhat would show it works?
TechnologyPick a process measure, an outcome, and a comparison group before launch.
- Test for harmWho could be misjudged?
TechnologyAsk the vendor for accuracy by student group and list who the data describes poorly.
- Govern & reviewApprove, pilot, sunset
TechnologyTake it to the governance body; set a pilot length and retirement criteria now.
Questions to ask any vendor
- What student success theory or research is the product built on?
- Which data fields does it use, and which are optional?
- What is its accuracy for each student group at institutions like ours?
- What happens after a flag, and who decides?
- Can students see why they were flagged and correct their data?
- Who owns the data, what can you do with it, and how is it deleted when we leave?
- Which results come from independent evaluations, and which are your own?
Red flags
- Impact claims with no comparison group
- Demographic traits used as risk factors
- No way for students to see or contest their data
- “Set it and forget it” messaging that replaces staff contact
- Results reported only as averages