Student Success by DesignA practitioner’s guide to theory, data, and AI in student success

9.1 · Practitioner Tools & Resources

Decision guides and checklists

A short path from “a vendor called” to “we know what this is for.” Use it in the first meeting, before a demo shapes the conversation.

Short shows the summary, takeaway, and questions.

Most technology decisions start with a product. This guide starts with students. Work through the five steps in order; if a step has no answer, stop and get one before going further.

Should we adopt this tool?
  1. Name the mechanismWhich theory, which lever?

    TechnologyWrite one sentence: which students, which need, and which theory explains why the tool should help.

  2. Define the responseWho acts, and how fast?

    TechnologyName the role that responds to what the tool surfaces and the time they have to respond.

  3. Plan the evidenceWhat would show it works?

    TechnologyPick a process measure, an outcome, and a comparison group before launch.

  4. Test for harmWho could be misjudged?

    TechnologyAsk the vendor for accuracy by student group and list who the data describes poorly.

  5. Govern & reviewApprove, pilot, sunset

    TechnologyTake it to the governance body; set a pilot length and retirement criteria now.

Questions to ask any vendor

  • What student success theory or research is the product built on?
  • Which data fields does it use, and which are optional?
  • What is its accuracy for each student group at institutions like ours?
  • What happens after a flag, and who decides?
  • Can students see why they were flagged and correct their data?
  • Who owns the data, what can you do with it, and how is it deleted when we leave?
  • Which results come from independent evaluations, and which are your own?

Red flags

  • Impact claims with no comparison group
  • Demographic traits used as risk factors
  • No way for students to see or contest their data
  • “Set it and forget it” messaging that replaces staff contact
  • Results reported only as averages