Stop Measuring Seats Filled

Sunday Morning Coffee

Here's an impressive-sounding program report: 347 people participated.

Okay.

And?

I don't mean to be difficult. Well, maybe I do a little. But after decades in correctional education, I've become increasingly skeptical of numbers that tell us how many people showed up without telling us what happened because they did. We do this all the time.

How many students enrolled?

How many attended?

How many instructional hours did we provide?

How many classes did we offer?

How many people completed the program?

Those numbers aren't useless. We need to know whether programs are reaching people and whether we're delivering what we promised, but they answer a very limited question: Did we do something? They don't answer the much more important one: Did it make a difference?

Activity Is Not the Same as Impact

Imagine I tell you that a correctional education program served 500 students last year.

Is that good?

Your instinct may be to say yes.

But what if 400 of them attended only once?

What if students spent hundreds of hours in class but made almost no academic progress?

What if people earned credentials that had little value in the labor market?

What if nearly everyone completed a workforce program but few participants could find or keep employment afterward?

Suddenly, 500 doesn't tell us very much.

This is the problem with relying too heavily on activity measures. They tell us what the program did. They don't necessarily tell us what the learner gained, and those aren't the same thing.

We Measure What's Easy

There's a practical reason programs rely on these numbers. They're easy to collect.

Enrollment is easy to count.

Attendance is easy to count.

Completion is relatively easy to count.

Learning, confidence, persistence, behavior change, skill development, employment stability, and successful reentry?

Those are harder.

They require follow-up, good assessments, data systems, and staff time. Sometimes partnerships between agencies that don't particularly enjoy sharing information with each other. And occasionally the willingness to discover that the program we love isn't working quite as well as we thought.

Counting seats is much safer.

Completion Matters. But Completion of What?

I'm a huge believer in credentials. GEDs. High school diplomas. Industry certifications. College credits and degrees. They’re important, of course. But even here, we need to ask better questions.

Did the student actually develop the skills represented by the credential?

Does the credential have value outside the correctional facility?

Does it connect to the learner's next educational step or career pathway?

Can the person build on it?

If it's workforce training, are there employers willing to hire someone with that credential, including someone with a criminal record?

A certificate hanging on a wall can represent a tremendous accomplishment. It can also represent the end of a program that never asked what came next. We shouldn't confuse the two.

Start With the Change You Want to See

When I help design programs, I like to work backward. Before deciding what we're going to teach, I want to know: What should be different for this person when we're finished?

If it's a digital literacy program, maybe success means a learner can independently navigate common workplace technology, communicate professionally, protect personal information, and approach unfamiliar technology with enough confidence to figure it out.

If it's a workforce program, maybe success means more than creating a résumé. Can the participant identify a realistic career pathway? Demonstrate workplace behaviors? Navigate an online application? Interview effectively? Obtain employment? Keep it?

If it's a reentry program, completing twelve lessons isn't the real goal. Being better prepared to navigate reentry is.

Once we define the change we're trying to create, we can figure out how to measure it. That's very different from designing a program first and asking afterward what numbers we can report.

Look at More Than One Kind of Success

Good evaluation doesn't require reducing a person's life to a spreadsheet. In fact, I think the strongest programs look at several kinds of evidence.

We can measure learning: What knowledge or skills did the learner gain?

We can measure progress: Did academic performance, attendance, persistence, or skill proficiency improve?

We can measure application: Can the learner actually use what was taught?

We can measure next steps: Did the person move into another educational program, earn a higher-level credential, obtain employment, or continue along a career pathway?

We can measure longer-term outcomes: Did employment last? Did wages increase? Did the person continue their education? Did they successfully navigate reentry?

And we can ask learners themselves: What changed for you?

That question shouldn't replace objective data, but objective data shouldn't replace that question, either.

Some Important Changes Happen Before the Big Outcome

There's another reason we need better measures. If we only count major outcomes, we can miss meaningful progress. Consider a student who enters an education program convinced that they're stupid. They've avoided school for years. They refuse to read aloud. They quit whenever something becomes difficult. Six months later, they haven't earned their GED yet.

Did the program fail?

Maybe.

Or maybe that student now attends consistently, asks for help, reads independently, persists through difficult work, and has moved two grade levels.

Those are changes worth taking note of. They're sometimes called intermediate outcomes, but I think of them as evidence that something is moving. The GED may come later. If our only measures are enrolled and completed, we miss the entire story happening in between.

Be Careful What You Measure

Measurement also shapes behavior. If an organization is rewarded primarily for enrollment, guess what it will focus on? Enrollment.

If funding depends on completion, there may be pressure to get people across the finish line.

If the only workforce outcome that matters is job placement, a program may prioritize the fastest available job rather than the job most likely to lead to stability or advancement.

We shouldn't be surprised when people optimize programs around the numbers we've told them matter.

That's why choosing measures isn't just a technical decision. It's a values decision. Our metrics quietly tell everyone what we consider success.

I’m constantly quoting the fabulous Shirley Garcia (a principal who worked for me at the TJJD and later at WSD and even later at LSJA): What gets measured gets done.

We should choose what we measure carefully.

Evaluation Isn't About Proving You're Good

I think this is where some organizations get nervous about evaluation, because evaluation can feel like judgment. Did we succeed or fail?

I prefer a different question: What are we learning?

Maybe participants are completing the program but struggling three months later. That's useful information.

Maybe one part of the curriculum consistently produces stronger outcomes. Useful.

Maybe students who receive mentoring persist longer than students who don't. Very useful.

Maybe something we were absolutely convinced would work...doesn't. Painful, but still useful.

Evaluation shouldn't exist simply to prove to funders that a program is wonderful. It should help us make the program better.

Keep Counting the Seats

I'm not suggesting we stop tracking participation. If 500 people enrolled, I want to know that. If 347 completed, tell me that too. But don't stop there.

Tell me what they learned.

Tell me what they can do now that they couldn't do before.

Tell me what happened next.

Tell me where they struggled.

Tell me whether the program worked equally well for different groups of learners.

Tell me what surprised you.

Tell me what you're changing because of what you learned.

Because ultimately, the question isn't how many people sat in our classrooms. It's whether sitting in those classrooms changed what became possible for them.

That's the number I'm interested in.

I hope that you’re counting your blessings, and maybe the number of cups of coffee you’ve had this morning. Or, you know…who cares about that really, as long as it makes you happy. :) Here’s to a great week, full of wonderful life events, moments, and love that are beyond measure. Cheers! ☕