The stakes around selecting high-quality instructional materials are as high as ever. School systems and educators are striving to accelerate pandemic recovery against a national backdrop of declining math and reading scores and a fast-approaching deadline to commit to spending ESSER III funds. Those funds represent an incredible opportunity to make a lasting investment in the growth and learning of a generation of students—by adopting high-quality instructional materials and providing teachers with the resources and support to get the most out of them.

But how can districts evaluate the potential effectiveness of a curriculum before they adopt? That’s a factor that educators and administrators are increasingly prioritizing. In a recent EdWeek Market Brief survey of K–12 education companies, 41% said multiple schools and districts had requested data on outcomes tied to their products, and 14% had been asked for results from a randomized study or experimental design. In this Q&A, EdReports Executive Director Eric Hirsch discusses ways for districts to consider evidence and efficacy when selecting materials, and why strong implementation planning is critical for teachers and students to be able to realize the benefits of high-quality curriculum.

Q: We’re seeing more attention, and in some cases state requirements, around securing evidence that a program is effective before it’s adopted. How should school systems think about this? 

Eric Hirsch: Of course, districts want to see evidence that programs work, and that they work within their context. So, assuming you’re meeting whatever state evidence requirements are in place, what quality of evidence or research do you need to feel confident beyond that? And how do you strike a balance between what’s ideal and what’s realistic and available in your timeframe?

There are many studies showing that high-quality instructional materials, when implemented well and paired with curriculum-based professional learning, help to drive improvements in student outcomes. But there are very few independent, rigorous studies of the effectiveness of any single program. “Gold standard” studies are expensive and take years to complete, and it’s a significant challenge to control for all the variables of implementing materials. Publishers often conduct or sponsor their own studies, but those aren’t independent by definition. And, even if you’re lucky enough to find more than one independent study of a program’s effectiveness, they may report contrasting findings.

None of this is to say that efficacy isn’t important or that we shouldn’t be asking these questions. But when you’re comparing specific programs against one another in a time-sensitive adoption process, it’s unlikely that you’re going to find robust efficacy research to inform that comparison in a meaningful way.

Q: How does EdReports consider evidence of a program’s effectiveness in its review process?

Eric Hirsch: Because there’s very little independent evidence of any single program being effective across different contexts, districts need to think about a range of evidence options. At EdReports, we believe that one of those options is looking for high-quality curriculum design. It’s absolutely not a guarantee of effectiveness on its own, but it’s both a promising signal and an essential precondition.

EdReports’ review process asks, first: What alignment to college and career ready standards does the publisher claim for this program? Second: What evidence in the materials supports those claims of alignment and other dimensions of quality? And third: What’s the evidence that these materials are usable and that they set up educators to implement the materials successfully? We also give every publisher the option to submit a “publisher background” document that we post alongside our review of the materials, where they can give supporting information on the program design, relevant research or efficacy studies, and professional learning supports.

To summarize: EdReports reviews materials as they’re designed, not as implemented. That’s because selecting high-quality instructional materials is just the first step. They also have to be implemented with integrity in order to benefit students, and there are many factors that impact implementation which are beyond the scope of a print or digital curriculum.

Q: So how should districts incorporate evidence and efficacy when they’re considering new materials?

Eric Hirsch: First, before you even look at any specific materials, you need to understand as much as you can about your current landscape and starting conditions. That means asking questions like: What’s your vision for instruction in this subject area, and how well-aligned are your existing materials to that vision? What’s the current marketplace for the subject and grade band for which you’re selecting? (EdReports reviews are a great resource here). And are there any independent efficacy studies of programs for this subject and grade band?

Then you can consider the specifics of where to set the bar for evidence in your selection process. What evidence do you need to see from publishers? Is it sufficient to know that a single school saw growth in learning outcomes after implementing a program, or do you need more examples than that? Do you want evidence of success with certain types of audience or demographics, for example? And how will you prioritize efficacy as a factor in your selection decision? That is, if a program were to meet all your other adoption criteria but not your pre-established bar for efficacy evidence, what would that mean?

Finally, you should think about how to monitor effectiveness while you pilot shortlisted materials and when you implement the ones you’ve selected. Effective implementation planning and ongoing, curriculum-based professional development are fundamental to creating the conditions in which materials can be used well, and can therefore help students learn and grow. So educational leaders need to invest in the necessary implementation planning, resources, and support in order to be able to learn anything about the effectiveness of the materials in practice. Assuming you’ve covered those essentials of implementation, you can then consider what benchmarks to set at different stages in order to judge whether the program has been effective in your context.

Q: How can states and districts use EdReports as a resource to help increase the adoption and use of materials that are high quality and meet students' needs?

Eric Hirsch: EdReports has published over 1,100 free, independent, grade-level reports of K–12 English language arts, math, and science programs. Each report contains a wealth of evidence to illustrate whether the materials meet expectations against our review criteria. Using these reports is one critical aspect of selection, but it has to happen in the context of a comprehensive, teacher-led adoption process. We’ve created an extensive set of resources to support that process in our free adoption guide.

We also work directly with states and school systems to support selection. For states, that can mean helping to identify and shape policies, requirements, and incentives to create the conditions for districts to select and implement materials successfully. With districts, we can develop a tailored adoption process that reflects local policy and priorities (including evidence requirements), provide professional learning, consultation, and tools to support your adoption committee, and advise on communication strategy to get all your stakeholders as engaged in the process as possible.