Q&A: Heather Weiss on Evaluating Connected Learning


8.5.11 | Heather Weiss, the founder and director of the Harvard Family Research Project (HFRP), is working with the MacArthur Foundation to evaluate its work at the intersection of digital media and learning. For the past five years, the Foundation has been examining how digital media are changing the way young people learn, play, socialize and participate in civic life. 


Heather Weiss.

Weiss, who is also senior research associate and lecturer at the Harvard Graduate School of Education, has broad experience working with foundations and organizations to use evaluation to improve the lives of children and families and improve learning outcomes both in and out of school. Here she shares some of the lessons she’s learned in the past and discusses how an inclusive, informed evaluation can help move the conversation into the future.

Spotlight: You’re tasked with evaluating the effectiveness of the Digital Media and Learning Initiative. What does that evaluation entail?

Heather Weiss: The evaluation will assess the influence MacArthur’s support has had on building the digital media and learning field. Specifically, we are assessing whether it, in partnership with others, is on the path to creating influential principles, practices, infrastructure, leaders, networks and prototypes for more connected learning environments with demonstrable outcomes for youth. The evidence will come from document reviews, interviews, surveys, blogs and other interactive tools. We will also use the findings to identify key indicators to track as the strategy moves ahead into its next phase.

Spotlight: How did you come to this project?

HW: The MacArthur Foundation reached out to the Harvard Family Research Project team, and we were intrigued for several reasons. Past experience as a research network member and in other roles with the Foundation gave me deep respect for its commitment to funding and using research to inform policy and practice, and I was interested to see how this commitment was in play in shaping the DML strategy and its ambitious field-building goals.

Also, as President Obama tweets on policy initiatives and digital media become one of the fastest growing global businesses, efforts to positively shape digital media to support new ways of learning and engaging in civic life are of enormous consequence for children, youth and families, and just about every social and educational institution. There are also huge challenges to be addressed if digital media are to live up to their potential to support learning and these transformational changes. As evaluators, we recognize both the potential and the challenges and bring a commitment to gather data to surface the tensions, advance the conversation and promote discussion.

Spotlight: What kinds of markers of success do you look for in such a broad initiative?

HW: One of the first goals of the independent evaluation is to fully understand the overall strategy. Then we look at the key strands, and if, how and why they align and have evolved over time.

The DML’s first five years focused on four key questions about how youth are using and are changed by digital media and how learning environments and institutions might need to change in light of this. We will examine how this research did and did not affect the DML strategy and the field, and how it has set up the next stage of the strategy from the perspective of multiple internal and external stakeholders.

Other key retrospective evaluation questions include multiple stakeholders’ views about the value-added of the Foundation’s investments, its leadership in this emerging field, and about its efforts to create the conditions for a new DML field. We will also help to build an inclusive and informed conversation about how to move ahead.

Spotlight: How do you design an evaluation like this, when the DML program has so many moving parts?

HW: Our evaluation design involves interviews and surveys of DML grantees, field leaders across government, the digital media industry, the academy, and philanthropy, and of supporters and critics of digital media and learning. We expect that key tensions and issues for the field will surface, and we will focus on these in a series of public, interactive conversations via webinars, blogs, and reports.

One important tension, for example, is if and how digital media and learning help to improve 21st-century skills and learning outcomes, which of those skills should “count” and be assessed, and whether digital learning comes at the cost of fewer basic skills or reduced learning standards.

Spotlight: Are there similar projects you’ve evaluated in the past? What did they reveal about the program or initiative that was helpful to its designers?

HW: HFRP has evaluated a number of large, complex, and multi-year foundation change strategies, including the Packard Foundation’s efforts to go to scale with pre-K programs, the Mott Foundation’s strategy to go to scale with high-quality afterschool programs as drivers for expanded learning opportunities, and Atlantic Philanthropies’ effort to connect school and afterschool learning to close achievement gaps.

A skyscraper of research shows that expanded learning opportunities beyond the classroom, including family and community engagement and afterschool and summer learning opportunities, are key to closing achievement gaps and ensuring that all children have the necessary skills for 21st-century success.

– Heather Weiss

Our evaluation approach emphasizes compiling and using data for reflection, learning and course corrections, as well as for assessing progress toward the program’s goals. We have found that in some cases grantees are unaware of their role in the foundation’s strategy or that their perspectives are not brought to bear. The evaluation process can change this to the benefit of all involved. The evaluation provides evidence that the strategy is or is not creating the desired changes and either validates it or opens up possibilities for course corrections and additional or alternative pathways to achieve outcomes.

Spotlight: What is “complementary learning” and how does digital media support such an approach?

HW: A skyscraper of research shows that expanded learning opportunities beyond the classroom, including family and community engagement and afterschool and summer learning opportunities, are key to closing achievement gaps and ensuring that all children have the necessary skills for 21st-century success. HFRP argues that access, quality, and individualized learning will be maximized if all learning opportunities are linked or connected and complement one another. 

Digital media, when managed well, is another type of expanded learning environment with the potential to turbo-charge engagement in and access to expanded learning opportunities. All players in complementary learning systems, including children and youth themselves, will be key to managing the opportunities and challenges of digital media in supporting expanded learning.

Spotlight: What’s your favorite form of technology?

HW: My favorite form of technology at the moment is my BlackBerry because it keeps me connected in multiple ways. I admit to having been a technophobic Luddite. My BlackBerry is my personal path to the digital media world, and I use it to connect in-person to 4- to 12-year-old digital natives who delight in teaching me about all the great apps I have and can get, and about other things I could do if I just had an iPod and iPad. It is a lot of fun. As I get more comfortable with all of this, who knows what’s next.

Carousel photo by Dan Callahan.


evaluation, q&a


Leave a comment

Comments are moderated to ensure topic relevance and generally will be posted quickly.

Commenting is not available in this section entry.