The Center for Data Insights at MDRC, a nonprofit, nonpartisan research organization, recently partnered with state agencies to develop and execute a variety of data projects. We learned that barriers to success are not primarily about technical issues or analytic methods. Rather, data projects need three essential ingredients to be successful: people, perseverance and project scoping.
PEOPLE
For example, MDRC worked with the New York State Office of Temporary and Disability Assistance to explore factors associated with long-term cash assistance receipt. The agency gathered an 11-person cross-functional team that included researchers, programmers, employment experts and operational staff members who worked regularly with local offices. Team members who did not have technical expertise provided content expertise and contextual information that were instrumental for both data quality assurance and interpretation of the analysis. The collaborative process prompted the technical staff to ask questions such as “How can different local offices use this analytical information in a practical way?” as they conducted their data analysis.
PERSEVERANCE
Perseverance is essential to success in any data project. Teams using new data techniques often go through a “hype cycle” in which high expectations for exciting results from a planned data analysis are frustrated by an analytic challenge. Successful teams persevere and adjust their original plans as needed.
The Colorado Department of Human Services was exploring the use of supportive payments, which are additional cash payments that can be used for the basic needs of the most vulnerable families who participate in the Temporary Assistance for Needy Families (TANF) program. They first wanted to know how the timeliness of certain types of supportive payments were related to employment outcomes, but the way the data had been recorded and tracked did not allow them to analyze data by payment type. Once they adjusted their research question to investigate the relationship between payment receipt and employment, they found selection bias issues that led to misleading findings about supportive payments. The team then tried several different ways to reduce the bias before identifying the approach that more accurately estimated the positive contribution of supportive payments to employment outcomes.
The team was able to draw insights from their analysis because they had the determination to persevere through data quality, selection and precision challenges. They also plan to continue both supplementing their analyses and improving their agency’s data collection and documentation practices to better support Colorado families who rely on TANF services.
PROJECT SCOPING
Project scoping is a way to set boundaries on your project by defining specific goals, deliverables and timelines. Designers should make room to be agile as they determine the scope of their data projects. The idea is to start small and then use what you learn to build more complex and nuanced analyses.
For example, the Washington Student Achievement Council (WSAC), the agency that oversees higher education in the state of Washington, wanted to learn about whether the United Way of King County’s Bridge to Finish campaign, which provides services to college students who may be at risk of insecurities for food, housing or other basic needs, could help students persist and earn a degree. The project scope began with a simple task: specify demographic and service use characteristics of students that may be associated with academic persistence and determine if these characteristics are measurable with the available data. This allowed the team to focus on the questions that were answerable based on data quality and completeness: Did the program recruit and serve students from historically marginalized groups? Was the program model flexible enough to address students’ most pressing needs?
If instead the project had been scoped to begin with more complexity, like building a predictive risk model to identify students who might not persist or complete college, the project would have been stymied because of insufficient data and an incomplete analytical tool. For the Bridge to Finish campaign, the simpler approach at the outset, with a project scope that was flexible enough to change as data challenges emerged, ended up leading to findings that were much more useful and actionable.
Setting up data projects for success is not primarily about data itself. Instead, it is about people who are planning designing, and pushing through challenges together. Projects that are scoped effectively and that encourage project teams to persevere through challenges yield better results, richer findings and ultimately help government agencies fulfill their missions.
Edith Yang is a senior associate with the Center for Data Insights at MDRC, a nonprofit, nonpartisan research organization.
