Boston Data Proves Revitalized Youth Jobs Programs Can Have Far-Reaching Impacts

This story was originally published by Data-Smart City Solutions.

In 2015, Pew released a staggering report highlighting the drastic decline in summer youth employment in the U.S., with less than one third of American youths holding a summer job in the preceding year. In 2016, the City of Boston’s youth employment rate dropped below that national average, with only 29 percent of youth employed. Entering the labor force is becoming an obstacle for many youth, and while the root causes vary, urban summer youth employment programs (SYEPs) are being revitalized to help curb this trend.

Bolstering youth employment program performance has been a strategic priority in the City of Boston, both within and beyond the current mayor’s tenure. Nationwide, participating in a SYEP has a proven positive impact on student performance after graduation, to say nothing of the profound link between summer youth programs and reduced crime. Early in his administration, Mayor Martin J. Walsh set a bold agenda to address impediments to economic mobility in Boston, including improving SYEP performance and participation; using digital services and new technology is a critical part of that objective.

On May 18, 2018, Mayor Walsh announced that the city had made improvements in workforce development and increased economic mobility within the city. Among the key indicators supporting this finding was improved student outcomes following participation in the SYEP. Mayor Walsh noted that “participants in the City's Summer Youth Employment Program had 35 percent fewer violent crime arraignments and 57 percent fewer property crime arraignments in the 17 months following program participation.”

While a number of policy and programmatic factors contributed to Boston’s gains in economic mobility, summer youth employment experience is a key component in career development for young people and Boston’s recent efforts to redesign and digitally-enhance SYEP performance is worth a closer look. Thanks to a partnership funded by philanthropy, the city-administered portion of Boston’s SYEP, known as SuccessLink, underwent a recent digital overhaul using technology and data to improve the application process and, ultimately, get more youth into summer jobs.

Recognizing a need to improve program performance and address low-tech and ad hoc management practices supporting SuccessLink, Boston’s Center for Youth and Families Division of Youth Engagement and Employment (YEE) with the support of Boston’s Department of Innovation and Technology (DoIT) partnered with the Boston Metropolitan Area Planning Council (MAPC) to design Youth Jobs Boston, a more effective, accessible youth employment system. This two year partnership running from 2015-2017 was initiated through a grant via the national Civic Tech and Data Collaborative, a joint initiative of Living Cities, Code for America, and the Urban Institute’s National Neighborhood Indicators Partnership (NNIP). MAPC is the Boston area NNIP partner organization. The Collaborative brought together local government officials, civic technologists, and community data organizations across seven communities, including Boston, to explore how to harness data and technology to benefit low-income residents. The John D. and Catherine T. MacArthur Foundation provided funding to the Boston team through the national project, and BNY Mellon provided the local matching funds. 

As part of the Civic Tech and Data Collaborative, the city’s YEE and DoIT partnered with MAPC to design a new platform supported by an algorithm to optimize the SuccessLink’s summer youth jobs application process, creating the open source platform Youth Jobs Boston. The improvements made to the city’s summer youth employment website and application management processes by the Youth Jobs Boston project tell an important story about how digital tools can support better governance and better opportunities for city residents who need them most.

SYEP in Boston

Before delving into the technical side of the Youth Jobs Boston project, it is important to give some context on the ‘state’ or history of SYEP in Boston. The city was an early leader in SYEPs; starting in the 1980’s, Boston began funding summer jobs for youths. The city’s SYEP continues today subsidizing more than 10,000 youth jobs each summer. Of that 10,000, the city administers approximately 3,000 summer jobs.

Boston’s SYEP operates across sectors with private, nonprofit, and governmental entities administering different programmatic areas. From John Hancock’s MLK Scholars, to Boston PIC for Boston Public School students, to ABCD (Action for Boston Community Development), Boston’s SYEP offers a wide variety of opportunities for students to gain meaningful work experience prior to graduation. The City of Boston’s YEE runs the SuccessLink Employment Program, administering 3,000 summer jobs with limited government resources. This is where Youth Jobs Boston comes into the mix.

The Youth Jobs Boston project was undertaken to help the city improve access and participation for students and employers (and to make more efficient use of the city’s staff time and resources). Boston’s SuccessLink application process needed a tech- and data-enabled redesign as it operated with ad hoc application management processes, relied only on telephone communications to notify students of a job offer, and collected limited data on the program; a digital revamp was a low-hanging fruit method for the city to achieve significant performance improvements without a major funding reconfiguration.

Boston’s Interim Chief Information Officer, Patricia Boyle-McKenna noted “One of the reasons this project was so successful was Mayor Walsh’s commitment to summer jobs for youth in the City of Boston. The City has played a crucial role in introducing pathways that positively impact youth employment. What better way to do that than through the summer jobs program that exists within the Department of Youth Engagement and Employment?” With the support of the bully pulpit, BCYF’s YEE, DoIT, and MAPC were able to focus on improving the performance and digital infrastructure of an existing program, creating innovation from within city hall instead of establishing a new program all together. 

Digital Redesign: Creating Youth Jobs Boston

MAPC’s digital services team was the technical lead for this project, but the groundwork for Youth Jobs Boston was initially structured during the grant development phase with critical support provided by the local volunteer-led Code for America brigade, Code for Boston. Code for Boston participated in the initial proposal along with MAPC and project staff was recruited, in part, from Code for America. For further detail on the Civic Tech and Data Collaborative’s Boston project see NNIP’s recent case study available via Medium.

It is important to note that Boston’s DoIT is well-established, with nationally recognized data and tech capacities, and while MAPC was the technical lead for Youth Jobs Boston, MAPC worked closely with DoIT and YEE, making course corrections and adjustments along the way to ease the handoff process when the project concluded.

When the Youth Jobs Boston project began in 2015, project staff started by interviewing YEE staff, youth applicants, and employers to develop a holistic assessment of the job application and in-office administrative processes. That assessment took the form of an end-to-end process map, see image below, revealing barriers to entry and bottlenecks in the youth summer jobs application cycle and management systems.

 

Key challenge areas included, but were not limited to, limitations in how the city notified students that they had been accepted by a position, general administrative process management across the various departments involved in supporting SuccessLink, challenges in student acceptances and retention, and limited data collection over the course of the annual application cycle. Youth Jobs Boston used the parameters identified during the end-to-end process analysis as a guide for how to make SuccessLink more effective and more accessible for the city, the employers, and, most importantly, the students.

Youth-Centered Design

To design an application process that is able to deliver the best job offers for students, MAPC and project staff convened several UX-style focus group sessions with youth to better understand their preferences when considering a summer job position. MAPC in coordination with the city convened groups of approximately a dozen students at a time to gain insights into their preferences, needs, and interests related to summer employment.

These UX sessions allowed project staff to incorporate youth-centered design into Youth Jobs Boston to help overcome challenges on the student-user side of the job application process. Patricia Boyle-McKenna noted that “We engaged young people early, we got their feedback about what’s important to them when they pick a job and then we did some market research in regards to what pathways look like for different positions and went from there.”

The youth-centered design sessions revealed three key priorities for how students select a job: (1) access to transit, (2) duration of travel time, and (3) what subjects, issue areas, or skills are most important to a student when considering a summer job position. This feedback provided important context for designing the Youth Jobs Boston platform.

(Click here to view the video.)

While these UX sessions were a critical feature in the design process for Youth Jobs Boston, it is not the first time that the city has used this participatory tool to incorporate the public’s perspective into policy and program design. DoIT Director of Product Management, Reilly Zlab, noted that other city projects have also put UX at the forefront in the last year or two. “In the City of Boston we really embrace user-centered design in our methodology and approach to the tools we create for residents. You can see this in long term projects like Imagine Boston 2030, but it’s also evident in projects like our new death certificates purchasing website, new iterations of our 311 non-emergency reporting tool, and our fiber connectivity project to connect emergency service networks to ensure that users have the best experience across agencies,” said Zlab.

From an academic perspective, it is no small feat to engage high school students in a policy or program design process. Zlab stated that youth-design sessions “were a great opportunity for the city to connect with high schoolers, because conducting research with people who are under 18 is normally quite challenging as they are ‘a vulnerable population’ by research standards and are normally a lot harder to test with. We were conscious of that while bringing them along in the process and excited to expose them to the world of UX.”

Building an Algorithm for the City

The ‘original’ SuccessLink job application process relied upon a lottery to assign 50 percent of the program’s job openings, and while this helped streamline the limited hours available to staff to make focused, case-by-case application reviews, the fit or suitability of the lottery’s outputs were not catered to any particular preferences or context for the student applicants. Following the initial end-to-end process analysis, project staff saw this lack of context in the lottery as an opportunity to improve the program’s performance and worked to design an algorithm. Based on the end-to-end process analysis as well as the priorities and preferences expressed by students during the UX sessions, MAPC, with the support of researchers at MIT, decided to use an algorithm to connect the student feedback to the behind the scenes application review process in the Youth Jobs Boston platform.

Youth Jobs Boston uses the Gale Shapely algorithm, a widely used deferred-acceptance algorithm that is most notably employed to assign medical students to their residency. MAPC used the Gale Shapely algorithm to build a new assignment process that takes the many moving parts of job-to-student matching and connects students with a job that they want, that they’re qualified for, and, most importantly, that is close by and accessible. MAPC’s Digital Services Manager, Alicia Rouault, argues that “By embedding youth feedback into the algorithmic design, the job matching process greatly improved the quality of the offers made to young people, and provided staff with an automated, data-driven hiring process.”

According to Rashad O. Cope, Director of BCYF Division of Youth Engagement and Employment, “The primary goal of the algorithm was matching young people to the hundreds of jobs that we’re offering. We have young people that live in various neighborhoods across the City of Boston, our jobs are scattered across the city, and young people have different interests. So the algorithm was really an opportunity for us to take all of that into consideration and find a convenient way to match young people to jobs that would truly benefit them.”

The algorithm developed for the Youth Jobs Boston platform tells an important story about how significant performance improvement can be achieved in government when familiar, tried-and-true data and computer science tools are brought into the public sector. Furthermore, it is important to note that this is not the first time that the Gale Shapley algorithm has been deployed to help students in the City of Boston.

Similar to the Youth Jobs Boston application process, Boston schools also use an algorithm to assign students based on their applications and preferred school options. Leading scholars in the fields of economics and education have written extensively on the efficacy of the various algorithms used to support public school selection processes both within and beyond Boston. Most notably, MIT’s Professor Parag A. Pathak’s research on ‘the Boston Mechanism’ has helped identify opportunities for improvement in the city’s school choice algorithm. In 2005, Pathak et al. wrote that the algorithm used by Boston in the early 2000’s, known as a priority matching mechanism, tended to be susceptible to gaming because if an applicant ranked a less-popular school as their top choice that would help increase the likelihood of getting that top choice. Pathak et al. note that “it is costly in the Boston mechanism to list a first-choice that you do not succeed in getting because, once other students are assigned their first-choice places, they cannot be displaced even by a student with higher priority.”

Using a deferred-acceptance algorithm like Gale Shapley, Pathak et al. argue, eliminates that gaming aspect and provides results maximized for all students regardless of preference. Thus “there is no student who loses a seat to a lower-priority student and receives a less-preferred assignment.” (In this instance, higher- and lower-priority means how the algorithm organizes student applicants based on a variety of criteria, it does not imply a varied degree in importance, but rather is a technical distinction internal to the sequencing of the algorithm. For example, in the SuccessLink lottery when a student does not accept an offer they are moved to the end of the lottery thus becoming ‘lower-priority.’) In 2006, the Boston School Committee voted to adopt a deferred acceptance mechanism, and since that time, Boston’s school application process has undergone a series of adjustments and, ultimately, a similar digital overhaul as with the SYEP’s SuccessLink. For instance, in 2016, Boston adopted a newly designed, online common application portal and a revamped lottery process supported by its own iteration of the Gale Shapley algorithm.

New Processes, Better Results

Beyond conducting the end-to-end process analysis and UX sessions, as well as designing a tailor-made algorithm, MAPC and project staff had to also build the Youth Jobs platform. In other words, they had to build the front-end interface, the guts behind that to support the algorithm, and other components to make Youth Jobs Boston a functioning job application portal. This was no small feat as they had to design a platform that is simultaneously user-friendly for the students, valuable for employers, and more efficient for government administrators, not to mention compatible with city systems and compliance requirements.

Before MAPC partnered with the city, there was no centralized process for tracking the various steps in the hiring process prior to when the human resources department was notified of a finalized job offer. YEE staff managed applicants and positions using ad hoc methods including a varying combination of Google Docs and Excel spreadsheets and when it was time to finalize a student hire they had to also use the city’s human resources’ talent acquisition management system. From the front-end interface and guts behind it to the API that integrates with the city’s existing software, designing and building the Youth Jobs Boston platform was where the user input and market research collided with the nuts-and-bolts of municipal digital infrastructure.

The Youth Jobs Boston website provides a centralized, one-stop-shop for both employers and students to track application statuses and visualize job and applicant location. The website provides detailed hiring process updates and has a search function that allows applicants and administrators alike to view and request information on jobs based on industry and location. On the employer side, community based organizations who are offering employment opportunities can see what type of positions youth are most interested in; and city staff can manage the job assignment lottery. Youth Jobs Boston codified all of these components creating a website that can evaluate performance, track and communicate an application’s status, and for the first time, capture data to help the city continue to improve and adapt the program.

Introducing a Digital Notification Process

Beyond refining and centralizing the application process, MAPC also looked at the job offer notification process to identify how to improve this critical and time-sensitive phase. In the old SuccessLink application process, the city called students to make job offers giving them a 72 hour response window to accept or decline before cycling them to the end of the lottery waitlist. Maintaining that limited notification-to-acceptance timeframe, MAPC piloted digital communications, e.g., email notifications, with Youth Jobs Boston to help expedite the job offer phase. The results were staggering: Email-based notifications saved the equivalent of 19 work weeks or 95 days that city hall staff would have spent making job offer phone calls.

But project staff drilled down further. While email notifications were initially a revelation for government process improvement, during the UX sessions, students noted that they treat their email as a ‘school-year resource,’ and many do not regularly check or have mobile access to their email during summer break. Timing is key during this phase of the application process, and by switching gears to not only save staff time but also to connect with students on their preferred mode of communication, the city could place more students in summer jobs. Youth Jobs Boston emphasizes ‘meeting students where they’re at,’ and where are students? Texting.

In 2016, Youth Jobs Boston was piloted with email-based job acceptance notifications and student job acceptance increased by 20 percent. Then, in 2017, Youth Jobs Boston supplemented email communication with text-based notifications: the job and acceptance rate spiked by 56 percent.

The use of SMS notifications in Youth Jobs Boston offers a similar lesson as the algorithm—text notifications are certainly not new, but by utilizing a familiar tech-tool that has been widely adopted by other services and sectors, government performance can achieve drastic improvements. The success of Youth Jobs Boston text notifications has helped the city consider what other programs could benefit from this or other future tech innovations. Boyle-McKenna noted that the success of SMS in Youth Jobs Boston has helped the city consider whether to implement this technology in other projects, too.

Scaling What Works with Data to Support

During her presentation at BARI’s Spring Conference 2018, MAPC’s Rouault noted that the Youth Jobs Boston platform is documenting processes in a way that the SuccessLink program had never documented, making future insights and program improvements more likely than ever before. For example, by using IBM Watson Sentiment Generator, Youth Jobs Boston was able to determine that the qualitative experience increased delight and understanding among students using the platform.

The possibility of further innovation in the SuccessLink project remains open. For example, in 2017, Boston’s Department of Transportation made student transit passes available over the summer. While preliminary uses of the program were lower than expected, given the high priority of job location and distance from home expressed by students during the Youth Jobs Boston UX sessions, cross-referencing data on transit pass use with SuccessLink data could provide valuable insights into how city subsidized opportunities enable students from all neighborhoods and demographics to make the most of their summer break.

In just two years, Youth Jobs Boston achieved significant, measurable improvements to the performance of the SuccessLink program with a jump in job acceptance rates, streamlined staff time, and strong positive feedback from employers and students alike. Youth employment program performance is typically measured with massive, multi-year evaluation studies (and Boston’s SYEP is no exception), and it is a rare and important moment for municipal governance when concrete, replicable performance improvements are identified within such a finite timeframe.

The Civic Analytics Network at Harvard tracks some of the most innovative municipal data-driven projects from across the U.S., but the lag of program evaluations and analysis can make it challenging to identify a truly effective new policy or methodology. With that said, time and again the dramatic gains achieved by government projects that adopt basic digital infrastructure to optimize administrative performance break that mold. Youth Jobs Boston is just that, an example of data-smart innovation.

Handoff to the City

The Youth Jobs Boston project concluded in 2017 and the platform has since been taken on by the city. At handoff, the city’s digital service team had to make some tweaks and adjustments to integrate Youth Jobs Boston with the city’s systems. For example, the city and MAPC use different email communications systems, but because these two systems were so similar, integrating the Youth Jobs Boston structure into the city-run SuccessLink system was relatively easy. For the text notifications, MAPC used a different SMS notification account, but, thankfully, this account was on the same system that the city uses so it was a quick fix to switch the accounts.

It is important to note that SuccessLink still operates across two platforms—the newly designed Youth Jobs Boston site as well as the human resources talent management system, which maintains various compliance requirements. Cope noted, “We’re working on finding ways to do a better job to align both of these systems so that our partners have one entry point to track progress. For the most part, it’s been a monumental shift in how this employment program has existed in the past, and we’re very excited to continue the conversation with the partners to make sure that we are continuing to enhance this interface to work better for everyone.”

By working with the city throughout the development of this project, MAPC and the city were able to preemptively adjust and course-correct to make the handoff to the city easier. The city-run 2018 SuccessLink cycle is underway thanks to the support from the new platform. Reflecting upon the Youth Jobs project, Zlab stated, “What was really valuable in this situation is that the partnership between MAPC and the City was collaborative from the beginning. That’s usually where we have experienced challenges, in the past—when partners are not brought on from the beginning. This partnership was so successful because we have been continuously negotiating with each other to determine which tools to use and engaging one another in a productive manner that was focused on the user experience the whole way through.”

Prior to the 2018 SuccessLink application cycle, the city convened focus groups of employers to get their feedback on using the Youth Jobs Boston website. Employers indicated that it was an easily navigable interface that they had the ability to locate students interested in their opportunities, and that they were better able to select students. Student feedback indicated that while they received offers from jobs that best matched their expressed interests, those jobs were, at times, too far from home.

In preparation for the 2018 cycle, the city adjusted the Youth Jobs Boston algorithm to re-balance how interests and location were weighted in determining student placements. Boston’s Analytics Project Manager, Elizabeth Kazakoff described this process, stating that “When we were in the process of doing the handoff for this project, we asked a lot of questions about the algorithm and how it was working out with placements. We heard some feedback from our stakeholders and from the constituencies of the stakeholders about placements that were matched to their interests, but disproportionately were prioritizing the interests over the distance. So we made some tweaks to better account for the distance of the job from the student’s home. That’s a major overhaul to the algorithm for this year.”

The outlook for summer 2018 application cycle is positive, according to DoIT Product Manager Rachel Braun. “Even though conversion to actual job acceptance rate cannot be calculated yet, mid-way through the application cycle, early figures are encouraging. The City sent 5,777 job offer emails with a 1.7 percent bounce rate, meaning that 91.5 percent of job offer emails were sent successfully. Of those successfully sent, 69.5 percent were opened. 87 percent of opened offers were on mobile and 97 percent were Gmail accounts. This is a great start for gauging the 2018 cycle’s performance,” said Braun.

What this Means for Your City

Youth Jobs Boston provided better access to summer jobs using data, technology, and youth-centered design, and the future of the SuccessLink program will be supported by that foundation. “The mayor is committed to summer jobs and how we can continue to provide great experiences for young people so we are confident that programs like this will continue, it’s just a matter of making the tools continue to adapt to students’ needs,” said Boyle-McKenna.

The Civic Tech and Data Collaborative brought digital innovations to cities across the country, and replicability was a critical objective for the Collaborative’s various projects. Youth Jobs Boston is open-source and available via GitHub for other cities to replicate this new model.

The stakes are high for youth summer employment. According to Brookings, “Summer jobs programs offer youth a paycheck, employment experiences, and other organized activities in the service of multiple goals: increasing participants’ income, developing young people’s skills and networks to improve their labor market prospects, and offering constructive activities to promote positive behavior.” Recent research into the long-term impact of SYEPs has also shown that these programs help reduce inequality across demographics--meaning that summer youth employment programs support economic mobility.