Sandboxes for Learning Final Report

Sandboxes for Learning Final Report

Written by Paul Allison, Christina Cantrill, Joe Dillon, Dawn Reed and Chris Sloan

Our question at Youth Voices ( for over fourteen years now has been how do we create spaces where youth can create and enter into conversations with peers from around the country about issues that matter to them. As educators working with youth, we’ve developed careful processes where youth start with their own questions, see where these intersect with other youth and with important civic issues, and develop inquiries over time.

With the support of the DML Playlist for Learning grant, we have tried to capture these processes in our Sandboxes for Learning playlists:

Sandboxes for Learning engaged four teams of teachers to work on Playlists, XPs, and Badges over the course of the last nine months. These teachers draw from the New York City Writing Project based at Lehman College, the Red Cedar Writing Project based at Michigan State University, from the Oakland Public Schools in California as well as a distributed group of colleagues that met online including educators in Montana, Utah, Texas, and Colorado.

Here is a presentation we shared about this work in the late winter that gives more background. Below are our reflections at the conclusion of this project in May 2017. Many thanks to the DML and HASTAC communities for their support of this project.

Who were you addressing with your design objective?

Our design objective focused broadly on the Youth Voices community, a network of teacher and student participants who use Youth Voices as a place to connect and share their work.

Through the fall we worked together in local groups to design playlists for learners we knew in our own contexts. We began with our own passions, ie. we started by designing playlists that picked up on a successful activity that we found fun to do with our students and that we also thought could support their connection to the Youth Voices social network.

By December, when we had a set of draft playlists to look at and consider together, a representative group of us came together to think across our classrooms and to develop a set of personas. These personas are amalgam/representatives, they came from various backgrounds (ELL, ESL, at risk, high-needs, gifted, various SES) as well as locations (urban, rural, suburban, home-schooled, etc.). We also considered what would drive student learners to the spaces in the first place, based on choice and interest as well as for various purposes from individual curiosity to classroom learning to clubs to credit recovery. We named them Angel, Anthony, Precious, Mary, and Jordan.

The personas helped us think together about how we might engage those types of learners in the community of Youth Voices as well as through LRNG playlists. Through this process, we learned how important it was to design for these personas as they supported us seeing commonalities and differences across our learners as well as better identifying “hooks” in our playlist work. The personas surfaced a range of interests that youth could be bringing across contexts.

This work on persona development was then brought back to the four groups of teachers by the teachers who gathered in December and integrated into the rest of our work moving forward. As important as these specific personas were to our work with Sandboxes for Learning, it was equally important to add this process of persona development to other local and national NWP projects.

What are the three essential questions the field needs to answer to move learning playlist design and implementation forward?

  1. About Teaching: How does the educator’s role change when youth are engaged in interest-based, choice-rich, peer-to-peer inquiries on platforms like LRNG and Youth Voices?

    Related ?s: Moving beyond the false dualities of online learning and face-to-face learning with or without mentors, let’s assume that youth have all of this available to them. What does this blended learning experience look like? How does it shift/change within and across different in-school and out-of-school contexts? What kinds of teaching practices foster this work in these different spaces? How will educators’ changing practice and roles inform their curriculum design in the future?

  2. About Learning: How are playlists not just a replication of external expectations for youth but instead support youth agency and connected learning?

    Related ?s: How can we truly foster youth in pursuing a personal interest with the support of peers, mentors,   and caring adults? How can we involve youth in developing playlists, XP’s and badges themselves for other youth?

  3. About Assessment: How does having external assessors on badges change the game for youth and educators?

    Related ?s: The public sharing of requirements and assessments can encourage perseverance and polishing, and it legitimizes the work at times. How can value be built into an ecosystem that supports youth agency and connected learning that also sees youth levelling up to learning opportunities that further college or career planning and community connectedness? In what ways can badges support this process? How can we involve youth in the assessment process? Once they have earned a badge, can they become assessors of that badge? How can all of this not be left to the end, but become more central to the work?

Have the goals for your project changed at all throughout the design process?

From the beginning in 2003, the community of teachers who built the Youth Voices platform for youth to publish and have conversations about their passions also worked together to create guides and student-facing curriculum for youth who used the site. We have created over a hundred “missions” for youth on the site over the years, designed pathways through these online experiences, and issued digital badges for the work submitted. However, the number of teachers developing and using these missions was small. Sandboxes for Learning has changed this for us.

The Youth Voices ecosystem now includes about twenty teachers who have been actively creating curriculum for youth on the site, not just their own students. There is a growing understanding of how we can use the LRNG platform and Youth Voices together to motivate learners and to build a stronger learner community for connected learning for both youth and educators.

Sandboxes for Learning has brought more clarity to our theory of practice as well. We have just begun in a lot of ways, but our study of game theory, which was introduced to us Sheryl Grant and Lucas Blair, challenges us to understand what motivates youth in online learning experiences. This impacts what we create and how we foster the learning spaces where students experience a high degree of agency and teachers are present to facilitate connected learning. We’ve also worked on how to create curriculum across spaces and applicable for national assessment.

During the implementation process, we had to remind ourselves about the essential goal of the project, which was to grow the Youth Voices community. Our attention switched back and forth from low-weight, innovative “curricular” work to logistical, human work. We’ve learned to keep focused on those things that strengthen the conversations about passions on Youth Voices.  

What did you learn through the design process? What would you do differently if you were to start over?

Working from the theory that youth need an openly networked place online to show and talk about their work on a daily and weekly basis, Youth Voices was created. Therefore, whenever possible in our Sandboxes work, we ask youth to respond to XPs by publishing their work publicly in places that invite comments from peers and caring adults (including Youth Voices, but also via related open tools/forums such as, Now Comment, Voicethread, etc.). Then we ask them to submit their web links on LRNG to the place where they posted their online, openly-networked, peer-oriented work. Therefore a core thing we’ve been learning is how to relate playlists to a social network for youth; what works and what questions are raised.

Now that we are testing these playlists with youth, and we look now at the different kinds of artifacts that kids are designing, we are reminded how important it is to learn from the work itself. By looking at the work we can start to think through and talk together about the different “habits of heart, mind and work” (Coalition of Essential Schools) we’d like to support learners to practice through these playlists and Youth Voices. These conversations support us in thinking more deeply about design and assessment.

We have learned/have been reminded that curriculum design is complex work that requires time and space to do and that we do it best when we work together as colleagues and alongside learners. The shifts in role and approach that educators notice when we start designing and then teaching with playlists and badges (ie. online curriculum and assessment) takes time to integrate into current practices. These shifts then impact what educators and learners want in the playlists and badges. It’s a wonderfully iterative process of praxis.

We have learned that the playlist, badges and XP design process requires reviews of genres and tools, explorations of audience and purpose, considerations of national learners using playlists, persona work, evaluations of criteria based on artifacts received from learners, and considerations of the impact of having remote designers and assessors in support of local practice. We are really excited about the design of playlists that start to support cross-classroom practice and break through the more traditional space/time constraints. And we have been thinking about the use of XPs in more nimble ways and doing more of this design work directly with our students.

If we were to start over, we’d do the following:

  • Start with an expanded timeline. We found that focusing on playlists one semester and testing the next was a really fast turnaround and did not sufficiently allow for us to fully test and revise our work. More time on the front-end would have helped us better clarify the connections between Youth Voices and LRNG for all teachers, do more persona work, as well as map out stages for individual teams more clearly. More time on the back-end who have supported a more thorough process of playtesting and the development of shared assessment practices.
  • Get more granular in our designs. We tended to find, overall, that opportunities we were designing ultimately weren’t granular enough for the learning situation we were creating and and that our Level 1 was a little too challenging for teachers and students. If we were to repeat the design process again, we anticipate that we would, for example, design more granular XPs and then use XPs as activities to test playlist ideas and designs before compiling entire playlists themselves.
  • Engage Youth Ambassadors. If we were to do it again we imagine starting with a few youth “ambassadors” and have them get through the whole process of doing the work and then ask these youth to support peers. Peer design and assessment is, in general, is a direction we’d ultimately like to take some this work, and this would have supported us in thinking in that direction more clearly. As well as had us working alongside youth from the beginning.
  • Make badges count. Having now spent some time considering the ways that badges might be thought of/designed to give alternative pathways to traditional, in-school credits, we can see the value of starting there with some robust discussions about how to make this happen. In this scenario, what’s new for youth might be the path, not the reward… at least at first. Somehow, badges need to count in their context and we have been exploring different ways that this can happen in our settings.
  • Expand our tools and ways of working. Maybe unavoidable, we realize, but if we were to start over we would have liked to start with an expanded set of tools and ways of working. For example, starting with personas, rather than developing them half-way through the process, would have supported our design thinking more generally. We also developed a protocol to review playlist designs with each other early on. Upon reflection we realize we could use a similar expanded protocol to look at student work/artifacts and what we can learn from them in the context of play-testing and revision.

What are the 3 most important things about designing your system or solution that you would share with another organization just getting started?

  1. Spend time with teams to understand your aims and what you can do with design; consider systems for assessment. Keep an eye toward developing systems that are sustainable. Not every learning activity needs a playlist and a badge, but youth who apply for your badges deserve speedy replies. Plan for that.
  2. Begin with XPs that can be successfully completed in reasonable chunks of time; for example, XPs that can be done in an hour and playlists can be completed in five hours. Use the open-ended framing of playlists as an opportunity to design smaller.
  3. Start anywhere, but don’t forget to work together on badges and the criteria you want to use. The process of assessing badges is a project of its own, requiring its own set of documents, workflow, and agreements.

What is left to do? What is left unanswered? What might help you continue to succeed?

One of the key things we learned is how complex this work really is. So while we succeeded at the basics that we had planned in our proposal we do feel that we just started to scratch the surface on what is possible here.

The priorities still left to do are to more fully test the playlists and review the artifacts submitted through the playlists, then revise the lists while refining the assessment and integration of assessment in our work.

Across our playlists too, we realized that we were asking youth to use a variety of technology tools, most of which will be brand new to any learner at any time. And this is in addition to using the platforms of Youth Voices and LRNG themselves! We realize that these pieces need their own support and that will help learners wherever they enter to use these platforms and tools successfully. We also noted that as curricular designers, we could use our own set of resources about these spaces and tools while developing shared protocols to support our work and iterations along the way.

This summer, therefore, the NWP will be supporting two of our Sandboxes for Learning teachers to attend a national Resource Development Retreat and create a basic toolbox resources and set of playlists to support this work. What we identified is that while the list of tools and spaces is diverse, it is also finite. And while we realize it’ll change over time, at this moment there is a set that we often use and can identify as helpful for learners to use creatively. Youth Voices too is now part of the larger NWP ecosystem in the LRNG platform, and these resources will also be made more generally available to the NWP community.

What are the three essential questions the emerging field of connected learning playlist design needs to answer or make happen in order to move your work forward or scale it?

  1. How does connectedness improve learning? Connected Learning Playlists need ways that youth can support each other as peers and role of mentors in the mix (Youth Voices in this design taught us things about this) – this is always about management and balance too, as well as continual support of motivation
  2. How does social and participatory cultures improve learning? Connected Learning Playlists need ways that youth participation with playlists drives more youth participation with playlists. Is there a way to make the system more openly transparent and social? This seems to be a strength of YV- which still has room for improvement- that is lacking in the current technology.
  3. How does making improve learning? Connected Learning Playlists need to focus on creative opportunities for learners that support a development of rich artifacts along the way. Ideally there is an opportunity through playlists to deepen creative processes and fluencies.

What parts of the playlist platform technology are working well? What limitations are you experiencing?

The LRNG platform itself is missing core connected learning functionality around relationships and connecting. At this point the platform is individually focused without ways for peers and/or mentors to interact and/or collaborate.

Since we have also be exploring national assessment of the work our students do, the lack of ability to see the feedback youth are getting over time when asked to revise/resubmit becomes a problem. We have concerns on the youth side of things — ie. they could receive circular or counter feedback that would frustrate their efforts; on the teacher/mentor side of things, this cuts off the ability for assessors to act in true mentorship roles with youth across their revisions. We also realize there could be privacy issues here; we recognize that the youth own their own data. But we think that movement towards social learning and more connectedness with the platform could start to help/address some of these issues.

Our teams haven’t really had a chance to play with revisions of content yet as this functionality is fairly new. The desire to revise within the flow of the process was a strong desire as tweaking curriculum in real time and in responsive ways is common practice in one’s own work. Therefore the design of a platform that involves iterative versioning is a new thing for many of us. We need to test and tinker with the new capabilities in the system and understand them better overall.

Similarly, we found that our students often wanted the ability to unsubmit and resubmit work before they obtain feedback from an assessor. As anyone publishing online knows, sometimes you don’t realize something you want to/wish you had changed until you hit that “submit” button. One teacher suggested an intermediate option where editing is still possible and a student can look across their set of work before a final submission. As assessors, we’d be interested in supporting student revisions of submissions in this more dynamic way too. We often notice that some content isn’t quite ready for assessment, and therefore more learners had control of this aspect the better.

Another request that came up was for notifications that playlist creators receive (and can control) when participants interact with content they designed (some options in the user dashboard that say “notify me when someone starts the playlist” or notifications when an XP is done or a playlist is completed. Or an option to not get inundated with notifications.). This could support mentorship in some potentially more dynamic ways.

What have you done, or do you plan to do, to evaluate the efficacy of your learning playlists in your community/communities?

In the Sandboxes for Learning project overall we have been looking to understand how to design for connected learning and teaching. We have therefore been looking at playlist design, development and use within that context, and the ways that the playlists work with and supported youth in connecting to peers and a range of mentors through Youth Voices was a key point that we have been discovering together.

To evaluate the efficacy of this work we did a variety of things — from using protocols with each other to talk through our designs and design questions; interviewing students about their experiences; surveying students to get feedback on playlists they worked on, etc.. We have also have been discussing ways to bring the artifacts that youth create more central into our discussions and thinking.

Going back to our personas is another place where we’d like to evaluate what we are doing — we haven’t quite gotten there yet. We begin to evaluate based on the one-on-one learning space where the moments of mentorship from student to student or teacher to student are easier to see in these case. We can evaluate end products there and we can also see growth in revisions and through conversation. When we move to national assessments though, this becomes much more difficult and we have been thinking it through. Returning to the persona activity, as well as look at the work submitted, feel like important elements in evaluating our work and designing forward.

What are the 3 main challenges to widespread adoption or scale of these learning playlists for your organization?

  1. Shifting roles: Youth Voices and LRNG are a welcoming platforms. Youth can use them to publish immediately, anything they are thinking about or making in a class or out-of-school program. At the same time, the educators using these platforms have high aspirations for each other to make important shifts in our roles and in our approaches that give more and more time to self-directed learning and peer-supported inquiry that takes advantage of online tools for civic participation. It takes time for educators (all of us) to understand what it means when we are no longer the purveyor of curriculum and assessment, and this many of us feel is a shift we’ve experienced with connected learning and with LRNG playlists and badges. Once educators make these changes, use of the platforms and design of curriculum for them becomes a more natural process.  
  2. New practices and tools: Additionally, there are distinct practices and tools to learn in this mix, ie. learning the new platform; understanding the approach to assessment; integration with various other technologies also calling teacher attention; required curriculum; time to teach each other and teach students; conceptualizing badging and it’s potential in context, etc.. For many of the educators involved in this project, most of whom are very experienced, innovative teachers who are ready to adopt new approaches, there was still a struggle with the multiple steps and having the time/space to  integrate. Of course this is always the challenge when it comes to integrating any work and a key question in program design; ie. how do you get folks on board when the learning and time curve is tricky?
  3. Time intensity: Overall we know that we needed more time to review, understand and iterate on our work. The grant focus had us pushing pretty hard to complete tasks that we had to later admit were overly ambitious given our timeline. Our shrunken timeline probably was a positive (we got a lot made in a short period) but it also felt like we were racing the clock at every step, which became a particular challenge when it came time to implement.

What plans are in the works, or do you plan to put in the works in order to sustain your system?

A priority for us is to develop our assessment team and assessment processes. We need clarity on how the assessment team will work and time to work through and reflect on integration of the work and assessment within our current practices.

We also need a team of youth and adults to continue the building of playlists and badges. We need a team to take responsibility for identifying the curriculum behind any post on Youth Voices. If an XP exists for that post, this group of adults and youth would embed it at the bottom of the post. If an XP does not exist yet, members of this team would reach out to the teacher to create an XP, and eventually a playlist and badge on LRNG. Also there are scores of missions on that could usefully be revised into playlists.

Further testing and revision of playlists is necessary. We plan to do this for our own purposes at Youth Voices, particularly in preparing for school in the Fall, but also in order to make one of our playlists available at the national level for NWP and Cities of LRNG. Opening up our playlists more nationally beyond our Youth Voices team is another experiment that will push on our assessment and revision practices.

And, as previously noted, NWP is supporting a couple of teachers to come to a retreat to create resources to support this work in the form of playlists and toolboxes. This will support sustainability and also support the overall NWP ecosystem within LRNG.

How are you getting institutional buy-in, or adoption from your learners or other stakeholders?

From learners: We are noticing definite buy-in from learners/students when they know it is people other than their teacher are actually assessing them. They demonstrate angst when their “judges” do not get back to them and let them know if they have earned a badge. They are frustrated when they do have to revise their work and re-submit, but so many of them actually do end up revising because they want the full credit for earning the actual badge. One teacher noted that some of her most challenged learners were willing to re-submit for a badge, even if it was their third or fourth time doing so. She wrote that there was something about getting that email from LRNG that told them they had earned the badge that made them feel like their efforts were worth it.

The worth of badges themselves, however, have been a discussion among both student and teachers involved on this project. Teachers designing playlists generally were unsure of what badges might unlock and among our diversity of students, they were not always motivated by XP completion or badge completion per se. That said, conversations around badging seem to have been interesting. In addition to the example above, another teacher reported that a student respondent in a survey said when asked whether he would share the badge in a job or college application: “the badges are most likely a foreign and somewhat meaningless award to them, but I could explain the meaning behind them, which could be helpful.”

From teachers: The connection to Youth Voices was of value to the teachers on this project as was the ability to integrate work that we found exciting and supportive of our students/learners was useful and motivating. Over time we all discovered the value in national response to student work. And discussions we started about rethinking our own approaches to badging in our teaching are exciting and motivating to us.

From our schools and local sites: The connections to Youth Voices and a national team of other educators and their students is of interest to both our school communities and local writing project sites. School leaders, in a couple cases, we open to experimenting with playlist and badges in areas of high school need, such as credit recovery and grade repair. Previous work and the reputation of the National Writing Project also supported buy-in among administrators and leaders.

From our national network: NWP has organizationally been involved in the assessment of national badges for Cities of LRNG so we brought with us an interest in learning from this process from a curriculum design side. As we are also committed to supporting Connected Learning and connected teaching, we were interested in what was possible when playlists such as LRNG were linked to a related community, such as is found at Youth Voices and through our local writing project sites and partners.

What outreach strategies will you employ to communicate and support your playlists?

We have been using multiple ways to communicate about this work and will continue to do so. For example, teachers from the Red Cedar Writing Project team have been blogging about this work at (soon to be The Current @ Educator Innovator), New York City Writing Project teachers facilitated a workshop at their Teachers Teaching Teachers (TTT) conference at Lehman College in April. Paul Allison also runs a Wednesday night TTT webcast that invited many of us to talk about our work and those archives have been shared online and disseminated specifically to the NWP and Cities of LRNG communities.

See a YOUtube playlist of these TTT discussions.

To support our playlist work we will continue to work on playtesting, revision and assessment at vis a vis Youth Voices as well as piloting some related work this summer through local writing project summer camps. The team will also present this work at the NWP Annual Meeting and sharing via CLTV @ Educator Innovator.

How are you going to, or did you, incorporate the feedback from the workshop into your plans going forward?

Incorporating feedback from other grantees, as well as from each other and the youth we work with, was a key way of working throughout. For example, at the very start of this grant at DML meeting in October, the activity around personas and an introduction to Lucas Blair and his work with badges and game-based thinking turned out to be a very influential element of the persona work we were able to do together at our meeting that December. The workshop in Chicago was a nice window into a range of work and gave us a better sense of the different ways that playlist design and development was being undertaken. The questions that we brought to the meeting were quite broad and valuable time was spent thinking about the core ideas about learning, teaching and assessment alongside other grantees.

In addition and throughout this grant, a weekly Teachers Teaching Teachers webcast that Paul Allison hosts brought us together as designers to talk through our work. He also invited in several grantees into these discussions – including SciGirls, Playground City and the Music Experience Design Lab team – which allowed us to spend more time to learn from others questions and thinking.

Have you budgeted adequately to materialize the design work you imagined?

The grant amount covered what we needed but could have been more strategically used to cover work over a long period of time. Ideally a project like this would both span the school year as well as a summer.

[edit: move financials to a separate budget narrative]

Is there anything else you would like to share with us about your progress?

We are super excited about this work and want to continue. The Youth Voices community is a great way to work with the integration of playlists in more depth and in a clearly purposeful way. Our start also solidified a leadership team and some genuine challenges of practice with both Youth Voices and with LRNG playlists. We are learning a great deal and are ready for a second iteration.

Thank you for this opportunity.

The Sandboxes for Learning Team