Expanded story: Data dashboards

Written by Mica Pollock and Jedd Cohen for the dashboard project, with initial dashboard development by local technologist Seth Woodworth and next development for piloting by David Lord of San Diego.

Click here for the Summary on this project; click here for the Overview and key findings on this project.

The Details of the Work

The expanded story behind our efforts, our communication and implementation ¡Ahas!, and our turning points!

Starting off, and developing our goals:

From start to finish, the dashboard project was a saga of not fully understanding the extent of work involved in creating tech tools from scratch. Our first technologist early in the project argued convincingly that simple, clear "dashboards" were key to including ordinary people in data discussions and, quite possible to create from scratch, so the non-technologists in the group jumped headfirst into a tech development area we knew nothing about. In fact, as we began the OneVille Project back in summer 2009, we actually proposed to create a “dashboard” displaying basic data linking multiple youth- and child-related databases in the city. That was because many policymakers and researchers have a sense these days that the more data seen by more people, the better (see our Research base page). But we quickly became unsure that “seeing everything” on young people was necessarily good – and especially, not clearly necessary at the level of the individual student and family (does a teacher actually need to know a student’s police record in order to serve him better? Who exactly should see health data on a child?). And anyway, the district was first interested in getting all of its own basic data viewable, quickly -- and that’s what we ended up working on in the end. Now that we understand more about tech tool development, it's likely that this multiple database project would have been very difficult to get close to completing with part-time, individual developers.

We also moved away from any effort at multi “sector”-database linking because SomerPromise, the Mayor's new site-based initiative to provide comprehensive youth services, was also interested in tackling the issue of linking databases across agencies, and we felt they were better positioned to pursue that goal even as we worked to do R & D supporting the effort by creating administrative and family-level views of school data alone. So, the goal became to create a simple data display that could help educators and families of youth in a single school see some basic data from district’s data warehousing software, Aspen X2, in a single view. This included creating a translated display easily understandable by an immigrant parent.

¡Aha! In addition to having the ability to quickly see and sort such basic data, diverse partners in young people’s lives need supports to communicate ABOUT basic data.

Throughout, we also became convinced that a dashboard shouldn’t just show data more clearly – it should help people communicate about it. Our rationale: Data displays in schools have traditionally been a) on paper and b) one-way. Think a report card or a quarterly report on one’s “scores”: schools or districts just “display” student scores to students and parents or show parents their child’s absences. Since OneVille’s goal is to support diverse partners in running communication about pursuing the success of young people, we wanted to make sure that parents could communicate back ABOUT data, to teachers -- and that tutors, teachers, and parents could over time communicate with one another. So, we designed the individual view to allow users to upload commentary, and future iterations of the admin and teacher views would also allow for qualitative notes.

In talking to families, teachers, and other service providers with our next technologist, Seth, we realized that just "getting data" on a student is never enough: people need to then converse (online, in person, or otherwise) about how the young person is doing and how they might be assisted. Just knowing how many days a child is absent is the first step, but then you need to have a conversation about why and what to do about it!

We ultimately found the project of building a finished data dashboard for actual ongoing use by a district's educators and families to be beyond the capacity and resources of our single-technologist staffing and two-year funding plan. The future of open source dashboard development for districts and communities may well lie in large, more experienced developer teams constructing and then troubleshooting free dashboards together for pay. (If we had focused our Ford grant fully on the dashboard's development, we perhaps could have supported such a team.) But having almost finished usable dashboard products relying on two individual, part-time young developers in sequence, we consider our work to be evidence that the goal is worth pursuing and can be attained -- with the resources described in our final section below, "Development process and future prospects." In the following sections, "The Community's need for the work," "Design process," and "Feedback on the design," we describe the tool as it was designed to work.

The community’s need for the work

Our work was shaped by a number of insights about necessary communications about "basic data" with students and their families, as individuals and across classrooms or schools.

¡Aha! One-Stop Shopping: It seems crucial to be able to see different kinds of student data at the same time, in a single display -- and to be able to sort that data to check for patterns.

Our dashboard was intended to sit “above” Aspen X2, the district’s current database, and displays its data in a more easily readable format for more people. (The dashboards are also designed to be automatically updated, currently through emails of data from the district's IT staff. For full adoption, a small amount of programming could afford further automatic upload.)

The value of a dashboard on top of X2 became clear as parents, teachers, and afterschool providers all shared a range of concerns about the accessibility of the data as stored in X2:

Once some of these users would get to X2, they found the format of the data there hard to understand. First of all, information isn't translated for non-English speakers. In addition and particularly importantly, when a teacher logged into X2, it was hard to see any student’s growth over time (i.e., test score growth). As Josh pointed out, test scores are kept in pure chronological order and since students take many tests, it was hard for anyone looking at X2 to see growth on a single test from year to year. Further, the “fields,” or “boxes,” keeping data in X2 didn’t actually have calculations like test score growth: they just showed one score, then the next. Seth first spent lots of time building “tubes” to make the dashboard automatically calculate this growth; rebuilding and then tweaking these "tubes" for final 100% accuracy was David's last task.

From an administrator’s perspective, it was also hard to compare different aspects of student data simultaneously. For example, a request to the central office was required to link a table of attendance by student name with a table of MCAS score by student name. Principals also said they had to click through numerous choices within X2 before seeing any data at all. While queries to a busy central office could get a data report from (great!) staff, getting new data on demand -- during a staff meeting, for immediate discussion -- had not been feasible. 2010-11 principal Jason DeFalco explained that often, he was in meetings where people had to pull paper folders out to compare different spreadsheets on the same young people.

As our dashboard conversations with Principals DeFalco and Vadhera showed us, part of this problem was that some important data fields were not kept in X2 at all yet (e.g., afterschool enrollment and attendance, specific in-school tutoring services students received, and some disciplinary records). We ultimately had to leave out some data folks wanted on the dashboards because it was not yet kept in X2. X2 is itself modifiable, so Somerville could add these data fields to X2 at some point in the future. But in addition, while some in-school afterschool providers have direct access to X2, many running afterschool programs elsewhere in the community didn't, meaning they had trouble knowing basic information like whether students who were coming to afterschool were going to school. When we began the project, even some in-school afterschool providers typically kept their information in separate computer databases or on paper.

Again and again, people voiced the need to see all the data in one place. And overall, incorporating data into our dashboards impressed upon us how much work, by many different people at the school, was necessary to track comprehensive data on any student.

Finally, perhaps most interesting, the existing X2 system was set up only to house student data -- not to help people talk about it. In talking about the individual-view dashboard prototype with parents, the major feature everyone emphasized was the ability to immediately comment ON data, rather than simply “look at it.” While the District's new report card (which came out on paper, printed from an online database) importantly had sections for comments by teachers, those comments -- on particular skills listed on the report card -- only could be chosen from a drop-down list. Teachers could add longer summary comments on student progress only once per quarter, in the days when report cards “open” for updating and before they “close.” X2 also cuts off teacher comments at a certain length. So, the individual-level dashboard we designed has comment boxes next to each issue on the dashboard, for users to say whatever they want.

Design Process

From the very beginning, in the 2009-2010 school year, we collaborated with a range of Healey community members to design the dashboard’s user interface. As it turned out, Greg Nadeau, Somerville resident and Healey dad, had already made a color-coded Excel spreadsheet for the Healey Principal the year before we began work on the dashboard. It was an early Excel version of what we came to call the Admin View. Here’s the earliest version of our Admin View:

Before we worked on making the dashboard version of this view “live” by connecting it initially to X2, the school’s main need at the time was entering updated data into the Excel spreadsheet -- by hand. We knew that we would eventually develop a dashboard to automatically transfer data from X2, but the school needed to integrate the data sets immediately. So Susan Klimczak, from the South End Technology Center at Tent City, who would later become the lead organizer for the eportfolio project, did some valiant handiwork at the principal's request, assisted by colleague Al Willis, the original proponent of the OneVille dashboard project (who also ended up later helping to realize the eportfolio project). They cleaned up the current year’s spreadsheet and added new data typically not kept in X2 that the principal also wanted to see, like disciplinary referrals and afterschool enrollment. We also had some regular meetings via phone conference with Greg and the principal to consider the patterns the principal wanted to see and the new "fields" for data that the principal felt needed to be created permanently in the district student information system (e.g., attendance in the afterschool programs. Afterschool programs weren't keeping this data in Somerville's core data system, X2, and still don’t.)

Turning point: The Teacher View

Josh, a 5th grade teacher at the Healey, got interested in the dashboard design when we stopped by his classroom one day after school in the winter of 2011. Looking together at his computer and printouts, we realized he was already creating spreadsheets of student data from X2 by hand. He was interested in quickly displaying and sorting basic data, to supplement his face-to-face and phone conversations with students and parents. His work showed us the need for what would become the “Teacher View,” a version of the Administrator View in which all the students come from the same homeroom.

Between this stage and the latest version of the admin and teacher views (below), Mica, Seth (our project technologist, who had stepped up to tackle dashboard development in 2010 while assisting with many other OneVille projects), and Jedd met with Principal DeFalco several times during the 2010-2011 year and new Principal Vadhera beginning during the summer of 2011 to figure out the highest priority data fields for the admin view and to brainstorm potential uses. Based on DeFalco’s feedback, and with Seth’s programming skill, we developed a dashboard to transfer data from X2 to a user-friendly view. Seeing Greg’s initial prototype, above, sparked DeFalco to suggest additional data fields: years at Healey, score growth on the MAP, ELL status, MEPA scores (English language learner assessments), IEP status, and afterschool program name (in the end, we did not add program name to the dashboard, because it was never made a real field in X2.) From this feedback, we developed the view below (names are fictional to ensure anonymity):

All data fields are visible on one screen – so there is no need to click through multiple windows to view the desired data. Viewers can sort up to three columns at a time simply by clicking at the top of each while holding down the shift key. With Vadhera, over the summer of 2011, we brainstormed additional uses for the admin staff, which are described below in the second-to-last ¡Ahas!, “Data really can launch a conversation.”

Turning Point: Making the report card online and interactive Stories of parents and afterschool providers trying to communicate with teachers about students’ report cards prompted us to push forward on an “Individual View” that included the report card instead of just absences, grades, and test scores. Mica, Seth, Josh, and Jedd had many meetings in the spring of 2011 to brainstorm the design for this view, considering parent needs for accessibility and common communication challenges between teachers and parents.

We had also been inspired by the New Visions for New Schools project, which has reported success with a view that gives parents and students a quick understanding of student performance on quantifiable measures. (http://www.hfrp.org/var/hfrp/storage/fckeditor/File/9thGradeTracker.pdf) Compared with New Visions' view, we wanted to include the the report card's qualitative assessments and also, allow and encourage parents - and afterschool providers - to comment on any child's progress, send these comments to the teacher, and begin a conversation about how to support the student at home and in school. New Visions emphasizes a verbal agreement between family and teacher, called an “Improvement Plan,” so we created space on the dashboard page to write down this agreement. In our initial prototype, we did this by adding actual “Improvement Plan” text boxes in the right-hand side of the screen:

As we continued to develop the dashboard and add new types of basic data for viewing, we wanted to keep all the info on a single page for easy access, but new developments convinced us to spread the info out across several pages. At the time we were designing the “Individual View,” the District was ceasing to assign its elementary students numerical grades, or their equivalents on the typical A, B, C scale, and moving to the standards-based K-6 report card that asked teachers to grade students as proficient on a range of grade-level skills. So, we made Somerville's new K-6 report card (handed out on paper to families) online and color-coded. We also decided that we wanted the text boxes to prompt more of an ongoing conversation, rather than a formal quarterly agreement. So, we put these text boxes next to each chunk of data, with a prompt embedded in the “save comment” click button to encourage parents to write comments or questions. With the help of local technologist Evan Burchard, whom Seth recruited, we developed a revised version. The image below includes a sample tutor comment. That’s because in the months to come, we would realize that any approved viewer of the dashboard could comment:

As we added the report card to the individual view, we realized it was substantial enough that it needed its own page (above). In order to make the individual view as accessible as possible, even to families that are not used to looking at lots of quantitative data in one place, we spread the rest of the individual view’s information across several other pages, the names of which are visible as tabs at the top of each page. The individual view is now organized like a slideshow: Clicking on different tabs allows the viewer to see and comment on different parts of each student’s profile. We combined info about the student’s attendance and MCAS and MAP scores into the Individual View, which also includes the teacher’s quarterly summary comments from the report card:

At this point, the Individual View offered many text boxes for parents and other service providers to enter comments. So, we created a “Comments” page, which captures all the comments entered for review before sending to the teacher. In thinking this through with Josh, we figured that the main (“homeroom”) teacher really had to be the “point person” for younger students in particular; so, all comments go to him/her as a starting point. On the dashboard’s final page, the parent or afterschool provider can also request that the teacher reply to their comments or make an appointment with them:

Parents or other service providers can specify any new contact info and convenient meeting times. After receiving these comments, the homeroom teacher can forward any relevant parts to the appropriate subject area teachers via email. (Rather than have parents automatically “reply to all” on comments perhaps best designed for the homeroom teacher, Josh felt that homeroom teachers would like to take the lead in responding to and informing other involved teachers or service providers about families’ comments. Testing how these conversational dynamics actually went was to be an important piece of the pilot.)

To enable easy translation for piloting, the view has hardly any words added onto the language already in the report card. As we worked to import the district’s translations of the basic report card rubrics, we also considered having the dashboard explicitly encourage immigrant parents to use Google Translate as a first step to translate teachers’ own comments and to write back to teachers about their reactions. In the completed views, we would have offered a translated version of the basic individual view in Spanish, Portuguese, and perhaps Haitian Creole.

Overall, knowing all the work that families and schools face on a daily basis, we designed these tools to spark specific kinds of interaction around particular chunks of student data. How people used the tools would be up to them – but rather than have the tools just “display” data, we wanted the individual view, in particular, to also prompt and encourage communication about data.

Feedback on the design

¡Aha! Many parents welcome an invitation into a conversation with their child’s teachers, and even if these parents are unfamiliar with technology, these parents often see technology as an opportunity for connection, rather than an obstacle.

We asked for feedback repeatedly on the dashboard views, showing it to administrators, families, and afterschool providers, including doing focused interviews with parents and students from Josh’s own class (parents participating in the pilot would continue to be co-designers). In recent interviews, several immigrant parents emphasized the way the individual view dashboard could spark parent involvement: Smiling, one said, "Parents are not just left out of the school. With this, you are bringing them in, sucking them into the school curriculum!" When asked whether the dashboard might feel like extra work, another parent articulated his/her vision of parent involvement: “Not extra – you have children, you spend time to communicate. The more time you spend, the better students do.” One English-speaking parent with three children at the school explained that the dashboard’s comment and scheduling features solved a long-standing problem for her: After being a Healey parent for 11 years, she had only ever had time to meet with each of her children’s core academic teachers during PTA nights, but never the specialty teachers, e.g., music, art, or support room teachers. She said our dashboard could enable and encourage parents like her to submit their questions, requests for meetings, and updated contact info to the student’s homeroom teacher, who could forward it to the specials teachers. Another parent was especially enthusiastic about online access: “I do everything on the computer now.” And another immigrant parent said he does “everything” on his smart phone!

¡Aha! Data really can launch a conversation.

In a recent meeting with OneVille staff, Principal Vadhera described the potential value of the integrated dashboard tools, in contrast to the old system of requesting info from many different people: “Right now, in just five minutes, I have seen a complete picture of the kid. Without even checking in with folks [other staff]. Normally, I would have to wait for them to get back to me, and bring charts and graphs to meetings. What a great way to launch conversation.”

Online access to this data also could help close an even more basic communication gap, as Vadhera noted: “Even having this [individual view] up there [online] for parents to go back to,” could help when “the report card didn’t get in the backpack, or whatever.” She could see it being a powerful tool in conversations with parents who come in to see her, as there were often crossed wires about things as basic as students’ absences.

Principal Vadhera explained that on the admin view, she was interested in “anything that shows a gap.” Similarly, sorting attendance data by grade, for example, would save hours for staff who had typically printed the daily reports and then spent hours across “a week” looking for patterns among the different pages.

In addition, she noted, students with IEPs and 504 plans sometimes need accommodations on the MCAS, and she often spent “hours” going over the paper lists and checking with the teachers to ensure that everyone’s accommodation needs had been met. Our tool could allow her to sort by IEP and 504 status, so that all these students appear together, and, as she said, “so we don’t have moments when things fall through the cracks.”

Principal Vadhera and Josh both suggested that the dashboards could enhance teamwork among educators: In staff team meetings, access to each view could allow teachers and administrators to collaboratively assess a student’s needs, design and discuss targeted interventions, and, if desired, record their plan by submitting it as subject-specific comments that would get archived in the homeroom (lead) teacher’s email. Such team conversations could involve the school’s “student support team,” a standing group of educators that Purnima described as “the central nervous system of the building,” including Purnima, the Vice Principal, Nurse, and Adjustment/Redirect Counselor. Josh explained that another advantage to the individual view was that in contrast to his whole-class spreadsheets, it could allow him to present a single student’s data in one of these team meetings without revealing all the other students’ grades unnecessarily (a breach of confidentiality). Finally, reviewers noted that another relevant team that could use the dashboard to look at data together (even remotely) was each student’s individualized group of supporters, e.g. their homeroom teacher, Special Ed or ELL specialists, and reading/math resource room staff. Josh expressed interest in piloting the dashboard this way.

¡Aha! The families whose children most need assistance are often the hardest to reach with technology, but they are also the most in need of such rapid access to information.

We planned to continue substantial parent outreach during the pilot phase to show parents how to use the dashboard. We would face the same challenges with the individual view as anyone working to enhance collaboration around students across barriers of income, racial/ethnic background, language difference and tech literacy. Not all parents have home access to computers and internet (though phones with internet access are increasingly popular – and our dashboard can be accessed through a smartphone), and some parents are not functionally literate in their home language. The same work schedules that make parent-teacher meetings hard also make it hard for some parents to coordinate their schedules with the computer labs at local libraries or in the housing projects where some families live. (Also, to look at an online data display together, educators too need internet access -- not always easy if people meet in a room without a computer, wireless, or a laptop to plug into an ethernet cable.)

Several years from now, the proliferation of smartphones and iphones will likely shrink this challenge dramatically, making it easier than ever for partners to join the conversation about student data.

Development process and future prospects

We planned to pilot our three “views” at the Healey in fall 2011 and report out what we learned. As Principal Vadhera explained about our planned incremental approach to implementation, “A lot of ideas start like THIS (gestures big with hands). And then they fail. This is a guinea pig, Josh could always share back, move forward in small increments. Teachers might just want to get on board with this!”

While we asked for our local technologist to be done with development for piloting by September 1, the tools were still in development by mid-October, 2011, after the Ford funding had ended. The prototypes of the administrative and teacher views appeared to be nearly complete pending new data from the District (which required building final "tubes" to the district's student information system), and the word on the individual view was that it was "nearly complete." These delays prompted a district administrator to reconsider the pilot, even as Josh, families, students, and supporting teachers were still ready to test the dashboard in fall 2011, as were their principal and local afterschool providers. By that time, our budget for development was spent down, and we were fortunate to find another technologist, David Lord, based in San Diego, to bring the admin and teacher dashboard views to near completion for free.

David spent 200 hours rebuilding the three dashboard views our original developer had worked on in order to clean them up, i.e., to make them easier to combine and debug/modify in the future, and develop them closer to completion. The goal was to create work and code that other developers elsewhere could also ultimately use for new projects (one of OneVille's ultimate goals with tech development). He also "cleaned up" work-arounds in the original code and designed both the admin and teacher views to be compatible with an eventual individual view.

The biggest challenge was simply the scale of the work. Listening to local community members' desires to see a variety of data types quickly and clearly in one place, we had expanded any original vision of a single, hyper-simple dashboard to three more complex views; after the original budget ran out, the project still required several months of full-time development work, rather than a few days or week, and David did the work pro bono after his actual job ended. A secondary challenge was that both the prior OneVille PI, Mica, and the current dashboard project manager, Jedd, understood education data but had no significant experience with technology project management -- i.e., understanding the scope of developing tools from scratch. The full series of technologists working on the project, in turn, didn't know details about education data -- the format of student data in the district's system, information necessary in order to finish the "tubes" to the system 100% accurately. All this too had to be learned from scratch across our team. Features that a non-technologist would consider trivial, such as creating logins for different teachers, ended up requiring substantial reworking of the code. Similarly, while we knew what subset of student data people wanted to see, we learned that the district's data set had many more kinds of information to track and parse in order to make sure that no unknown values caused errors in the dashboard display.

David estimates that 60-100 hours of work remain to finish building the "tubes" from Somerville's data system to our teacher and admin dashboard views, for 100% accuracy in the data display. Given that the teacher and admin views are now ready to integrate with an individual view, David estimates that finishing the individual view, which is not as far along, would take 220-330 hours. Adapting the tool to another district's student information system (if not X2) would involve rebuilding most of the tubes, and some elements of the display, so it would likely take closer to 480 hours.

The ultimate lesson here involves the timeline of software development, and the risks associated with community-based design research if one is literally designing a new technological tool from scratch. Developing a software application from scratch means lots of work on the developer's end before any immediate use by the community partner; it also means that the developer's work assignments can shift according to community desires. The reality is that part-time technologists creating these products to community specifications, working alone on our limited budget, couldn't quite create a financially sustainable tech solution for Somerville's data viewing. We might lose confidence in local open source development, but we've learned the hard way that the basic need is sufficiently budgeted and planned development hours; the future of such open source dashboard development may lie in large, experienced teams designing and troubleshooting such tools together. And, while building "from scratch" is not truly free, it may remain far less expensive than storebought tools, and so more attainable with district budgets: the reality is that a tool for easily seeing data "all in one place" otherwise may not exist in Somerville for some time due to the expense of "off the shelf" tools. We're optimistic that next developers can pick up right where we left off, rather than starting from scratch like we did. That's how open source development works, in fact. The code for these dashboard products is now available online and free to the next developer. See Technological How-Tos in Overview and key findings ).

Our goal has always been to launch an open source dashboard design that could contribute not only in Somerville if it proved useful, but elsewhere through iterative development. While our "product" in the end is code instead of a finished piloted product, we believe that the community design process we began will serve next design efforts for "data display."

Click here for the Summary on this project; click here for the Overview and key findings on this project.