School and district staff likely to face various roadblocks related to using data for decision making. Below is a list of common roadblocks as well as suggested solutions for overcoming them. If you are experiencing roadblocks not addressed below and/or would like more feedback on a specific roadblock below, please contact us at DDI@wested.org.
Educators are inundated
with too much data
Educators are flooded by the proliferation of data from a variety of sources (e.g., multiple assessments, student records). The data are often too much for an individual teacher to handle.
Educators should form data teams to work on the wealth of data in a collaborative manner, apportioning the data sources among them. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008.]
Being led by guiding questions and framing hypotheses helps to streamline the data by identifying the essential data sources, and eliminating the less relevant. By asking targeted questions around what insights need to be gained in order to make specific instructional changes, educators can get a sense of the importance of particular pieces of data. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008.]
Educators should triangulate among different data sources. Triangulation helps to bring together disparate data in a more manageable and understandable manner. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008.]
Roadblock: Educators do not have enough relevant data to make informed decisions (for example, in non large-scale tested subjects and grade levels)
A major focus related to data for decision making is placed on subjects and grade levels that are tested through large-scale assessments, usually at the state level. However, the data for those subjects and grade levels are often not informative enough to make instructional modifications. Furthermore, many subjects and grade levels are not tested in this manner, and obtaining other relevant and insightful assessment data is difficult.
Though common assessments add more testing, which is a not ideal, new common assessments could be developed and aligned to the curricula, providing educators with useable information.
The development of common metrics can provide a key foundation. For some subjects—such as art, music, physical education, and industrial arts—educators can think creatively about the kinds of metrics and indicators that reflect student performance. Other possible solutions include the development of student learning objectives (SLOs) and learning progressions. [For more information, see Datnow, Park, & Wohlstetter, 2007; Wayman, Cho, & Johnson, 2007.]
Roadblock: Educators do not have enough time to interpret and use data to inform their decisions
Time is an educator’s enemy. There simply is not enough of it. For educators to say that they don’t have time to use data reflects a misunderstanding of data use. As Steven Katz (Early & Katz, 2006) noted, if people think using data is an isolated event, then they are not using data properly.
Data use should become an integrated part of educators’ repertoires of practice. Data should be used constantly, formally or informally, to inform practice. Through common planning time or data team meetings, time should be set aside in the school calendar for educators to discuss data. Data use should also become integrated into a school’s improvement plan. Thus, data will become an expected part of the school’s culture. [For more information, see Halverson, Pritchett, & Watson, 2007.]
Roadblock: Educators focus too much of their attention on one group of students (“bubble kids”) at the expense of other students
“Bubble kids” are those students on the cusp of either passing or failing, on the “bubble” of either being deemed proficient or not proficient. Because of the imprecision of assessments, particularly of using only one measurement, it is risky and inequitable to focus solely on these students. Students on either side of the cusp may not be that different, in terms of their performance or understanding. Given another test, the scores may well change along with the proficiency designations.
All students should receive equitable education attention, based on their academic profile and their learning strengths and weaknesses. Data should be used to differentiate instruction, which will allow the teacher to better provide equitable attention to all students. [For more information, see Booher-Jennings, 2005.]
Roadblock: Educators find data systems, including dashboards and student information systems, difficult to use
Educators often state that they received training on how to use a data system long before they had the need to use the system, resulting in the data system becoming dated and/or educators not remembering how to use the system. Furthermore, receiving training on a data system is not tantamount to receiving training on data use.
The provision of ongoing training on data use is an essential component of rolling out a data system. Successful training will:
- Be timely and ongoing
- Be conducted with proximity to intended use
- Include readily available assistance on an as-needed basis
- Be not just about the technical aspects of the system, but also about data use more generally. [For more information, see Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 2009; Wayman, 2005, 2007.]
Roadblock: Schools and districts often lack the technological infrastructure for a data system
A data system is imperative to the development of a data-driven culture. Yet, many data systems are expensive and complex even though districts must provide data to their state departments of education.
All schools and districts should have in place some level of technology to support data use, even if it is a simple spreadsheet. Some state departments of education are building state data systems that have the capacity to serve as proxies for local systems for districts too small or not solvent enough to operate their own systems. Discussing options with state education staff or neighboring districts is a good first step. [For more information, see Wayman, Cho, & Johnston, 2007; Wayman, Stringfield, & Yakimoski, 2004.]
Roadblock: There is a lack of qualified personnel skilled in data use
Though data literacy is an emerging skill set among educators, schools of education are not yet widely preparing teachers and administrators to use data and there remains a shortage of qualified personnel skilled in data use.
Schools and districts should look across their staff to determine who might have the skills and the proclivity toward using data. Creating a data team and appointing a data coach to facilitate data use can help spread data use throughout the school.
Using a distributed leadership model helps to infuse data use in the school and minimizes the risk that data skills reside solely in one individual. A turnkey model, in which teachers train other teachers, can be used to help train teachers. Special education teachers often receive data training in their preparation programs. Schools can look to these educators for assistance. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008.]
Roadblock: There is a lack of support for or interest in building a data culture
Data use can come from all levels, from the classroom to the district and beyond. As with any program or intervention, having sound leadership that can provide a vision, support, and the needed resources is important. Without leadership, either at the building or district level, enculturating data becomes more difficult. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008.]
Schools should encourage educators to gain training through incentives such as stipends, release time, and other perks.
Teachers can develop a bottom-up strategy in which data are used, despite a potential lack of leadership in the building. This is more difficult to do without the structural and emotional support from leadership. However, a push from faculty members may help to convince leaders that data use is important and should be made a priority.
If the leadership supports and believes in data use, and faculty are skeptical:
- Leaders should provide positive examples to the staff of how data can enhance their practice
- Leaders can begin hiring staff who are more data-oriented, slowly transforming the faculty into data users
Roadblock: School and district administrators face many competing priorities in addition to data use
Data-driven decision making has not traditionally been a priority for districts to expend sparse resources on funding. Other high-stakes disciplines, topics, and skill sets typically take precedence.
Administrators should realize that data literacy is a skill set that can potentially impact all grade levels and subjects, and must make it a priority.
Data literacy, when combined with content expertise and pedagogical content knowledge, can be a powerful and informative tool for educators to help them identify students’ strengths and weaknesses. Thus, the data analytics or data literacy skills combine and interact with disciplinary knowledge and can be beneficial to all domains and content areas. [For more information, see Gummer & Mandinach, in press; Mandinach, Gummer, & Friedman, in press; Mandinach & Jackson, 2012.]
Roadblock: Educators fear or mistrust data and how those data are used
In the current environment, many teachers are particularly fearful of the growing use of data linking student performance and other new metrics with how they are evaluated. [For more information, see Henig, 2012.]
Policymakers who are developing the teacher evaluation metrics should make clear what indices are being used and how they relate to the realities of instructional practice. Policymakers should also make clear that data can help educators enhance their practice through continuous improvement processes if the data are sufficiently aligned and used formatively and constructively, rather than punitively.
Roadblock: Educators and others have difficulty finding research that provide compelling evidence on the benefits of data use
Despite the logic and appeal of data use to enhance decision making, there is little rigorous research evidence of the impact of data use on educational outcomes. [For more information, see Hamilton, Halverson, Jackson,Mandinach, Supovitz, & Wayman, 2009.]
There is a wealth of results from accumulated research on data-driven decision making implementation and from case studies, as well as other aspects of data use. Research study results indicate that data use may impact student performance for some grades and some topics [For more information, see Carlson, Borman, & Robinson, 2011; Konstantopoulos, Miller, and van der Ploeg, 2013.]
Research also exists about the components of data use such as leadership, vision, data teaming, data coaches, and technology systems. Those results can help educators and policymakers make decisions about implementation components.
In terms of showing that data use impacts what teachers do and ultimately student performance, we must keep in mind that research must mirror practice. Until there is enough implementation of practice, rigorous research evidence will be difficult to obtain. That said, some evidence is beginning to emerge that shows data use has a positive impact on achievement in different content areas and grade levels. [For more information, see Carlson, Borman, & Robinson, 2011; Konstantopoulos, Miller, and van der Ploeg, 2013.]
Our foundational research library includes some of the research. [For more information, see Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 2009; Mandinach & Jackson, 2012.]
Roadblock: Schools and districts have difficulty using relevant research
Many districts, due to lack of resources and funding, have disbanded their research offices, leaving staff without the capacity to conduct even the simplest of studies and share those results with others.
Districts should tap into knowledgeable administrators who can lead small studies.
Districts can look to local universities for possible collaborations with professors and graduate students or to the nation’s Regional Educational Laboratories. Our data use in action library contains more information on this topic. [For more information, see Copland, Knapp, & Swinerton, 2009; Halverson, Gregg, Pritchett, & Thomas, 2005; Knapp, Swinerton, Coplan, & Monpas-Huber, 2006.]
Roadblock: It is difficult to find professional development that fits a school or district’s needs
A number of professional development providers specialize in data-driven decision making. Some use “canned” models, whereas others customize to the needs of particular clients. Some providers conduct only face-to-face training, whereas others may provide virtual or hybrid models. Some models focus on training only data teams, administrators, or teachers. Some professional development materials may be available online and useable by district personnel.
With so many options available, how can schools and districts determine what professional training is best for them?
If a district is considering engaging a professional development provider, the model selected must align with the district’s education objectives. Educators should think critically about their district’s needs and discuss those needs with the provider to make sure the fit is right for the district. The collaborators and services pages provide more information on training and development assistance. [For more information, see Love, Stiles, Mundry, & DiRanna, 2008; Mandinach & Jackson, 2012; Means, Padilla, & Gallagher, 2010.]