February 24, 2020
Dr. Ellen Mandinach (WestEd) and Dr. Kim Schildkamp (University of Twente) have co-edited a special issue of Studies in Educational Evaluation. The article “Misconceptions about Data-based Decision Making in Education: An Exploration of the Literature” examines the landscape for data-driven decision making and stimulates discussion in the field.
Additional articles will be released as they are compiled.
September 18, 2019
A new article to be published in Teachers College Record explores the types of information parents seek as they make decisions about their child’s education and the types of data displays that help make information accessible and understandable.
The authors (Dr. Ellen Mandinach, Dr. Ryan Miskell, and Dr. Edith Gummer) conducted focus groups with parents across the state of Missouri. Focus group participants shared how they make educational decisions and the data they like to have access to when making those decisions.
The research found that parents seek qualitative data sources that supply descriptions of schools and districts. They want not only test scores and school grades but also information that helps them understand the schools. They want details about the teachers, leaders, and programs offered and information about safety and processes. Parents want the information presented in accessible and understandable formats that include informative graphics with easily understood details.
January 3, 2018
The WestEd Data for Decisions Initiative is pleased to announce the publication of the Data Literacy Assessments Project. These resources are available to educational professionals, professors and students in college and graduate courses, and to researchers.
The Data Literacy Assessments Project at WestEd, in collaboration with Using Data Solutions, has produced four scenario-based exercises that can be used to measure data literacy. The materials can be used as assessments, as supplemental instructional materials to develop data literacy for educational professionals in professional development or college and graduate courses, and for research purposes. The scenarios are designed to depict typical instances when teachers need to use data to make decisions in the course of their practice or personal experiences. They will allow users to demonstrate a variety of data literacy skills as outlined in the construct Data Literacy for Teachers (DLFT) developed by Ellen Mandinach and Edith Gummer.
The resources are available through the DDI website or by typing the URL into your internet browser: https://datafordecisions.wested.org/data-use-in-action/data-literacy-assessments/
August 9, 2017
A new book focuses on data use according to the Data Team™ Procedure, which schools throughout the Netherlands have been implementing in a supervised manner since 2009.
August 9, 2017
Essential to improving teacher quality is ensuring that the right data are available to inform the policy and practice changes needed to continuously improve educator preparation program (EPP) quality, teacher effectiveness, and ultimately student learning. Unfortunately, that data are not uniformly available today.
This paper describes how data sharing among states, EPPs, and K-12 leaders can help ensure quality teaching and learning. The paper:
- Discusses challenges to using data to improve educator preparation
- Presents state policy recommendations
- Profiles policy in action in Massachusetts, Tennessee, and Missouri
WestEd’s Ellen Mandinach served as a member of the working group for this paper, developed collaboratively over one year with a group of national, EPP, and state-level experts.
April 15, 2017
Ellen B. Mandinach contributed a blog post, which can be found below, on data literacy in early education to the National Institute for Early Education Research (NIEER).
Data have been used in education for many years. Good teachers and administrators have been using data to inform their practice and make decisions. Why is data use important? Because it is no longer acceptable for educators to solely use anecdotes and gut feelings to make decisions. Educators need hard evidence. To that end, there has been a growing emphasis for the past 15 years to make education a more evidence-based and data-driven discipline.
Data have ranged from accountability and compliance data to data for continuous improvement at all levels of education, but one issue that has loomed large is the conflation of data literacy with assessment literacy. The two constructs have been confounded for many years (Mandinach & Gummer, 2016a; Mandinach & Kahl, 2014). When educators and the general public think about data, they typically think about test results and student performance. They fail to think about all the other sources of data that help educators to inform their practice. Until fairly recently, there has been no clear definition of data literacy and certainly no analyses of the skills, knowledge, and dispositions that are needed to use data effectively and responsibly (Data Quality Campaign, 2014; Gummer & Mandinach, 2015; Mandinach & Gummer 2016a, 2016b). The work of Mandinach and Gummer, based on several years of research, has attempted to lay out what it means for educators to be data literate.
A foundation of data literacy is the consideration and use of diverse sources of data, not just the limitation to only student performance data. For educators to have a comprehensive understanding of their students, they must look to behavioral, attitudinal, motivational, medical, attendance, home context, and other kinds of data. Even though measures of teacher readiness such as the edTPA (SCALE, 2013) contain an assessment rubric, it makes clear that teacher candidates must be able use contextual information, “assets,” to inform their understanding of student performance.
Diverse sources of data are particularly important in early childhood education where teachers often must look beyond student performance results to understand a child. As Dwyer (2015) notes, some of the data most relevant in early childhood settings other than assessments (formative, summative, and diagnostic) include screening results, informal check-ins, child characteristics and experiences, attendance, health information, family language/education experiences, family conditions for learning, classroom observations, participation, walkthroughs, and staff experience and education. In workshops conducted across Pennsylvania for early childhood educators, Dwyer, Mandinach, Nunnaley, and Saylor (2015) noted several purposes for data use in early learning:
- Improve child outcomes
- Improve teachers’ skills
- Identify gaps in achievement
- Realign resources
- Facilitate parental engagement
- Improve program quality
- Increase access to high quality programs
- Change adult behavior
Dwyer and colleagues (2015) reflected on why data use is important in early learning settings, recognizing that evidence is important. They noted that there needs to be realistic expectations for how data use can inform and improve daily practice. Through effective data use, educators can:
- Reflect on practice
- Check assumptions
- Get others’ views
- Commit to new actions
- Attend to the effects of changes in practice
- Make practice public
But how does this happen? More than eight years ago, the Institute for Education Sciences (IES) commissioned a comprehensive review of the literature that existed at the time, recognizing that data-driven decision making was only then emerging as a hot topic in educational research (Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 2009). Some 3,000 research and implementation studies were identified with only a handful meeting the strict criteria for rigorous research laid out by the What Works Clearinghouse. Five recommendations were noted. For there to be effective data use at any level of education, schools and districts must:
- Make data part of an ongoing cycle of instructional improvement
- Teach students to examine their own data and set learning goals
- Establish a clear vision for schoolwide data use
- Provide supports that foster a data-driven culture within the school
- Develop and maintain a districtwide data system (p. iii)
These five recommendations have stood the test of time. The growing research in data use further confirms the recommendations. Much of the work firmly espouses the need for the introduction of data teaming, leadership, and the creation of data cultures within schools and districts. Data systems have morphed from data warehouses to dashboards and apps that provide real-time data for instructional decision making.
Yet despite having much of the infrastructure in place, particularly the billions of dollars spent on technology at the federal, state, and local levels, attention to the human infrastructure remains problematic. Fulfilling all of the five recommendations from the IES practice guide is an important step forward. However, if educators do not know how to use data both effectively and responsibly, the investment in attaining the recommendations will go for naught. Even though the field recognizes the importance of data use, the delivery of consistent and comprehensive professional development is often lacking and falls below other competing priorities.
As Means, Padilla, and Gallagher (2010) noted, professional development for data use must be ongoing, not sporadic. As Mandinach and Gummer (2016a) note, waiting until educators are in practice to acquire data literacy skills is too late. They must begin to acquire such skills at the earliest stages of their professional careers, that is, during pre-service preparation. Because of this growing need, WestEd and its collaborator, Using Data Solutions, is working toward the development of curriculum materials that can be used in teacher preparation programs to teach data literacy. The objective is for teacher preparation programs across the country to begin to integrate the construct, data literacy for teachers (Mandinach & Gummer, 2016a, 2016b) into their curricula. The ultimate objective is to create a teaching corps that knows how to use data.
January 3, 2017
An interesting book is now on the best seller’s list about decision making and pertains to all fields, not just education. Michael Lewis of Moneyball fame has written The Undoing Project. It is the story of two psychologists who studied decision making in fields such as medicine and sports. The two psychologists, Daniel Kahneman and Amos Tversky wrote ground-breaking articles on how decisions are made, the assumptions, and fallacies. For example, the hot hand phenomenon in basketball is really a myth. Kahneman was awarded the Nobel Prize for Economic Sciences for this work. A good and interesting read for those involved in decision making.
September 15, 2016
A new report to the Bill and Melinda Gates Foundation titled “Blended Learning and Data Use in Three Technology-Infused Charter Schools” examines the affordances of the technologies in three blended learning environments and their impacts for teaching and learning activities. A particular focus of the work was to examine whether the blended learning environments provided access to and more diverse data sources for teachers and students from which to make educational decisions.
Key findings include:
- Blended learning environments provide data to teachers and students that may not be readily available in more traditional classes;
- Blended learning environments provide for any time and any where virtual learning opportunities;
- Teachers were able to address the needs of particular students through various media and diverse learning experiences;
- Students were engaged through flexible and customizable learning activities; and
- The schools exhibited strong leadership, an explicit vision for the use of technology and data, the engagement of students in the teaching and learning process, the enculturation of data use through data teams and data coaches, and the provision of professional learning opportunities.
There is much that can be learned from these three schools about how the alignment and practice of research-based recommendations can create blended or personalized learning environments that have the potential to reach even the most challenged students and help them to succeed.
August 11, 2016
DDI is proud to announce the release of two articles as part of a special issue of the international journal Teaching and Teacher Education. The special issue’s theme is on how countries are improving the capacity of teachers to use data.
The first article is titled “What does it mean for teachers to be data literate: Laying out the skils, knowledge, and dispositions.” The authors, Dr. Ellen Manindach (WestEd) and Dr. Edith Gummer (Ewing Marion Kauffman Foundation), were invited to write an article on a framework for a construct called data literacy for teachers. The article lays out the framework, identifying the specific knowledge, skills, and dispositions teachers need to use data effectively and responsibly. It concludes with a call to schools of education and teacher preparation programs to begin to integrate data literacy into curricula and practical experiences.
The second article is titled “Teacher learning how to use data: A synthesis of the issues and what is known.” The authors, Dr. Ellen Manindach (WestEd) and Dr. Jo Beth Jimerson (Texas Christian University) were invited to synthesize the articles in this special issue on data use. The synthetic piece contextualizes how the articles contribute to the knowledge base of how teachers use data.
January 27, 2016
WestEd and the Michael & Susan Dell Foundation conducted four case studies of teacher preparation programs considered to be ’emerging ‘ in terms of incorporating data literacy into their programs. Dell and WestEd have just released the case studies. The four programs – Western Oregon University, Boston Teacher Residency, Relay Graduate School of Education, and Urban Teachers – help shed light on what constitutes a quality program and how each approaches data literacy.
The case studies can be downloaded together or as individual files: 1) Training Data-Literate Teachers – Insights from Pioneer Programs; 2) Boston Teacher Residency; 3) Relay Graduate School of Education; 4) Urban Teachers; and 5) Western Oregon University.
October 16, 2015
Ellen Mandinach (WestEd and the DDI) conducted a workshop at the 2015 CAEP Conference. The workshop, entitled “Using Data for Programmatic Continuous Improvement and the Preparation of Data Literacy for Educators,” engaged deans of schools of education and other administrators about the use of data. It is a resource to help administrators understand why data literacy is so important and to think through how the construct can be integrated into teacher preparation programs. The workshop also helped institutions think through how to use data for programmatic improvement.
August 19, 2015
Ellen Mandinach and Andrea Lash of WestEd have written a chapter on assessment and data entitled “Assessment Illuminating Pathways to Learning.” This chapter describes trends in assessment and how different types of data can be used to inform practice. The chapter was released in the third edition of the Handbook of Educational Psychology.
June 1, 2015
As in international schooling contexts, talk about data-driven practice has become ubiquitous in schooling dialogues in the USA, and with the pending reauthorization of the No Child Left Behind Act (the main driver of increased data use in American schools), educators in the USA should expect even greater calls for formalized data use. Yet, the field lacks readily accessible tools that allow school district leaders and evaluators to examine educator perceptions related to data-informed practice. This paper outlines the process used in the development, testing, and validation of one instrument that district leaders and evaluators may use to learn more about the ways in which data are used and perceived in their respective contexts. Potential applications as well as limitations of the instrument are outlined.
Jimerson, J. B. (2015). How are we approaching data-informed practice? Development of the Survey of Data Use and Professional Learning. Educational Assessment, Evaluation, and Accountability. DOI 10.1007/s11092-015-9222-9.
April 2, 2015
Data-driven decision making is one of the strategies most widely endorsed by policymakers and educational leaders as an effective way to improve instruction.
“Yet, when we show up to do workshops on data-driven decision making and I ask teachers how many of them have had a course on using data, I don’t see many raised hands,” says Ellen Mandinach, a senior research scientist at WestEd. “Schools of education, which never send their students into the classroom without a methods course, too often don’t ensure that their graduates have similar preparation in understanding and using data.”
Mandinach believes strongly in the potential of data-driven decision making (DDDM), which research is beginning to show has the power to transform educators’ practice and improve student achievement. As Director of WestEd’s Data for Decisions Initiative, Mandinach and her colleagues work with a variety of stakeholders to advance the use of data to improve teaching and learning. She is encouraged that policymakers at the highest levels, starting with U.S. Secretary of Education Arnie Duncan, see the importance of DDDM.
“They talk about the importance of arming educators at all levels with data,” says Mandinach, “and that to be more effective, teachers should tap into a range of information about their students, beyond just student assessments.”
Turning Information Into Action
Educators who apply DDDM to their practice can reap significant results. Consider, for example, a group of teachers in one rural school district trying to uncover the reason behind poor behavior and academic performance in a particular group of students. “They had looked at every test score and performance measure imaginable but still couldn’t figure out why the kids were having so much difficulty,” says Mandinach. It was only after looking more deeply at all the data available to them, and specifically at transportation data, that they discovered that the struggling students were in fact those who spent the most time on the school bus. “They then were able to take that raw data — time on the bus — to create learning opportunities.”
In this case, bus schedules were modified to shorten students’ commutes, and Wi-Fi was installed on buses to make travel time more productive for the students. Mandinach calls this data literacy in action — the ability to understand and use multiple forms of data (e.g., data on attendance, student demographics, behavioral referrals, school surveys, classroom artifacts, and observations, as well as student performance data) to inform targeted solutions.
Mandinach contends that teachers need support to become data literate so they have information to inform their decisions, change their practice, and boost student achievement.
“States and districts are getting pretty good at collecting data,” says Mandinach. “But until you learn ways to take the raw data — whether qualitative or quantitative — and put them into some kind of education context, you don’t really provide much useful information. It’s that information that helps you determine the best instructional steps to take in the classroom.”
Mandinach remembers working with a high school math teacher in Arizona who had a student struggling to complete his homework assignments and who often fell asleep in class. The teacher learned that the student’s father was incarcerated and that he had two after-school jobs to help support his mother and four siblings. Armed with a clearer understanding of the student’s multiple, competing responsibilities (likely the case for many other students as well), the teacher made changes to his lesson plans — including adding more small-group activities and structured classroom discussions — to keep all his students more alert and engaged during class. He also found more classroom-based ways in which the struggling student and others could learn and succeed in the course, knowing that completing homework after school would continue to be a challenge.
Need for Data Literacy in Teacher Education Programs
Mandinach led a WestEd study designed to examine the extent to which schools of education are teaching their students how to engage in data literacy. The study found that although most schools reported that they included data literacy in their curriculum, a closer examination of the syllabi revealed that the focus on data literacy was almost always cursory and usually concentrated solely on assessment data — the results of standardized tests or other assessment measures of student progress. While assessment data is critical to helping teachers understand their students’ academic performance, Mandinach says that it’s only one piece of a much larger body of knowledge that educators can, and should, tap into.
“True data literacy,” says Mandinach, “is the ability to connect a deep understanding of multiple forms of data — anything from how many students ride the school bus to students’ scores on attitudinal scales — with performance standards; curricular, content, and pedagogical knowledge; disciplinary practices; and an understanding of how children learn.”
According to the study’s findings, a greater emphasis on data literacy at schools of education could result from external pressures: new credentialing or licensure regulations, as well as demands from local school districts that new teachers possess such skills. To be sure, many school districts try to provide their teachers with DDDM skills through professional development opportunities. But the process is costly, time-consuming, and often ineffective. “There are a few really good providers out there, but even if a district finds one, the information can’t be delivered in a one-shot session. It has to be ongoing,” says Mandinach.
She also believes that because understanding data can affect virtually all instructional and classroom management decisions a teacher makes, “training on data literacy has to be introduced as early as possible in a teacher’s education — ideally at the teacher preparation stage — so it becomes an embedded skill set.”
Schools of education are beginning to realize that school districts increasingly want teachers skilled in DDDM, says Mandinach. Furthermore, she notes that the tests teacher candidates need to pass for licensing and certification, such as the PRAXIS, are developing data literacy components. Given such powerful incentives, some schools of education are starting to rethink — and beef up — their data-related coursework.
University of Delaware Among Schools of Education Leading the Way
That’s the case at the University of Delaware, where educators are working to better align the School of Education’s teacher preparation program with the edTPA teacher licensing test. According to Elizabeth Farley-Ripple, an assistant professor of education at the university, the edTPA seeks to evaluate, among other things, how well teacher candidates can use data about their students to differentiate education or assess whether or not students have met instructional objectives.
“To prepare our students for the edTPA and to meet new state regulations that make data literacy part of the agenda, we began to look for ways to teach them how to better use data related to the academic, behavioral, and social-emotional components of teaching,” says Farley-Ripple.
Such external incentives, however, tell only part of the story. Faculty members meet regularly to engage in an internal improvement process. “As we discuss how well our courses are meeting our objectives, we increasingly find ourselves talking about the importance of data literacy for making our teacher candidates more pedagogically effective and well-rounded,” she says. “Which pushes us to consider which of our courses are touching on data literacy and how.”
Currently, the School of Education at Delaware does not offer a stand-alone course in data literacy. Instead, faculty members are addressing the subject themselves within the context of the subjects they teach. For example, in a math methods course, teachers typically learn how to identify errors in students’ mathematical reasoning. Ideally, says Farley-Ripple, they would simultaneously learn how to use data related to such errors to come up with targeted lesson plans to address students’ misconceptions.
Another example: Students taking a course in classroom management might learn not only how to record data on classroom behavior in the form of running records, but also how to turn that data into practical information to inform classroom strategies. For instance, a student in Delaware’s teacher leadership program took running notes on her third graders who were frequently having a hard time staying seated and paying attention. From analyzing this classroom behavior data, she was able to identify times of day when her students were the most restless so she could intervene accordingly; she also researched kinesthetic learning and began introducing movement into her classroom. Notes Farley-Ripple, “Teaching data literacy in a classroom context helps ensure that it becomes part of what teachers do, which means they are more likely to use it day to day.”
Lessons Learned, Next Steps
Mandinach is encouraged by what is happening at teacher preparation programs like Delaware’s, and by “small pockets” of data-literate educators around the country. Supported by strong school and district leaders, and equipped with user-friendly data-related technology, they are finding ways, she says, to effectively use a range of educational data far wider than standardized test scores to transform their teaching.
However, Mandinach remains frustrated that so many schools of education offer very limited data-literacy training and support for their teacher candidates, though she acknowledges there are challenges, such as tight budgets and personnel constraints.
“We need to find a route around the impediments,” says Mandinach. “If you train teachers to use data in meaningful ways, it changes what they do in their classrooms and improves student performance. That’s a goal well worth pursuing.”
August 20, 2014
School improvement is complex work that requires multiple sources of information, including from students themselves. What students say about their student experiences can be used to understand and address school-related topics and problems and rethink policies and practices.
This resource provides educators with a purposeful and systematic way to gather and analyze student experiences to inform school improvement efforts.
March 24, 2014
Amanda Datnow and Vicki Park’s new book entitled Data-Driven Leadership is an essential guide to meeting the challenges of high-stakes accountability, building performance-based schools, and improving student outcomes.
Incorporating narrative reflections drawn from real educators and administrators, the authors refine their observations and interviews into practical conclusions that leaders can put to use immediately. This book empowers leaders to support inquiry, build trust in data-based initiatives, establish goals for evidence use, and provide educators with the skills they need to mobilize data for the good of all stakeholders.
November 5, 2013
Ellen Mandinach authored three guest posts on the Dell Foundation blog. The posts address the difference between data literacy and assessment literacy, using licensure to drive data literacy, and the capacity issues schools of education face in attempting to teach data literacy.