Data Literacy Assessments

The Data Literacy Assessments Project

The Data Literacy Assessments Project at WestEd, in collaboration with Using Data Solutions, has produced four scenario-based exercises that can be used to measure data literacy. The materials can be used as assessments, as supplemental instructional materials to develop data literacy for educational professionals in professional development or college and graduate courses, and for research purposes. The scenarios are designed to depict typical instances when teachers need to use data to make decisions in the course of their practice or personal experiences. They will allow users to demonstrate a variety of data literacy skills as outlined in the construct Data Literacy for Teachers (DLFT) developed by Ellen Mandinach and Edith Gummer.

Teachers adjust their instruction (moment-to-moment, daily, etc.), based on how well students are learning curricular material. The first scenario focuses on short-term data. This is the Formative Assessment scenario. The second set of materials is built around an end-of-year assessment that can be used for multiple purposes, including curriculum planning. This is the Summative Assessment scenario. Third, we have developed a set of materials around indicators that provide evidence for which students may be at risk of failing to graduate, an increasingly problematic issue in education. This set of materials is called the Early Warning Indicator scenario. Academic achievement and student performance are not the only things about which educators care. Decisions are often made by teachers using other sources of data. We therefore have developed a scenario about teacher career decisions based on diverse educational data. It is called the Workplace Decisions scenario.

The scenarios are designed to be used in multiple ways. First and foremost, they are assessments that teacher preparation programs and professional development providers can use to determine how well the students or participants can demonstrate various data literacy skills and knowledge. The scenarios also can be used for self-assessment by practicing teachers or students in teacher preparation programs. Second, they can be used as instructional devices for educators to learn about and discuss how data can be used to inform practice. Finally, the scenarios can be used as research devices. The scenarios are intended to be used in classroom settings, professional development sessions, by individual educators, or by researchers. They can be used as a team exercise or for individual use. A pre-service educator may respond to either some or all of the problem scenarios. The materials are designed to complement one another but may be used separately.

The scenarios have accompanying materials. Data sets have been created for the first three scenarios. The Workplace Decision scenario relies on access to existing online databases. Answer Keys have been constructed for all the scenarios. There also is a List of Data Literacy Skills tapped by each scenario. The Early Warning Indicator scenario includes two articles to supplement understanding of the topic.

General information about the materials and data literacy can be found in the Introductory Materials section. These include:

Please contact Ryan Miskell (202-471-2463 or rmiskel@wested.org) or Ellen Mandinach (202-674-9300 or emandin@wested.org) for more information or assistance on accessing these free materials.

For additional information about Data Literacy for Teachers, please see:

  • Mandinach, E. B., & Gummer, E. S. (2016). Data literacy for educators: Making it count in teacher preparation and practice. New York, NY: Teachers College Press.
  • Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60, 366-376.

Instructions for the Data Literacy Scenarios

The data literacy scenarios were developed with three purposes in mind. First, they can be used as measures of data literacy. Second, they can be used for instruction in college courses or in professional development settings. Third, they can be used for research.

For each of the four scenarios, there is the actual scenario that sets the context and an answer key for each scenario’s activity. For the Formative Assessment, Summative Assessment, and Early Warning Indicators scenarios, there are accompanying data sets. For the Early Warning Indicators scenario, there are two articles that can be used for supplementary understanding of the topic. For the Workplace Decisions scenario, the scenario contains URLs to active websites and databases for the user to access. Access to these websites is needed to complete the questions. The answer keys include suggested or idealized responses to open-ended questions embedded in the scenarios. A matrix is provided to show a list of the data skills each scenario is intended to target. These skills are linked to the Data Literacy for Teachers construct developed by Mandinach and Gummer.

We have also provided additional materials to help users understand data literacy and make more effective use of the scenarios. These include:

These are to help instructors, users, or researchers understand data literacy and how to use the scenarios and the accompanying materials.

For Assessment Purposes

These are only suggested uses. The materials are meant to be used flexibly. The primary materials needed for assessing data literacy will be the scenarios and the data sets. The articles for the Early Warning Indicator scenario can supplement understanding. Access to the URLs for the Workplace Decisions scenario is necessary to complete the scenario. The Formative Assessment and Summative Assessment scenarios each have one data set. The Early Warning Indicator scenario has a data dashboard and three other data files. We have included for each scenario a List of the Data Literacy Skills that may be demonstrated by the user.

Background information about data literacy is contained in the additional materials. They can help the instructor and student/educator to understand more about data literacy and how the scenarios address the data literacy skills.

The scenario, data set(s), and reading materials (for the Early Warning Indicator) should be given to the student or educator. They can work through the documents at their own pace. Within each scenario are sets of questions to be answered. Some require an objective answer; others are open-ended and require a descriptive response. We have provided Answer Keys. For the open-ended responses, we have prepared possible answers to help users understand what kind of responses might be expected.

We have also provided a List of Skills practiced within each scenario and the List of Skills Across the Scenarios. In the ideal situation, these skills will be demonstrated in the responses. Each scenario should require slightly different subsets of skills. The Answer Key, in combination with the Objectives and the List of Skills, should help determine whether a skill has been demonstrated. The objective responses are either correct or incorrect. The open-ended items are more subjective. Note that we decided not to provide a scoring rubric to provide gradations of expertise and did not tie each potential response to each possible expected skill. The expected skills will become obvious as users examine the responses and link them to the List of Skills.

For Instructional Purposes

The scenarios can be embedded into instruction or professional development to help students and educators learn about data literacy through the demonstration of skills in different types of contexts found in education. The Objectives document outlines the expected overarching goals for each of the scenarios. Along with a discussion of the inquiry cycle and the types of uses for data and the skills required, instructors would give students a scenario and the accompanying materials and data set. Have the students work through the materials and respond to the questions. Use the List of Skills and the Definitions of the Data Literacy Skills documents to determine if the skills have been demonstrated in the responses. As noted above, we have not provided an explicit scoring rubric that links responses to each skill. The possibilities are infinite. But the skill descriptions should help the instructor or professional development provider understand whether a student has shown any evidence of a particular skill targeted in each scenario. Of course, there will be gradations of demonstration. Some students may show sophistication in their responses, whereas others may demonstrate only a bare minimum of knowledge. These differences can be used to facilitate further instruction and discussion.

For Research Purposes

The scenarios can be used for research purposes with proper attribution. The scenarios are linked to the Data Literacy for Teachers construct. Note that there has not been a formal psychometric study, but one researcher and one professional development provider independently rated each scenario for targeted skills. There was 93% agreement. Minor adjustments were subsequently made to insure complete agreement across raters.

We are happy to answer any interpretive questions. Please contact:

Ellen Mandinach, emandin@wested.org

Diana Nunnaley, diana.nunnaley@gmail.com

Ryan Miskell, rmiskel@wested.org


Scenario Objectives

For each of the scenarios, we have provided three overarching objectives that are embedded in the materials. These objectives are combined with the specific data literacy skills and knowledge to create rich opportunities for teaching, learning, and assessment.

Formative Assessment

  • Understand the need to examine individual student learning and the learning of groups of students. Examine student understanding and misconceptions tied to specific items and standards.
  • Demonstrate an understanding of the need to examine item distractors and keys to identify learning strengths and weaknesses to translate outcomes into actionable instructional steps. Demonstrate the knowledge that it is important to drill down beyond a simple right/wrong distinction to understand what students know or do not know to help surface the “why” and what to do about it.
  • Understand how the results can inform the transformation of instruction for the class, groups of students, and individual students based on the matrix of responses to each item and all the items.

Summative Assessment

  • Understand school-level performance relative to other schools.
  • Understand the role that annual assessment data have on informing instruction and other sorts of decisions, such as curriculum planning, professional development planning, and the provision for special services.
  • Understand how to examine how different students and different groups of students performed.

Early Warning Indicator

  • Understand and appreciate the value of research on the context to inform about a problem of practice in a district, school, or classroom.
  • Appreciate that there is more to data than only student performance results; non-cognitive data can be helpful in providing a more comprehensive understanding of students.
  • Recognize the importance of triangulating diverse and disparate sources of data to address a problem of practice and shed light on possible solutions.

Workplace Decisions

  • Recognize that there are many different kinds of data, not just student performance data in education.
  • Recognize that there is more than a narrow focus on just one’s own classroom, but that there is a larger context in which to situate decision making.
  • Understand the need to vet the credibility of data and data sources for accuracy, relevance, and timeliness.

Formative Assessment Scenario

The Formative Assessment Scenario will allow participants to: 1) Understand the need to examine individual student learning and the learning of groups of students. Examine student understanding and misconceptions tied to specific items and standards; 2) Demonstrate an understanding of the need to examine item distractors and keys to identify learning strengths and weaknesses to translate outcomes into actionable instructional steps. Demonstrate the knowledge that it is important to drill down beyond a simple right/wrong distinction to understand what students know or do not know to help surface the “why” and what to do about it; and 3) Understand how the results can inform the transformation of instruction for the class, groups of students, and individual students based on the matrix of responses to each item and all the items.


Summative Assessment Scenario

The Summative Assessment Scenario will allow participants to: 1) Understand school-level performance relative to other schools; 2) Understand the role that annual assessment data have on informing instruction and other sorts of decisions, such as curriculum planning, professional development planning, and the provision for special services; and 3) Understand how to examine how different students and different groups of students performed.


Early Warning Indicators Scenario

The Early Warning Indicators Scenario will allow participants to: 1) Understand and appreciate the value of research on the context to inform about a problem of practice in a district, school, or classroom; 2) Appreciate that there is more to data than only student performance results; non-cognitive data can be helpful in providing a more comprehensive understanding of students; and 3) Recognize the importance of triangulating diverse and disparate sources of data to address a problem of practice and shed light on possible solutions.


Workplace Decisions Scenario

The Workplace Decisions Scenario will allow participants to: 1) Recognize that there are many different kinds of data, not just student performance data in education; 2) Recognize that there is more than a narrow focus on just one’s own classroom, but that there is a larger context in which to situate decision making; and 3) Understand the need to vet the credibility of data and data sources for accuracy, relevance, and timeliness.


Resources

A list of all available resources is provided below.

General:

The Formative Assessment Scenario

The Summative Assessment Scenario

The Early Warning Indicators Scenario

The Workplace Decisions Scenario