 |
TEACHERS & STUDENTS - BARRIERS
While we propose that all states collect as much data as possible on students and teachers, there are formidable barriers to data collection. When it comes to VAM statistical data, few states have created the data systems necessary to conduct the research (such as the kind conducted by Dan Goldhaber, Bill Sanders, Eric Hanushek, and others). Specifically, only five states—Florida, North Carolina, Tennessee, Texas, and Utah—have the necessary data elements to link longitudinally students and teachers for any districts as Goldhaber did in his analysis of the impact of NBCTs on student achievement. Specific barriers are listed below:
- Student assessments that have data appropriate for value-added analyses are expensive to develop. However, all tests should be constructed using the appropriate psychometric properties.
- States such as California do not have a method to connect teachers across academic years. In some states, teacher unions or organizations are wary of the use of such data and work to block collection of such data. In other states, the state education agencies simply fail to invest the monetary resources required for creating useful data sets.
- While most states assemble data on teacher background information, credentials, and experience, they often do not have data on the kind of teacher preparation program they attended and do not link their teacher test scores accurately to the university or college they attended.
- Most states do not have robust data systems that accurately can track students over time and determine why they drop out of the database. The state must have a data system that can verify why a student disappeared from the sample. Such a system would help ensure that there is nothing systematic about those students for whom there is not complete pre- and posttest information (e.g., all the students are of the same ethnicity or district, indicating a problem with the data).
- In higher elementary grades as well as middle and high school, students typically switch classes throughout the multiyear period in which researchers are investigating “teacher effects,” and thus researchers cannot always determine who is teaching whom. Thus, states and districts must invest in student enrollment data systems that have the ability to capture this complex movement of students (and teachers). Many of the larger school districts utilize scheduling software that allows students to be matched to teachers. However, even in these districts, the scheduling software is often incompatible with the software used by researchers inside and outside the school district.
- Most states, even those with “sophisticated” TQ data systems, cannot link teacher information from different databases. While local school districts list a teacher’s name on each student’s record, not all use the same format for the teachers’ names (e.g., computers cannot recognize that “jon a smith” should be linked to “Smith, Jonathon A.”). Thus, states must utilize unique numeric identifiers for both teachers and students. As stated previously, the state can encrypt the teacher’s SSN so that a researcher can merge the data sets without actually identifying the individual teacher.
- While teachers hold multiple certificates and teach multiple classes, students are often taught by multiple teachers. Some state data systems simply cannot handle such complexity. Thus, the state must maintain a database with additional fields for cases of such complexity.
- All state data sets have some degree of missing data. The state must set an absolute standard for how much missing (teacher or student) information is “too much.” Without such a standard different analyses will vary in both validity and reliability.
- States often do not keep detailed records of how data are collected or the detailed definitions used to construct particular variables. In such instances, researchers must spend countless hours searching for the appropriate state agency employee to answer questions about the variables. Thus, state agencies must make public the details of data construction and the variable definition.