TEACHERS & STUDENTS - USING DATA

Unfortunately, creating data sets that would allow researchers to calculate the aforementioned indicators of success is not particularly easy. Gathering and using both quantitative and qualitative data in reliable and valid ways present serious challenges. Nevertheless, longitudinal student–teacher data can permit estimation of sophisticated statistical models that can inform both policy and practice. Dan Goldhaber and Emily Anthony recently completed a study of the effects of National Board Certification on student achievement, using a comprehensive data set from North Carolina. Goldhaber completed an analysis of how he and his colleagues had to build data sets in the state in order to conduct the necessary VAM TQ models.

The following are the some of the key issues in using standardized test scores to measure teacher effects:

1. States’ student test scores must be vertically aligned so that researchers have an appropriate base (a prescore) by which to judge student learning gains (the differential between a pre- and postscore). This requires states to invest heavily in appropriate test development.

2. Student test scores must be scaled correctly, so that a measured test gain for students scoring near the bottom of the test scale may not be equivalent in learning terms to an equivalently sized test gain for students scoring near the top of the scale. Again, this requires states to invest heavily in appropriate test development.

3. The state must maintain a fairly standard set of student background information such as race/ethnicity, gender, learning disability, free or reduced-price lunch status, and English proficiency status. States already should collect such data or be in the process of collecting such data in order to comply with the reporting requirements of the No Child Left Behind Act.

4. The state must maintain accurate data on test exemptions (e.g., special education, English language learner status). The percentage of students exempted from testing can heavily impact any VAM analysis.

5. The state must maintain a data system that allows tracking prospective teachers from the point when they enter a teacher preparation or alternative certification program (or no teacher prep program at all) through their eventual classroom assignments. This is a difficult task because it requires the cooperation of each and every teacher preparation program in the state. Such cooperation should be a requirement for the accreditation of such programs.

6. The state must maintain fairly detailed information on its teachers, including standard teacher background information (e.g., race/ethnicity), credentials (e.g., licensure status and area, degree level), years of teaching experience, the type of preparation program they attended (if any), and their performance on various tests (e.g., licensure tests such as Praxis I or II or college entrance exams such as the ACT or SAT). Much like in Texas, states could require that these data be submitted in order for teacher education graduates to be recommended for certification.

7. The state must ensure that the data are as accurate as possible. For example, states must ensure that a teacher’s years of experience increase by 1 each year and that a teacher’s test scores fall within an appropriate range (e.g., between 400 and 1600 on the combined SAT score). While states such as Texas and North Carolina collect fairly detailed data on students and teachers, the teacher data are often not as closely scrutinized as student data, since typically no stakes are attached to the teacher data. Thus, states need either to attach some stakes to the proper reporting of teacher data or to invest in ensuring the accuracy of the data through careful review and “cleaning” processes.  

8. The state must be able accurately to match a teacher’s name to each student test record and be able to identify that the teacher actually taught the subject matter on which the student was tested. To our knowledge the only states where this individual student–teacher linkage can be done statewide are Florida, North Carolina, and Tennessee (Other states, such as Texas, are approaching having the ability to link teachers to particular schools and grades. Currently, such states can match teachers and students in particular districts, but not for all districts.) —and this only can be done with a limited number of teachers and students. Ideally, the matching should occur through a unique numeric identifier such as teacher and student Social Security numbers (SSNs). While the state would hold the SSNs necessary to match teachers and students, the state could use various algorithms to encrypt the SSNs to maintain the confidentiality of students and teachers.

9. Researchers need at least 3 years of student–teacher matches in order to estimate the type of statistical models (student fixed-effects models) that account for the likely nonrandom match between teachers and students that arises from classroom assignment. (For example, we might imagine that parents who provide a great deal of educational support in the home also would try to enroll their students in classes with experienced teachers.  We would not want to misattribute the impact of parental support to teacher experience.  Form more information on these types of models, see Goldhaber and Brewer, 1997.)

10. Researchers must have the ability to link all of the above data to other school and community characteristics, including household and economic data. (A more ambitious research agenda would require more comprehensive tracking of teachers. For example, many researchers and policymakers are interested in the value of particular types of teacher training or teacher-training institutions.  Research on these issues would require the ability to track teachers from the point when they enter a teacher preparation program all the way to the eventual classroom assignments.)  This requires states to ensure that common identifiers are in all of the various data sets. Texas, with some of the most detailed data on teachers and students in the nation, collects data in various data sets, all of which can be merged. Table 1 describes the data sets and the common elements that allow linkages of the data sets. While the Texas data sets could be created and merged more efficiently, the data sets do allow various researchers to conduct VAM analyses.

Table 1
Data Sets and Elements That Allow Linkages

Data element	Entity	Identifier 1	Identifier 2	Identifier 3
Teacher assignment data	SEA/ district	Teacher SSN	Campus number	District number
Student test score data	SEA/ district	Student SSN	Campus number	District number
Teacher certification data	SEA	Teacher SSN
Student enrollment data	District	Student SSN	Teacher SSN

Each of the universities working with and supported by Carnegie’s Teachers for a New Era initiative has been assembling evidence on the effects of its teacher education programs on student learning:

For example, Bank Street College has created Action-Oriented Inquiry teams where partners from other institutions representing the Arts and Sciences and practicing K-8 teachers are examining the effects of Bank Street College graduates on K-12 student learning. Data sources include interviews about teacher thinking, classroom observations, and student work samples. Evidence gathered and analyzed across the teams will be used to generate issues and ideas for consideration among the Bank Street programs for admissions, program renewal and postgraduate induction into the profession:

The University of Virginia has launched a series of studies, examining the effect of the university’s teacher education students on K-12 learning. One effort includes assessing classroom quality as demonstrated by teachers’ “clinical practice” (i.e., teaching processes) in K-3 classrooms and examining the relation between classroom quality and children’s engagement. Another effort includes measuring the effect of elementary preservice students' tutoring on “struggling” second- and third-grade students’ achievement over a semester.

Stanford University is also conducting a range of teacher education effects studies, assembling evidence from teaching portfolios (the PACT, a National Board-like assessment) as well as teaching practices and growth in K-12 students’ learning and then compare those learning gains with those of beginning teachers who have been differently prepared or unprepared. Stanford researchers are developing and refining instruments that draw upon student teachers’ pre- and postassessments of pupil learning from units they have taught.

Other institutions are making use of the Western Oregon University Work Sampling System, which involves student teachers in collecting systematic evidence of student understanding at the beginning and end of a unit of study and then assessing student learning gains.  The process includes the prospective teacher’s defining the teaching context (description of situation and community and effects of these factors), goals and objectives (considering school, district, state, and national), rationale (explaining the purpose and significance of the lessons), pre- and postassessments, lesson plans and modifications, data analysis, and reflective essays.

Last updated: February 22, 2006