Education 351B  Autumn 2013
    Statistical issues in testing and assessment

David Rogosa Sequoia 224,   rag{AT}stat{DOT}stanford{DOT}edu
Course web page:
For Autumn 2012 complete materials go here

9/30 Updates to this website are    here

Registrar's information
  EDUC 351B: Statistical issues in testing and assessment
  Units: 2-3
  Room:  160-325
  Schedule: Monday 3:15-5:05pm
  Grading Basis: Letter-S/NC
  Course Description:
  The new book by Howard Wainer, "Uneducated Guesses: Using Evidence to Uncover 
  Misguided Education Policies" is the basis for this seminar. Also included will 
  be supporting research literature and data analysis activities for topics 
  such as college admissions, methods for missing data, assessment of 
  achievement gaps, and the use of value-added analysis.

Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies.   Howard Wainer (Author)
amazon page    available in paper and Kindle
Publisher: Princeton University Press (August 21, 2011), ISBN-10: 0691149283 ISBN-13: 978-0691149288
Wainer youtube video from PUPress--Howard Wainer critiques misguided education policies

Week 1  9/23.   Organization; meet and greet.
In the news, September 2013
1.   California Voters Strongly Support Student Testing According to Rossier/PACE Poll     New USC Poll: Public Approval for Testing and Evaluations
2.  College Admissions Officers Ready for Revamped SAT     Hints on the New SAT

Week 2  9/30.   SAT and College Admissions:
In the news
1. STAR (Calif) teast cheating. See also Wainer Chap 8
   Schools lose academic ratings after claims of cheating     SD County teachers caught cheating on state tests    San Jose teacher helped second-graders cheat on STAR test
2. PACE Accountability Conference

Class content (also see "Unit 1" grouping of materials)
1. Wainer Ch1 (SAT optional per NACAC)
    Wainer posting of early form (Feb 2009) of Chap 1 materials
    Source Material: " Report of the Commission on the Use of Standardized Tests in Undergraduate Admission" September 2008, National Association for College Admission Counseling.
2.Empirical Research on SAT Use and Usefullness
Questions: Does SAT help (enough) in college admissions?    Would something less noxious (e.g State tests, HS grades) do as well?    Isn't it all SES?
ETS reports: 2012 SAT Report (see displays pp. 6, 10, 22)
Mantra of UC researchers:
 High-school grades provide a fairer, more equitable and ultimately more meaningful basis for
admissions decision-making and, despite their reputation for "unreliability," remain the
best available indicator with which to hazard predictions of student success in college. 

Unit 1 Materials Core and background
Wainer Chapters 1-3 (and Chap 5, continuation of Chap 2)-- Rebuttal to Report of the Commission on the Use of Standardized Tests in Undergraduate Admission September 2008
In the words of Wainer:
  a report, commissioned by the National Association for College Admission Counseling, 
  that was critical of the current college admission exams, the SAT and the ACT. 
  The commission was chaired by William R. Fitzsimmons, the dean of admissions and 
  financial aid at Harvard.
  The report was reasonably wide-ranging and drew many conclusions while offering alternatives. 
  Although well-meaning, many of the suggestions only make sense if you say them fast.
  Among their conclusions were:
      Schools should consider making their admissions "SAT optional," that is allowing 
      their applicants to submit their SAT/ACT scores if they wish, 
      but they should not be mandatory. The commission cites the success that pioneering 
      schools with this policy have had in the past as proof of concept.
      Schools should consider eliminating the SAT/ACT altogether and substituting instead 
      achievement tests. They cite the unfair effect of coaching as the 
      motivation for this -- they weren't naive enough to suggest that because there was no 
      coaching for achievement tests now that, if they became more high stakes 
      coaching for them would not be offered. Rather, they argued that such coaching would be 
      related to schooling and hence more beneficial to education than is 
      coaching that focuses on test-taking skills.
      That the use of the PSAT with a rigid qualification cut-score for such scholarship 
      programs  as the Merit Scholarships be immediately halted.
    Wainer posting of early form (Feb 2009) of Chap 1 materials
    comparing the incomparable early version (Dec 1999) of Chap 5
Source Material: "Report of the Commission on the Use of Standardized Tests in Undergraduate Admission" September 2008, National Association for College Admission Counseling.
More NACAC    Preparation for College Admission Exams National Association for College Admission Counseling    National Association for College Admission Counseling Foundations of Standardized Admission Testing
Some commentary on the NACAC report:
Dramatic Challenge to SAT and ACT
In Defense of the SAT, Columbia U
Standardized Tests: Fair or Unfair?

Dick Atkinson on College Admissions testing:
Reflections on a Century of College Admissions Tests  Educational Researcher, Vol. 38, No. 9, pp. 665-676  
      cited Univ of Calif report Validity Of High-School Grades In Predicting Student Success Beyond The Freshman Year: High-School Record vs. Standardized Tests as Indicators of Four-Year College Outcomes
The New SAT: A Test at War with Itself   invited presidential address at the annual meeting of the American Educational Research Association held in San Diego, California on April 15, 2009

A more substantial regression exercise with SAT and GPA: SAT Scores, High Schools, and Collegiate Performance Predictions Jesse Rothstein Princeton University
Most recent comprehensive item on SAT, SES etc
Psychological Science 2012 23: 1000 originally published online 2 August 2012. Paul R. Sackett, Nathan R. Kuncel, Adam S. Beatty, Jana L. Rigdon, Winny Shen and Thomas B. Kiger. The Role of Socioeconomic Status in SAT-Grade Relationships and in College Admissions Decisions
This has cites to the Rothstein paper and to earlier Geiser et al UC Presidents Office studies NOTE: remarkably, all links at UC president are broken, they calim they reorganized their website: I have the Geiser and Studly, may just post it, try a google search: Geiser Studley "UC and SAT" for some proprietary postings.
12/7/11 for Chap 1,2. PACE report: State Standards, the SAT, and Admission to the University of California
For better or worse, these analyses are taken seriously by the University of California administration and Regents:   ADMISSIONS TESTS AND UC PRINCIPLES FOR ADMISSIONS TESTING: A Report from the Board of Admissions and Relations with Schools (BOARS)
The following seems to be the source data analysis document: Agronow, S., and Studley, R., 2007, Prediction of college GPA from new SAT test scores - a first look. Annual meeting of the California Association for Institutional Research (CAIR), Nov 16, 2007