It's that time of year..

.. in the next week or so, I expect to receive 3 email messages from U.S. News & World Reports containing the links to their annual survey of undergraduate institutions. While the data provided in these three surveys - the Main Statistical Survey, the Finance Survey, and the Financial Aid Survey - comprises the bulk of the data used by USNews to calculate their annual college rankings the largest single factor in the ranking - coming in at a whopping 25% of the final ranking - is a so-called "repuational" survey of University administrators. This survey is rather highly subjective in that it asks top administrators to provide their opinion on the reputation of their peers. This reputation survey, as well as USNews's method of statistically imputing missing data such as average admissions test scores, is now under attack by a small group of college presidents who feel that it's time for higher education institutions to "fight back" against USNews.

The text below is excerpted from a newsletter I prepared for distribution on my campus last fall to try to shed some light on the USNews Rankings and what they might actually *mean*. Given the current kerfuffle, I thought it might be interesting reading, though I am just about 100% positive that my colleagues at other higher ed institutions won't find anything particularly new in it.

The rankings are based on data collected by US News from a variety of surveys and sources in “15 areas related to academic excellence”. US News participates in the Common Data Set Initiative, which means many of their survey items are drawn from a common core of questions used by many of the larger publishers. The Common Data Set saves colleges and universities time by allowing them to answer one set of questions on a given topic, rather than slightly different variations from each publisher. This consistency also means that consumers — students and parents — will find similar information about each college even if they look at different publisher rankings. Data not provided by institutions via survey response are either computed from other sources, such as the US Department of Education, or statistically imputed.

US News assigns each indicator a weight determined by their editors. Weights reflect which of the various indicators US News feels provide the best measure of the quality of a school. Rankings are based on the total weighted score for each institution.
... Understanding how each factor is weighted explains why some institutions rank higher than others in any given year and also why US News provides a ranking for Public institutions separate from the full rankings.

Factors included in US News Rankings, including their weights and sources

Academic Reputation (25%). Determined by the Academic Reputation Survey completed by the President/Chancellor, Provost, and Dean of Admissions at peer institutions. Each individual is asked to rate peer schools' academic programs on a scale from 1 (marginal) to 5 (distinguished). The survey is administered by Synovate and had a 58% response rate for the 2006-07 rankings.

6-year Graduation Rate (16%). The Department of Education requires institutions to report on the percent of students who start at their institution and complete their degree within 150% of expected time. For most baccalaureate institutions, 4 years is the expected time to degree completion and 6 years is 150% of expected time.

Average Educational Expenditures per Student (10%). Educational expenses are reported to the Department of Education annually; US News uses the same data supplied to the Department of Education and available in the Integrated Postsecondary Educational Data Systems (IPEDS).

Incoming First-Time, First-Degree Seeking Student ACT/SAT Scores (7.5%). First-time, first-degree seeking students are what are considered “traditional” undergraduate students. This group includes only students who have enrolled in college credit courses for the first time with the intent to complete a degree-granting program. High school students participating in post-secondary options and students who transfer from a prior institution are not included in this group.

Average Faculty Salary (7%). Average faculty pay, plus benefits, during the two previous academic years, adjusted for regional differences in the cost of living (using indexes from the consulting firm Runzheimer International).

Opinions on the usefulness of the rankings vary. For many looking for a simple way to compare institutions, the rankings provide a quick and easy reference. Critics claim the rankings simplify the value of higher education too much by focusing on inputs, such as funding sources and student test scores, when the real focus ought to be on how much and how well students are learning. Of the top 5 factors by weight, the only output measure is graduation rate. While other “outcome” measures — such as 1st-to-2nd year retention rate (4%) and a value added measure that compares actual graduation rates to the graduation rates predicted based on various input measures (5%) — are included in the total ranking, they are dwarfed by the effect of academic reputation and financial resources.

Sources: U.S. News & World Report ranking methodology website
V.M. Conley & G. Fink. “Using National Data Sets in Institutional Research”, Association for Institutional Research Foundations Institute.