Q&A: The Man Behind the U.S. News College Rankings

  • Share
  • Read Later
Blend Images / Getty

School may still be out for the summer, but all eyes are on college this week: the 2010 U.S. News & World Report college rankings hit stands today, with Harvard and Princeton tying for first place among national universities and Williams ranking first among liberal-arts colleges. TIME spoke to Robert Morse, director of data research at U.S. News and a two-decade veteran of the controversial rankings, about how the list is put together and how it could be better, plus a look at this year's rising stars.

TIME: For readers who might not be familiar with it, what's the methodology behind the rankings?
Morse: [They're] based on 15 indicators, [including] a reputation survey, admissions data, faculty data, financial-resources data, alumni giving and graduation and retention rates. We're not comparing all 1,400 schools. We're dividing them up into 10 categories, like national universities and liberal arts. We assign a weight to each of the variables. The peer survey, or the academic reputation, is the highest-weighted variable — it's 25%.

What tells you how heavily to weight each factor?
Our accumulated judgment. Our rankings aren't social science in the sense that we're not doing peer-reviewed rankings; we're not submitting our conclusions and our weighting system to a group of academics and letting them decide if they are right or wrong. We do meet regularly with academic experts about the relative importance of the factors that we use. And we have been doing this for 25 years.

You mentioned the reputation survey. How do you respond to the criticism that some people charged with filling it out may not have direct experience with the schools they're rating, so they may just be going on rumors?
I think there is a small group of schools, mainly in the liberal-arts category, that have strong feelings about the reputation survey. Generally speaking, our response rate did tick up a little bit this year — it went to 48% from 46% — so there's some indication that this boycott [among schools that are refusing to fill out the reputation survey] is losing some of its potency. But U.S. News is not expecting people to have knowledge or be able to rate each school in its category. It's based on the premise that since we have a big enough respondent base, enough people have some knowledge of enough schools that we get a statistically significant number of respondents for each school. There are subjective parts of education, parts that can't be measured by just quantitative data. The peer survey tries to capture that part of it.

The rankings don't really seem to change much from year to year. Do you ever have the desire to change the methodology and shake things up?
I personally don't. I wish that we were able to measure things like outcomes in learning — that there was comparative data on what people say is missing in the rankings and missing in our education in general. What are students learning?

What keeps you guys from measuring that?
There just isn't any data available. The schools themselves aren't measuring learning. What's the difference between your knowledge when you start as a first-year student and when you graduate? What do you feel about the teachers? What's the rigor of the academic program? How engaged are you on campus? Information like that is just not available from all schools.

Have you considered something along the lines of the reputation survey, but to assess some of the things you're talking about?
You need the schools to cooperate. We would have to get access to enough students to have a statistically significant sample. I personally don't think schools are going to work with U.S. News on something like that.

Another criticism you often hear is that the rankings encourage schools to add unnecessary perks, such as climbing walls. Is that fair?
Some schools or college presidents or boards have used wanting to improve in the rankings as an administrative goal. Some schools are targeting their academic policies toward improving in the rankings. But I don't think that's really hurting students. The factors that you cited aren't really part of the rankings. Many people at the schools don't understand the ranking methodology and say things as an excuse vs. the truth. Generally, targeting the rankings doesn't hurt students. If schools are targeting ranking factors like improving graduation rates and improving freshman retention and paying faculty more money and having more small classes and fewer large classes and having faculty with better credentials, those are all U.S. News ranking factors, and the students are going to benefit from that.

A couple of months ago, it was reported that Clemson University was trying to manipulate the rankings through strategic voting — giving competitors a below-average rating. The university denied the claim. Is that something you try to police?
We definitely do. We throw out the two highest ratings and the two lowest ratings for each school, so we have some statistical safeguards to prevent any strategic voting from impacting any school's score. And the fact that the schools' scores have been stable — there's been academic research to show that — is proof in and of itself that schools aren't able to game the scores.

Let's talk about specific schools. Any surprises this year — a school whose ranking has jumped from last year?
There wasn't a lot of change from this year to last. Harvard was No. 1 last year, and now Harvard and Princeton are [both] No. 1. People are going to write about that. But they were very close before, and now they're tied. That's not really a big change. Schools are pretty stable, and the top schools have the resources to continue to draw the best students and graduate them at a high rate year after year. It's hard to move big institutions to a great degree at the top.

Is there a school that's been moving slowly and steadily up the list that may strike new readers as being surprisingly high up?
Schools like Washington University and the University of Southern California have had slow and steady climbs up the national-university list by making across-the-board improvements. One school that's going to surprise people is the University of Maryland–Baltimore County. It's both a top Up and Coming school and No. 4 in the Commitment to Undergraduate Teaching [list]. That's a school that isn't one of the name brands but is up there near the top of these two lists, just like George Mason, Northeastern and Drexel are near the top of Up and Coming.

Have you considered adding alumni earnings to the rankings formula, along the lines of the PayScale rankings?
Yes. We have earnings in two of our rankings. In our MBA rankings, we have earnings data. In our law rankings, we have [job] placement data. There were issues about that, that [PayScale] was giving credit for somebody's salary for their undergraduate vs. their graduate degree. That's a problem with using it. But obviously people care. It's important how much money people make and the value of degrees. Students are graduating with debts, and they have to pay them back.

There are so many college rankings these days. What sets you guys apart?
I think our rankings make the most sense to the public because our rankings are the most transparent. Our rankings have the methodology that makes the most sense. When the public sees that the schools are wanting to do better in our rankings, they say, well if the schools want to improve in these rankings, they must be worth looking at. So, in essence, the colleges themselves have been a key factor in giving us the credibility.