Indiana Alumni Magazine
Everybody Wants to be No. 1: A Look at College Rankings
In this and the next issue of the Indiana Alumni Magazine, we examine college rankings. This part considers these questions: What are rankings? Are there more than one kind for colleges and universities? How do rankings work, how valid are they, and how seriously should they be taken as true measures of institutional quality? The November/December issue will look more specifically at Indiana University's rankings at the undergraduate and graduate levels.
by Don Hossler
The production of college guidebooks and rankings has become a growth industry. In 1997 researchers at UCLA reported that total revenue from the publication of college rankings and guidebooks would reach $17 million that year. Time, U.S. News & World Report, Business Week, and the publisher of the Princeton Review publish annual college guidebooks and rankings. And each year another magazine or book purports to identify the best universities or colleges, the best graduate programs, or the best undergraduate majors.
Although it is commonly assumed that high school juniors and seniors and their families are the most avid consumers of these publications, college and university alumni also pay close attention to rankings. After all, the stature of their alma mater can be a source of pride or a cause for disappointment. In business, starting salaries for new employees can be influenced by the rank of graduate MBA programs.
With many alumni, an interesting phenomenon seems to occur. No matter what their academic record was when they entered the university, these same students, when they become alumni, are tempted to want only the best and brightest to be admitted because this will increase their university's prestige and provide them with more bragging rights.
It's not only alumni who want to point to rankings that make their degrees look better. Colleges and universities are tempted too. Even though university presidents may publicly rail against rankings, there is so much misunderstanding about their true value among the general public that admissions and public relations professionals often tout rankings that appear to "make us look good."
Americans appear to have an obsession with rankings. We rank sports teams and identify eighth-graders who are the top collegiate basketball prospects. We rate hospitals and retirement cities. We rank the cities with the best business climates. And now we rank colleges and universities and their academic programs. A school may be the best national university, a best buy among national liberal arts colleges, or the best regional comprehensive university in the Southwest.
This penchant for wanting to know "who is No. 1" is not new. Efforts to identify the leading graduate programs in many disciplines and fields of study have existed for many years, but the attempt to determine which college or university provides the best undergraduate education is a recent phenomenon.
THE PROBLEMS WITH RANKINGS
Efforts to rate graduate programs have been undertaken since the first half of the 20th century. Scholars and educational policymakers wanted to know which graduate programs generated more research, graduated more students who became distinguished scientists, or garnered the largest number of research grants.
Ranking graduate programs is more straightforward than ranking undergraduate institutions or undergraduate majors. If students are seeking a doctoral degree in physics or a master's degree in creative writing, they are likely to take most of their classes from faculty in physics or creative writing. As a result, the publication record of the faculty in creative writing, or the number of grants garnered in physics are good indicators of the quality of the program.
At the graduate level, students take most of their classes with other graduate students who are enrolled in the same program. Because the experience of a graduate student is more self-contained and focused, ranking graduate programs makes more sense than ranking undergraduate programs. Rankings publications do not attempt to develop global, summary rankings for all graduate programs or for an entire graduate school. Such rankings, it must be assumed, would be meaningless. At the undergraduate level, most students don't even start taking classes from professors in their area of interest until the junior year. Even during the junior and senior years, students take classes in elective areas. They may never have a class with an award-winning faculty member in their major.
Because the intensity and focus of students in undergraduate majors vary considerably, it is difficult to rank the quality of an undergraduate major. It is even more difficult to find an effective means for developing a summary ranking for all undergraduate programs. Is it really possible to add the in-class experiences of students majoring in chemistry, drama, music, sociology, classical studies — keeping in mind that one of these students may be in student government, another an athlete, another in choir, and another a commuting student — and still come up with a summary rank for all undergraduates at the University of Michigan or for Amherst College?
Various criteria are used to rank colleges and universities at the undergraduate level. U.S. News & World Report considers such factors as admissions selectivity, graduation rates, student-faculty ratio, the percentage of alumni donating to their alma mater, and peer rankings. To rate reputation — the so-called "peer ranking" — the publication circulates a list of colleges and universities to college administrators or faculty and asks them to rank each institution on the list. This assumes that the administrators or faculty know enough about the institutions to accurately rate them.
The Princeton Review purports to rank the social life of campuses by surveying currently enrolled students. One year the social life at Indiana University Bloomington was ranked on the basis of 50 students who were willing to fill out a survey administered in the Indiana Memorial Union. Representatives sat in the IMU all day trying to induce more students to complete the survey, but only 50 took the time to do so. This gives new meaning to the phrase "a random sample."
Another ranking publication that has been around for many years is the Gourman Report, which professes to rank all undergraduate majors at most four-year institutions in the United States.
Jack Gourman insists that there are individuals on each campus who provide him with information about the quality of each major. No one seems to know any of these individuals, yet Gourman is able to rank majors down to the level of two decimal points. Still, one issue of the Gourman Report ranked highly an IUB major we do not offer.
In addition, rankings can be manipulated. One institution, for example, developed a two-part application. The first part requires no application fee and elicits little in the way of formal action from the admissions office, but it gets counted as an application. Admissions professionals at the institution acknowledge that the purpose of the two-part application is to increase the number of applications the campus can report in rankings surveys. The reason? Larger differences between the number of students who have applied and those who have been admitted make a school look more selective.
Another university has started to provide less information on admissions standards. Administrators at the campus hope that this will induce more students to apply. That way the campus can reject more applicants. Given the formula for determining institutional rankings, being more selective will help the university move up in the rankings.
Part of the formula a major publication uses to rank business schools is how students rate teaching. A dean at one business school came up with a strategy to improve his school's rank. He posts teaching evaluations outside a faculty member's classroom so everyone can see them. He reasons that faculty will pay more attention to their teaching when their evaluations are publicly displayed. And improved teaching will mean a higher ranking.
In 1995, Steve Stecklow, a reporter for the Wall Street Journal, documented that some colleges and universities were using false data. He compared the information these colleges provided for rankings and guidebooks to information about enrolled students and finances in annual reports prepared for trustees and donors, or for the sales of bonds to build new campus facilities. These comparisons showed conflicting information. A campus might report one set of data for average class size, average SAT scores, or campus financial information for a rankings publication and a different set of figures for its annual report. Stecklow concluded that as part of their marketing strategies, some campus administrators appeared to intentionally provide incorrect or misleading information for guidebooks and rankings.
WHAT CAN RANKINGS TELL US?
Despite their inherent flaws, rankings publications like U.S. News & World Report can provide some useful information. Higher education scholars have used many of the variables — library holdings, faculty salaries, the average SAT score of undergraduates — as indirect indicators of institutional quality. In fact, knowing that an institution has been consistently ranked with other highly regarded institutions for the past 10 years in U.S. News & World Report can be a reasonably good indicator that this college or university may be better than many other institutions that consistently have been ranked lower.
But shifts over one year or even two years from 25th place to 50th or from 60th to 40th are probably not the result of actual changes in the quality of an institution. The basis for the strength of a college or university rests in the quality of its faculty, the extent to which students are engaged in their courses, the quality of the libraries and how much students use the libraries, and so forth. These institutional attributes don't change much from year to year.
In recent years, the quality of universities appears to change annually because of subtle changes in the formulas that rankings publications use in calculating who's No. 1 (or No. 101). Thus, Old Main University, and every other institution ranked behind Old Main, might submit the exact same data as they had the previous year, but OMU might drop in the rankings. Why? This year's formula might give more statistical weighting to the cash value of fringe benefits for faculty, adjusted for regional variations in the cost of living. Or it might give greater weight to the level of expenditures for programs in science and engineering. Informed alumni need to recognize that changing formulas can result in headlines — and more sales of the publication — but may not indicate a change in the quality of their alma mater.
WHAT REALLY MATTERS?
Another perspective on institutional quality can be garnered by trying to determine what students actually gain from earning a college degree. Much of the research on the college experience and outcomes has found that the extent to which students are involved with the college they are attending, both in class and out of class, determines the outcomes of a college education. In their 1991 study How College Affects Students, E.T. Pascarella and P.T. Terenzini reviewed thousands of college-impact studies and concluded that the amount of energy and effort students invest in their education is much more important than the college or university they attend. When the background characteristics of students are taken into consideration, the research on college outcomes has found no systematic or convincing evidence that student outcomes are related to the traditional measures of institutional quality used in rankings.
The best indicators of the quality of a college or university are outcomes and assessment data that focus on what students actually do after they enroll, their in-class and out-of-class experiences, and the quality of their effort. Indiana University is in the forefront of efforts to identify more meaningful measures of institutional quality.
George Kuh, a Chancellor's Professor in the School of Education, is leading two projects that provide more empirically based efforts to assess the quality of student effort and thus indirectly capture evidence about institutional quality. Kuh directs the College Student Experience Questionnaire, a survey instrument that assesses the quality of undergraduate effort and involvement. He is also leading a project funded by the Pew Trusts to develop an annual survey of college students that would focus on the Indicators of Good Practices in Undergraduate Education, a handbook published in 1996 by the National Center for Higher Education Management Systems. This new survey, the National Survey of Student Engagement, asks students to rate their involvement in their coursework and campus life. The vision for this project is to develop a national database on all institutions that would provide more direct measures of the college experience. If this project is successful, it may eventually provide information on student outcomes that could be used in guidebooks and rankings.
These efforts are important for several reasons. First, they have the potential to provide more direct indicators about the quality of education at our colleges and universities. Even more important, they could provide insights into substantive educational interventions that could enhance the quality of the college experience.
Will recalculating the value of their fringe benefits change the ways faculty teach, reduce the student-faculty ratio, or improve the quality of the educational experience? No, but it can raise or lower a college or university's rank in U.S. News & World Report.
What would be more useful? How about knowing the frequency with which students discuss ideas with faculty members or receive prompt feedback on their homework? These are the kinds of faculty and student behaviors that are known to improve learning outcomes. Wouldn't it be nice if colleges and universities were competing to see which campus could engender the highest levels of student-faculty interaction? 
Don Hossler is IUB vice chancellor for enrollment services. Also a professor of education, he has written several articles about ratings and rankings. For further information, he recommends Alexander Astin's Achieving Educational Excellence (1985) and George Kuh et al's Involving Colleges: Successful Approaches to Fostering Student Learning and Development Outside the Classroom (1991). In the next issue, Hossler looks at rankings and indicators of quality at IU.