Judith M. Nixon, Editor
E. Stewart Saunders, Guest Columnist
Print version (Adobe Reader required)
For my second column as the editor of the new Management column, I decided to focus on library service assessment. It is certainly one of the most important activities we need to do; however, often we, as librarians, have little training on or knowledge of how to evaluate and assess our service. We are better at collection evaluation than service evaluation. LibQUAL+, one of the most important assessment tool libraries are using right now, was the first assessment method that came to mind. My long-term colleague, Stewart Saunders, is the Purdue Libraries statistics expert. He analyzed the LibQUAL+ data for us, and so was the logical choice to write this article.
What next for the column? I am on the lookout for ideas and writers on the broad range of topics that relate to running a reference or public service department. I encourage you to suggest column topics and to become an author and write on any successful reference programs or services.–Editor
The Internet and Google have changed the information landscape. Libraries now compete for a share of the information market, and library patrons are now referred to as customers. As libraries become businesses, they must take care of their customers in the same manner as does the private information sector. Private firms seek to satisfy customer needs, so libraries must do likewise. As libraries attempt to meet competition from other information providers, managing resources and having a sense of strategic direction become all the more necessary.
In the past, library management decisions were based on data and intuition. Data was in-house data–circulation statistics, reference activity, budget figures, and so on. But intuition? Well yes, what managers knew about patron needs was a consequence of casual conversations, rumor, and the squeaky wheel. Occasionally libraries would try a patron survey, but this was the exception, not the rule. Academic libraries, after all, had a clearly defined educational mission: they, better than students (or faculty), knew the needs of their clientele. Those needs were shaped by the curriculum and research enterprises of the college or university. To guide us, we had Books for College Libraries, Choice, and the professional research literature in library science. Isn’t that what we meant when we put forth the ideal of a professional librarian: someone who has been educated in the principles of collection development, reference, and the organization of knowledge? Why should we ask patrons about their needs? They have no training in these areas.
All Other Judgments Are Essentially Irrelevant
In the last decade, that scenario changed. Despite confidence in our professional knowledge, we have turned to our customers for their input. (Note they are no longer patrons.) We are not alone; even General Motors has learned that what customers think is important. Universities and colleges have come to realize that the crucial measure for an educational institution is impact, not input. We no longer talk about “research,” we now use the term “discovery”; we no longer use “teaching,” we now use “learning.” University and college administrations now expect libraries to prove their value in terms of the learning and discovery that results from library use. This has become all the more important, as the Internet now offers an alternative to libraries as a source of information. In order to demonstrate the library’s superiority to other information providers, we need to show our impact with a measure that stakeholders can understand. What better way to measure value than to ask library customers for their views about the library’s impact on their learning and discovery. The new wisdom has become: “Only customers judge quality; all other judgments are essentially irrelevant.”1
That still leaves unanswered the question of how best to attain this measure. For years, the service sector of the United States economy–banks, restaurants, hotels, and so on–collected information on patron satisfaction using questions with a simple Lickert scale: “On a scale of one to five please rate your satisfaction with our accommodations.” And so the customer rated general satisfaction with services. It wasn’t until Parasuraman, Zeithaml, and Berry proposed that there needed to be a second question that a true scale of measurement was developed: “On a scale of one to five please tell us what level of service you care about.”2 The true measure of satisfaction is the difference between the level of services received and the level expected; thus gap analysis was born (known as a ServQual measure). During the 1990s, several university libraries surveyed patrons using the ServQual instrument. This effort evolved into the creation of a survey instrument to measure gaps in library service, LibQUAL+.
What Is LibQUAL+?
LibQUAL+ was developed by the Association of Research Libraries (ARL) in conjunction with several faculty members at Texas A & M University. They wished to develop a survey that allowed local libraries to discover their particular strengths and deficiencies but, at the same time, was standard across all libraries. Standardization would allow libraries to compare themselves with peers. It also would allow them to turn to those peers with high performance ratings for help with best practices.
The survey consists of twenty-two core questions that measure patron satisfaction with the:
- quality of service provided by the staff;
- extent and quality of information resources provided by the library, including ease of access to information; and
- quality of the physical space provided by the library.
In addition, there are a number of general satisfaction questions as well as questions on demographic characteristics of the respondent. Finally, the survey includes an opportunity for respondents to express their satisfaction through an open-ended, general response.
Each of the twenty-two core questions actually contains three questions. A question would read something like this: “When it comes to service from the library staff 1) my minimum expectation is _____, 2) my desired expectation is _____, 3) my perceived level of service is _____.” The answer to each of the three questions is a mark on a nine-point scale indicating a level of expectation and a level of perceived service. The difference between the minimum and desired level of expectation is referred to as a zone of tolerance. If the perceived level of service falls between these two expectations, it is within the zone. If the perceived level falls below the minimum expectation, this indicates a serious shortfall of service. Differences between the perceived level of service and the two levels of expectation are measured as gaps to indicate the strength of satisfaction, if the gap is positive, or of dissatisfaction, if the gap is negative. It is quite normal for the gap between the perceived level and the desired level to be negative. A positive gap on this measure would indicate perfection for the library, an unlikely result. On the other hand, the gap between the perceived level and the minimum expectation should normally be positive if the library is minimally meeting customer needs.
After the survey has been completed, ARL gives the participating library a document containing charts and summary data. An SPSS file of raw data also is made available for further analysis. When my own institution, Purdue University, did the survey in 2005, we used the raw data to create many additional charts and data subsets. By this means, we were able to see not only how the faculty or the students responded to the survey, but we were able to do data sets and charts for each college at the university and for certain subsets of the faculty and students. By drilling down into the data, we discovered patterns of responses that were not apparent from the aggregated data. ARL also creates norm tables for each year. A library can then determine its percentile ranking against other institutions for different questions and groups of users. Just recently, ARL added a new feature that allows the library to break out responses by status and discipline from surveys done at other institutions.
LibQUAL+ has enjoyed great success as an assessment tool. As of March 2007, 1,025 libraries have administered this instrument. It has been used by university, college, medical, special, and even some public libraries. Library consortia also have used it. The survey has been translated into other languages, and it has been administered to libraries in more than twenty other countries. To date, 764,785 respondents have completed the survey form. What has made LibQUAL+ the predominant tool for academic library assessment? ARL wanted a survey instrument that would be standard across all academic libraries. The advantages are two-fold: 1) individual libraries can compare their results with results of peer institutions; and 2) libraries can use a proved and tested survey instrument, thereby foregoing all the expense and work of developing their own survey.
Creating a survey is not an easy task. The survey must be both reliable and valid. LibQUAL+ was developed with the help of a grant from the U.S. Department of Education. Experts in the field of test development drafted the first version, which was administered to a small group of respondents. The results from the first trials were then analyzed in order to revise and improve the survey instrument. The team that designed the survey was looking for the underlying three or four factors that define library satisfaction. They used factor analysis to this end. Questions in the survey were then grouped around these general factors.
LibQUAL+ does not claim to be the end-all for library assessment. After seeing the results from the survey and pinpointing specific issues, libraries are advised to focus on specific questions by using smaller surveys, doing interviews, or having focus groups. A number of libraries have used a first iteration of the survey to create a benchmark. After a few years, the LibQUAL+ survey is repeated to determine progress toward a greater impact. ARL describes the LibQUAL+ survey as one tool in a kit of tools for performance measure.
Does LibQUAL+ Work? What Do Libraries Do with the Results?
Does assessment work? In particular, does LibQUAL+ work? Does LibQUAL+ deliver the kind of data that are useful in improving the services offered by the library? Does it offer data that nudge the library in a new strategic direction? There is a fairly large library and information science (LIS) literature base on how individual libraries have responded to their LibQUAL+ results. If I were to characterize these responses, I would say that much of what libraries do falls in the category of very specific changes to very specific services. There have been attempts to use the survey to formulate strategic plans, but these broader objectives seem less amenable to the utilization of assessment data.
The University of Pittsburgh Library epitomizes the kind of changes that can take place as a result of LibQUAL+ data.3 Based on their results from a survey administered in 2002, this library carried out a large number of specific changes. A major complaint of faculty and graduate students was lack of complete runs of journals. To address this perception, the Pittsburgh Library purchased electronic backruns for many journals, instituted document delivery to faculty and graduate students, and moved the off-site storage facility nearer to campus and provided it with a shuttle connection. Undergraduates were more turned off by the food-and-drink policy. The library changed policies, allowing covered drinks in the library and food in certain designated areas; they also installed a coffee bar within the library. LibQUAL+ and subsequent focus groups revealed a lack of confidence in library staff. Users wanted not just assistance, but competent, professional assistance. This resulted in restructuring the staff training program. Focus groups in conjunction with LibQUAL+ revealed that: 1) no matter where patrons ultimately found their information, the search nearly always began online; and 2) students were more comfortable obtaining assistance from peers than from librarians. To deal with the first issue, the library instituted a chat reference service. To deal with the latter issue, the library set up a system of student consultants. The consultants were undergraduate students who offered assistance in the library and in residence halls. Finally, the Pittsburgh Library created an online system for renewing books.
Libraries in the OhioLink consortium did the survey in 2002. When the composite scores for the consortium were compared to scores of peer libraries or national averages, the consortium ranked higher than the average of its peers or the national average.4 OhioLink points to this as evidence that there is value added by being a member of the consortium. The consortium makes possible a level of service that each individual library could not attain. In this instance, LibQUAL+ results are used to justify the continued funding and political support needed for the consortium.
Libraries also have looked to LibQUAL+ for help in strategic planning. The argument is that libraries need reliable data as a basis for planning. In a sense this is true; in another sense, LibQUAL+ does not quite measure up to this task. What is evident is that libraries are using LibQUAL+ results as a repository of information from which aspects of the strategic plan can be implemented. What is more tenuous is using this information to actually map out the strategic plan. Purdue Libraries discovered this when it undertook to create a new strategic plan in 2006. LibQUAL+ was administered in 2005 with the idea that the results could be used to formulate the plan. As they progressed, the planning team realized that the shortfalls in library service as revealed by LibQUAL+ were focusing attention on the sins of the past and not on the possibilities of the future. This does not mean that LibQUAL+ was a futile exercise. On the contrary, its measures turn up frequently in the plan as a metric for determining progress toward the goals of the strategic plan; what LibQUAL+ did not do was set the goals of the strategic plan.
Bowling Green State University also used LibQUAL+ to support their strategic plan.5 Again, its role was more one of implementing specific service features to meet the plan’s goals than an attempt to formulate goals for the library, although this did happen. One goal of their plan was to reorganize library space in order to provide one-stop shopping. LibQUAL+ indicated that graduate students wanted a graduate study area, while other students wanted small group study areas, larger computer labs, and possibly a coffee bar. One strategic goal that seems to have come from the survey is the need to market library service more effectively. LibQUAL+ indicated a widespread lack of such knowledge. This resulted in new services to inform students and faculty about new information products as well as older services that previously had gone unnoticed.
LibQUAL+ Defects
LibQUAL+ is not without its defects. Patrons complain that it is too long (thirty-nine questions), or that all questions have to be answered before the survey will be accepted. There also is tension between the need for local information and the standardized information provided by the survey. Many libraries would like to tailor the questionnaire to find out information that is specific to their library clientele or local problems. This is difficult to do without making the survey too long or removing some of the standardized questions. As with any survey that uses self-selected respondents, the resulting estimates are always going to be somewhat biased. This means that any attempt to use statistical analysis for confidence intervals can only give ballpark figures, as one always has to guess the size of the bias. Despite the survey’s emphasis on the representativeness of the sample, this does not remove the fact of bias; a correspondence between sample and population concerning the representativeness of such variables as gender or discipline does not indicate a lack of bias. These variables are seldom the source of the response bias or are correlated with the source of that bias.
Pages: 1 2