Another example of how the results can improve our understanding of reference service was that the providers’ perceptions were lower than those of the users by about one half point on the five-point scale. The differences were not immense, but the gaps were consistent and statistically significant. This could be a reflection of the different perspective of the learner and the teacher, or of either over-confidence on the part of the user and under-confidence (perhaps realism?) on the part of the provider, or both. Whatever the case, it provided a good opportunity to raise morale. Reference providers can be pretty hard on themselves. The nature of the job—dealing every day with struggling users and repeat questions—can give us the impression that we are making no progress. Showing how satisfied our users were and that we consistently rated ourselves lower was a positive lesson for our reference providers.
One final example from our test: the users’ mean responses for transactions involving a librarian were consistently higher than for those involving staff. Again, from a management perspective, this is a good thing to know. Librarians are expensive. This result may help us justify why we have them on the reference desk instead of less formally qualified staff. Our users got more help and were more satisfied with the results in transactions involving librarians rather than staff.
Conclusion
As noted earlier, this is a study of perceptions. The questionnaire cannot be used to objectively evaluate the help the user received or the quality of the teaching involved in the transaction using external criteria. The results of this survey in Pittsburgh academic libraries confirmed LibQUAL results—users generally give public-service personnel high marks for service. More specifically, this study indicates that our users perceive that they get the help they need and are satisfied with reference service. Users are more positive about their interactions with librarians than staff, though they are positive about both. But they are slightly less satisfied with our teaching and particularly evaluative assistance. Reference providers (in our test, librarians most of all) are harder on themselves and users (or perhaps they are more realistic) than users are. If administrators rely on reference providers’ perceptions of service satisfaction and effectiveness, they will have a distorted view of the service. Gathering responses from both sides of the reference desk provides a more nuanced picture of reference service as a whole.
This method for evaluating reference could easily be extended to synchronous online reference transactions, delivered to user and provider at the end of the transaction, and even to e-mail reference, but it would be more intrusive and time consuming to expand it to phone reference. It has been tested in a variety of academic libraries, but there is no reason why it cannot be used by public libraries that follow an instructional model of reference service. The combination of perspectives from users and providers adds value to the instrument, but it is possible to use the user portion alone as an even simpler evaluation tool. Once developed, it was relatively easy to administer and did not unduly burden reference service providers or users. It was also relatively easy to process the data and analyze the results. If used to test online reference, data-processing time could be reduced even further with automatic collection of digital responses using readily available software and systems.
Like many other libraries, the University of Pittsburgh Libraries conducts the LibQUAL survey on an annual basis. More specific evaluations of reference service should be conducted on a similarly regular basis, if not continuously. As a manager, I would also like to be able to evaluate the service performance of individual reference providers. This would enable the annual or tenure reviews of reference service personnel to be based on something other than observation, anecdote, and the occasional thank you note from a user. Such individual evaluation would make reference service evaluation more comparable to the student evaluations that are a common component of classroom faculty review processes. Obviously there are delicate personnel issues involved in such a step. Also, individualizing the questionnaire might add to the problem of a bias towards positive transactions. But I think the librarian profession needs to move beyond anonymous evaluation and have the confidence to assess the service quality provided by individual librarians, particularly if we can find a way to do so quickly and easily.
Acknowledgements
The author wishes to thank Jean Alexander, Head of Reference at the Hunt Library at Carnegie Mellon University; Jill Ausel, Director of the Library at Chatham College; and Elaine Rubinstein of the Office of Measurement and Evaluation of Teaching; Rachel Shackelford, Hillman Reference and Instruction Office Manager; Kate Thomes, Engineering Librarian; Eve Wider, Hillman Reference Area Supervisor; all of the University of Pittsburgh. This article is based on research conducted while the author was head of Hillman Public Services at the University of Pittsburgh and first presented at the Reference Research Forum at the ALA Annual Conference, June 25, 2006.
Corresponding concerning this column should be addressed to Judith M. Nixon, Head, Humanities, Social Science & Education Library, 504 West State Street STEW, Purdue University, West Lafayette, IN 47907; e-mail: jnixon@purdue.edu.
Jonathan Miller is Director of Libraries, Olin Library, Rollins College, Winter Park, Florida.
References
- Reference and User Services Association, “Definitions of a Reference Transaction,” www.ala.org/ala/rusa/rusaprotools/referenceguide/definitionsreference.cfm (accessed June 6, 2007).
- Reference and User Services Association, “Guidelines for Behavioral Performance of Reference and Information Service Providers,” www.ala.org/ala/rusa/rusaprotools/referenceguide/guidelines behavioral.cfm (accessed June 6, 2007).
- LibQUAL Charting Library Service Quality, “Welcome to LibQUAL,” www.libqual.org (accessed June 1, 2007).
- Matthew L. Saxton and John V. Richardson Jr., Understanding Reference Transactions: Transforming an Art into a Science (New York: Academic Pr., 2002).
- Jo Bell Whitlatch, Evaluating Reference Services: A Practical Guide (Chicago: ALA, 2000).
- Jo Bell Whitlatch, The Role of the Academic Reference Librarian (New York: Greenwood, 1990).
- Marjorie E. Murfin and Gary M. Gugelchuk, “Development and Testing of a Reference Prediction Assessment Instrument,” College and Research Libraries 48 (July 1987): 324–25.