Tina M. Neville and Deborah B. Henry
Print version (Adobe Reader required)
In 2005, the authors tested the consistency and ease of use of a skill- and strategy-based reference question classification system published by Warner in 2001. Results of that test indicated that the Warner system was a significant improvement over the traditional resource-based system. In this study, reference librarians from other institutions were invited to compare the technology-sensitive Warner system to the traditional Katz classification system. The results of this larger test mirror the findings of the original study. Overall, classification was more consistent using the Warner system.
Libraries of all types depend upon use statistics for planning and managing reference services and for assessing the value and usefulness of the library’s collection. Historically, reference statistics have been troublesome to collect because of the qualitative nature of questions and the difficulties in assigning a wide variety of these questions into a minimal number of strict categories. The classification of questions must be distinguishable and consistent if librarians are going to be able to use the results effectively for planning and evaluation. Online resources and new technologies have altered the types of questions received at library reference desks, leading some institutions to reconsider the classification systems used to document reference service. During selected periods from 2004 to 2005, the authors recorded every question they received while they were staffing the reference desk of a small academic library. The questions were classified two ways—using both the traditional method described by Katzand a new classification method proposed by Warner.1 The results were compared for ease of use and consistency in classification. The Warner method worked better at the authors’ institution and was incorporated by all reference librarians beginning in July 2006.2 The authors performed the 2005 study, like many studies reported in the literature, at their home organization. They sought to test those results by conducting a similar comparison using participants from multiple organizations.
Literature Review
Classification and analysis of reference questions has intrigued librarians for years. As early as 1951, Lawrence Thompson encouraged colleagues to evaluate existing categories for their usefulness and to construct new classifications as needed.3 During the mid1960s, the American Library Association (ALA) and the National Center for Educational Statistics cosponsored a national conference aimed at standardizing library statistics.4 Later studies examined reference question classifications aimed at improving collections, refining staffing needs,or analyzing chat reference services.5 The evaluation of the Reference and User Services Committee of RUSA’s Reference Services Section (RSS) provides a detailed bibliography of articles that traces the evolution of reference service and evaluation.6 It is unlikely that the collection of statistics relating to reference service activities will ever be completely uniform among all libraries. However, reference librarians and their administrators will continue to collect and compare these data. What, then, can be done to improve the methods that are currently used?
Literature evaluating various types of reference desk activity as well as electronic reference (e.g., chat, e-mail, and instant messaging) abounds.7 As part of the evaluation of the overall effectiveness of these services, authors have also tried to categorize the nature of the questions received. Katz described traditional reference-question categories in detail as directional, ready reference, specific-search questions, and research.8 In 2001, Warner suggested a new classification system for reference questions that includes skill-based and technology-related categories (defined as nonresource-based, skill-based, strategy-based, and consultation).9 Her institution, a health sciences library, was undergoing a physical redesign and consolidation of the circulation and reference service desks and, concurrently, was assessing staffing and collection needs. Reference librarians at Carnegie Mellon have also created their own classification study to address this issue.10 Their six-point READ Scale bears some resemblance to Warner’s classification; it includes effort and time along with a skill-based assessment and allows for higher levels of classification for research assistance that is conducted beyond the reference desk.
Several studies note that the consistency of classification is also an issue of concern.11 Do librarians interpret the defined categories in a uniform manner? Two reference librarians might perceive the same question differently, leading them to classify it in different categories, particularly if the categories are not clearly defined. National reporting agencies and organizations continue to collect reference transactions as part of their multi-institution statistical reports.12 Yet, if librarians within institutions have difficulties with consistency, how can cross-institutional statistical comparisons be expected to provide meaningful information?
Method
The University of South Florida St. Petersburg (USFSP) is a small public institution serving approximately five thousand students and offering both undergraduate and graduate degrees. The library has a staff of seven professionals (MLS) who staff the reference desk for seventy-three hours a week, including evenings and weekends. After an extensive analysis of questions received at the desk in 2004 and 2005, the authors presented the Warner system to the entire reference staff of the USFSP Poynter Library as a means of providing a more accurate portrayal of reference activity.13
The authors wanted to verify that the new method is truly easier to use and provides a more consistent means of classification from one librarian to another. Testing of the Warner system for this study was performed in two stages. The authors recorded every reference desk interaction they performed while on desk duty during the spring 2007 semester. As they recorded each question, they coded it with the Warner classification that seemed to be the best fit. Librarians must record reference statistics quickly so that they may assist other patrons, so an intuitive system is essential. Therefore the authors made every attempt not to overanalyze each question; the Warner classification that came to mind immediately was recorded, and no attempt was made to see whether it corresponded to previous choices for similar interactions.
At the end of the semester, each author transferred her list of questions with their accompanying classification codes to a Microsoft Excel spreadsheet, and then hid the coded column. The authors then exchanged spreadsheets and proceeded to classify each other’s questions without seeing the original code assignments. Finally, the spreadsheets were combined and the two sets of codes compared for discrepancies—both for consistency of the original coder and for variation between coders. The intent of this portion of the study was to determine whether librarians who are very familiar with the Warner process would find the coding to be easier and more consistent.
The second part of the study involved a comparison of the Warner and traditional categories using a survey that was distributed on three library discussion lists. The survey was approved by the Institutional Review Board (IRB) at the University of South Florida prior to its dissemination. The LIBREF-L and PUBLIB discussion lists were selected as the main means of dissemination because they are both lists frequented by reference librarians and aimed at a national and even international audience. LIBREF-L counts more than two thousand reference librarians as subscribers.14 Although subscription figures were not available for PUBLIB, it is known to be a popular list reaching a large number of librarians. It should be noted, however, that PUBLIB does not limit content to reference issues. The survey was also posted on the Florida Library Association discussion list (more than eleven hundred subscribers) in an effort to reach additional reference staff at all types of libraries.15
Participants in the survey were asked to classify forty questions: twenty questions using the traditional reference categories defined by Katz and twenty questions using the Warner system (see the appendix for a list of the survey questions). After a brief explanation of the coding systems, the authors asked the participants to code each question into one of four categories as described previously. Participants also received options for “I am unable to place this into a category” and “I would not record this as a reference question.” The questions covered a broad range of reference interactions. Although no two questions were alike, matching questions appeared in both sections of the survey to allow for comparisons between the two systems. For example, both the Katz and the Warner sections of the survey included questions requiring a basic library catalog search, a question for help with specialized software, a complex research question, and so forth.
