Beginning in July 2004, all public service points began to categorize and record questions using the Warner model. The 45 percent variation in the classification of questions noted by Kesselman and Watstein prompted MLK Library planners to add sample questions and definitions on the survey forms.14 Data thus collected could be used to increase service efficiency and to inform decisions on staffing of service desks. The remainder of this article deals specifically with the data collected within the Reference unit and what it indicates about the ability to fulfill patron needs.
Results
Analysis of data for the Reference unit falls into four sections: initial period (September 2003 to February 2005), combined telephone and reference (March to September 2005), in-service (June 2005), and the post–in-service experience (October 2005 to May 2006), as reflected in tables 1–5 and figures 1–3. These four sections reflect MLK Library’s attempt to analyze operations, more efficiently allocate librarians and staff, apply the Warner model to a different type of library, and begin to address the needs of new technologies.
Initial Period: September 2003 to February 2005
Table 1 shows the initial use of the Warner model in identifying Level I–IV questions received at the reference desk and in the Reference Connection room.
The data reflect both in-person and telephone questions answered by librarians, support staff, and student assistants. During this period, 100 percent of telephone reference questions were answered by librarians in the Reference Connection room.
The results demonstrate that the percentage of Level I–II questions versus Level III–IV questions handled were consistent with the 80 percent–20 percent pattern established by Warner. Baseline use figures for both semesters and break periods were established for the total number of questions received. While the total number of reference questions has remained high, more Level III–IV questions were addressed during the fall and spring semesters than during winter or summer breaks (e.g., January, June, July, and August).
Because of the library’s goal to implement tiered reference, one area of special interest was the number of lower-level questions answered by librarians rather than support staff or student assistants. With this percentage consistently near or over 60 percent and sometimes nearing 80 percent, the survey data was examined more closely for periods of low question volume, particularly for telephone reference that was handled by librarians in the Reference Connection room. It was felt that, with librarians answering 100 percent of the telephone reference questions, the number of lower-level telephone questions could be easily reassigned to support staff.
Combined Telephone and Reference: March to September 2005
As a result of the examination of the data, the separately scheduled telephone reference service was moved out to the reference desk from 9–10 a.m. (Monday–Friday) and from 5–9 p.m. (Monday– Thursday). In this new service model, support staff and student assistants answered the telephone first, fielding Level I–II questions, and referring Level III–IV questions to librarians at the desk.
The data reflecting the combination of telephone reference with in-person reference during limited hours can be seen in table 2. While the percentage remained fairly constant for Level I–II questions, the percentage answered by librarians dropped below 60 percent, and the usage of librarian time at the reference desk was reduced by 20 percent. The change in telephone service also reduced the percentage of telephone questions answered by librarians from the 100 percent experienced before March 2005 to an average of 86 percent. A small function and staffing change had a greater effect than expected.
In-service: June 2005
The SJPL has had a tradition of holding in-service events for training and other purposes. In June 2005, the entire Reference unit devoted its in-service to the evaluation of tiered reference service. Librarians and support staff from the public library’s General Collections unit joined the librarians and staff of the Reference unit to share their collective experience with the data-gathering process. To ensure that everyone understood the Warner model, definitions and examples of Level I–IV questions were reviewed. The forty-five librarians, support staff, pages, and student assistants in attendance were asked to assign fifty-five questions (e.g., “My laptop doesn’t connect,” “I need a video on exporting. How do I find one?” “Can I reserve a room for my group’s meeting next week?” and “I need some information on the German court system”) to Level I, II, III, or IV on the basis of examples used on the Individual Activity Count form. Small breakout groups discussed the coding and reported their results to the larger group, with an overall agreement rate of 69 percent.
Table 3 shows the nine questions that generated the most confusion. Seven of the questions reflected the difficulties encountered in identifying Level II–III questions. An analysis of the results found that some librarians and staff had coded questions on the basis of their personal knowledge of the subject or the answer rather than on the level of the question. This result is similar to results found by Kesselman in which personnel used their own definitions when tallying reference questions by category.15 Based on the in-service experience, the examples for the Level I–II questions were modified for subsequent survey periods (see table 4) in hopes of avoiding future confusion about categorizing questions. The revised form was distributed to all library units that participated in the survey week each month.
Post–In-Service Period: October 2005 to May 2006
Because of the data reported in tables 1 and 2 and the in-service experience (tables 3 and 4), the Reference unit decided to pursue four goals: better triage of questions, ability to handle more telephone calls, use of librarian time for more complicated questions, and the potential for staff adjustments. To accomplish these goals, the telephone reference service was fully combined with the in-person reference service in October 2005.
In October 2005, reference desk staffing consisted of two librarians and a support staff person or student assistant. The support staff or student assistant welcomed new patrons, answered their simple directional, technical, or catalog-search questions, and directed those with complicated questions to librarians. A modest redesign of the reference desk was done to facilitate the triage of telephone questions. A wall of paneling was added to a section of the reference desk to discourage in-person questions at that location. The librarians scheduled for telephone reference moved there, with a support staff or student assistant added during the busiest hours. The support staff or student assistant would answer the telephone first and forward more difficult questions to the designated librarian.
Table 5 and figure 1 show the results of these changes. Table 5, a continuation of the data reported in tables 1 and 2, shows a reduction of 35 percent in the number of questions received between October 2005 and May 2006 when compared with the library’s inaugural year (October 2003 to May 2004) and a reduction of 20 percent when compared with October 2004 to May 2005. While the percentage of Level I–II questions remains high (74–81 percent), the percentage of the lower-level questions answered by librarians was reduced significantly from the 60–80 percent to 34–48 percent. The changes resulted in increased opportunities for better triage of reference questions and more effective use of librarian time.
Figure 1 shows that librarians answered an average of 43 percent of Level I–II telephone reference questions between October 2005 and May 2006. This is a significant reduction from the 100 percent of telephone reference questions answered by librarians from September 2003 through early 2005 and the 86 percent answered by librarians from early 2005 to October 2005. The staffing and service adjustments that have been made have had a positive effect on question triage and the effective use of staff.
The categories defined in the Warner model were also used to categorize e-mail and virtual reference questions. At the MLK Library, the responsibility for e-mail reference service is shared equally by academic and public librarians. Figure 2 shows the percentage of Level I–II e-mail questions from October 2003 through May 2006. The results are similar to the percentage of Level I–II questions received for in-person activity. The total number of e-mail questions received per month (16–118) fluctuates, but is higher in the months at the beginning of a semester, such as September–October and February–March (85–119 questions). Over the thirty-month period, the number of Level I–II questions dipped below 70 percent only five times and below 60 percent only once. The data show that the Warner model applies to e-mail reference and suggest that the merged library should examine an approach to e-mail reference that employs support staff in handling more basic questions.
Live online reference is not a merged function. SJPL librarians monitor a public library queue while SJSU librarians monitor a California State University system queue. With data only available for selected months for the public library queue and comparable data not available for the academic queue, it is impossible to draw conclusions about the effectiveness of the Warner model for evaluating live online reference.
[...] Developing a Model for Reference Research Statistics: Applying the “Warner Model” of Reference Q… [...]