We decided to study the perceptions of users and providers of reference services with all the limitations that perception studies imply. The model we used to explain to people what we wanted to do was the simple customer satisfaction postcard used at restaurants, but in our case a postcard would be completed by both the customer and the server.
We developed a short questionnaire (see figure 1), tested it, and made some minor adjustments to wording. The questionnaire is printed here so any library can modify and use it. The two-part form is perforated (using a simple manual paper perforator available from any office supply company) and is separated when handed to the user. Each copy of the questionnaire includes a control number on each part so responses can be linked during data analysis (that is, the same number appears on the top [user] and bottom [provider] portion of the form. A different number appears on each sheet.) This was achieved with a simple mail merge in Microsoft Word. The use of a sequential control number has the added advantage of enabling us to calculate how many surveys had been distributed at any one service point and thus calculate the response rate. The questionnaire asks some basic demographic information about both user and provider and also provides an opportunity for users to make comments and to identify themselves if they wish to be included in future, perhaps qualitative, assessment of reference service (for example, focus groups and interviews) The core of the instrument is the paired questions that seek the user’s and the provider’s perspective on whether each of the outcomes were met.
Testing the Instrument
In our test of the questionnaire, we could have spread the sample out, handing the form to every tenth user, for instance, but we decided that this would be overly complicated and confusing for reference providers. So we simply administered the survey for a three week period during our busiest times (in the middle of the fall semester). Some participant libraries in the test were not interested in the statistical significance or in generalizing the results of our test. But it is possible to calculate samples based either on the total population served by any library or on the total number of reference questions asked at a service point. For instance, working with a statistician at the University of Pittsburgh, we calculated an appropriate sample size that was based on both the total number of reference transactions at the Hillman Library and on the number of faculty and students at the university. In effect we considered two possible definitions of the population: the total number of reference transactions, or the total number of potential users of reference. In both cases the sample sizes required to have some confidence in the representativeness of our results were small. To achieve results within a 5 percent margin of error at (95 percent confidence) we would need responses from only thirteen University of Pittsburgh undergraduates, graduates, and faculty. In this case it was necessary to assume that who asked questions (undergraduate, graduate, and faculty) and also the subject and complexity of those questions was random over time.
Most of our outcomes, except perhaps for outcomes 1 and 4, are not relevant to directional questions, such as “where are the bathrooms?” We did not want to ask users and providers to complete a questionnaire each time they asked such directional questions for which reference services are not designed and organized. Although reference providers certainly answer many such questions because these often lead to more substantial questions, and answering them reinforces the important lesson that reference providers are here to help; they are not the raison d’etre of reference service. To exclude such questions, which form forty-three percent of the total transactions recorded at the main Hillman information desk, and to ensure consistency between libraries, we created the following rubric to help the reference providers decide when to offer the questionnaire to users. We asked them to only offer the questionnaire if the interaction meets the following criteria:
The question or need for service is greater than an answer to a simple directional question or a request for a simple query of the library catalog or database to locate a known item. As a general rule, if it takes longer to complete the questionnaire than it did to complete the transaction, don’t ask them to do it.
We developed this statement after discussion with the reference providers who would have to administer the survey. We also gave providers some form of words they could use to introduce the questionnaire to the user and to meet the requirements of our institutional review boards.
We printed more than the number of questionnaires we estimated might be needed at each service point and distributed them to all service points. We reminded all reference providers that the study would be conducted for the specified period and asked them to hand the top part of the form to the user at the end of the reference transaction. We also asked them to complete the bottom portion as soon as possible after the transaction and deposit the completed questionnaire in a box located conveniently at or near the service point. At the end of the survey period, the responses were collected from both users and providers and entered into a Microsoft Excel spreadsheet. Each row consisted of the data from surveys with a single control number, bringing together the responses of both the user and the provider to a single reference interaction for analysis.
Results
The purpose of this column is not to report on a particular research study but to introduce a method for evaluating reference service that captures both the user’s and the provider’s perspective. If the reader is interested in a full report on our use of the instrument in Pittsburgh academic libraries, the PowerPoint presentation to the Reference Research Forum at the 2006 ALA Annual Conference can be found at http://web.rollins.edu/~jxmiller/Refevaluation.htm.
This questionnaire measures user and provider perceptions of the success of individual reference transactions as measured by whether users received the information they needed, learned something about how to find information, learned how to evaluate information, and whether both parties were satisfied with the reference interaction. With enough responses it is possible to drill down into the results to explore distinctions between the perceptions of various groups of users or providers (in our test, for instance, the mean responses for those transactions involving a librarian were consistently higher than those for staff) and also, with appropriate statistical software, to be able to explore the relationships between the data. For instance, do undergraduate students perceive that they are learning how to evaluate information during the reference interaction? Or does the provider’s perception of a successful reference interaction correspond to the user’s perception of the same interaction?
One example of what can be learned from the questionnaire concerns the role of information evaluation in reference. The lowest mean results we received in our test were for questions three and seven, concerning the outcome “the user learns something about how to evaluate information,” and that was also the question with the lowest or number of paired responses. This result refl ects the fact that we got the highest number of “not applicable” responses to this question (and these came largely from providers, not users.) We can interpret this result to mean that our users perceived that they were learning how to evaluate information during reference transactions, even when our reference providers did not think that any teaching, learning, or modeling of information evaluation was taking place. Or if such instruction was taking place, providers did not perceive that users were learning how to evaluate information. Evaluation is a major component of information literacy, and as such is a major part of what we do at the academic reference desk. In this case our users seemed to recognize that, but our providers did not. Perhaps the lower mean was a result of that lack of conscious attention to the issue of evaluation by providers. This result was an opportunity to discuss the role of information evaluation during the reference interaction and to find ways to improve our ability to explicitly raise issues of evaluation during reference transactions.