The results of these six transcripts achieved what Fliess termed a “fair” level of agreement between two pairs of coders (Cohen’s kappa = 0.49 in both pairs) and “good” agreement in one (kappa = 0.65).19 There is no consensus on a minimal level of agreement in most if not all disciplines, but these numbers do fall below the kappa score of 0.787 in the Shachaf and Horowitz study. Partly because of this nontrivial level of disagreement between the coders, the normality of the distribution in the data cannot be assumed, and the data are treated as ordinal rather than ratio-level. The statistical analysis used to compare the data sets was then a nonparametric Kolmogorov-Smirnov test for two unrelated samples.20
Once the instrument was finalized, 106 transcripts from the previous studies were identified as having significant enough conversation and length to be appropriate for in-depth analysis of be-havior, and another 90 transcripts from the same time period were chosen as a control group. Each coder was assigned approximately one-third of both the test and control groups, and the transcripts were scored independently using the instrument. Analysis enabled comparison of all thirty-two measures, as well as the five larger dimensions that included subordinate levels.
Results
As has been demonstrated in other studies, these data show a relatively low level of adherence to the RUSA Guidelines.21 Of the five major aggregate categories, librarians serving both appropriately and inappropriately behaving patrons scored in the average range (between 3 and 4) on four dimensions, and librarians serving inappropriately behaving patrons scored below average on the “searching” dimension (4.0).
Comparing grouped median scores for those dimensions for which an ordinal scale was used, as well as the five broad areas, which are aggregates of all subordinate categories, shows that of these thirteen areas, librarians serving appropriately behaving patrons scored better on nine. The four in which librarians serving inappropriately behaving patrons scored better were 2.0, 2.6 (qualitative and quantitative), and 4.9.
The differences between these medians for most dimensions, using the Kolmogorov-Smirnov test, were not significant (see table 1).
Of those that did show significant differences (3.0, 3.1, 4.0, 4.5, and 4.9), librarians serving appropriately behaving patrons scored better in listening (3.0), cordiality (3.1), searching (4.0), and explaining how to use sources (4.5). Interestingly, librarians serving inappropriately behaving patrons scored better on 4.9: offering pointers to patrons.
Discussion
Although adherence to the RUSA Guidelines was only average for both groups of librarians, the fact that scores for librarians serving appropriately behaving patrons were significantly better than those serving inappropriately behaving patrons in listening/inquiring (3.0) and searching (4.0) is encouraging. This result could be interpreted as meaning that adherence to the RUSA Guidelines related to listening and searching yields an effective virtual reference transaction. However, it also could mean that the librarian serving the appropriately behaving patron is simply more likely to have an opportunity to conduct a successful reference interview and embark on a satisfactory search strategy. In Sample Transcript A (figure 1), the only communication that the librarian shares with the patron is a canned message from the service and the forwarding of two webpages. It should be emphasized that while the 3.0 category is titled “Listening/Inquiring,” a large part of the category is communication skills. Not surprisingly, this impersonal transcript devolved into an inappropriate transcript. However, this transcript is complicated by the fact that the librarian does not appear in the transcript until four minutes have passed for the patron. It is possible that the librarian felt rushed and thought that the best strategy for dealing with the patron was to merely send information as quickly as possible, rather than taking time for personal interaction. If the librarian had the opportunity to enter the interaction earlier, perhaps there would have been more positive communication from the librarian.
In Sample Transcript B (figure 2), the librarian is more effective at communicating with the patron. However, no information is ever shared with the patron. It is difficult to tell from a mere examination of the transcript whether this is because the librarian lacks knowledge of sources for this information or whether it is merely a fact that he or she has not had enough time to conduct a search. This study does not attempt to suggest a causal relationship between effective communication and searching by the librarian and appropriate behavior by the patron. However, the significant difference for listening (3.0), cordiality (3.1), searching (4.0), and explaining how to use sources (4.5) shows that there may be some type of connection between appropriate behavior and librarian application of these RUSA Guidelines.
Conversely, it appears that this study found adherence to the RUSA Guidelines on approachability (1.0), expressing interest (2.0), and follow-up (5.0) is inconsequential with respect to serving inappropriately behaving patrons. This result, to some extent, corroborates other findings. Kwon and Gregory, for instance, found that adherence to guidelines on welcoming, interest, and inquiring did not correlate to patron satisfaction.22 But Kwon and Gregory did find that using a patron’s name is related to satisfaction, whereas this study did not specifically consider the use of a patron’s name. Kwon and Gregory also separated listening from inquiring and found that listening was in fact correlated to satisfaction.23 Because adherence to the RUSA Guidelines is only partially helpful in satisfying patrons—and serving them even when they misbehave—perhaps speaks to the ineffectiveness of the guidelines in virtual settings. Indeed, Shachaf and Horowitz found that overall adherence to both RUSA’s and the International Federation of Library Association’s corresponding reference guidelines did not significantly correlate to patron satisfaction.24
Perhaps these initial studies, then, suggest two major implications for librarians practicing virtual reference and bodies that provide behavioral guidelines on that practice: (1) adherence to guidelines needs to be improved in virtual settings, especially with respect to conducting reference interviews and successful search strategies; and (2) the guidelines themselves are not well defined in some places and should be improved. If following guidelines does not assist librarians in satisfying patrons or in mitigating (or at least coping with) inappropriate behavior online, perhaps the guidelines need improvement.
The RUSA Guidelines that may need the most improvement lie in the outset of the interaction—approachability (1.0) in this study and welcoming in the Kwon and Gregory study— and in expressing interest (2.0). The RUSA Guidelines provide very detailed instructions on expressing interest in face-to-face settings (maintaining and reestablishing eye contact during the transaction), and though they also provide corollaries for virtual reference services (maintaining and reestablishing “word contact” with the patron), it could be that more specificity is necessary in this emerging form of service. Questions that arise from this example, and possible future research, would include: How often should word contact be initiated? What sort of language is most effective? How can a librarian compensate for the lack of nonverbal queues in virtual environments? Research is emerging that could inform such specificity, such as Radford’s promising work on interpersonal communication in chat reference.25 Understanding greeting and closing rituals, relational facilitators, nonverbal communication in verbal environments, and other factors is critical to providing good service online and to writing guidelines for it.