The transformation of the existing three questionnaire items into a composite variable involved a series of conversions from a natural language answer choice to a numeric value. By carefully analyzing the wording of the answer choices, a set of logical numeric values were assigned for answer choices of each of the three questionnaire items on the same five-point Likert scale. First, for the satisfaction with the answer item, the numeric value assigned for “unsatisfied” was 1, 2 for “not sure,” and 5 for “satisfied.” For the perceived staff quality item, the value assigned for “poor” was 1, 3 for “average,” 4 for “good,” and 5 for “excellent.” For the willingness to return item, the numeric value assigned for “never” was 1, 3 for “maybe,” and 5 for “very likely.” This value assignment procedure is presented in figure 6.16
In order to determine the composite variable’s reliability and validity, a classical theory alpha reliability and factor analysis were performed, respectively. First, the reliability test generated a score for the sample that had a Cronbach’s alpha reliability coefficient (a) of .845 for user satisfaction. This score is greater than the customary threshold value of .70, indicating that the composite variable is reliable. Second, the factor analysis result showed that the three questionnaire items were loaded on one-factor solution, with factor loading scores of .868, .916, and .876. All three items explained 78.64 percent of variance in the factor. This result indicates that the composite variable is measuring a single construct with a high score validity, suggesting that the composite variable is a valid measure of user satisfaction.17 The mean of the composite variable, user satisfaction, was 12.69, with a standard deviation of 3.44 in the range between a maximum value of 3 for “highly dissatisfied” and the minimum value of 15 for “highly satisfied.” This composite variable was used to examine the relationship between the use of RUSA behaviors and user satisfaction below.
The Presence of RUSA Behaviors in Chat Reference Interviews
Research question 1 of the present study asks to what extent reference staff members employ each of the ten RUSA behaviors during their chat reference transactions. Figure 7 presents the findings.
As shown in figure 7, the presence of the ten selected RUSA behaviors ranged between 28.7 percent and 63.7 percent (N = 422). The three most frequently observed behaviors were come-back (63.7 percent), interest (63.0 percent), and welcoming (53.8 percent). A complete comparison of this finding with those reported in the previous studies is not feasible because none has used the complete revised guidelines as a coding scheme. A couple of studies allow a partial comparison in the area of follow-up. According to Ross and Dewdney’s research, follow-up behaviors were observed in approximately one-third of the total in-person reference transactions.18 Also in the physical library setting, Gatten and Radcliff reported that 29 percent of the proxy patrons were asked by the librarians if their questions were answered.19 In the present study, however, the three follow-up behaviors appeared with greater frequency, ranging between 46.9 percent and 63.7 percent. Their frequent appearance may reflect the use of the scripted words stored in the chat reference software program (for example, “If you need further assistance, please contact us again”).
In reviewing the above findings, it should be noted that not all RUSA behaviors are expected to appear in a single reference session. For example, probing is not generally expected to occur in response to directional or circulation policy questions. Similarly, offering alternatives or making referrals are expected to occur mostly when reference staff is not able to provide a complete answer. Therefore, behaviors with a low frequency in figure 6 do not necessarily indicate inadequate behavioral performance on the part of reference staff. Rather, the information about the RUSA behaviors should be more useful in its relationship with user satisfaction, which is described below.
RUSA Behaviors and User Satisfaction
Research question 2 relates to whether chat reference users would be more satisfied with the service when librarians demonstrate a RUSA behavior than when they do not. The findings are presented in figure 8, where the means of the user satisfaction are compared between when librarians showed the behavior and when librarians did not.
Across all but two behaviors, user satisfaction was found to be higher when librarians performed the RUSA behaviors than when they did not. A series of statistical analyses was conducted to determine the statistical significance of the difference in user satisfaction between when each RUSA behavior was present and absent. Both parametric statistics (specifically, independent two-group t-tests) and non-parametric statistics (specifically, the Mann-Whitney U test) were considered in our study to determine the most appropriate statistical technique for the tests. In general, parametric statistics are recommended when a test variable: (1) is measured on an interval or ratio level; (2) is measured on a sample size of sixty cases or more; or (3) is normally distributed. When the data was examined with respect to these three criteria, the answer was not clear-cut. The test variable, user satisfaction, is an interval-level variable measured on the sample size of 422, but is not normally distributed.
Hence, the data were examined using both parametric and non-parametric statistical techniques, and the test results were compared. If the two statistical techniques generate identical results, parametric statistics are recommended to present the data, which is the case in the present study.20 When tested using SPSS 14.0, it was revealed that test results from both techniques were identical across all ten RUSA behaviors. This result indicates that the Mann-Whitney U test results corroborated the results from the t-tests. The findings are presented in table 1.
[...] Behavioral Performance on User Satisfaction in Chat Reference Services (pdf) (source: RUSQ via Stephen’s [...]