Feature

The Thank You Study: User Feedback in E-mail Thank You Messages

Lorri Mon and Joseph W. Janes

Print version (Adobe Reader required)
This exploratory study of unsolicited thank you messages from e-mail digital reference users analyzed the information provided in these messages for user perspectives on digital reference success, outcomes, and quality elements in answers. Digital reference interactions receiving thank you messages were also compared with nonthanked interactions. Results indicated that librarians who used more words in answers were more likely to receive a thank you response from users and that many other factors, such as e-mail or Web form use or the librarians’ expressing thanks to the user, did not appear to impact the thank you rate.

Librarians who answer questions in e-mail digital reference services are familiar with the intriguing phenomenon of the e-mail thank you message. In most question-answering interactions via e-mail, librarians send an e-mail answer to a user’s question and then never hear back again from the user, leaving the librarian wondering whether the answer was satisfactory or deficient in some way. However, occasionally the librarian receives a spontaneous, unsolicited e-mail thank you message from the user. Is it possible that these user thank you messages contain feedback that might benefit efforts in digital reference service evaluation, such as indications of digital reference interactions that were successful from the user perspective? This research examined thanked and nonthanked e-mail digital reference transcripts, and explored the textual content of users’ thank you messages in evaluating the feedback provided by users in their digital reference thank you messages.

The setting for this study was the Internet Public Library (IPL), an entirely virtual library based at the University of Michigan (UMich) that has provided e-mail question-answering services for users around the world since March 1995.¹ Questions are submitted to the service via e-mail or Web forms and are answered primarily by volunteer professional librarians and graduate students in librarianship training who participate from universities across the United States and around the world. IPL transcripts of e-mail digital reference interactions between users and librarians include the user’s initial question, the librarian’s answer, internal system notations such as time and date stamps, and any subsequent responses by the user or the librarian. From January through December 2002, the period for which transcripts were sampled in this research study, 5,400 questions were answered by IPL and thank you responses were received from 861 users–an overall thank you rate of 15.9 percent for the twelve months of 2002.

Literature Review

Although thousands of libraries are now actively engaged in online question-answering via chat, e-mail, instant messaging, and other forms of digital reference, the research field is still very new. An early call for libraries to experiment with e-mail appeared in 1981, and by the mid-1980s the first librarians were engaged in chat and e-mail digital reference services at libraries such as those at University of Washington and the University of Maryland, and at the librarian-staffed Winstar Telebase chat service for online users of Prodigy and other Internet providers.² However, it was not until ten years later that the first research study explored the nature of the e-mail digital reference interaction in depth.³

Digital reference research primarily has focused on the interaction as occurring between question submission by the user and answer transmission by the librarian. However, a 2000 study of e-mail interactions at IPL noted the phenomenon of subsequent e-mail thank you messages from users sent back to the service after the digital reference interaction was already completed. Out of more than 2,300 e-mail questions to IPL for January-March 1999, users were observed to have sent back to the librarians 458 subsequent e-mail thank you messages–an approximately 20 percent overall thank you rate.⁴

The research literature in computer-mediated communications suggests that a communications mode such as e-mail with reduced sensory cues lowers awareness of others in the interaction and tends to produce more impersonal behavior.⁵ In digital reference, the chat user who suddenly logs off during the interaction or the e-mail user who fails to respond to a clarification question might be seen as examples of the greater impersonality of reduced sensory cues in online communications. Thus, the 20 percent of e-mail users observed by Carter and Janes who made an extra effort to write back and thank the librarian even after the question-answering interaction was over may represent a potential window of insight into true digital reference user satisfaction. In providing a further exploration of users’ e-mail digital reference thank you messages, this study explores possibilities for using thank you messages in the evaluation of digital reference services.

Method

This study analyzed 810 e-mail transcripts from IPL, including 558 thanked and 252 nonthanked digital reference interactions. The primary focus of this study was to examine the properties of thanked interactions, but a smaller sample of nonthanked interactions was also drawn for comparative purposes. Systematic random sampling techniques were utilized to ensure that random proportional samples of both thanked and nonthanked digital reference transcripts were drawn from each month in the dataset for January-December 2002. In phase 1 of the research study, the transcripts for thanked and nonthanked e-mail digital reference interactions were analyzed and compared for a variety of quantitative factors including answer speed, information provided by users, and response length by librarians. Contingency tables and chi-square statistics were utilized in determining the statistical significance of differences observed between thanked and nonthanked digital reference interactions. Where statistically significant differences were found, the Cramer’s V statistic was calculated in order to test the strength of association between the variables.⁶ The Cramer’s V statistic was used as a measure of association strength because it norms for sensitivity of the chi-square test to large sample sizes.

In phase 2 of the research study, the sample of 558 thank you messages was also examined using qualitative coding for users’ comments and assessments regarding their digital reference experience. A codebook for user-identified quality factors in digital reference service was designed based upon an extensive literature review of the research on reference evaluation and reference assessment. Intercoder reliability testing for the codebook was conducted during August and September 2003, and Cohen’s Kappa was computed to correct for chance agreement, finding scores at or above the 70-percent satisfactory level for all three intercoding sessions (.70, .83, and .77).⁷ Grounded theory was also used in open coding for themes and concepts emerging from the data.⁸

Results: Phase 1

The portrait of IPL users emerging from the first phase of analysis of the thanked and nonthanked transcripts indicates that the majority of users in the 2002 sample who described reasons for asking questions indicated that their information-seeking was not related to a school or academic assignment. Among the nonthanked sample, the proportion of education-related assignments was higher, possibly reflective of the lower thank you rate observed in the Carter and Janes study in 2000 for questions submitted through IPL’s youth question form, but this difference in thanking rates for academic and non-academic questions was not statistically significant.

In textual comments regarding their reasons for asking questions, users reported their reasons for information-seeking as including projects in writing or speaking, such as reports, articles, essays, presentations, and teaching; desires to acquire itemsby purchase or loan; and other-directed activities such as dispute resolution or helping others. Imposed queries, or questions being asked in order to gather information for other people, were reported by 10.2 percent of the thanking users (n=57) and 6.7 percent of the non-thanking users (n=17).⁹ Users described seeking information on behalf of friends and family members, including geographically distant acquaintances.

The majority of both thanking users (n=386, or 69.2 percent) and non-thanking users (n=183, or 72.6 percent) chose to submit their digital reference questions through IPL’s Web-based question form, although a “plain e-mail option” was also available. Rates for thank you responses in the sampled transcripts were similar, showing no statistically significant differences for users submitting questions via plain e-mail or Web forms.

IPL’s Web form is more detailed than forms used by most public and academic libraries and is designed to ask up-front the full range of questions recommended by Robert Taylor including question subject, user’s goals and motivation, personal details, and preferences for answer types and formats.¹⁰ In submitting and negotiating their questions with digital reference librarians, more than 60 percent of both the thanking and non-thanking users provided information equivalent to answering six or more of the fields on IPL’s Web form. No statistically significant differences were observed in the extent to which the thanking and non-thanking users provided information to the librarians about their questions.

Users specified a deadline date in about half of the questions, and librarians were able to meet the requested deadlines in most of these cases (approximately 96 percent for both thanking and non-thanking users.) Differences in the patterns of deadlines requested by thanking and non-thanking users were not statistically significant. The majority of users specifying a deadline also indicated willingness to wait for at least one week for their e-mail answers. Expectations for speed of answer from the other half of the digital reference users who did not specify a preferred response time are unknown.

Overall, it was found that for both thanking and non-thanking users, at least half of the answers had been sent by librarians within the first three days. For thanking users, 50.2 percent had been sent an answer by the third day after submitting a question, while 57.5 percent of non-thanking users had been sent an answer in the same period. Differences in thanking versus non-thanking user behavior by answer speed were found to be statistically significant. To explore the strength of the relationship between thanking behavior and answer speed, the Cramer’s V statistic was calculated, in which a finding of .10-.19 would indicate a weak relationship, .20-.39 would indicate a moderate relationship, .40-.59 would indicate a relatively strong relationship and .60 or above would indicate a strong relationship.¹¹ For answer speed and thanking behavior, the Cramer’s V of .16 suggests a weak association between these variables.

Users were also observed to have a greater tendency to thank librarians who avoided using prewritten standard answers or FAQ responses. IPL has standard answer FAQ responses for some commonly asked questions that librarians can copy into an answer. In this study, twenty-five of thirty-four standard answer FAQ responses or three-fourths of all the standard answers failed to receive a thank you message from users. IPL uses subject codes to classify answer types, with FAQ answers as one subject-code grouping. In comparing the FAQ answer group (n=34) to three other answer subject-code groups of similar size in the sample, Humanities (n=39), Biography (n=47), and Business (n=50), statistically significant differences were found between thanking and non-thanking user behaviors with a relatively strong association to the subject-code answer type.

This is an intriguing result as FAQ or standard answer responses are very common among busy digital reference services. These FAQ answers also are often sent out quickly by digital reference service administrators as part of the initial triage process of sorting through incoming questions. In the Thank You Study sample, 34 of 810 answers were FAQ standard answers, while in general during 2002 at IPL, more than 600 questions, or approximately 11 percent of the 5,400 questions answered had received FAQ responses.

Users also appeared to have a greater tendency to thank librarians who used more words in their answers. More than half of the thanked librarians had exceeded two hundred words in their answers, as compared to only about one-third of the nonthanked librarians.

The difference in thanking behavior was found to be statistically significant, but with a weak association to the number of words used by the librarian. Other factors (for example, short FAQ answers with lower numbers of librarian words and other issues in the content of librarian responses) may play a role in these results.

As a side note, it was observed that most of the users expressed their questions in two hundred words or less, with only forty-one thanking users and only three non-thanking users exceeding two hundred words. The smallest number of words observed in a user’s question was eight words in a question submitted by plain e-mail rather than the Web form, and the smallest number of words used by a librarian was thirty-four words in a nonthanked original answer.

Users generally responded quickly with their thank you messages upon receiving the librarian’s answer. Nearly half of the thanking users responded with their thanks on the same day that the answer was sent, and another 30 percent responded the following day. Within three days from the time that the librarian’s answer was sent out, 85 percent of the users had sent back their thank you messages, a result that raises questions as to whether users have thoroughly evaluated answers before sending back the thank you response.

Librarians thanking users in their answers (e.g., “thank you for your question”) did not appear to have an impact on the thanking rate. Approximately 90 percent of both the thanked and nonthanked librarians were found to have used some variant of thanks or thank you in their answers to the users. This high number of librarians thanking users in their answers is consistent with IPLs proposed, and generally followed policy and guidelines for providing a consistent and friendly style of answers, including a recommended answer structure that incorporates thanks to the user.

Results: Phase 2

In the second phase of the study, the 558 user thank you messages were analyzed for specific comments about the digital reference interaction. Through an extensive review of the literature in reference assessment and reference evaluation, the researchers developed a list of possible digital reference quality factors and prepared and tested a codebook for qualitative coding of users’ textual comments. The quality factors developed from the literature review and intercoder reliability testing were: clarity, completeness, expertise, helpfulness, instruction, precision, and speed.

Pages: 1 2 3

Comments (One comment)

Post a comment