Qualitative research mandates that researchers remain open to additional categories that may emerge from the data during coding and analysis. In this process, researchers added two major additional categories for users’ actions and outcomes and also for users’ socioemotional comments on the digital reference interaction. In the area of user actions and outcomes, codes were added for: user-reported outcomes, user actions, willingness to return to the service, and willingness to recommend the service to others. In the area of socio-emotional content, codes were added for user comments regarding: emotion and affect, concerns aboutbothering the librarian,comments aboutthe librarian as a person,and comments aboutmaintaining contact with the librarian.
The first question to be answered in this research was whether user thank you messages contained any specific or evaluative user comments at all regarding the digital reference interaction. In analyzing the content of the messages, 157 of the 558 messages were found to contain no further information beyond basic expressions of “thanks” or “thank you.” In some of these messages, a flatter affect was observed (lower-case letters, no punctuation, no capitalization) while others appeared more emphatic through the use of upper case, multiple exclamation marks, magnifiers such as “thanks a hundredfold” or interjections such as “WOW!” However, since no other specific user comments on the digital reference interaction appeared in the messages, the researchers did not further explore these nondetailed messages.
For the remaining 71.9 percent of the thank you messages that included more detailed user feedback regarding their experiences, the researchers observed 367 user comments on answer-quality factors, 209 comments about user actions and outcomes, and 149 user comments on social or emotional aspects of the digital reference interaction.
Helpfulness or usefulness of the answer was the quality factor most often mentioned by users, with 148 users referring to this aspect of the answer in their thank you messages. Specific ways in which users described the helpfulness of answers included “getting started,” “direction,” and “strategies,” as well as “confirmation” and “verification.”
Expertise of the librarian was commended by sixty-nine users, who spoke of the librarian as “exemplary” and “professional,” and described the services received as “great research,” “invaluable help,” and “excellent service.” Praise for the librarian’s skill was also occasionally coupled with defensive or apologetic formulations such as “apologies for my lack of Web browsing skill,” “I didn’t look hard enough,” “I didn’t dig deep enough,” and “I didn’t allow enough time.”
Speed of the answer was the third most frequently mentioned answer-quality factor, with fifty-six users commenting on the quickness of the answer. Users described the answer as “prompt,” “timely,” and noted that it allowed them to move forward “without delay.”
Precision of the answer in meeting information needs was described by forty-four users who noted the answer was “right on target,” “just what I needed,” or “exactly what I was looking for.”
Completeness of the answer was noted by thirty-two users who described the digital reference answers they had received with terms such as “thorough,” “exhaustive,” “detailed,” “in-depth,” and “very complete.”
The two answer-quality factors least mentioned by digital reference users were “instruction,” involving showing techniques and methods by which information was found, and “clarity,” referring to clear and understandable communication. Instruction was mentioned by seventeen users who spoke of learning “how to find sources,” “how to do a better search,” “search terms,” and the ability to “use this approach in future.” Only one user mentioned Clarity as a factor in describing an answer as “accessible.”
In addition to commenting on quality factors in answers, users also provided feedback in their thank you messages regarding actions and outcomes. These included: user-reported outcomes, user actions, willingness to return to the service, and willingness to recommend the service to others.
Users discussed their actions or intended actions in response to answers in 104 of the 558 thank you messages. Actions described by users included plans to “follow up” and “take it forward,” “go to the library” and “get in touch” with referral sources.
Outcomes resulting from information received in the answers were described by sixty-five users. A wide range of outcomes were reported, including academic achievement, dispute resolution, completion of projects, and improved skills. Users completed speeches, projects, and reports; prepared for teaching; described how they “found the book,” “found the source,” “got an A+”; and reported success in “settling a family argument” and even in “baking a better pie.”
The user satisfaction measure of willingness to return has been previously tested in reference evaluation studies.12 Willingness to recommend the service to others similarly has been considered suggestive of user satisfaction. Users expressed willingness to return to the service in thirty-one of the thank you messages, while another nine users reported willingness to recommend the service to others. Users described having bookmarked the site or added the site to their Favorites, and commented on how the service had once again “come through” for them.
Users also commented in their thank you messages on the interpersonal interaction with the librarian and discussed their feelings about the interaction process. User feedback on the socio-emotional aspects of the digital reference interaction included comments about the librarian as a person in the interaction, affectivecomments about emotions and feelings, comments about maintaining contact with the librarian, and concerns about bothering the librarian.
E-mail has been described as a communications medium with reduced social presence cues in which a user’s awareness of another person in the interaction may be diminished, leading to more impersonal behavior.13 In this study, eighty users were observed to comment on the librarian in the digital reference interaction, describing the librarian as “kind and thoughtful,” “wonderful,” “considerate,” “an angel,” and “a saint.”
Users also expressed in the thank you messages their emotional reactions to the interaction and the answer received, with forty-seven users describing feelings such as being “very happy,” “thrilled,” “delighted,” and “pleased” and finding the digital reference process and results to be “fascinating” or “interesting.”
Some users also mentioned continued communication with the librarian in their thank you messages, with sixteen users expressing an interest in “staying in touch” or “letting you know” the final results of the information-seeking process. Also, six users expressed their concerns about bothering the librarian–a response that librarians often encounter in face-to-face reference services. These online users employed words such as “trouble,” “hassle,” “bother,” and “nuisance” in describing themselves and their digital reference questions.
Discussion
In this study, digital reference transcripts were analyzed to explore user perspectives regarding online interactions with librarians, and to better understand the phenomenon of user thank you messages in e-mail reference services. Users were observed to incorporate a variety of evaluative feedback in their thank you messages, including comments on outcomes, actions, answer-quality factors, and socio-emotional aspects of the interaction with the librarian. Transcripts for thanked and nonthanked e-mail interactions were also analyzed and compared.
Researchers noted an overall thank rate of 15.9 percent for e-mail questions answered by the service in 2002. This suggests the possibility of comparing thank rates across institutions, services, and user groups. If user satisfaction with digital reference services correlates with user thank you messages, these thank rates could be used as an outcomes measure for evaluation of digital reference services. Additional research with digital reference users is needed to determine whether a user thank you is an effective indicator of user satisfaction as well as success in meeting information needs.
Nearly half of the thanking users in this study were observed to have sent their thank you response on the same day that the answer was received, and more than 80 percent had responded by the third subsequent day. This raises the question of whether users have taken sufficient time to evaluate the answer before sending their thank you responses. Future research is needed to more fully evaluate whether a thanking user’s satisfaction with the librarian’s answer might change with the passage of additional time: How does time affect user satisfaction with digital reference services? The “imposed query” represents an additional area in which a user’s judgment could change as further feedback is received over time from the original questioner. In this study, sixty-four users (10 percent of the thanking and 6.7 percent of the non-thanking users) had volunteered the information that their question was asked on behalf of others, suggesting that the actual number of imposed queries among digital reference users could be higher.
Results of this study indicated that users most often cited helpfulness or usefulness as an important quality factor for digital reference answers, and that thanked librarians tended to use more words in answers. Standard answers or FAQ responses were also observed to receive fewer thank you responses from users. Additional research in exploring these dimensions of quality as users perceive them could provide valuable insights for digital reference service providers. Is a longer answer more likely to contain information that directly helps a user, or does the user perhaps perceive a longer answer as evidence of a librarian’s hard work? Are pre-written FAQ standard answers truly not considered helpful by users, or do other factors affect users’ responses to the answer?
Observations of users’ socio-emotional content in this study also raise questions regarding users’ perceptions and expectations of digital reference services. In a face-to-face reference desk setting where a librarian can be seen working on the computer or shuffling papers, it is not unusual for users to express concerns regarding bothering the librarian. However, in the online environment with no such visual cues present, users continued to express this concern as well as other defensive or apologetic formulations. Some users also referred to continued contact with their librarian, which is reminiscent of previous research by Southwick in which digital reference users described their preference for sending e-mail questions to a librarian whom they already knew.14 Future research involving interviews with digital reference users could provide useful clarification in context to further illuminate how users perceive digital reference interactions.
Conclusion
This research, which was funded by the 2003 OCLC/ALISE Research Grant, observed the thank you rate at an online digital reference service and explored the information behaviors of digital reference users and the range of evaluative feedback they provided to librarians in their thank you messages. A codebook developed for analyzing thank you messages is available for download at the Thank You Study Web site.15
The results of this study raised questions about the use of FAQ standard answer responses, suggesting that further study of the efficacy of these responses in meeting users’ needs would be of value. Higher thanking rates for librarians who used more words in their answers also raises questions as to whether users prefer longer answers, whether longer answers had a better chance of including the information that users needed, whether users perceived an extra effort from the librarians, or whether other factors such as short FAQ answers influenced results. The speed with which users sent back thanking responses also brings into question whether users had fully evaluated the answers before sending back their thank you message. Does a thank you indicate satisfaction from the user perspective, do users’ indications of satisfaction correlate with success in meeting information needs, and is there a potentially important distinction to be made here between users’ immediate versus long-term satisfaction with the answer received?
Additional areas of exploration for future research would include an examination of thank rates to ascertain whether these rates vary or are consistent among different types of services and institutions. A key need is to investigate thank you messages in conjunction with user interviews, in order to explore this phenomenon more fully from the user perspective and to further illuminate how users perceive and express their satisfaction with digital reference services. Bringing users’ perspectives into the assessment of services will enrich our understanding of online information-seeking and improve our ability to meet digital reference users’ needs in the future.
Lorri Mon is Assistant Professor, College of Information, Florida State University, Tallahassee. Joseph W. Janes is Associate Dean in the Information School of the University of Washington in Seattle.Submitted for publication February 17, 2005; accepted for publication October 12, 2005.
Detailed and precise answers do help the users, whereas a quick reply will not get any responses from the users. This happened with me recently when I filled out an online form to contact a doctor for an aunts treatment. I took out time to write an email, fill out the form and other requirements, whereas the answer I got was that ‘I am unable to understand your request’. The reply looked more like an auto generated response rather than a personl one where the doctor had spent time reviewing the patients submission entry. After reeceiving the response, I never bothered replying back to them, as to me it looked like a waste of time.