Making Unmediated Access to E-Resources a Reality: Creating a Usable ERM Interface

Evaluation: Rounds Two and Three

Following redesign, two more rounds of usability testing were conducted, with two pretests conducted between rounds. The pretests were used to assess whether changes to the site improved user performance (and therefore warranted a full round of testing) or whether further changes to the site were needed. Round two involved four participants— two undergraduate students, one graduate student, and one faculty member. Results from this round showed a radical improvement in participant performance. The main subject display and the tabular access to “By Database Name” and “Not Sure Where to Start?” were intuitive to users. The pared-down lists of databases by subject area and database descriptions also proved a success, enabling users to select an appropriate database for the given topic most of the time. Additionally, allowing direct access to the database from the subject list was a resounding success. During Round 2, a total of twenty-eight tasks were presented to participants. Only four of those (14 percent) were not completed, and another four were completed with difficulty (14 percent). Twenty (71 percent) were completed easily. In all, twenty-four tasks (86 percent) were completed, whether easily or with difficulty.

We identified a few areas still in need of improvement, which we attempted to correct while being cautious not to make changes that would reduce the efficacy of the components that had tested well:

• Access to formats. Several popular nonarticle, multidisciplinary formats had been added to the subject display. We had thought our users would seek out format types within the unified alphabetical arrangement, yet, when asked to find a newspaper article on the use of performance-enhancing drugs in sports, all four participants looked under an academic subject (i.e., sports, medicine, and nutrition). It may have been possible that users believed newspapers were included in the subject databases, but when questioned posttest, several indicated that newspapers did not fit in with the other subjects. To remedy the situation, we broke out format databases into a horizontal list beneath the academic subjects.
• “Not Sure Where to Start?” screen. The “Not Sure Where to Start?” tab was composed primarily of links to a small selection of multidisciplinary databases. Two undergraduate students clicked on this tab. For one, expectations were met while the other lamented, “I didn’t expect this.” The team revised this screen slightly, in part because of the comments of the dissatisfied student, but also because of our own lack of confidence that the screen supplied the necessary support for our user base. A new screen continues to provide access to the general databases, but also provides an overview of what the Research Database Locator is and how users could get research support from library staff.
• Links to subtopics. As an affordance to users, links to subtopics were provided within the subject display and were repeated in a sidebar on the databases by subject pages. The sidebar also included links to subject liaisons and guides. In both places, links to subtopics and the sidebar lacked visibility. As a corrective measure, we made graphical design changes that gave them more visual force.

The third and final round of testing involved four undergraduate students (one from a regional campus) and one graduate student. The results from the test were very similar to those of the earlier round. Of the forty tasks presented to participants, seven (18 percent) were not completed, eleven (28 percent) were completed with difficulty, and twenty-two (55 percent) were completed easily. In total, thirty-three tasks (83 percent) were completed, whether easily or with difficulty. Small improvements were seen in navigation of formats and the visibility of the side bar. “Not Sure Where to Start?” continued to disappoint, but this time for different reasons: We had designed a full screen of information, which we thought was brief and well-segmented into two columns; however, we found that users were only perusing the left column.

Following the third round of testing, we made more graphical changes to increase the effect of the formats, sidebar, and to further edit and rearrange “Not Sure Where to Start?” Edits to the “Not Sure Where to Start?” page included reducing the amount of text, placing the most critical information in the upper-left corner, and adding three screencasts for basic operation, advanced features, and how to use the Virtual Private Network for off-campus access.

Current Status and Future Plans

During UConn’s spring break in March 2007, the new database locator was rolled out to the UConn Library’s website. See figure 4 (redesigned interface), figure 5 (redesigned “Not Sure Where to Start” page), figure 6 (sample “best bets” page for business), or visit our website. The collective response to the redesigned site has been extremely positive, and we are pleased with the results. However, as with any iterative design process, the cycle of testing and improving does not naturally come to an end; there are always more problems to discover and more improvements to be made. To that end, we have created a wish list of items for future development that address known shortcomings and introduce new functions. These include improved visibility of navigational elements (formats and sidebar), improved effectiveness of the “Not Sure Where to Start?” page, cross-referencing between subjects, a search box for database descriptions, a mechanism to redirect commonly misspelled database names (e.g., PsycINFO and LexisNexis), database descriptions that can be customized for each subject list, and a “My Databases” feature that allows expert users to create pages with their favorite databases.

The one-year redesign process was time-consuming, occasionally overwhelming, and left us with more to be done; however, we are certain that the iterative design approach was essential for the project’s success. A great many websites are expertly and attractively designed, following established design principles and with knowledge of and input from users. Yet, when launched, these sites may still fail. The iterative design process recognizes that usability can only be determined by users. It makes failure, though not ever welcome, beneficial by allowing it to occur in a testing environment where problems can be identified and later corrected. Our experience has borne out that well-planned sites, which seem intuitive to librarians, may simply not work for our users. At the UConn Libraries, we were able to create a more user-friendly and accessible database locator by partnering with our users in a test-to-success process. The process required a heavy investment from staff, but the feedback we have received thus far from patrons and staff alike has made it well worth the effort.

Kate Fuller is Learning Commons Research Coordinator, Jill Livingston is Reference Librarian and Liaison to Allied Health and Kinesiology, Stephanie Willen Brown is Electronic Resources Librarian and Liaison to Communications, Susanna Cowan is Undergraduate Education and Outreach Librarian, and Thomas Wood is Applications Developer, Homer D. Babbidge Library, University of Connecticut Libraries, Storrs. Leslie Porter is User Education Librarian, Agnes Scott College, Decatur, Georgia. Submitted for review October 3, 2007; revised and accepted for publication March 14, 2008.

Reference & User Services Quarterly,
vol. 48, no. 3, pp. 287–301 © 2009 American Library Association. All rights reserved. Permission granted to reproduce for nonprofit, educational use.

References and Notes

1. Martha Kyrillidou and Mark Young, eds., ARL Statistics 2004–05: A Compilation of Statistics from the One Hundred and Twenty-Three Members of the Association of Research Libraries (Washington, D.C.: Association of Research Libraries, 2006), www.arl.org/stats/annual surveys/arlstats (accessed June 12, 2007); Elaine Kroe et al., Data File, Public Use: Public Libraries Survey: Fiscal Year 2004 (Washington, D.C.: U.S. Department of Education, National Center for Education Statistics, 2006) (accessed June 12, 2007).
2. The UConn Libraries maintains SFX as an OpenURL resolver as well as Serials Solutions for both it’s A–Z journal list and for adding MARC records of journals into the catalog; the ERM is not a comprehensive list of our electronic journal holdings.
3. Jeffrey Rubin, The Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (New York: Wiley, 1994).
4. Jakob Nielsen, “Alertbox: Current Issues in Web Usability” (accessed June 12, 2007).
5. Darlene Fichter, “The Age of Darwinian Design,” Online 31, no. 2 (2007): 52–53.
6. Laura Cobus, Valeda Frances Dent, and Anita Ondrusek, “How Twenty-Eight Users Helped Redesign an Academic Library Web Site: A Usability Study,” Reference & User Services Quarterly 44, no. 3 (2005): 232–46.
7. Brenda Battleson, Austin Booth, and Jane Weintrop, “Usability Testing of an Academic Library Web Site: A Case Study,” Journal of Academic Librarianship 27, no. 3 (2001): 188–99.
8. Laura Manzari and Jeremiah Trinidad-Christensen, “User-Centered Design of a Web Site for Library and Information Science Students: Heuristic Evaluation and Usability Testing,” Information Technology & Libraries 25, no. 3 (2006): 163–69.
9. Jennifer L. Ward, “Web Site Redesign: The University of Washington Libraries’ Experience,” Systems & Services 22, no. 3 (2006): 207–16; Dawn Holland, “Practical Experiences of Using Formal Usability Testing as Tool to Support Website Redesign,” SCONUL Focus 36 (Winter 2005): 31–35; Robert L. Tolliver et al., “Website Redesign and Testing with a Usability Consultant: Lessons Learned,” OCLC Systems & Services 21, no. 3 (2005): 156–66; Carole A. George, “Usability Testing and Design of a Library Website: An Iterative Approach,” OCLC Systems & Services 21, no. 3 (2005): 167–80; Leanne M. VandeCreek, “Usability Analysis of Northern Illinois University Libraries’

   Website: A Case Study,” OCLC Systems & Services 21, no. 3 (2005): 181–92; Dominique Turnbow et al., “Usability Testing for Web Redesign: A Case Study,” OCLC Systems & Services 21, no. 3 (2005): 226–34; Heather Jeffcoat King and Catherine M. Jannik, “Redesigning for Usability: Information Architecture and Usability Testing for Georgia Tech Library’s Website,” OCLC Systems & Services 21, no. 3 (2005): 235–43; and Jennifer L. Ward and Steve Hiller, “Usability Testing, Interface Design, and Portals,” Journal of Library Administration 43, no. 1–2 (2005): 155–71.

10. See for example the articles at Jakob Nielsen’s “Alertbox: Current Issues in Web Usability,” a column published within his website, useit.com: “Usability 101: Introduction to Usability” (accessed July11, 2007); “Top 10 Mistakes in Web Design” (accessed July 11, 2007). Nielsen’s “top ten” first appeared in a chapter of a book he coedited with Robert Mack: Jakob Nielsen, “Heuristic Evaluation” Usability Inspection Methods, ed. Jakob Nielsen and Robert L. Mack (New York: Wiley, 1994): 25–62. The list itself (not the chapter) is reproduced at Alertbox: Jakob Nielsen, “Ten Usability Heuristics” (accessed July 11, 2007).
11. John Kupersmith, “Library Terms That Users Understand” (accessed July 11, 2007).
12. Nielsen, “Top 10 Mistakes in Web Design.”
13. Kupersmith, “Library Terms That Users Understand.”
14. Not all actions are searches. We logged what the users did from this page: ran a keyword search, linked to a title letter, or chose a subject from the category pull-down menus.
15. We cannot say how many of these knowledgeable users are library staff and other “expert” users. We did not limit our collection of usage data to nonstaff IP’s. It is hard to say at this point whether librarian use of the interface significantly affected these numbers.
16. Jakob Nielsen, “Quantitative Studies: How Many Users to Test?” (accessed July 23, 2007).

Appendix. Questions Used in Testing

[back to text]

Preliminary Questions

1. Please indicate your status at UConn.
   • Undergraduate student
   • Graduate/Post-graduate student
   • Faculty Member
2. What is your home campus?
   • Storrs
   • Regional
3. How long have you been at UConn?
   • Less than one year
   • One–two years
   • More than two years
4. Are you now, or have you ever been, employed by the UConn Libraries? n I am currently or was formerly employed by the Library n I have never been employed by the Library
5. Are you at least 18 years old?
6. What is your major or department?
7. Which of the following best describes your experience with the UConn Library website:
   • I never use it
   • I seldom/hardly ever use it
   • I use it on average once a month
   • I use it once a week or more
8. Have you ever participated in a library instruction class?

Session Questions

1. What does that word “databases” mean to you? Talk about what you think it means or where you have seen it before. What do you think is the overall purpose of this page?
2. Your nursing professor asked you to find articles on diabetes, your hypothetical patient’s condition. What would you do from this page? Alternate question for graduate students/faculty: Your friend has recently been diagnosed with diabetes and you want to research her condition. What would you do from this page?
3. You’ve got to do research for one of your classes on the topic of college students and depression. Where would you start from this page? Alternate question for graduate students/faculty: You’ve got to do research on the topic of college students and depression. Where would you start from this page?
4. Search for a database named ERIC (scholarly articles in education) and connect to it. (Ask: why did you select that particular ERIC?)
5. You’re doing research on the history of Apartheid in South Africa. Where would you start from here?
6. Find a list of databases by subject and choose biology. Specifically, you need to find some articles in the field of biopharmacology. Where would you go?
7. You’re looking for a newspaper article on the use of performance-enhancing drugs in sports. From this page, what would you do?
8. You need to find a newspaper article written on the day you were born. Where would you go/what would you do from this page?
9. Have you written a paper or article [depending on participant; use “article” for faculty] recently?
   • What was the topic?
   • What sources did you use, if you remember?
   • What did you use to find sources?
   • [If they didn’t use the Research Database Locator, ask ... ] Now that you’ve seen this, how would you have used it for your topic?

Post–Evaluation Questions

1. What is a good name for this resource that you’ve just used? What are some terms you’d use to describe it?
2. If you had to tell a friend or colleague what this site does, what would you say?
3. How easy is it to find information resources from this site?
   Not easy Very easy
   1 2 3 4 5
4. How would you rate your experience using the site?
   Frustrating Very pleasant
   1 2 3 4 5
5. What else did you like or not like with this site?
6. What suggestions do you have for improvement?
7. [If user tried “Not sure where to start”] You used the page “Not sure where to start?” What did you expect to find there?
   [If user did not use “Not sure where to start”] There is a part of the site called, “Not sure where to start?” What would you expect to find there?
8. Would you use this site again? Why or why not?
9. Do you have any additional comments or questions about this study?

[back to text]

Pages: 1 2 3 4 5