Before administering the test with our representative sample of users, we conducted two pretests with undergraduate students to ensure the test and our administration methods were on target. Based on what worked and what did not, the test and procedures were revised.
Selection and Recruitment of Participants
In 2006, UConn had a total enrollment of 28,481 students over six campuses; nearly 75 percent of that number was undergraduate enrollment. UConn offers 8 undergraduate degrees in 105 majors, 16 graduate degrees for 90 fields of study, and 5 professional degree programs. In addition, UConn employs nearly 4,500 faculty. To be sure we were gathering accurate data, we were careful to use a representative sample of our users rather than a random sample. As often as possible, we would recruit three undergraduate students, one graduate student, and one faculty member for each test, ideally from various academic departments in the humanities and sciences. We also aimed to hold tests at as many of UConn’s six campuses as possible.
The team solicited volunteers through e-mail messages sent to faculty and graduate student distribution lists on campus. The response from graduate students was resounding; faculty were not as responsive, but we still received a sufficient number of volunteers. Undergraduate students were recruited on the fly from the café adjoining the library and from library computer areas just prior to each test. These locations are popular social areas that draw undergraduate students who otherwise do not use library resources or tools. In fact, many of the undergraduate volunteers had never used or seen the database locator before the test.
It was imperative to our goals that library staff be involved in the testing process to see firsthand how users navigated our database locator. Many librarians and staff teach classes using the database locator and are involved in populating the back-end. Watching students and faculty use the tool without librarian mediation would be important to understanding the inherent problems with the existing interface. Volunteers from the library staff were recruited to assist in the test process. One member from the testing team would administer the test and serve as a contact and recruiter for participants. Staff volunteers would record everything the tester said or did (e.g., mouse clicks, keystrokes) during the test, and one person would keep track of the amount of time each task took to complete. As an incentive, $10 gift certificates to the university bookstore were given to participants.
Performing Formative Evaluation
The first test was conducted on the existing research database locator that had been in use for over a year. As illustrated in figure 1, users were presented with a screen that gave them several choices for locating a database. We were interested in observing how users navigated these choices, how (or if) they found an appropriate database for their topic, and if they were able to expand a specific topic out to a broad subject or format type. We also wanted to ensure that they were able to access the appropriate database once they identified it. This first round of testing was conducted at the Storrs (main) campus with five participants—three undergraduate students, one graduate student, and one faculty member.
Following this first round, we saw several common problems with the interface. The keyword search caused the most confusion and frustration for the majority of the testers, even though this search was the most attractive method for answering the task questions—it appeared first on the page and was highlighted in a colored box. For users unfamiliar with database behavior, specific database titles, or just looking for the quickest and most direct way to retrieve information, the keyword search was the first method attempted, despite its low rate of success in producing useful results. Terms entered in the keyword search box were run against database titles, descriptions, and subjects, but this was not clearly noted; searches on topics or misspelled databases resulted in a completely blank screen. This type of limited keyword search ran counterintuitive to participants’ expectations of a broad, Google-like keyword search and resulted in failed searches and palpable frustration. Participants consistently tried to search using keywords directly related to their topic rather than by broad subject, not understanding that a database must be selected before searching. We repeatedly saw them type in “diabetes,” “depression,” or “apartheid” in the keyword search box to answer the test questions and retrieve no results.
Participants attempted to overcome these problems by using the “subject grouping” option, which consisted of four pull-down menus labeled for broad disciplines. Participants were generally able to expand from the topic (diabetes) to a broader subject (medicine); however, they often had difficulty finding the appropriate subject page from the pull-down menus. One completely missed seeing the link for Medicine, Nursing, and Allied Health due to the way the list of subjects displayed under the Sciences pull-down. A more serious problem was that users were unable to guess which subject grouping their topic would fall under; users did not know if psychology would fall under Science or Social Science. They often perused the lists of two to three subject grouping pull-downs before finding a topic that made sense to them.
Another significant problem was that once participants were successful in navigating to a subject, the number of available databases listed related to that subject was overwhelming. We observed participants staring intently at a screen with more than twenty databases displayed, clearly unsure which to choose. In addition, the descriptions of the individual databases included too much text. Users frequently commented that it was too difficult to read the descriptions or that they were scanning for relevant material. Regarding text on the page, one said “cut the words down. I just scan them [the descriptions]. I don’t actually read … when there are paragraphs to read.” The participant commented, “I will say it would be easier to use a different Web. It’s not like I have time to figure out where it would be.”
Once they had identified a database, users were brought to a description page from which they could access the database itself. Many participants were unaware that this intermediary page was not the database itself and looked fruitlessly for a way to search, or were unsuccessful in finding the access point—a small button at the top of the screen reading “Access this Database.” While attempting to find a link to the database, several users scrolled to the bottom of the database description page, where, still looking for the access point to the database, they clicked on links to subject guides. The subject guides were a quagmire, offering the user additional lists of databases related to the subject area.
Overall, testers failed to complete almost half the tasks during this first round. Undergraduates fared the worst with this design, failing to complete the tasks given to them 66 percent of the time; the faculty member missed only one question and the graduate student was able to complete all the tasks during this first round—perhaps because they were familiar with conducting research at the library. However, across the board, 43 percent of the tasks were not completed and 29 percent were completed with difficulty. Only 29 percent were completed easily.
After analyzing the results from the test on the existing interface, we identified several common problems with our database locator:
- The keyword search box was mistakenly be ing used as a Google-like tool; users typed in specific topics rather than database names or subject categories.
- Typing a database title in the keyword search box often did not bring up that database first (e.g., ERIC appeared nineteenth in the list of databases generated).
- The subject pull-down menus were not intuitive.
- The number of databases listed in many subject results were so long as to be confusing.
- Database descriptions were too long for users to easily scan; they weren’t easily able to identify a relevant database from skimming text.
- Users did not know how to access the database once they found it.
Quick Fixes
The rate of failure for the existing database locator interface was so high within the undergraduate testing group that we developed a set of provisional quick fixes (see figures 2 and 3) we hoped to roll out immediately to improve the patron experience until the formal redesign process was completed. The changes were largely cosmetic because we did not want to make drastic alterations to the functionality of the product without further testing. Many of the common problems detailed in the previous section were addressed:
- General improvements to the visual design were made by removing extraneous marks (borders, lines, icons, colors, etc.) and adjust ing whitespace.
- The “Keyword” search label was changed to
“Search For Databases.” - Hyperlinked examples of successful keyword searches were included below the search box.
- A strategy guidance page was created to help locate resources if a keyword search produced no results.
- “Access this Database” image was replaced with a bright green “GO” button.
- A search box was added at the top of each subject page.