Assessment Partnering on Locally Based Surveys
Saunders points out that LibQUAL+ contains a “tension between the need for local information and the standardized information provided by the survey.”15 Indeed, surveys like LibQUAL+, while powerful and highly standardized, may not conform to the needs of a specific campus or library environment. It is to a library’s advantage to gain a more locally relevant picture of student perceptions and needs. Optimally, a local survey could be designed to easily correlate with national findings.
In two instances in recent years, the Penn State Libraries found that partnering with others has enabled a better understanding of student and faculty use of library resources. Through partnerships with Information Technology Services (ITS) and the Office of Student Affairs Research and Assessment, the Libraries have been able to take the pulse of our students in a survey devoted exclusively to library topics and in another survey in which the Libraries’ questions were integrated with technology questions. One incentive to survey students was to clarify how our students were doing with regard to library use and acquisition of information literacy. Recent national surveys indicated that people in general, and students in particular, were not using libraries and library resources as much as they had in the past; in fact, they were using them at a very low rate. These reports created significant discussions about the future of libraries. But, in examining the results, it seemed that there was considerable ambiguity in at least some of the questions. Were the surveys asking the questions so the responder knew what they meant? Were the results a reflection of an actual problem? While the Libraries could not answer the questions on the scale of the national survey, we could ask questions our own way and discover to what extent students were using our webpages, databases, and online resources. Penn State ITS has a history of surveying students and faculty with regard to campus IT issues and trends. Two surveys in particular—the Penn State Pulse surveys and the Faculty Advisory Committee on Academic Computing (FACAC) survey of Penn State faculty and students—have yielded powerful longitudinal data comparisons for nineteen Penn State campuses and provided an opportunity to ask library-focused questions.
The Penn State Pulse Surveys
The Pulse surveys began at Penn State in 1995 as a method for acquiring student feedback, usage trends, views, and behaviors. As of May 2008, 156 Pulse surveys have been administered to Penn State students by the Office of Student Affairs, Research and Assessment. The surveys are conducted by phone or online, and a typical survey draws several thousand valid responses from students across Penn State’s nineteen campuses. In 2005, a Pulse survey on information literacy was conducted in partnership with the Penn State Libraries, garnering 2,003 responses from undergraduate students.16 The survey focused on students’ knowledge of the Libraries’ services, collections, and resources, as well as on their information-seeking behaviors. Questions were developed by two librarians (the head of Library Learning Services and the head of Public Services), the senior director of Teaching and Learning with Technology, and the head of the assessment team from the Office of Student Affairs, Research and Assessment. The team allowed the librarians to create the direction and content for the questions, which was done in consultation with other librarians. Rough questions and areas were then discussed as a group to look for things such as broad understanding and clarity of question content and wording. The advice provided through this consultative process was invaluable and resulted in questions that were asked differently than we librarians might have asked them but were clear enough to elicit the information desired from responding students.
The results of the survey highlighted Penn State students’ self-reported achievement of information literacy competencies (88 percent felt confident in their ability to find and retrieve information, 73 percent were familiar with library guides and databases, and 81 percent felt able to find needed library resources), research competencies (88 percent indicated the ability to assess the validity and authority of Web information), use of specific library resources, and exposure to a library course-related instructor (over 70 percent reported having a librarian as a guest lecturer in class).
The FACAC Survey
In place since 2001, Penn State’s FACAC survey of technology use has targeted a variety of different campus technology users—students, TAs, faculty, and staff. The survey has grown to a response pool of nearly three thousand users. When the FACAC survey began, the focus was entirely on technology-related questions, including the use of computers, adequacy of campus technical support, and preferred methods of technology training. Since then, the terrain covered by the survey has grown to include the use of electronic and mobile devices, frequency of specific Web-based activities, use of technology in teaching, and active participation in the social Web.
In 2007, the Penn State Libraries had the unique opportunity to contribute questions for the FACAC survey. While the FACAC survey questions are a tightly controlled group— as they represent the needs of many ITS units on campus—a number of library-related questions were included in the final version. The Libraries felt this collaborative survey was especially important because so much of what students do is aggregated under the general heading of “technology,” and nearly everything students do to seek and find information— and to ultimately integrate it into papers and projects—is done through and with the use of technology. It made sense, and we felt sure it would make sense to students, to have library and IT questions seamlessly included in one survey. The Libraries contributed a variety of questions to the survey, nearly all of which were used.17 In an effort to draw upon and compare responses with national statistics, specific questions were designed to correlate with OCLC survey questions. This enabled later direct comparison with the national OCLC findings.
The following are a few of the questions contributed by the libraries to the 2008 FACAC survey:
- Please indicate if, in the last year, you have used any of the following online resources and services.
- My Library Account
- The CAT (Online library catalog)
- Library databases (ProQuest, Lexis, etc.)
- Online Reference Materials
- Google Scholar Electronic Books Electronic magazines and/or journals ASK! Online library help service
- During the last year, how often did you access the Libraries webpage?
- How frequently do you require your students to use online library resources? (faculty only)
FACAC Survey Findings: Comparison with National Survey Data
The library questions included in the 2008 FACAC survey presented thought-provoking results when compared with the results of College Students’ Perceptions of Libraries and Information Resources. While the FACAC survey’s response pool of 1,771 undergraduate students and 231 graduate students was much larger than the OCLC pool of 396 student responses, conclusions could still be drawn. Looking at the two studies together yields some positive information regarding Penn State students’ information literacy skills. In particular, major library research and discovery tools (the online catalog and library databases) were used at a dramatically higher rate. Worldwide, 38 percent of college students used the library catalog at least once annually, while 79 percent of Penn State students used the library catalog (the CAT) at least once last year. Worldwide, 33 percent of college students reported using library databases at least once. At Penn State, 64 percent of students used databases at least yearly. Worldwide, 61 percent of college students used a library website annually, while nearly 75 percent of Penn State students used the Libraries’ website at least yearly. Use of e-journals by Penn State students is lower; 40 percent compared with almost 60 percent of students worldwide. It was hypothesized that the current labeling in use on the Libraries’ website affected this statistic. While some libraries call their list of databases “e-journals,” the Libraries most frequently use the terms “databases,” “resources,” and “articles” on their webpages. Another question on the FACAC survey addressing a specific local concept may also assist in explaining the lower response to the e-journal question. The Libraries’ “Get it @ PSU” button appears in many databases and in Google Scholar, which leads students to full-text articles in other licensed databases. Half of Penn State students report using this feature to get “full-text journal articles,” a phrase that may be more familiar to them. Use of e-books is nearly equivalent between the two groups: just over 30 percent of college students worldwide and 28 percent of Penn State students. Use of ASK! (the Libraries’ help service) was higher locally: 14 percent of Penn State students compared to only 8 percent of college students nationally.18
Using FACAC Survey Results for Planning and Improvement
The planning of new service initiatives, including the Libraries’ future Knowledge Commons, has been affected by the findings of these surveys. The FACAC survey reported that 88 percent of students on campus owned laptop computers. This impressive statistic helped reinforce the importance of robust wireless access and a multitude of power outlets throughout the libraries, and the creation of more laptop-centered areas in the Knowledge Commons to encourage students to bring their laptops to campus by providing comfortable and convenient places to use them. Nearly half (46 percent) of undergraduate students self-reported that instructors have required a multimedia project as a class assignment during this academic year. Based upon this and other anecdotal evidence, the Libraries began planning in collaboration with ITS for a multimedia creation suite to be housed in the libraries.
Social networking made a significant showing, with 83 percent of students, 34 percent of faculty, and 23 percent of staff having Facebook accounts. This information encourages us to continue development on our existing Facebook library applications and to consider developing new ones. In questions asking for likelihood of participation, more than 50 percent of students indicated they would “post your favorite library materials on your Facebook profile,” confirming that students not only use Facebook but would use it for desirable library applications. Since this is not one of the current features available in the Penn State Libraries Facebook application, it gives direction for the Libraries’ next project. In ranking their preferences for ways to get help, “getting help in person” was given the highest rating by 55 percent of students, with e-mail as the next most preferred, encouraging us to retain these two services even as we expand our other forms of virtual reference service. Even though students communicate in a wide variety of ways, 84 percent still used e-mail daily while 26 percent updated their Facebook and MySpace accounts daily. Nearly all (93 percent) did online research and searches, and 83 percent used online resources, such as Wikipedia. One-fourth of the students reported having taken at least one online course in the past year. Another result—that 42 percent of students accessed library materials through ANGEL, our course management system (CMS)— indicates the success of our custom-designed CMS library tools: electronic reserves, custom library guides, and our ASK! virtual reference service (which includes IM, chat, e-mail, and phone options).