Coding into the Great Unknown: Analyzing Instant Messaging Session Transcripts to Identify User Behaviors and Measure Quality of Service

Abstract

After one year of providing virtual reference service through an instant messaging (IM) service, Binghamton University (BU) Libraries, under the purview of its Digital Reference Committee (DRC), undertook a study of collected session transcripts. The goals of this work were to determine who was using the IM service and why; if staffing for the service was adequate and met our in-person reference standards; and if improvements to the libraries’ existing reference services were needed. The findings revealed that 31 percent of identifiable users were students and 5 percent of users were campus community members. The analyses also revealed that many used the service for complex questions and not just ready reference, policy, and directional questions as had been expected. The most common question types were Web site navigation help (29% of all sessions), research assistance (22%), and instructional questions (23%). The American Library Association Reference & User Services Association (RUSA) Guidelines for the Behavioral Performance of Reference and Information Service Providers were used to measure quality of service. The findings reveled that approachability, showing interest, and listening were each demonstrated in over 80 percent of sessions, indicating these activities can be demonstrated effectively in a virtual environment. The study also found that questions were correctly answered 84 percent of the time. The study provided valuable insight into how patrons approach and locate information on our Web site and demonstrated a need for additional training, improved site design and navigational aids, and future discussions of staffing alternatives for the IM service.