|
Search | Back Issues | Author Index | Title Index | Contents |
 |
D-Lib Magazine
|
|
Anne R. Kenney, <ark3@cornell.edu> |
|
IntroductionIn April 2002, the dominant Internet search engine, Google™, introduced a beta version of its expert service, Google Answers, with little fanfare. Almost immediately the buzz within the information community focused on implications for reference librarians [1]. Google had already been lauded as the cheaper and faster alternative for finding information, and declining reference statistics and Online Public Access Catalog (OPAC) use in academic libraries had been attributed in part to its popularity. One estimate suggests that the Google search engine handles more questions in a day and a half than all the libraries in the country provide in a year [2]. Indeed, Craig Silverstein, Google's Director of Technology, indicated that the raison d'être for the search engine was to "seem as smart as a reference librarian," even as he acknowledged that this goal was "hundreds of years away" [3]. Bill Arms had reached a similar conclusion regarding the more nuanced reference functions in a thought-provoking article in this journal on automating digital libraries [4]. But with the launch of Google Answers, the power of "brute force computing" and simple algorithms could be combined with human intelligence to represent a market-driven alternative to library reference services. Google Answers is part of a much larger trend to provide networked reference assistance. Expert services have sprung up in both the commercial and non-profit sector. Libraries too have responded to the Web, providing a suite of services through the virtual reference desk (VRD) movement, from email reference to chat reference to collaborative services that span the globe [5]. As the Internet's content continues to grow and deepen — encompassing over 40 million web sites — it has been met by a groundswell of services to find and filter information. These services include an extensive range from free to fee-based, cost-recovery to for-profit, and library providers to other information providers — both new and traditional. As academic libraries look towards the future in a dynamic and competitive information landscape, what implications do these services have for their programs, and what can be learned from them to improve library offerings? This paper presents the results of a modest study conducted by Cornell University Library (CUL) to compare and contrast its digital reference services with those of Google Answers. The study provided an opportunity for librarians to shift their focus from fearing the impact of Google, as usurper of the library's role and diluter of the academic experience, to gaining insights into how Google's approach to service development and delivery has made it so attractive. What is Google Answers?Google Answers is an expert answering service that is staffed by over 800 freelance researchers who have been vetted by Google. For fifty cents, users can pose questions on Google Answers, and like Ebay®, they determine what the product is worth to them. They set the amount of money they are willing to pay for the correct answer (minimum of $2) and the amount of time they will wait for a reply. Any registered user may offer comments on the question but only Google-affiliated researchers may provide an answer. When a researcher indicates an interest in responding, the question is "locked in" for an hour. If the response is satisfactory, the user pays Google the agreed-upon fee — 75% of which goes to the researcher and 25% of which the company pockets. If the user is not satisfied with the response, s/he can request clarifications or reject the response and the question goes back into the queue for other researchers. The user can request a full refund (minus the fifty cent registration fee) if ultimately not satisfied with the results. Google Answers also allows users to rate responses. Researchers who receive too many negative reviews may have their privileges revoked by Google. The service provides an FAQ, a searchable and browseable database of questions, commentaries, and answers to previously asked questions; tips for users; a training manual for researchers; and terms of service. The Cornell Taste TestOf all the expert services available on the Web, Google Answers represents a good subject for comparative review with academic VRD services for many reasons. The search engine enjoys a large and growing market share of users, including those who rely on academic libraries. The service is well-documented and the question database contains many research questions and responses that parallel those answered by reference librarians. The information available about Google Answers supports a comparison of costs and response attributes and provides the means for assessing quality based upon external ratings. Google Answers is not specific to a domain, which makes it broadly relevant as a counterpoint for considering virtual library reference services. Finally, use of Google Answers has increased steadily over the past year, indicating a certain staying power for this type of service [6]. Like many research libraries, Cornell University Library has been experimenting with digital reference for a number of years. The offerings include:
A review of the Google Answers service indicated that it parallels more closely email reference than chat reference. Like email reference, Google Answers may involve a sequence of exchanges, but it is still asynchronous. Chat reference is an interactive session during which a reference librarian may respond to questions, but the software also enables reference staff to demonstrate the use of library resources to patrons. Our study was designed, therefore, to compare Google Answers to the email reference service used at Cornell. The study consisted of three stages: posting a set of questions to both Google Answers researchers and to Cornell reference staff; conducting a blind review of each pair of responses by reference staff from across the CUL system; and evaluating the results. In comparing the nature of the responses, we were interested in posing and answering the following set of questions.
Study MethodologyWe selected a set of twenty-four questions for the study, representing the range of research questions, from simple to complex, typically encountered by reference librarians [7]. The study did not include directional or how-to questions. The questions came from three different sources. The first batch (Maryland set) consisted of the twelve questions used by Neal Kaske and Julie Arnold in their study of online chat services, which they kindly agreed to allow us to use [8]. This test set had already been assessed as study-worthy and had been drawn from real questions posed at a variety of reference desks. All of the questions could be answered readily using basic reference materials, including web-accessible resources, but each required some effort to answer correctly and completely. The second set (Cornell set) comprised six questions that Cornell library users had asked over the past six months and that were deemed by the Cornell reference staff to be representative of questions they routinely receive. The third set (Google Answers set) consisted of six comparable questions selected from the list of unanswered questions on Google Answers. At Cornell, a senior reference librarian, a beginning reference librarian, and a reference assistant each prepared answers for their allotment of the twenty-four questions and noted the amount of time spent on each question. Each librarian was asked to prepare their responses as if they were answering an ordinary email reference request. The Cornell reference librarians also assessed the degree of difficulty for each question based upon an informal rating of question complexity and anticipated time required to respond. They used a simple system ranging from one as the easiest to four as the most difficult. Table 1 lists the distribution of questions by difficulty rating, and Table 2 shows the difficulty ratings by source of the question.
To gather the Google Answers responses to the study questions, one of the reference librarians created email accounts using multiple free email service providers. She registered with Google Answers as a user, submitted the questions, responded to requests for clarifications, collected the responses, and completed the loop by rating the Google Answers researchers' answers. The answers to the questions that were posted to Google Answers were priced as shown in Table 3.
Google researchers declined to bid on two of the more difficult questions at the $10.00 fee. Both questions had been assigned a difficulty rating of three and were among the questions drawn from those received by Cornell reference staff. These two questions were ultimately resubmitted to Google Answers with an increase in price to $50.00 and $100.00 respectively. At these rates, both questions were then answered. Once the final set of 24 questions and completed responses from CUL and Google Answers were in hand, reference librarians across the CUL system blind-reviewed the responses. At least two reference librarians evaluated both sets of responses to each question, and no one librarian evaluated all of the questions. For several questions, we received three or four evaluations. Most evaluators reviewed two questions, but several reviewed as many as five questions. The evaluators varied in terms of levels of reference experience and academic ranking within the CUL system. They were not given extensive instructions, but were asked to evaluate and rate the responses using this scale: 1 = Poor, 2 = Fair, 3 = Good, 4 = Very Good, and 5 = Excellent. They were also aware that they were evaluating responses by a reference librarian and a Google Answers researcher, but they did not know which response came from which group. Study FindingsThis study was more exploratory than scientific in nature. It was intended to provide some insights into the value and utility of alternative information service providers in order to enhance the academic reference function and to identify areas for further study. The evaluations, therefore, reflect a research library bias as to what constitutes a good response. The study did not focus on or measure user satisfaction, one of the three ways quality service is generally defined — the other two being accuracy and utility [9]. For the purposes of the study, we assumed that users were satisfied with the responses they received. The evaluations implicitly reflect, then, reference librarians' notions of a quality answer, which includes accuracy and utility, but do not necessarily correspond to user satisfaction. For instance, we know that users who rated answers taken from the Google Answers database, assigned each a five-star rating (the maximum possible number). Cornell librarians, on the other hand, were more critical of the freelance specialists' responses, rating them on average at 2.7 out of a possible 5.00. In addition to using the simple rating scale, many reviewers provided comments that offer valuable insights into their evaluation process. Positive ratings were given to responses that:
Negative comments noted that responses:
In addition, the participating reference staff and several reviewers commented that the responses might well have been improved if a reference interview had been possible. Some noted that many responses were too Web-based, thus failing to provide the best references. They also took exception to Google Answers researchers who did cite resources beyond those freely available on the Internet but often conveyed a somewhat negative connotation on the necessity of having to visit a library to access the material. Evaluation scoresThe evaluations indicate that there was no clear "winner" in terms of responses to the questions posed. Cornell reference librarians fared better in the overall average and were rated slightly higher in answering the Cornell reference questions (study set 2) and substantially higher for the Google Answers questions (study set 3). Google Answers researchers led Cornell staff by a significant margin on the Maryland questions (study set 1), which represent half of the question set used in this study. The results of the evaluations based upon question source are presented in Figure 1.
Figure 2 presents the distribution of evaluation scores by level of difficulty. Cornell leads by a small margin in both the level 1 and level 3 questions, but is well below Google Answers for level 2 questions. The Maryland set of questions contained the majority of level 2 questions.
It is interesting to note that the reference librarians, whose strive to instruct users as well as provide answers, scored higher on level one questions, which tend to call for more straightforward factual answers, and level 3 questions, which require more in-depth research. Google Answers researchers do not experience such tension; their sole aim is to answer the question satisfactorily. This desire in some cases, the reviewers' noted, may have led some Google Answers researchers to offer responses based on opinions rather than facts. Another possible explanation for the difference in scores may be financial in nature. Google Answers researchers are paid by the response; reference librarians are paid by the hour. Perhaps the Google Answers researchers spent less time answering the simplest questions because they could earn only 75% of the average price paid for Level 1 questions ($2.20 in our study). The most difficult questions (level 3-4) paid the most (averaging $20.36 in our study) but required a lot more effort to prepare a successful response. Level 2 questions, netting on average $4.88, may have struck a balance between effort expended and reward earned. The number of questions in our study, however, was too small to draw any firm conclusions. Another interesting finding from the evaluations is that the ratings for Google Answers responses were much more consistent from one reviewer to the next than were the evaluations for Cornell responses. For instance, on one question, the two reviewers rated the Google Answers response at 3 and 4 respectively, a mean difference of 1. The Cornell response was rated 2 and 4, for a mean difference of 2. The overall mean difference in scores between reviewers for the same question was 1.3 for Cornell, but only 1.0 for Google Answers. The standard deviation for both was 0.8, indicating that the shape of the distribution of ratings was the same for both. However, the Google Answers ratings curve was shifted to a lower value (e.g., more responses rated the same by reviewers). Figure 3 presents a histogram of the differences in ratings between the responses from Cornell and Google Answers. Although freelance researchers may not offer better answers than Cornell librarians, their responses were considered more consistent.
A final point on the evaluations involves sources. Google researchers are experts at locating hard-to-find information on the Web. Their answers, therefore, tend to be limited to freely available networked resources. Reference librarians, on the other hand, are experts at locating and evaluating information in print and digital forms, and have access to a vast array of credible resources they physically own or license. Although reviewers complimented the reference librarians' use of non-Web materials, the rating scores themselves do not reveal any significant advantage being gained through their use. Again the number of questions is too small to draw any real conclusions, but the findings suggest that the quality of response based on resources used should be further studied. CostCost proved difficult to measure. Google Answers currently allows users to set a price for answers between $2.00 – $400.00, only 75% of which goes to the successful researcher. Early versions of the tips for users on the Google Answers site suggested using an hourly rate of $25.00 for factoring appropriate prices for questions based on anticipated time to complete the answer. Although it may seem logical that Google Answers researchers would be less likely to take on questions that would require more work than they would be compensated for, there is no explicit measure to evaluate. Hobbyists who serve as Google Answers researchers may well take on a question for the challenge. Research libraries, on the other hand, have overhead costs, licensing fees, salaries, benefits, and other expenses that would need to be included in realistic pricing. We looked at some relevant pricing schemes using Cornell examples. Hostline charges $135.00 per hour, including the delivery of citations, with a 10% discount for alumni of the Cornell Hotel School. Cornell's Ask-a-Librarian reference service for alumni and friends costs between $75 to $135 per hour, depending on the subject. Like a number of expert services and unlike Google Answers, the librarian provides an estimate for the user before undertaking the work. For the purpose of this study, we calculated costs two ways. For the Google Answers responses, we used the price set for each answer. For the Cornell cost, we divided the preparation time for each answer by the average salary plus benefits of the three participating reference librarians, which came to $42.24 per hour. Table 4 correlates level of question difficulty with time spent by reference librarians preparing a response. The average time spent on the easiest questions was 15 minutes, which equates to $10.56 in staff salary and benefits. The cost to respond to Level 2 questions averaged $16.05; Level 3 questions averaged $42.24.
Figure 4 compares the costs for Cornell and Google Answers based on the two methods of calculation presented above. Although the figures do not represent exact or comparable measures, they are nonetheless suggestive.
Figure 5 charts cost against average rating per response. This chart illustrates again that Google Answers prices were much lower than Cornell's, but for neither group is there a strong correlation between price and quality of response.
ConclusionThis study offered a quick, limited review of an emerging phenomenon — market-driven external reference services — that will ultimately affect the role played by reference librarians in academic settings. We see three immediate lessons learned from this study. First, the study revealed the importance of self-assessment. Although Cornell reference librarians scored higher overall than did the freelance researchers working for Google, their scores were not significantly better. Both groups were rated overall in the "good" category, but one might have expected that highly trained and comparatively expensive information professionals would have scored consistently higher. There are many plausible explanations for why the librarians did not score higher than the freelance researchers, some attributable to the flawed or limited nature of our study. In addition, studies evaluating reference service present a mixed picture. Peter Hernon and Charles McClure, among others, estimate that on average librarians provide correct answers 55% of the time as judged by other information experts [10]. But in a recent study, John Richardson suggests that librarians provide an accurate source or strategy in response to users' questions approximately 90% of the time [11]. And users themselves consistently rate their reference encounters as highly satisfactory [12]. The lesson to be taken from our study is not the relative rating achieved but the importance of ongoing review as part of a strategy of self-assessment. A recent SPEC Kit on reference assessment noted that only 3% of the 77 responding libraries to a 2002 ARL survey regularly assess the quality of transactions [13]. Just as public school teachers evaluate each other's performance throughout the school year, reference librarians could improve their services through peer review. In addition, Google Answers' practice of encouraging users to rate and publicly post evaluations of responses received should be considered. A similar precedent already occurs in the academy as student ratings of professors' classes are posted with the course description at some institutions. Second, academic libraries should make a practice of regularly monitoring developments in the broader information landscape. These developments will not only have a pronounced impact on information provision but they can also help reference librarians assess their own programs. Although it is still early days, we can envision a point in the future where some forms of reference service will be outsourced in a manner similar to the outsourcing of other library functions, such as copy cataloging. If for instance, an outside provider can adequately address simple reference questions at one-fifth the cost of doing so in-house, why duplicate the service? Reference librarians need to analyze more thoroughly how much time is spent by function performed. By freeing themselves from more routine tasks, they can focus their efforts on aspects of complex information discovery and use in which they clearly excel. Finally, what lessons can academic libraries draw from the ancillary services offered by commercial enterprises? It has already been noted that Amazon.com is used by many in lieu of public access catalogs and that their methods of recommending like materials, providing sample page images for review, and posting reader reviews encourages customers to purchase materials. Can academic libraries entice greater use of intellectually vetted material by adopting similar practices? What can we learn from the practice of customers assigning fair market value for products and services that is at the heart of Ebay.com or Google Answers? Are there other ways to quantify the value of information services to users beyond pricing? Answers to such questions will help research libraries justify their value to academic administrators who are responding to the current economic crisis by casting about for programs that can be cut or eliminated. Academic librarians must become more savvy in articulating their value to the educational enterprise in order to prosper in a rapidly changing information environment. Commercial enterprises determine their services in part by assessing their competitors and going one better. AcknowledgementsSpecial thanks go to those Cornell staff members who participated in this study, in particular Lynn Thitchener, and to Dr. John Kenney (Department of Physics, East Carolina University) for his assistance in data analysis. Notes and References[1] For an overview of reactions to Google Answers, see <http://www.library.cornell.edu/iris/research/google.html>. [2] Barbara Quint, "Some Advice for Google Answers," Information Today, June 2002, Vol. 19, Issue 6, pg. 8. [3] "Google: Reference Librarians can Keep Their Jobs," Library Journal Academic News Wire, May 13, 2003. [4] William Y. Arms, "Automated Digital Libraries: How Effectively Can Computers be Used for the Skilled Tasks of Professional Librarianship?" D-Lib Magazine, July/August 2000, Vol. 6, No. 7/8, <doi:10.1045/july2000-arms>. [5] For a discussion of the various options, see Joseph Janes, Chrystie Hill, and Alex Rolfe, "Ask-an-Expert Services Analysis," Journal of the American Society for Information Science and Technnology, 52(13):1106-1121, 2001. [6] For information on the growth of this service, see <http://www.library.cornell.edu/iris/research/google.html>. [7] For a listing of the questions, cost, time spent, ratings, and evaluation comments, see <http://www.library.cornell.edu/iris/research/google.html>. [8] Neal Kaske and Julie Arnold's study: "An Unobtrusive Evaluation of Online Real Time Library Reference Services," paper presented at the Library Research Round Table, American Library Association Annual Conference, Atlanta, GA, June 15, 2002 <http://www.lib.umd.edu/groups/digref/LRRT.html>. [9] John V. Richardson, Jr., "Reference Is Better Than We Thought," Library Journal, April 15, 2002, 127 (7), 41-42. [10] Peter Hernon and Charles McClure, "Unobtrusive Reference Testing," Library Journal, 111 (April 15, 1986) p. 37-41. [11] Richardson, Op. cit., p. 42. [12] Results from a fall 2002 survey of chat reference users at Cornell indicated that over 80% of them received a full or nearly full answer to the questions they posed; 94% indicated they would use this service again. [13] Association of Research Libraries, Reference Service Statistics & Assessment, SPEC Kit 268, September 2002, <http://www.arl.org/spec/268sum.html>. (On June 17, 2003, the name of Google's director of technology was corrected to read Craig Silverstein.) Copyright © Anne R. Kenney, Nancy Y. McGovern, Ida T. Martinez, and Lance J. Heidig |
||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||
|
Top | Contents | ||||||||||||||||||||||||||||||||||||||||||||||
| | ||||||||||||||||||||||||||||||||||||||||||||||
|
D-Lib Magazine Access Terms and Conditions DOI: 10.1045/june2003-kenney
|
||||||||||||||||||||||||||||||||||||||||||||||
