[PREVIOUS] [CONTENTS] [NEXT]
Issues in Science and Technology Librarianship Winter 2000

Measuring the Use and Value of Electronic Journals and Books

Linda S. Mercer
Associate Director, Digital Communications and Resources
Bernard Becker Library
Washington University School of Medicine
mercerl@msnotes.wustl.edu

Abstract

Much has been written on issues pertaining to licensing and archiving of digital information. Until recently, there has not been enough information to evaluate how these digital products, particularly journals, are being used. Furthermore, meaningful data are often difficult to obtain as some publishers and vendors supply little or no data or only information they feel supports the purchase of their products. As it becomes increasingly difficult to afford all digital content, librarians must be able to measure digital use of e-journals and books in order to make the best purchasing decisions for their institutions. Librarians must develop their own solutions as well as solutions in collaboration with publishers so that better evaluation of digital content use can occur.

Introduction

In a 1999 article from this journal, a group of five librarians summarized the "most relevant issues" surrounding electronic publishing of scholarly journals (Buckley et al. 1999). The issues were access, cataloging and indexing, pricing, archiving, and licensing. I would like to add usage statistics for performance measurement as one of the emerging, and most highly relevant issues librarians will face in the coming years.

Current Practice

As librarians, we often use statistics as performance measures: how many volumes are housed in our libraries; how many journals are subscribed to; how many people come into the library; how many times books and journals have been checked out; how many reference questions are asked. Most of us cite statistics as we submit our budget proposals each year. Important decisions about the nature of our individual libraries are made based on performance factors that often support what we intuitively believe to be true. Nevertheless, accurate and informed performance measurements have not always been possible, and more often, standard statistical observation does not always reflect or define what is "good" or "of value."

Understanding what is "of value" becomes even more difficult and complex as an increasing number of our resources become digitized. Different digital formats, interfaces, pricing structures, and access restrictions complicate our ability to evaluate journal resources using consistent measures. The very nature of information is changing; by virtue of desktop access, ability to manipulate information, and value-added features, we need to ask if the same content is as valuable today as it was five years ago. Considering the changing nature of our library collections, it is probably time to redefine the role of performance as well as the nature of what and how we measure it.

Here is a scenario that is similar to recent discussions at the Bernard Becker Medical Library at Washington University School of Medicine in St. Louis:

As the journal renewal process for the year 2000 takes place, a group of librarians gather to review a report listing journal usage statistics, the cost per use, and the subscription rate for all print journals housed in the library. A line is drawn at a certain use level based upon the available funding. Once the total is hit, all titles below that line are considered for cancellation. A title falling beneath that line is found to also have an electronic counterpart. However, there are no electronic statistics available for that particular title from the publisher. Thus, it is impossible to determine accurately the actual overall use of the journal. The librarians wonder what the level of electronic use was and whether that use would move that particular title above the cancellation line. The librarians also wonder about the title just above the cancellation line that does not have an electronic counterpart. Does the library retain it over the slightly lesser used title that possibly has electronic uses? For example, when evaluating Current Opinions in Psychiatry, librarians found that the journal was used only 16 times in print format. However, the corresponding statistics for Current Opinions in Psychiatry in Journals@Ovid revealed that it was accessed nearly 300 times. Based on this additional information the title was retained.

Additionally, another title with low use is provided as part of a bundled set of journals. Cancellation of this title, no matter how poorly used in any format, nullifies the library's contract resulting in increased prices for the other titles. Nevertheless, knowing the amount of electronic usage could help the librarians decide when it might be time to renegotiate or attempt to modify the contract.

Finally, the library group notices that the overall use for important print journals is falling off. The nature and historical importance of these journals are such that there is no doubt that they are not losing their scholarly presence. The answer is fairly obvious: they are being accessed online. With this increased online access, many researchers are accessing these journals from their desktops rather than visiting the library.

Generally most journal evaluation reports inform librarians how their print collections are utilized. They do not, however, supply any information on how often or what resources are being used on the client desktops. Moreover, the typical journal evaluation report does not identify an expanding web-based user base. It also does not help determine if the library is reaching previously underserved populations nor does it suggest whether users will forgo information found only in the library for the convenience of getting similar information in their offices.

Beginning Steps

How do we address these problems? The answers lie in many arenas. Answers must come from the market research which librarians should perform on the clientele they know best. But answers must also come from publishers who should be seen as partners in our attempt to provide the best service to our clientele.

So what should be our performance measures and market research questions?

This paper will focus on two reporting mechanisms that offer useful information for evaluative purposes.

We can begin with one set of measures available for some Highwire Press titles and determine what they tell us about performance. The statistics provided for these limited titles1 are among the most extensive on the market. An example of the December 1999 use-data for the Journal of Biological Chemistry (JBC) is included in Figure 1. As shown, the statistics include:

The Highwire statistics provide some basic data as well as some meaningful novel data. What is particularly useful is the number of unique IP addresses that access this site. We know for instance, that during the month of December there were over 550 unique IP addresses that accessed JBC over 1200 times. From this data we can conjecture that a wide range of researchers frequently use this resource. Thus, this type of reporting not only provides a sense of the volume, but also the breadth of use for this particular title.

Other statistics for JBC give us the opportunity to learn more about our users' habits. While this information may not correlate directly to purchase and cancellation decisions, it helps us ascertain how researchers are using these value-added resources. In addition, this information helps us identify areas in which training might be appropriate. For instance, during the month of December, 1,169 searches were performed in JBC. This suggests that users are doing more than going to the site to obtain a specific citation. They may be using this journal's special online feature to search for new information and use it as an information-gathering tool. It also gives us a window into the format preferences of our users. Many users tell us they prefer PDF, but for JBC in most months in 1999, most JBC users chose HTML format over PDF.

Content-usage statistics for
Journal of Biological Chemistry
December 1999

Washington University
Subscriber ID: xxxxx
Contact: Xxxxxxxx Xxxx
Associate Director for Technical Services
Phone: 314-362-6802
Email: xxxxx@msnotes.wustl.edu
Total-usage statistics
Access Events to: Home Page Current issue TOC All TOCS Searches Abstracts Full text HTML PDFs User Names IP addresses in use
Total Usage 1,262 389 604 1,169 436 1,608 1,420    
Unique Events 1   57   338 1,014 897 1 558
Article Usage by Section
Section: Total Articles Online Abstracts Full text HTML PDFs
Unique Articles Accessed Total Accesses Unique Articles Accessed Total Accesses Unique Articles Accessed Total Accesses
Mini-Reviews 442 0 0 46 103 38 116
Communications 1,042 13 24 50 70 50 88
Carbohydrates, Lipids, and Other Natural Products 1,024 14 17 30 36 23 27
Cell Biology and Metabolism 9,009 160 210 516 741 493 755
Enzymology 2,276 16 19 57 78 48 66
Membranes and Bioenergetics 1,776 9 12 50 60 34 59
Nucleic Acids, Protein Synthesis, and Molecular Genetics 5,397 49 63 152 198 102 142
Protein Chemistry and Structure 3,791 18 20 106 154 105 160
Additions and Corrections 235 0 0 5 6 2 3
Other Articles 44,935 59 71 0 0 0 0
Totals 69,937 338 436 1,012 1,446 895 1,416
Top 10 Articles in December 1999
Full text HTML PDF Abstracts Total Accesses Age of Article in days from 12/31/1999 Article
11 19 0 30 15 Section: Mini-Reviews
Moshe Oren
Regulation of the p53 Tumor Suppressor Protein
Dec 17, 1999 274: 36031-36034
[Full Text]
5 17 0 22 8 Section: Mini-Reviews
Mikhail Bogdanov, William Dowhan
Lipid-assisted Protein Folding
Dec 24, 1999 274: 36827-36830
[Full Text]
2 12 0 14 1 Section: Mini-Reviews
Michael D. Kaytor, Stephen T. Warren
Aberrant Protein Deposition and Neurological Disease
Dec 31, 1999 274: 37507-37510
[Full Text]
3 11 0 14 22 Section: Communications
Masatoshi Yamasaki, Noriyo Hashiguchi, Chiharu Fujiwara, Tsuneo Imanaka, Toshiro Tsukamoto, Takashi Osumi
Formation of Peroxisomes from Peroxisomal Ghosts in a Peroxisome-deficient Mammalian Cell Mutant upon Complementation by Protein Microinjection
Dec 10, 1999 274: 35293-35296
[Abstract] [Full Text]
2 9 2 13 22 Section: Communications
Inaki Azpiazu, Humberto Cruzblanca, Ping Li, Maurine Linder, Min Zhuo, N. Gautam
A G Protein gamma Subunit-specific Peptide Inhibits Muscarinic Receptor Signaling
Dec 10, 1999 274: 35305-35308
[Abstract] [Full Text]
7 6 0 13 862 Section: Cell Biology and Metabolism
Dayang Wu, Herschel D. Wallen, Naohiro Inohara, Gabriel Nunez
Interaction and Regulation of the Caenorhabditis elegans Death Protease CED-3 by CED-4 and CED-9
Aug 22, 1997 272: 21449-21454
[Abstract] [Full Text]
7 6 0 13 15 Section: Cell Biology and Metabolism
Yuping Yuan, Suhasini Kulkarni, Philippe Ulsemer, Susan L. Cranmer, Cindy L. Yap, Warwick S. Nesbitt, Ian Harper, Nayna Mistry, Sacha M. Dopheide, Sascha C. Hughan, David Williamson, Corinne de la Salle, Hatem H. Salem, Francois Lanza, Shaun P. Jackson
The von Willebrand Factor-Glycoprotein Ib/V/IX Interaction Induces Actin Polymerization and Cytoskeletal Reorganization in Rolling Platelets and Glycoprotein Ib/V/IX-transfected Cells
Dec 17, 1999 274: 36241-36251
[Abstract] [Full Text]
2 8 3 13 1 Section: Nucleic Acids, Protein Synthesis, and Molecular Genetics
Jennifer R. Saam, Jeffrey I. Gordon
Inducible Gene Knockouts in the Small Intestinal and Colonic Epithelium
Dec 31, 1999 274: 38071-38082
[Abstract] [Full Text]
5 7 0 12 22 Section: Mini-Reviews
Kostas Tokatlidis, Gottfried Schatz
Biogenesis of Mitochondrial Inner Membrane Proteins
Dec 10, 1999 274: 35285-35288
[Full Text]
3 5 4 12 22 Section: Communications
Onno Kranenburg, Ingrid Verlaan, Wouter H. Moolenaar
Dynamin Is Required for the Activation of Mitogen-activated Protein (MAP) Kinase by MAP Kinase Kinase
Dec 10, 1999 274: 35301-35304
[Abstract] [Full Text]
47 100 9 156 99 (avg age) Totals for Top 10 Articles in December 1999

Figure 1

What does this information tell us about our performance? First and foremost it tells us that this particular resource is heavily used. In addition, it identifies subject areas most frequently viewed, providing clues into what areas might be good targets for training. It also tells us that researchers frequently use the search feature. This information may support the development and implementation of classes devoted to searching full-text information. The report further suggests that our users will download an HTML document rather than a PDF document. This information may help us make decisions about other product formats. Most importantly, the report indicates that this online journal is a valuable resource for our users since it is accessed over 1,300 times a month. In comparison, the print version of JBC is used on average only 475 times a month.

The next example that helps define performance comes from the Becker Library. Before Ovid Technologies, Inc. included a statistical package with its system, members of library staff and Medical School Computing Services (MSCNS) wrote a statistical package that could use Ovid's log files. We are able to extract data from the log files into a Sequel Server database and write reports in Microsoft Access to answer usage questions. Using this software, we are able to identify:

We are also able to tailor a report for a particular need. For instance, when there was a billing problem concerning the usage of the Internal Medicine residents, we were able to generate a usage report for that group alone. Additionally, we can examine an individual's account to determine how many hours an individual has used Ovid for a given period of time. A portion of one report generated from library data is illustrated in Figure 2.

Percentage of EUCLID Use by Departments and Divisions FY99

Department Name Percentage
MEDICAL HOUSE STAFF-I&R 9.52%
PATHOLOGY-INSTRUCTION&RESEARCH 4.87%
GEN SURGERY ADMIN CLINC. PRACT 4.14%
CELL BIOLOGY AND PHYSIOLOGY I&R 3.67%
BIOLOGY 2.93%
PSYCHIATRY 2.85%
INT MED-CARDIOLOGY 2.79%
OCCUPATIONAL THERAPY 2.61%
MOLECULAR BIOLOGY & PHARMACOLOGY 2.58%
OPHTHALMOLOGY 2.56%
SURGICAL PATHOLOGY 2.40%
NEUROLOGY 2.33%
INT MED-PULMONARY DISEASES 2.24%
PEDIATRICS-Administration 2.20%
BIOCHEMISTRY & MOLECULAR BIOPHYS 1.94%
INT MED-GASTROENTEROLOGY 1.91%
INT MED-LABORATORY MEDICINE 1.83%
PHILOSOPHY 1.83%
ANATOMY & NEUROBIOLOGY 1.70%
INT MED-RENAL DISEASE 1.55%
INT MED-METABOLISM 1.51%
ORTHOPEDIC SURGERY DEPT. 1.46%
INT MED-INFECTIOUS DISEASES 1.45%
BARNARD CANCER CENTER 1.32%
INT MED-DERMATOLOGY DIV-ADMIN 1.31%
OTOLARYNGOLOGY-INSTRUCTION&RESEA 1.18%
DIVISION OF ONCOLOGY 1.07%

Figure 2

This information has been valuable for marketing services, identifying groups or departments for training, deciding how many licenses to buy for each database and journal title, and determining which databases and journals to retain or cancel. What is surprising is that for every item canceled from Ovid, we tend to replace it with another. We have not decreased our business with this information vendor; rather we have used this information to obtain the best and most economical products for our particular user population.

Contrast these two examples with the usage data obtained from other commercial vendors. One vendor sends statistics that list, for each day, the number of queries, the number of sessions, average length of session and the number of "turnaways" (i.e., the number of times users have exceeded licenses). Another vendor sends a counting of the types of pages accessed (e.g., "list of journals," "articles," "issues," "help," etc.), the top 20 titles accessed each month and the journal titles accessed but not subscribed to by the library. In addition the form and type of article downloaded are also summarized. This type of limited information can give the library a sense of the preferred format, the length of an average session, or what the most popular resources are. Moreover, this information can provide us with a sense of the volume of use, though often not the breadth. Unfortunately, this information is often incomplete, frequently generating more questions concerning usage patterns than it answers.

Moving Forward

What then should we expect from ourselves and from the publishers and vendors with whom we have costly relationships? What data is useful and necessary for us to project the future of information access? What data is merely "fluff?" A single librarian or even library will not be able to answer these questions. However, efforts on local, regional and national levels to begin a dialog to define the important questions and problems, and come up with possible solutions should be initiated and supported. We can work locally within our institutions to begin the process of designing software for obtaining useful data. Individual libraries can join together to lobby key publishers to supply comprehensive and useful data for their products. In addition to this lobbying, librarian can offer to collaborate with publishers in the development and/or testing of these data gathering tools. Together we should work with national and regional organizations and publishers to help establish a set of minimal standards for basic statistics.

Currently, the Association of Research Libraries is establishing a project on determining usage measures for electronic information resources; see http://www.arl.org/stats/program/newmeas.html. In addition, The University of Michigan University Library and The Program for Research on the Information Economy are sponsoring a research conference on the economics and use of digital library collections; see http://www.si.umich.edu/PEAK-2000/. Internationally, the UK Serials Group's 2000 conference will be holding workshops on monitoring e-journal usage and as well as a session on performance measurement for academic libraries. On a local level, the University of Colorado library system is expanding its pilot study to take a more rigorous and refined look at the use of online products for its four campuses.

A more organized and standardized approach to obtaining meaningful and useful data should not lead to mass cancellations of subscriptions. On the contrary, informed decisionmaking should lead to an increased trust between libraries and publishers and expand and enhance a library's ability to respond to information needs. This will only help maintain library budgets that in turn will help to maintain or increase purchasing power. Finally, will this improve our performance as providers of information? The answer is obvious. By placing ourselves firmly at the center of gathering data and determining the electronic use characteristics of our products and researchers, we position ourselves as the true knowledge managers for the 21st Century.

Resources

Buckley, C., et al. 1999. Electronic Publishing of Scholarly Journals: A Bibliographic Essay of Current Issues. Issues in Science and Technology Librarianship Spring 1999. Available: http://www.library.ucsb.edu/istl/99-spring/article4.html

Notes

1 Journal of Biological Chemistry, Journal of Experimental Medicine, Journal of General Physiology, Proceedings of the National Academy of Sciences, Science

FEEDBACK

We welcome your comments about this article. Please fill out this form for possible inclusion in a future issue.

[PREVIOUS] [CONTENTS] [NEXT]

W3C 4.0 Checked!