D-Lib Magazine
July/August 1998ISSN 1082-9873
From the Editor
Metrics and the Digital Library
A Guest Editorial by Barry M. Leiner
Dr. Leiner currently chairs the D-Lib Working Group on Digital Library Metrics. Among other achievements, he has a distinguished record of public service at the US Defense Advanced Research Projects Agency (DARPA) where he was instrumental in the development of the Internet and, subsequently, the first phase of the Digital Libraries Initiative.
Comparing different systems, or evaluating improvements in a system, is difficult at best without having some form of yardsticks -- criteria by which the evaluation is done. The effectiveness of having easily understood, simple to use, and broadly applicable metrics is illustrated by the experience of the text retrieval community. Through agreement on precision and recall as common metrics, coupled with a test suite used widely in the community (through the TREC and Tipster efforts), comparisons of different approaches were made possible as well as evaluation of improvements in the individual approaches. This allowed the community to "stand on each others shoulders", resulting in significant improvement to the technology base overall.
Although precision and recall have direct applicability to aspects of the digital library, and have the virtue of broad community acceptance, the digital library environment is much more complex. Metrics are required to deal with issues such as the distributed nature of the digital library, the importance of user interfaces to the system, and the need for systems approaches to deal with heterogeneity amongst the various components and content of the digital library.
To address this community need for widely accepted, usable, and useful metrics to support the assessment of different approaches as well as nurture progress in the field, a working group was formed under the auspices of the D-Lib Forum.
This Working Group is to develop a consensus on an appropriate set of metrics to evaluate and compare the effectiveness of digital libraries and component technologies in a distributed environment.Coupled with a parallel effort being organized to establish and maintain a test suite available to the community for evaluation of different approaches, the intent is to provide the tools and environment that will facilitate a community approach to such evaluations.
The Working Group was established in late 1997. One early result of the discussions was the recognition that we needed to take a very broad definition of a digital library. To that end, we are using as a working definition:
The Digital Library is the collection of services and the collection of information objects and their organization, structure, and presentation that support users in dealing with information objects available directly or indirectly via electronic/digital means.With this definition and recognizing the complexity of the digital library environment, we recognized that we have a formidable task ahead of us. Metrics are needed from a variety of perspectives ranging from system to user, from content to services, from local to highly distributed, from single use to use across a "session", and so on. The approach we are taking is a combination of top-down and bottom-up.
From a top-down view, we are trying to characterize some of the typical uses of the digital library and associate meaningful metrics with those uses. We are also trying to characterize the nature of the different types of metrics needed. From there, we hope to be able to abstract a set of useful and usable metrics that, while not completely characterizing the digital library, will provide a sufficient set of useful information to help both the selection of approaches and the progress of the developers.
From a bottom-up perspective, we are working to identify ongoing activities that are already using various evaluation criteria that are candidates for broader usage. An early step towards that end was a Workshop on Digital Library Metrics sponsored by the Working Group and organized by Bill Pottenger and Bob McGrath of UIUC/CANIS. We hope to encourage partnering of organizations to evaluate the metrics and propose them to the Working Group as candidates for more widespread community usage.
An important aspect of the overall community effort lies in the consistent use of metrics in comparing different technical, system, and organizational approaches. To that end, CNRI (with support from DARPA) is organizing a community effort to create and maintain a test suite for digital libraries. The intent is to maintain a distributed collection of material with sufficiently known properties (e.g., which documents should be selected for a specified query) so that a standardized measurement methodology can be developed. The metrics developed by the Working Group are intended to be input for that process.
The broad interest in the topic of metrics for digital libraries is reflected in the membership of the Working Group. The Working Group is open to people with an interest in the area and a willingness to contribute their efforts. There are members from academia, industry, and government. There are computer scientists, social scientists, and librarians. Some are interested in research, and others are interested in evolving their operational library. The Working Group already has approximately fifty members with people continuing to join. Most of the work is done by email and use of the internet, but we take advantage of opportunities such as DL98 to have face-to-face meetings. Roughly twenty have participated in the face-to-face meetings. For more information on the Working Group, see the web site http://www.dlib.org/metrics/public/metrics-home.html.
Having an agreed upon set of metrics and methodology will allow us to compare and contrast different technical, organizational, and system approaches to digital libraries and will help the community move forward. Such metrics will allow us to articulate the benefits of particular approaches, support digital library operators in selecting the "best of breed" approaches, and quantify the general progress in the field. We are hopeful that a community approach, such as that of the D-Lib Working Group, will move us all towards that goal.
Barry M. Leiner
Chair, D-Lib Working Group on Digital Library Metrics
Corporation for National Research Initiatives
bleiner@cnri.reston.va.usCopyright (c) 1998 Corporation for National Research Initiatives
Top
| Magazine
Search | Author Index | Title Index | Monthly Issues
Previous | Next
Comments | E-mail the Editor
hdl:cnri.dlib/july98-editorial