|
Search | Back Issues | Author Index | Title Index | Contents |
 |
D-Lib Magazine
|
|
Heting Chu Thomas Krichel |
|
AbstractNEP (New Economics Papers) is a current awareness service for the RePEc (Research Papers in Economics) digital library. Since its initiation in 1998, over 56 individual lists have been created to loosely represent subfields within economics. Those lists in total made more than 37,000 announcements of about 28,000 working papers that were added to RePEc in the past five years. With several kinds of data available for NEP from May 1998 through June 2003, this study examines the development of the NEP service. The performance of NEP is then measured in terms of timeliness, coverage ratio, and usage. In exploring the various NEP parameters and their relationships, we discuss the potentials and other perspectives of the NEP service. Although it can be further improved, NEP could become an innovative model for current awareness services of digital libraries technically as well as organizationally. IntroductionEven though no consensus has been reached about the definition of digital libraries [Schwartz 2000], a considerable number of such entities have been established in the past decade for different constituencies and communities. RePEc (Research Papers in Economics) [1], has been specially created to facilitate the dissemination of working papers, journal articles and software objects in the field of economics. RePEc, a collaborative effort of over 100 volunteers in 30 countries, provides over 208,000 items of interest, over 107,000 of which are freely available online. The founders of RePEc have provided detailed descriptions about the digital library in their writings (e.g., José Manuel Barrueco Cruz and Thomas Krichel [Barrueco Cruz & Krichel; Krichel 2000]). Digital libraries have been researched and written about ever since they were first developed [Fox & Urs 2002; Schwartz 2000]. For instance, Davis and Lagoze illustrated the design and development of a digital library of computer science research reports (NCSTRL) [Davis & Lagoze 2000]. Cole, Allen and Schmitz particularly explained the building of a digital collection of educational resources in agriculture at the University of Illinois at Urbana-Champaign [Cole et al., 2000]. Many more papers have been published with regard to standards and related topics, such as interoperability for digital libraries (e.g., [Suleman & Fox 2001]. Evaluation of digital libraries, a subject about which the information science community is always keen, has received proper attention as well. Saracevic, for example, discussed the challenges facing digital library evaluation and suggested a conceptual framework for evaluation derived from the systems approach [Saracevic 2000]. Choudhury et al. described an evaluation framework for digital library services by focusing on an existing digital library project [Choudhury et al., 2002]. More specific issues in evaluating digital libraries (e.g., usability) are also examined in the literature [Xie & Wolfram 2002]. Services offered by digital libraries vary from institution to institution. They generally fall into two broad categories: traditional services and services unique to the digital environment. Personalization, plagiarism detection, analysis and processing of digital information are some examples of unconventional digital library services [Fox & Urs 2002]. Reference services of digital libraries, on the other hand, belong to the traditional category. However, such services have evolved so rapidly and extensively that they have become an emerging domain called digital reference services (see [Chowdhury 2002; Sloan 2001; Stemper & Butler 2001]). Current awareness seems to be a service that borders the boundary between conventional and unconventional digital library services. While current awareness is not a major item on the traditional library service menu, thanks to the Internet technology, it could achieve its full potential in digital libraries. NEP (New Economics Papers) is a human-mediated current awareness service of the RePEc digital library. Moreover, NEP [2] goes beyond the legacy model of digital library services in two aspects: First, NEP breaks down the division between the user and service provider. One special group of users, the NEP editors, volunteer their time in providing the current awareness service to their colleagues. Second, the NEP service does not simply funnel documents newly added to the RePEc digital library to the end user. Rather, it adds value to the information it disseminates by organizing and distributing it in a manner beneficial to the user [Barrueco Cruz et al., 2003). Further description of the NEP service is provided in the following section. The NEP ServiceNEP, started in 1998, provides current awareness service to all those who have subscribed to the individual NEP lists (see Appendix) for distributing papers newly entered into the RePEc digital library. The history and operations of the NEP service are comprehensively reported [Barrueco Cruz, et al. 2003]. Unlike RePEc which holds both working paper data (i.e., recent research reports prior to formal publication) and article data (i.e., peer reviewed writings), NEP covers only working paper data. Papers recently added to the distributed RePEc databases have been selected and compiled by a general editor of nep-all (All New Papers), the overall list that functions as the source provider for individual NEP lists. Those individual NEP list editors then select papers from nep-all, compile the selected papers into an issue of a NEP report, and disseminate them via their own lists to respective list subscribers. Figure 1 illustrates the NEP service process.
Each year, nep-all on average distributes about 40 issues of new economics papers to the editors of individual NEP lists (see Figure 2). It should be pointed out that there is not necessarily a one-to-one relationship between each issue of nep-all and other individual NEP list reports. One individual NEP list report may announce papers from different issues of nep-all. In addition, one paper from a single nep-all issue can be announced by more than one NEP list or not selected for announcement by any NEP list. For this reason, the term "announcement" is used to refer to the inclusion of an individual paper from nep-all in a specific NEP list report. Current awareness, as indicated before, is not new as a service in disseminating the latest information to the user in brick-and-mortar libraries. Nor is it unheard of in the world of digital libraries. The Contents-to-Go of the TORPEDO Ultra Digital Library Initiative is a case in point [Stackpole & King 1999]. Namely, Contents-to-Go emails electronic tables of contents for all journals in its collection to its subscribed users. Other kinds of current awareness include the use of lists to share information among catalogers [Condron & Tittemore 2001]. PhysDoc [3] collects and organizes links to physics resources worldwide by continent, country and town [Severiens et al., 2000]. It has the component for current awareness but is built upon pull technology. Users have to visit PhysDoc in order to learn what is new in the field. In comparison, NEP is based on push technology. New economics papers are sent to subscribers via individual NEP lists. The basic NEP operation model was set up by Thomas Krichel in 1998. NEP has been running, although not free of problems, without fundamental changes until now [Barrueco Cruz et al., 2003]. It perhaps could become an innovative model for current awareness services of digital libraries. Among the many features distinguishing NEP from other current awareness services, NEP operates on a scale larger than Contents-to-Go [Stackpole & King 1999], as will be described below. NEP's Development and GrowthThe first issue of nep-all was distributed on May 4, 1998. By the end of June 2003, a total number of 215 nep-all reports had been issued. Figure 2 describes the total number of issues nep-all distributes (i.e., # of issues) and the average number of papers each nep-all issue carries (i.e., issue size) every year from 1998 to 2003.
As shown in Figure 2, nep-all issue size steadily increases over the years while the number of issues nep-all produces annually remains about the same. This implies that the NEP service grows considerably in terms of the number of papers it distributes to individual NEP lists. Each nep-all issue in 2003 contains on average 243 papers, more than quadruple of the issue size in 1998. The most noticeable change in issue size occurred between 2000 and 2001 when the average number of papers each nep-all issue carried jumped from 96 in 2000 to 172 in 2001. In the time period covered in this study, nep-all in total delivered 215 issues that all together consist of 28,433 papers. Once nep-all sends out an issue, it is the individual NEP list editors who ultimately select and deliver the papers contained within to their respective list subscribers. Altogether, there are 56 such individual NEP lists at the time of this writing, excluding nep-all (see Appendix). Figure 3 shows the number of new NEP lists created and the accumulated number of such lists available on a yearly basis.
The year of 1998 saw the largest number of new NEP lists created, a necessity for launching the NEP service. While 11 new lists were formed in 1999, and 10 more in 2001, only 2 new lists were set up in 2000. The NEP leadership at that time felt that the service had matured. In late 2002, NEP moved to a new list software system. No new list was opened while technical changes were taking place, and new list information for 2003 is not available at the time of this writing. Via the network of 56 individual NEP lists, currently a total of 37,588 announcements of nep-all papers are made to 21,911 list subscribers. Further analysis and discussion of the individual NEP lists will be reported in a separate paper. As explained previously, an announcement in NEP refers to the inclusion of an individual paper from nep-all in a NEP list report. Figure 4 displays the total number of announcements all NEP lists made of nep-all papers every year. In 1998, the beginning year of the NEP service, it is understandable that fewer nep-all papers were announced because of the small number of reports NEP issued then. A leap in the number of announcements was subsequently observed from 2000 to 2001, parallel to the growth of nep-all issue size (see Figure 2). It should be noted that the announcement number in 2003 only includes the data for the first half of the year. We will not be surprised if the final figure of announcements for 2003 turns out to be the largest of all since 1998.
In a little more than five years, NEP has grown and developed into a distinctive and fully functioning current awareness service for the RePEc digital library. We present on a yearly basis the growth and development of the NEP service in terms of the nep-all publication frequency, average nep-all issue size, evolution of the NEP lists, and total number of announcements the NEP lists made. The NEP service overall has been enjoying healthy growth since its creation in 1998, with substantial progress between 2000 and 2001. In addition to its growth, NEP's performance over the years is to be examined in the next section from the perspectives of timeliness, coverage ratio and usage. NEP's PerformanceTimeliness, coverage ratio, and usage are the three parameters we have chosen to measure the performance of the NEP service in this study. As NEP is a current awareness service for digital libraries, we have selected our evaluation criteria accordingly. The evaluation frameworks suggested by, for example, Choudhury et al. [Choudhury et al., 2002] and Saracevic [Saracevic 2000], although pertinent, do not appear relevant for evaluating the current awareness service of digital libraries. TimelinessFor any current awareness service, timeliness should be one of the first dimensions to be considered. The value of a current awareness service will diminish if the service cannot be provided in a timely fashion. With respect to NEP, we define timeliness as the difference between the time when an issue of nep-all is distributed and the average time when papers from that issue of nep-all are announced in individual NEP list reports. Timeliness in this writing is measured in hours. Figure 5 shows the average timeliness data from 1998 to 2003.
Figure 5 indicates that 1998 is the year when NEP performed the best judging by timeliness. Several factors contribute to the result. First, NEP was more manageable then as it was young and small in size. Second, the editors of individual NEP lists were more enthusiastic about the NEP service when it was new. Third, the task for individual NEP list editors has become more demanding as nep-all has grown in frequency and size. It requires more time and effort on the editor's side to wade through the piles of papers issued by nep-all [Barrueco Cruz et al., 2003]. As described by Barrueco Cruz, Krichel and Trinidad, a series of technical changes and problems that occurred between 2000 to 2002 also affected the timeliness of the NEP service. According to Figure 5, the biggest delay in announcing nep-all papers took place in 2001. The significant increase of nep-all issue size (see Figure 2) partially explains this. As in many other situations, human beings need time to adjust to changes. In a sense, the sudden burst of nep-all papers brought challenges to the NEP list editors. A close examination of the 2001 nep-all issue size data reveals that only 8 of the 39 issues that year have fewer than 100 papers each. The rest of the nep-all issues may have carried as many as 478 papers in a single report. Across the years, the average number of hours needed to announce a paper from nep-all is 443, which is equivalent to less than 19 days. This figure does not seem very impressive by itself. However, the NEP management has since been taking measures to ensure a more timely service. As a result, the mean timeliness for the year of 2003 has been reduced to 199 hours from 478 hours in 2002. CoverageThree possibilities exist for papers contained in a nep-all issue regarding the number of announcements they receive: none, once, and more than once. Based on this categorization, coverage ratio is defined as the number of papers that received at least one announcement in a NEP list report, divided by the total number of papers in each issue of nep-all. In fact, coverage ratio deals with nep-all papers that are announced once and more, representing the second and third possibilities we just enumerated. As for papers in nep-all issues that receive no announcement, we plan to explore them in a future paper to find out, for instance, the suitability of the general structure that all the 56 NEP lists represent in economics. Figure 6 exhibits the average coverage ratio of the papers of each nep-all issue by year.
The fluctuations in coverage ratio appear small over the years with 0.068 (or less than 7 percent if expressed in percentage) being the largest difference found from one year to another in the data series. The mean coverage ratio throughout the time is 0.7 or 70 percent. In other words, 70 percent of the papers distributed in nep-all issues subsequently get announced via one or more NEP lists. There are on average 30 percent of the nep-all papers that do not receive any announcements. It was anticipated, based on common sense, that coverage ratio would change as the nep-all issue size gradually increases over the years (see Figure 2). The reality does not appear to support the above anticipation (r = -0.16, p = 0.05) while presumably hiding two contrasting trends. First, as more individual NEP lists are created, all else being equal, the coverage ratio should improve. On the other hand, it seems tempting to theorize that as the size of nep-all increases, again all else being equal, there will be a decline in the coverage ratio. Barrueco Cruz, Krichel and Trinidad have empirical evidence to corroborate this idea, but have no theoretical explanation [Barrueco Cruz et al., 2003]. We think that the theoretical underpinning of this observation could be that individual editors have an implicit target report size. That is, for each issue of their report, they will try to have a target size of papers. If there are few papers in nep-all, their selection will be more lenient. If the nep-all issue is large, they will be more selective. While timeliness clearly associates with the quality of awareness service, coverage ratio affects both the service quality as well as service substantiality. A current awareness service will not be highly regarded if only a small fraction of new information in a given area is reported. In this sense, we believe NEP performs quite well although no analogous data is available for comparison purposes. Multiple announcements of a single nep-all paper, mentioned in the beginning of the discussion on coverage ratio, result from the unique distribution network NEP has developed for disseminating new working papers in economics. The distribution network is unique in that 56 individual lists are currently employed in the NEP service with little coordination among them. Each NEP list editor is not aware of how other NEP editor colleagues work and is not required to consider if a particular paper is also announced in other list(s). In addition, individual NEP lists are created without an overseeing mechanism to ensure their mutual exclusiveness. They are only intended to loosely represent the subfields within economics. It is therefore common, rather than unusual, to have a single nep-all paper announced in more than one NEP list. Overlaps in announcement of nep-all papers can be treated as a synonym for multiple announcements. The average multiple-announcement ratio over the years is 0.4 or 40 percent of all the nep-all papers. Multiple-announcement ratio refers to the number of times a single paper was announced by more than one NEP list, divided by the total number of papers in each issue of nep-all. Of all the papers included in nep-all issues, 40 percent of them are announced by more than one NEP list. We will elaborate on the ramifications of this finding later when we associate multiple announcements with other NEP parameters examined in this study. UsageUsage of the current awareness information NEP provides, given the availability of data source, is assessed in the number of papers downloaded from the RePEc collection as a result of NEP list announcements. Even so, data of such downloads are only available from the nep-all issue dated October 27, 2002 and on, or the last 34 out of the 215 nep-all issues considered in this study. We understand the limited nature of the usage data (e.g., downloading does not necessarily lead to actual usage such as reading). The data, nevertheless, should be able to help reflect NEP's performance in usage to some extent. Figure 7 lists the average number of NEP-announced paper downloads from each nep-all issue in a month because some months (e.g., March 2003) had five nep-all issues whereas others had three (i.e., June 2003) or four issues (e.g., November 2002). Since there is only one nep-issue with download data in October 2002, it is not included in Figure 7.
As shown in Figure 7, average downloads of NEP-announced papers per nep-all issue peaked in February 2003. A possible explanation for this outcome is that people are usually ready to read more research literature after the holiday season is over. The same rationale perhaps can be used to explain the low downloads from November to December in 2002 during which major holidays fall. With the exception of March 2003, the remaining months in the data set recorded similar download volume. In addition, usage data gathered for this study is cumulative over time, so papers announced earlier in the time sequence automatically get more downloads. Note that we also cannot distinguish downloads as a result of NEP announcements from those done directly from RePEc. Out of the 10,164 papers announced in NEP list reports during the time period from which download data is available, 10,799 downloads were made. This number indicates that every NEP-announced paper on average is downloaded at least once. The objective of the NEP service, namely, to provide subscribers with up-to-date information being added to the RePEc digital library, is therefore met, because NEP-announced papers are downloaded for possible use. Nevertheless, as the access statistics provided by the LogEc service [4] suggest, NEP represents only a small portion of the total usage of RePEc. The user dimension undoubtedly influences the usage of the NEP service. We plan to research that area and other human related aspects of NEP when we are able to gather data of such kind. Potentials and Other Perspectives of the NEP ServiceWhat has been presented so far demonstrates that the NEP service, despite of the difficulties and problems it encountered [Barrueco Cruz et al., 2003], has been thriving since its creation, thanks to the joint and dedicated efforts of many volunteers. Looking at the interrelationships among the parameters we have examined, we find that downloading of NEP-announced papers correlates significantly (p = 0.01) with nep-all issue size (r = 0.58), number of announcements (r = 0.75), and multiple announcements (r = 0.77). These findings suggest that nep-all should attempt, by all means, to increase its size in order to facilitate more downloads. Meanwhile, either single or multiple announcements of nep-all papers should be further encouraged. They have an even greater impact than nep-all issue size in enhancing the value of the NEP service because NEP list subscribers may not download any papers if they are not announced. The immediate implication of these results seems at least two-fold: One is that existing NEP lists ought to try announcing more papers from nep-all. The other suggestion would be to create more lists to cover the 30 percent of papers that are reported in nep-all but receive no announcements from any existing NEP lists. Timeliness, as a measurement of NEP performance, has little impact on the downloading of NEP-announced papers. Since peer review of one paper in economics on average takes four years, the NEP delays appear really tiny in comparison. Besides, one could assume a paper, downloaded as a result of the NEP announcement, is sought mainly for its relevance and less so for its currency. Nevertheless, timeliness should be upheld as a standard for having quality current awareness services as we discussed previously. The issue size of nep-all shows a weak association with timeliness (r = 0.28, p = 0.01) and a strong impact on both number of announcements (r = 0.88, p = 0.01) and multiple announcements (r = 0.82, p = 0.01). In other words, a large nep-all issue size would not automatically delay the announcement of papers in that issue if the NEP list editors do their job right. But an increase in nep-all issue size would help augment the announcement of papers, and consequently downloads of those papers. With the recent implementation of the Mailman list software [Barrueco Cruz et al., 2003] as well as some rationalization in the list management, the NEP service is prepared to further expand its portfolio of individual lists. NEP shows great potential in its future operation and development as a current awareness service. NEP is a technically simple, yet organizationally pioneering effort in providing digital library services. Although it can be further improved, NEP perhaps would become an innovative model for current awareness services of digital libraries. Notes[1] RePec (Research Papers in Economics), <http://www.repec.org>. [2] NEP (New Economic Papers), <http://netec.wustl.edu/NEP/>. [3] PhysDoc, <http://physnet.uni-oldenburg.de/PhysNet/physdoc.html>. [4] LogEc, <http://logec.hhs.se>. References[Barrueco Cruz et al.] Barrueco Cruz, José Manuel, Krichel, Thomas, and Trinidad, Jerimiah C. (2003). Organizing current awareness in a large digital library. Presented at 2003 Conference on Users in the Electronic Information Environments in Espoo, Finland, September 8-9, 2003. <http://openlib.org/home/krichel/papers/espoo.pdf>. [Barrueco Cruz & Krichel] Barrueco Cruz, José Manuel, and Krichel, Thomas. (2000). Cataloging economics preprints: An introduction to the RePEc project. Journal of Internet Cataloging, 3(3), 227 -241. <http://openlib.org/home/krichel/papers/shankari.html>. [Choudhury et al.] Choudhury, Sayeed, et al. (2002). A framework for evaluating digital library services. D-Lib Magazine, 8(7-8), <doi:10.1045/july2002-choudhury>. [Chowdhury] Chowdhury, Gorbinda G. (2002). Digital libraries and reference services: Present and future. Journal of Documentation, 58(3), 258-283. [Cole et al.] Cole, Timothy W., Allen, Robert S., and Schmitz, John G. (2000). Building an outreach digital library collection. Illinois Libraries, 82(4), 239-250. [Condron and Tittemore] Condron, Lyn, and Tittemore, Cecilia Piccolo. (2001). Listservs and listprocs for catalogers. Cataloging and Classification Quarterly, 32(3), 25-29. [Davis and Lagoze] Davis, James, and Lagoze, Carl. (2000). NCSTRL: Design and development of a globally distributed digital library. Journal of the American Society for Information Science, 51(3), 273-280. [Fox and Urs] Fox, Edward, and Urs, Shalini. (2002). Digital libraries. Annual Review of Information Science and Technology, 36, 503-589. [Krichel] Krichel, Thomas. (2000). RePEc: An open library for economics. (To appear as a book chapter). <http://openlib.org/home/krichel/papers/salisbury.html>. [Saracevic] Saracevic, Tefko. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49(2), 350-369. [Schwartz] Schwartz, Candy. (2000). Digital libraries: An overview. Journal of Academic Librarianship, 26(6), 385-393. [Severiens et al.] Severiens Thomas, et al. (2000). PhysDoc - A distributed network of physics institutions documents: Collecting, indexing, and searching high quality documents by using Harvest. D-Lib Magazine, 6(12), <doi:10.1045/december2000-severiens>. [Sloan] Sloan, Bernie. (2001). Reference service in the digital library: A report on the ready for reference project. Library Hi Tech News, 18(10), 14-19. [Stackpole and King] Stackpole, Laurie E., and King, Richard James. (1999). Electronic journals as a component of the digital library. Issues in Science and Technology Librarianship, (22), <http://www.library.ucsb.edu/istl/99-spring/article1.html>. [Stemper and Butler] Stemper, James A., and Butler, John T. (2001). Developing a model to provide digital reference services. Reference Services Review, 29(3), 172-188. [Suleman and Fox] Suleman, Hussein, and Fox, Edward. (2001). The Open Archives Initiative: Realizing simple and effective digital library interoperability. Journal of Library Administration, 35(1-2), 125-145. [Xie and Wolfram] Xie, Hong, and Wolfram, Dietmar. (2002). State digital library usability: Contributing organizational factors. Journal of the American Society for Information Science & Technology, 53(13), 1085-1097. Appendix: Individual Lists of the NEP ServiceCopyright © Heting Chu and Thomas Krichel |
|
|
|
Top | Contents |
| |
|
D-Lib Magazine Access Terms and Conditions DOI: 10.1045/december2003-chu
|
