D-Lib Magazine
November 1999

Volume 5 Number 11

ISSN 1082-9873

Project Soup

Comparing Evaluations of Digital Collection Efforts

Michael L.W. Jones, mlj8@cornell.edu
Geri K. Gay, gkg1@cornell.edu
Robert H. Rieger, rhr1@cornell.edu
Human Computer Interaction Group (HCI-G)
209 Kennedy Hall, Cornell University, Ithaca, NY 14853

Since 1995, the Human-Computer Interaction Group at Cornell University (http://www.hci.cornell.edu) has conducted research or evaluation studies of a number of prototype efforts to build digital collections in museums and libraries. Our involvement as independent evaluators of these projects has provided us with a unique perspective from which to view various aspects of digital library development. We found that many of the successes and challenges that seemed unique to specific projects actually transcend the individual project boundaries. Many of the issues in digital library research we identified depend on technological, educational and cultural challenges that are likely to be quite different or even moot five years from now. Other challenges are likely to remain, however. The real value of highlighting common challenges comes from constructively applying such analysis to future challenges. This paper attempts to engage in such a future-oriented analysis of the key findings from five of the digital collection efforts that we have conducted.

Our findings are organized around three core areas:

"backstage" concerns (e.g., metadata, copyright and intellectual property issues)

collection maintenance and access (e.g., decisions regarding collection scope and the maintenance of a consistent quality and fidelity of digital records)

usability findings (e.g., user skill levels and expectations, and the use of collections in formal and informal educational settings).

The paper concludes with a look at how this evaluation process will guide our future research in the field and how it may serve to contribute to digital library development. Our specific recommendations are found at the conclusion of this paper.

Projects

Our selection of projects is not a representative sample of all digital library efforts, nor is it a full inventory of digital collection efforts on which the HCI-G has collaborated. However, we feel that this sample does address many key issues in digital library research. The scope of these selected projects also makes this sample useful in highlighting issues pertinent to digital collections. All of the projects included in our sample involve collections of historical cultural artifacts. This relative degree of homogeneity reduces the chance that observations are, in fact, due to issues related to dissimilar collections.

The following table highlights various facets of the five projects in question. Pertinent details on these projects will be raised throughout the analysis. Where possible, we provide hyperlinks to project sites so that the reader can investigate a particular project further if desired.

 

Project	Description	Materials	Active?	Year of study	Methods Used - HCI-G	Sponsor
Making of America (MOA)	Cornell U/U Michigan collaboration to digitize and serve collections of American history artifacts	Documents and historical artifacts, 1850-1877	Yes	1995	Preliminary user studies, surveys, concept mapping, classroom experiments	Mellon Foundation
Global Digital Museum (GDM)	British Museum/IBM Japan collaboration to create an interactive collection of museum artifacts	Digital images of museum artifacts	Testbed, testing continues	1996-7	Classroom student test/ survey	IBM Japan
Museum Educational Site License (MESL)	Common database of cultural heritage products from seven US institutions	Digital renditions of a subset of participating museum collections	Testbed, now offline	1996-7	Project Staff And Expert User Survey	Getty Museum
Art Museum Image Consortium (AMICO)	Common system for 28 North American museums	Digital renditions of a subset of participating museum collections	Yes	1998-9	Lay and Expert User Groups Survey (online)	AMICO
Frick Art Reference Library /Cornell University Libraries	Needs assessment for prototype for the Frick Art Reference Library and Cornell's A.D. White collection.	Full digitization of two defined subsets of larger collections	Prototype, not active	1998-9	Expert User, Faculty and Project Staff Focus Groups	Private Donors

 

Each of the five projects outlined above address specific needs and user populations and can be seen as distinct efforts addressing context-specific concerns. The projects outlined above represent many of the goals, objectives, challenges and obstacles inherent in the design and implementation of digital archives and collections.

Backstage Issues: Laying an Appropriate Foundation for Effective Digital Collections

It is important to realize that the formation and maintenance of any given physical or digital library infrastructure is not predestined or inevitable. Rather, the foundation of any collection is influenced by a myriad of social, organizational and technical interests, norms and debates. Library collections, regardless of medium, are socially constructed artifacts that both shape and are shaped by the needs, values and practices of both administrators and users (Kilker & Gay, 1998).

In many collections, the infrastructure has been rationalized and reified to such a point that end users and administrators alike see it as relatively commonplace and unproblematic. Such infrastructures appear transparent to involved stakeholders. Digital collections, however, can pose significant challenges to the transparent infrastructures of traditional collections. Such collections frequently require that core principles (e.g., metadata standards, copyright and intellectual property management) of collection management be revisited. In the short-term, such a renegotiation of basic principles may make previously transparent infrastructures more opaque -- which in turn may raise significant difficulties for both end users and administrators alike. In our evaluation work with the Frick Art Reference Library, for example, administrators and users found themselves attending to basic cataloging and access protocols dating back to the formation of the library itself.

Metadata design and implementation

Much previous digital library research has shown that the introduction of digital artifacts and representations into a collection can highlight the limitations of traditional metadata structures. New mediated objects such as on-line texts and correspondence, digital images, Web sites, or digital referents/aliases to physical artifacts are not well represented, nor are they adequately indexed for search and retrieval (Rust, 1998; Smith, 1996). Traditional metadata is also not effective in addressing copyright obligations and restrictions (Lagoze, Fielding & Payette, 1998; Rust, 1998).

Planning and implementing a workable metadata standard for diverse digital artifacts is a complex issue with numerous potential solutions. One simple, albeit restricted, solution to metadata issues is to maintain a relatively homogenous collection, borrowing heavily from existing and compatible metadata standards. Our three testbed projects (GDM, MESL, and AMICO) each borrowed from available object metadata, adding only project-specific fields (e.g., original source of image, metadata regarding digitization). As the collections in question were limited arbitrarily in terms of size, media variety and scope, such an approach to metadata proved to be relatively satisfactory.

Universal standards provide a more robust and scalable solution to metadata issues in digital collections. As current social and technical research on such standards (Baca, 1998; Stephenson & McClung, 1998; Weibel, 1999; Weibel & Hakala, 1998) suggests, however, achieving such a common standard is not a simple process. In the Frick/Cornell project, it was determined that creating a mutual metadata standard most likely would not be fruitful, as the collections in question failed to share enough metadata classifications. The unattributed photoarchives of the Frick, for example, required significant information regarding established and suggested provenance, the authenticity of this information, and ownership history. Much of this information was not directly relevant to the A.D. White collection.

Incompatibility between collections was also a factor in the MOA project. Collection management staff at Cornell and Michigan avoided any controversy regarding metadata by agreeing to disagree early in the process. This led to the establishment and maintenance of two separate, yet related, collections at first, followed by post facto efforts to increase interoperability and compatibility between the two repositories of information. While this process did allow both efforts to develop into mature collections, it was accomplished at the expense of creating a transparent solution that would serve both collections and both communities simultaneously. It is telling that the next iteration of MOA, Making of America II, is primarily concerned with establishing a common, robust and scalable metadata framework.

Copyright and intellectual property issues

Like metadata, the issue of copyright and intellectual property protection is not a new one for collection administrators or end-users, nor is it an issue that is particular to digital collections alone. Digital technologies complicate the negotiation of intellectual property rights, as digital objects are copied and disseminated easily and rapidly at little to no per-unit cost to the user. Digital technologies, however, also offer collection managers new means of controlling licensed access to information through password protection, Internet address screening, user tracking, and data watermarks. On the other hand, these solutions may interfere with the transparency of the system and complicate legitimate end-user access to information (Gay, Rieger & Sturgill, 1998; Gay & Rieger, 1999).

As with metadata, the simplest solution is to structure the collection such that copyright and intellectual property concerns are negligible. In both the Frick and MOA efforts, the objects in question either a) were owned by the collection (Frick), or b) were already in the public domain (MOA). In both cases, the potential threat of unauthorized access, copying and dissemination did not prove to be a salient issue. This beneficial arrangement was primarily the result of voluntarily imposed limits on the scope of the collection designed to ensure compliance with copyright law.

It is interesting to note that even in instances where objects in a digital collection may appear to be in the public domain, there are still significant access and rights issues to be addressed. All three museum testbed projects (GDM, MESL, AMICO) were restricted to licensed users of the system, despite the fact that most of the material was of an age that creator-held copyright ceased to be an issue.

This situation was due, in part, to the fact that each collection was intended to be an experimental testbed and not a wide-scale implementation. Full public access would have caused strain on the prototype systems used and likely would have made research and development much more problematic. However, it is also important to take into account the intellectual property concerns of the stakeholders involved -- in this case, museums. While most museums exhibit and maintain their collections on a not-for-profit basis, their existence does remain intrinsically intertwined with the intellectual properties they manage. Museum collections also play a key role in sustaining the museum’s public image, which in turn can be leveraged to sustain a level of status and prestige in the research and academic community. Providing free access to digital renditions of a museum’s core collection may reduce museum attendance and revenues from the sale of related products such as collection catalogs. Understandably, museums may find it difficult to relinquish control over their core properties, especially to collection efforts like GDM, MESL, and AMICO that do not explicitly aim to place an individual museum’s identity and image front and center.

Such museums may choose to participate in larger digital collection efforts on a piecemeal basis, providing limited access to only a small slice of their full collection. Varying levels of organizational cooperation and commitment proved to be a consistent challenge in both the GDM and MESL projects, with some stakeholders being more open and co-operative than others. Such inter-organizational conflicts may create an uneven balance in the scope of the resulting digital collection. (Martin, Rieger & Gay, 1999)

To summarize, metadata incompatibilities may result in a series of related but distinct databases that must be accessed separately, frustrating any effort to create a cohesive collection. Copyright concerns may restrict or even bar access to contemporary material, and intellectual property issues may even complicate access to material in the public domain.

Establishing an Effective Content Base: Finding a Balance Between Quantity and Quality

Even after accounting for backstage issues, we are left with issues regarding the breadth and quality of the digital collection. In an ideal world supported by limitless human, technological and financial resources, users of a digital collection would receive quick access to all high-fidelity, high-quality digital records pertaining to any topic, no matter how specialized or obscure. Real world implementations, of course, must aim to strike an effective compromise regarding the scope and quality of records made available.

Quantity: Achieving Critical Mass

Striking an appropriate balance often means delimiting the scope of the collection such that it provides a critical mass of information for the intended user base. While the concept of critical mass is linked to the quantity of objects in a given database, it is difficult to determine some "magic number" that can be used as a quantitative benchmark. While a collection of 250,000 digital images may seem impressive at first, it fails to impress if the intended scope is "all available images from all time periods and cultures worldwide." Similarly, a collection of 500 popular European 18^th century paintings may be useless for advanced research, but quite appropriate and thorough for lay users. In other words, critical mass implies not simply quantity, but also quantity appropriate to the task.

Issues of appropriate scope were especially prevalent in the museum testbed projects. GDM, MESL and AMICO all established a very broad scope that suggested to users that all institutional holdings were accessible. While this approach does facilitate a wide range of potential uses and users, it may also lead to disappointment and frustration. While user satisfaction for MESL and AMICO was quite high, open-ended comments in the questionnaire noted a) a wide range of needs and objectives and b) numerous requests for more images to meet those needs. While just 8% of AMICO users noted that they did not find what they were looking for, only 35% of users noted that they found precisely what they were looking for, with the remainder of users noting either browsing behavior or finding something related to, but not matching, what they were investigating. This lack of critical mass suggests that these services may be more applicable for general-purpose uses than specific structured research. GDM users were relatively confident that they would use the site for "entertainment" purposes (3.4 mean on a scale of 5, with a median of 4) but were less certain of its use in educational situations (2.8 mean on a scale of 5, median 3).

The MOA project, on the other hand, aimed to achieve critical mass through explicitly and narrowly defining the scope of the project from the outset (e.g., Crane, 1996). The original MOA scope was restricted to American historical documents dated from 1850-1877. The initial scope of MOA was restricted to this period for a variety of administrative, legal and pedagogical concerns, including the public domain nature of the content base, the need to preserve rapidly decaying artifacts from that era, and sustained interest in this turbulent period of American history. This decision also served to make the digital collection more manageable and, by extension, made it plausible for administrators at Michigan and Cornell to create a critical mass of documents from that period. The necessary tradeoff, of course, is to deny service to users interested in documents dated before or after this arbitrarily-prescribed area. Making such design decisions explicit, however, helps mitigate any user disappointment by encouraging users with interests outside the delimited scope to look elsewhere.

While such arbitrary limitations may be necessary in order to make a digital collection both plausible and valuable, decisions regarding where such limitations are placed are difficult to make and ideally should be done only after careful deliberation and consultation. At the Frick Art Reference Library, administrators chose a sub-section of their unattributed photoarchive collection as a starting point for a larger digitization effort. This decision was based on similar concerns to MOA: the collection was a manageable size, in relatively poor physical shape, owned by the library, and in need of reorganization and classification.

Users, however, were initially unsure of the merits of beginning with the photoarchive, noting that other segments of the art reference library likely would be more popular with both habitual users and a wider audience. In this case, especially intriguing about the focus group process was how user opinion shifted. Library staff answered questions about their selection process and the relative merits and drawbacks of alternative starting points. The user group became aware of administrative details that were largely transparent from the perspective of the user. Upon realizing these details and considering their suggested alternatives in light of administrative concerns, most users agreed that the staff had made the appropriate decision regarding collection scope. Involving users, in this case, not only validated initial assumptions, but also increased the legitimacy of the decision, at least among this select group of users.

Quality: Ensuring Fidelity and Accuracy

Substantial research and development on digitization (Entlich, Garson, Lesk, Normore, Olsen & Weibel, 1997; Shaw & Blumson, 1997) may lead one to believe that achieving optimal technical quality is of paramount concern. While this may be an optimal solution, the importance of record quality in digital collections can also be seen as context-dependent.

It is possible to conceive many usage situations where optimal detail may be superfluous or excessive. The museum prototype sites included images that primarily were scanned from secondary sources (e.g., catalogs, slides, transparencies) and intended for delivery on the World Wide Web. Standard computer displays offer a resolution of 72 dots per inch (dpi) -- a relatively coarse level of detail, but fine for generic browsing. Higher resolution scans -- generally useful for printing and capturing fine detail -- can be scanned at higher resolutions. Doing so, however, dramatically increases storage and bandwidth requirements both at the server and for the end-user. On-line image databases must be designed with these constraints in mind. With respect to early on-line efforts such as GDM and MESL, a great deal of potential detail was sacrificed in the interest of making the system usable in an average usage setting. Users of both systems did not raise substantial issues with respect to image quality, save to note that "larger" images were more helpful and more impressive than smaller ones. For these general-interest user groups, low-fidelity imagery scanned to meet the boundaries of the average computer screen would likely suffice.

Similarly, it may not be necessary for textual records to be displayed at very high levels of resolution. Printed MOA documents displayed as 600 dpi TIFF files may provide optimal quality for printing purposes, but in many instances such fine detail may be more aesthetic than functional in nature. If one was interested in a particular artifact from a design or historical preservation perspective, high quality scanning may provide valuable information about the aesthetics and qualities of the physical documents. If one was interested only in the informational content of the document, however, the scan itself may be superfluous -- a plain text file would suffice. MOA accounted for both usage patterns by providing either a scanned document or a plain text version translated automatically using optical character recognition (OCR) technology.

In other situations, however, attention to detail can be quite important. In the A.D. White collection faculty focus group, one participant noted to our amazement that a photograph in the collection was heavily altered, likely in an effort to represent the building in a more pristine condition than it exists in reality. This discovery would not have been possible had the image only been made available at a low level of resolution. Frick users also noted that high-resolution scans would be necessary to facilitate detailed and advanced art research. Storing and providing images at the maximum plausible resolution available may best serve in-depth research.

Care must also be taken to ensure the highest standard of accuracy in digitization. Any conversion process -- whether automatic or manual -- is prone to some degree of data corruption or loss. Such errors are especially damaging in digital collections. Since digital collections are, in essence, a collection of data objects and defined relationships among objects, errors in an object’s data profile can cause that object to become and remain permanently lost to the world. Frick users in particular were quite sensitive to this concern, noting instances in both physical and digital collections where poorly marked, mistyped or misattributed data in a record caused it to disappear from the research community until found by happenstance or luck. Error checking and verification processes such as those in MOA are essential to ensure the accuracy of a digital collection.

Usability: Meeting the Goals and Objectives of the End-User

In the early days of networked information, "build it, and they will come" may have been a sufficient model of user interest and behavior. Increasingly, however, it is imperative that digital collection designers proactively address the particular needs and challenges of its intended users. A digital collection can contain a critical mass of high quality, copyright-cleared content all organized around a solid metadata foundation, and still prove to be a failure. A collection is not likely to meet the needs of its audience if the collection demands in-depth knowledge of computer programming or access to fiber-optic networks. Similarly, if the scope of the collection is so narrowly defined as to exclusively support the interests of a learning community of twenty people, it is unlikely that any attention to metadata, copyright, critical mass, record quality, and bandwidth concerns will increase the size of the audience. In an age where digital libraries are competing with commercial World Wide Web sites for attention, usability issues cannot be ignored.

Technical Challenges

As noted earlier, technical obstacles can be of central importance in the design and implementation of digital collections. In the GDM and MESL projects in particular, users consistently noted slow response and download times as barriers to access. This complaint is especially salient with respect to casual and lay users, who are less likely to cope with large lag times and technical glitches.

MESL and AMICO users also expressed concerns with the chosen method of licensing control. MESL users were asked for passwords on a regular basis, presumably in an effort to determine whether the same licensed user was still in charge of a particular active session. While an effective technical solution to the issue of licensing, users found the repeated queries for passwords quite frustrating and noted that it detracted from the learning experience. The AMICO solution to licensing issues, site-specific access, is quite common in other on-line digital collections (e.g., ACM Digital Library, CIOS). Access in this model is tied to a specific domain name or, in some instances, a specific group of IP (Internet Protocol) addresses. This is effective in institutional settings (e.g., universities, corporations) provided that users consistently access the collection from "approved" locations (Pinfield, Eaton, Edwards, Russell, Wissenburg & Wynne, 1998). As the use of mobile computing solutions and third-party Internet service providers increase, however, this solution becomes less tenable.

Meeting a Wide Variety of User Needs and Abilities

An effective digital collection must explicitly account for the skill level of its intended user base. Early on-line efforts, in particular, had to address the possibility that users may be expert in their chosen area of study but may be relative neophytes in computing technology and computer networks. While it may be tempting to believe that today’s users are more computer literate, this cannot necessarily be assumed. For example, the Frick Library user group -- the most sophisticated user sample from the perspective of content mastery -- reported that they were relatively unskilled with respect to computing technology. Supporting users of all levels by providing clear instructions, an elegant user interface design, and context-sensitive help assists in transcending obstacles created by lack of computer literacy.

A robust digital collection should aim to meet a wide variety of user behaviors with respect to collection materials. Effective collections are not mere repositories for information. In many instances, the use-value of a particular collection is only indirectly related to issues of content and organization. Active interaction with both collection material and with other users of the collection makes truly special collections.  Ideally, a digital collection should be an active social and pedagogical space in which users interact with items in the collection, develop their own personal interpretations and views, collect and organize their favorite items in a manner appropriate to their needs, and share all the above with other users.

Even when there are no explicit provisions for an interactive social and pedagogical space, active participation with the collection is still common. The MESL end-of-project survey notes that the system actively encouraged faculty and student use of images in research and teaching. Similarly, AMICO users noted a variety of uses for images found in the database, including integration into class presentations and papers, personal research databases, and printouts for personal, non-academic use. MOA reports note that the collection has not only addressed anticipated historical uses, but also related uses such as tracing the ontology of specific words, investigating genealogy, and more general interest use.

Explicitly designing such interaction with collection materials and other collection users is challenging but worth the effort. The GDM prototype explicitly supported such interactions by allowing users to collect a personal scrapbook of favorite objects, annotate specific works, and reach a forum in which to interact with others regarding objects in the collection. This feature was quite well received by users of the system.

Similarly, the prototype Frick/Cornell database allowed users to connect to annotations of records directly -- a common practice in the physical Frick Library collection. Such a system would allow the digital collection to foster continued scholarly debate concerning attributions and provenance issues pertaining to the unattributed photo archive. Frick users, however, did raise questions regarding the use of such a feature, fearing that it would be misused in practice and lead to an avalanche of commentary of dubious merit.

Conclusions and Future Research Directions

Comparative analyses of past research findings are instrumental in noting how seemingly particular and local efforts can inform common salient themes and recurring debates. The point of comparative analysis is somewhat moot, however, unless we use information gleaned for the effort to help shape future research and development efforts.

Some conclusions we can draw include:

• Effective digital collections are complex sociotechnical systems: An effective collection requires consistent and simultaneous attention to a variety of social, organizational, administrative, and technical concerns (Kilker & Gay, 1998; Kling, 1999). This, in turn, suggests that a diverse team of people spanning a wide range of skills and abilities must manage the establishment and maintenance of such a collection. Ideally, the necessary human and financial resources will support this organization.
• Involve stakeholders early: Early in the MOA study, we asked participating faculty, librarians and designers about a range of issues relevant to themselves and their position (Kilker & Gay, 1998). While there were some similarities among the lists, there were also marked differences. If we had not discovered these discrepancies early in the process, development would have continued based on incorrect or incomplete visions of the needs of end-users and librarians.
• Backstage, content and usability issues are highly interdependent: Decisions regarding one facet of a digital collection effort may indirectly constrain or enable various choices in other areas. In making any major decision, attention should be paid to the foreseeable potential consequences for background, content, and usability concerns.
• Background issues should be "translucent" vs. transparent: Complex, confusing and excessive displays of metadata or access barriers interfere with use and frustrate users. Transparent systems are preferable, but systems that are too invisible do not provide the user with any information regarding the structure, purpose and limitations of the collection. A "translucent" solution would allow enough light to shine through while consistently, yet unobtrusively, reminding users of the system’s abilities, affordances and constraints (Norman, 1990).
• Determine collection organization, copyright, quality and quantity goals around social, not technical or political, criteria: Collection size and quality should be shaped around present and near-future use patterns and user behaviors, rather than technically optimal or abstract political goals.

• Design around moderate but increasing levels of hardware and user expertise: Expect that many of today’s technical and skill obstacles will be addressed as computing technology matures and literacy increases. That said, be aware that operating at the cutting edge of technology may alienate less sophisticated users who may be using less developed technical infrastructures.
• "Market" the collection to intended and potential user groups: Actively solicit information on the needs and objectives of intended user communities, and aim to meet and/or surpass those needs. Actively encourage potential users of the collection to discover what the collection has to offer. User studies can be seen as fulfilling this role implicitly by encouraging user "buy-in" throughout the design and implementation process.
• Look elsewhere for new directions: Borrow existing metadata standards and practices from related fields (e.g., World Wide Web metadata and technical standards, relational database design, object-oriented programming). Borrow relevant findings from computer-supported collaborative work and human-computer interaction research. Take inspiration from emerging Internet trends (e.g., personalized information services, push technologies, Internet clubs).

As the field of digital library research matures, it will be increasingly difficult to treat Digital Library (DL) projects as distinct, short-term efforts structured by technical and organizational research questions alone. While metadata, copyright, access, and digitization issues will continue to be of importance in future efforts, exogenous developments in technical infrastructure, shifting user skill levels, expectations and habits will play a strong mediating role in determining the long-term viability of a DL effort. Integrating users into the design and implementation process early and using their input to structure research priorities may change the course of future DL research, but it will likely do so in a way that will better facilitate the creation of sustainable and functional DL sociotechnical systems.

Bibliography

Baca, M. (1998) (Ed.).Introduction to Metadata: Pathways to Digital Information. Los Angeles: Getty Information Institute [On-line]. Available: http://www.getty.edu/gri/standard/intrometadata/msctbl1.htm

Crane, G. (1996). Building a Digital Library: The Perseus Project as a Case Study in the Humanities. ACM Digital Libraries Conference 1996, Bethesda, MD. ACM Press.

Entlich, R., Garson, L., Lesk, M., Normore, L, Olsen, J., & Weibel, S. (1997). Making a Digital Library: The Contents of the CORE Project. ACM Transactions on Information Systems, 15(2), 103-123.

Gay, G., Jones, M. & Rieger, R. (1999). Cornell/Frick Partnership Focus Group Report. Internal Report.

Gay, G. & Rieger, R. (1999). AMICO University Testbed User Survey. Internal Report.

Gay, G., Rieger, R., Martin, W. & Orlov, M. (1998). Global Digital Museum Prototype Evaluation. Internal Report.

Gay, G., Rieger, R. & Sturgill, A. (1998). Findings of the MESL Casual User Survey. In C. Stephenson & P. McClung (Eds.). Delivering Digital Images: Cultural Heritage Resources for Education. Los Angeles: Getty Information Institute.

Kilker, J., & Gay, G. (1998). The Social Construction of a Digital Library: A Case Study Examining Implications for Evaluation. Information Technology and Libraries, June, 60-70.

Kling, R. (1999). What is Social Informatics and Why Does it Matter? D-Lib Magazine, January [On-line]. Available: http://www.dlib.org/dlib/january99/kling/01kling.html.

Lagoze, C., Fielding, D., Payette, S. (1998). Making Global Digital Libraries Work: Collection Services, Connectivity Regions and Collection Views. ACM Digital Libraries 1998, Pittsburgh, PA, ACM Press.

Martin, W., Rieger, R., & Gay, G. (1999). Designing Across Disciplines: Negotiating Collaborator Interests in a Digital Museum Project. Proceedings of ICHIM'99, Washington, DC, September.

Norman, D.A (1990). The Design of Everyday Things. New York: Basic Books.

Pinfield, S., Eaton, J., Edwards, C., Russell, R., Wissenburg, A. & Wynne, P. (1998). Realizing the Hybrid Library. D-Lib Magazine, October [On-line]. Available: http://www.dlib.org/dlib/october98/10pinfield.html.

Rust, G. (1998). Metadata: The Right Approach. D-Lib Magazine, July/August [On-line]. Available: http://www.dlib.org/dlib/july98/rust/07rust.html

Shaw, E.J. & Blumson, S. (1997). Making of America: Online Searching and Page Presentation at the University of Michigan. D-Lib Magazine, July/August [On-line]. Available: http://www.dlib.org/dlib/july97/america/07shaw.html.

Smith, T.R. (1996). The Meta-Information Environment of Digital Libraries. D-Lib Magazine, July/August [On-line]. Available: http://www.dlib.org/dlib/july96/new/07smith.html

Stephenson, C. & McClung, P. (1998) (Eds.). Delivering Digital Images: Cultural Heritage Resources for Education. Los Angeles: Getty Information Institute.

Weibel, S. (1999). The State of the Dublin Core Metadata Initiative: April 1999. D-Lib Magazine, April [On-line]. Available: http://www.dlib.org/dlib/april99/04weibel.html

Weibel, S. & Hakala, J. (1998). DC-5: The Helsinki Metadata Workshop. D-Lib Magazine, February [On-line]. Available http://www.dlib.org/dlib/february98/02weibel.html

Copyright © 1999 Michael L.W. Jones, Geri K. Gay, and Robert H. Rieger

(Some HTML coding was changed 11/18/99 to replace em dashes with double dashes to accommodate users of earlier versions of web browsers.)

(On 12/20/99, the following note was added at the request of the authors: The authors wish to clarify that AMICO is a broad and active service that has conducted a wide range of research on metadata issues, access, and the economics of digital information. Information on the full project can be accessed at  <http://www.amico.org >. Information on AMICO's data specifications are available at  <http://www.amico.org/docs/dataspec.html >. Readers may also find helpful the following articles related to museum rights and the economics of digital imaging projects: 1) Models for Digital Libraries of Primary Resources," invited contribution, #4, 1998, pp 71-91. Available online at  <http://www.archimuse.com/publishing/amico/ >. 2)David Bearman and Jennifer Trant. "Museums and Intellectual Property: Rethinking Rights Management for a Digital World," Visual Resources, Special Issue, Copyright and Fair Use, The Great Image Debate, Vol. XII, no. 3-4, 1997, 269-280.)

Top | Contents
Search | Author Index | Title Index | Monthly Issues
Previous story | Next story
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/november99-jones