D-Lib Magazine
November 2005

Volume 11 Number 11

ISSN 1082-9873

Questions & Challenges Arising in Building the Collection of a Digital Library for Education

Lessons from Five Years of DLESE

Introduction

Education has long functioned as a cottage industry, in which individual craftspeople handcraft individual products for distribution to their local communities. The National Science Digital Library (NSDL) [1] and its component Digital Library for Earth System Education (DLESE) [2] offer the potential of a fundamental paradigm shift, in which educators craft educational resources in their area of greatest expertise and interest, and draw upon a national library of existing, high-quality materials for teaching and learning about other topics.

It's easy to write these words, but much harder to build a library that lives up to this promise. This article articulates some of the major questions and challenges that we, and our many collaborators, have faced while building the DLESE collections. The first part of the article describes questions for which DLESE has found at least tentative answers; the latter part describes tensions and challenges with which DLESE is still grappling. The intended audience includes builders of other education-oriented digital libraries who wish to know what lies ahead, and users of DLESE who wish to understand how DLESE works and why.

Resolved Questions

This section of the article concerns questions for which the DLESE community has agreed upon at least tentative answers. For each question, we articulate DLESE's answer, recognizing that different answers may be appropriate for other libraries. [3].

What is the goal of your collection-building effort?

DLESE's collection-building goal is: "To assemble and foster the creation of educational materials from the community that are rich in science content; diverse in format, content, and topics covered; fresh, dynamic, and interactive; instructor-ready; and integrative across traditional disciplines." [4]

Who is your audience?

A public or academic library typically begins with a pre-defined, semi-captive user base, for whom the library collections and services are specifically developed. In contrast, a publicly funded digital library like DLESE begins with no user base and must attract its user base – in large part, by building a collection that is of value to its audience.

DLESE aspires to provide resources for everyone who wishes to learn or teach about the Earth or environment "from K to grey." In practice, DLESE avoids highly specialized papers written for scientists seeking information in their area of research, feeling that such needs are well-served by GeoRef [5] and other indices into the scientific literature. In this we differ from NSDL, which does include collections of scientific papers [6].

On what range of topics will your library collect materials?

The scope of resources that are appropriate for inclusion in your digital library should be spelled out in a scope statement. DLESE's scope statement says "...the DLESE collection provides access to Earth system education materials with particular emphasis on interdisciplinary areas....All geographic areas on Earth are included in the scope of the collection, as well as the atmosphere and the solar system" [7]. "Earth System" is further defined as: "processes, states, cycles and interactions among the atmosphere, biosphere, geosphere, hydrosphere and space from a global to local perspective, across time scales. This includes the application of physical and chemical laws with mathematics to describe physical, chemical and biological processes and interactions of the spheres. DLESE's vision of the Earth system also includes environmental issues in policy, law, economics, health sciences, and engineering."

In practice, DLESE has welcomed resources that deal with constituent disciplines of Earth Science, resources that deal with interactions between natural systems and humanity, and resources that deal with educational matters of interest to the DLESE community (Figure 1). In other words, DLESE has welcomed resources that can be aggregated to craft an interdisciplinary education, not just resources that are interdisciplinary when considered in isolation.

Figure 1: There has been much debate over the appropriate scope of DLESE's content domain. In this Venn diagram, the upper right circle represents resources in the classic science disciplines that deal with natural Earth processes. The upper left circle represents resources that deal with the Earth system, including interactions between humanity and the environment. Some of these are science resources (in the cross-hatched overlap), but others may focus on non-science aspects of human-environment interactions, such as law, policy, or economics. The bottom circle represents educational resources, which may or may not pertain to the Earth explicitly. In practice, the DLESE Collection has accessioned resources in all three fields – while trying especially hard to find resources in the overlapping areas.

Will you have sub-collections within your collection?

A "collection," in the DLESE context, is defined as a set of metadata records for "a group of related resources that reflect a coherent, focused theme" [8]. All educational resources accessible through the DLESE Discovery System are collectively known as "The DLESE Collection." The DLESE Collection can be subdivided along two axes (Figure 2): according to degree of review scrutiny (i.e., Broad vs. Reviewed) and according to the means by which the resources entered DLESE (i.e., Themed vs. Community).

Figure 2: The DLESE Collection (symbolized by the grey field) can be subdivided along two dimensions: according to degree of review scrutiny (the DLESE Broad Collection versus the DLESE Reviewed Collection), and according to the means by which the resources entered the collection (the Community Collection versus themed collections).

The DLESE Reviewed Collection consists of resources that have been rigorously reviewed against seven selection criteria:

1. scientific accuracy,
2. pedagogical effectiveness,
3. completeness of documentation,
4. ease of use for teachers and learners,
5. ability to inspire or motivate learners,
6. importance or significance of the content, and
7. robustness as a digital resource.

There are multiple pathways to the Reviewed Collection, but all pathways must address these same selection criteria [9]. The rationale for maintaining the Reviewed Collection is to help library users find exemplary teaching and learning materials and to help resource creators achieve academic career recognition. The DLESE Broad Collection comprises resources that are relevant to Earth System Education and function reasonably well, but that have not been scrutinized against the seven selection criteria of the Reviewed Collection. The rationale for maintaining the Broad Collection is to provide an extensive variety of resources and to create a forum in which users' feedback helps creators iteratively improve the quality of individual resources.

The DLESE Community Collection comprises resources that have been individually contributed and cataloged, either as part of funded gathering/cataloging efforts or as voluntary contributions from individuals [10]. DLESE themed collections are distinct named sub-collections within the DLESE Collection, built around an organizing principle, such as topic or audience, and accessioned into DLESE as a whole [11]. The DLESE Discovery System permits users to confine their searches to specific themed collections. A themed collection may be part of the DLESE Reviewed Collection or part of the DLESE Broad Collection. The advantages of having themed collections are that the cataloging effort is distributed broadly, and specialized communities are mobilized to contribute in their areas of deepest expertise and interest.

Are you building a library of resources or pointers?

DLESE does not hold or disseminate the actual educational resources. DLESE's collections consist of pointers to, and information about, web-accessible materials. However, some thematic sub-collections within DLESE do hold and serve the actual educational resources, for example the Vision Learning Collection [12]. Regardless of the nature of the content, it's important to make decisions regarding long-term storage, version tracking and refreshing of data as standards and technology change. DLESE archives the metadata and tracks changes in versions of the metadata, but does not take responsibility for long-term access to the full content indexed.

What should be the granularity of items in your collection?

Bricks and mortar libraries collect physical objects, such as books, videotapes, and maps. Physical objects have physical boundaries that separate one library resource from another in a fairly obvious way. Where the boundaries are less obvious (e.g., journal articles), librarians can draw on long-standing policies and procedures. Not so in a library based on the World Wide Web, where one bit of web content can be nestled inside another, with boundaries that are ambiguous or arbitrary.

Early in DLESE's history, there was intense discussion about the desirable "granularity" (coarseness or fineness) of individual items appropriate for cataloging. This question has been settled in favor of creating separate catalog records for each entity that has "distinctive educational, technical or pedagogical information" [13], because that seems to best serve the needs of DLESE users. A "resource," in the DLESE context, is "any web-accessible teaching or learning material" [14]. DLESE resources may be lesson plans, computer activities, data sets, audio files, reference materials, syllabi, tutorials, assessment instruments, or any other material of use for teaching or learning about the Earth and environment.

How do you describe an educational resource?

DLESE is a library of metadata. A library with poor metadata is a library in which users cannot find what they want and library administrators cannot assess the scope and balance of their collection. Early decisions about metadata (for example, DLESE's controlled vocabulary for "topics") can become long-term fixtures of the library, because it is so difficult to revise metadata for a library of thousands of resources.

To describe individual resources, DLESE uses the metadata framework summarized in Table 1 [15]. The resource description focuses on what's in the resource, who made it, and what circumstances may limit who can use, or would wish to use, it. The latter includes grade level, copyright, language, cost, technical requirements, and alignment with educational standards. DLESE's metadata framework is extensively documented at the DLESE web site [16].

How do you associate accurate metadata with each resource?

DLESE has developed an online tool (the DLESE Collection System or DCS [17]) that embodies the DLESE metadata framework and Cataloging Best Practices guidelines [18]. Catalogers use the DLESE Collection System to populate the resource metadata fields with specific information pertaining to a particular resource. Before a resource is accessioned into the library, the quality and consistency of the metadata are checked by means of a series of automated and manual steps [19].

Alternatively, if a themed collection already has its resources cataloged in a metadata format that differs from DLESE's framework, the DLESE Program Center supports transforming or "cross-walking" the resource metadata from the native format [20]. The advantage of providing this cross-walking option is that it has allowed DLESE to welcome valuable heritage collections containing resources of interest to our users. On the other hand, providing cross-walking is labor intensive, and it introduces inconsistencies in the metadata collection in those cases where the fields and vocabularies of the native framework don't map exactly onto DLESE's framework. Such inconsistencies can limit precision in searching and browsing, and can distort results in collections assessment.

Who will do the cataloging?

Many hands and eyes have been involved in the cataloging of the resources in the DLESE Collection. Effective catalogers have included information professionals (e.g., at the American Geological Institute), students hired and trained as part of funded collecting efforts (e.g., Montana State), resource creators cataloging their own resources, and educators recruited and trained via a funded collection-building effort (e.g., the DWEL Collection).

One approach that did not work well for DLESE was "community cataloging." The idea behind community cataloging was that educators would contribute to the library by cataloging a few of their favorite on-line educational resources through an easy-to-use web interface. In spite of considerable effort spent on developing the web-interface, guidelines and best practices documents, this approach yielded few resources and the community-cataloged metadata often turned out to be incomplete or incorrect. The community cataloging functionality has been replaced by a simple "Suggest a Resource" web form [21].

The bottom line, in DLESE's experience, seems to be that dedicated and motivated people of various backgrounds (information professional, educator, student, scientist) can create good metadata, but only after a substantial learning curve, usually via formal training.

Will your library offer material other than educational resources?

DLESE also contains several other types of metadata collections. The DLESE News & Opportunities Collection contains metadata to describe events or time-sensitive resources that have specific start and end dates, for example, grants, scholarships, and conferences. An annotation collection contains annotation metadata records, which provide information about an educational resource that is not directly found in the educational resource itself. DLESE's Community Review System (CRS) serves annotations that point to teaching tips, comments, editor summaries, quantitative scores on review rubrics, and recommendations for use of the resource with specific audiences [22]. Annotations have also been suggested as a mechanism for marking critical thinking resources or history of science resources [23].

How do you decide if a resource is good enough to include?

The two subdivisions of DLESE, the DLESE Broad Collection and the DLESE Reviewed Collection, promise two different levels of control on resource quality.

The Reviewed Collection offers a relatively small number of resources that have been reviewed by an approved, rigorous process against a challenging set of well-defined criteria.

The Broad Collection promises a wide range of resources, with the trade-off that resource quality may be more uneven. Over the history of DLESE [24], the level of scrutiny applied to resources entering the Broad Collection has ratcheted up. In the earliest planning discussion (August 1999), some community members favored a completely open collection, incorporating all suggested resources from the community, with no censorship or judgment. When the library opened to the public, two accession threshold criteria were in place: relevance to Earth System education, and basic functionality. Following the 2003 Quality Workshop [25], one additional accession threshold criterion (attribution) and several additional guidelines (sound scientific principles, current scientific knowledge, obvious and intended utility for teaching or learning, lack of distracting advertising, and free or low cost to educational users) were implemented.

In keeping with the original vision of the Broad Collection as a venue for constructive feedback by resource users, DLESE offers several self-correcting mechanisms that can lead to the improvement or removal of resources in which users have found flaws. The description of every resource in the DLESE Discovery System contains a link to "Submit a Comment or Teaching Tip." Comments submitted here are vetted for profanity, obscenity, and gratuitous insults, and are then forwarded to the resource creator, the DPC metadata staff, or the Collections Committee for appropriate action. For most pedagogically oriented resources, the Discovery System offers a second link to "Submit a Review." These detailed user reviews are forwarded verbatim, but anonymously, to the resource creator for use in revising the resource. Finally, if a user thinks that a resource should be removed entirely, he or she can invoke the DLESE Deaccessioning Policy [26], which calls for a review of the challenged resource by the Collections Committee.

Will you reach a point of diminishing returns in resource gathering?

The DLESE Collection opened to the public with 815 resources in the summer of 2001, and has grown by more than an order of magnitude, to 10,904 resources in the summer of 2005. The Community Collection has grown steadily, while thematic collections have been accessioned in bursts (Figure 3).

Figure 3: Growth of the DLESE Collection, in terms of total number of resources. When the collection opened to the public in the summer of 2001, it had 815 resources; the collection has grown by more than an order of magnitude.

Even though the collections have grown impressively, we have reason to think that we have not reached the point of diminishing returns in collecting. Collectors report no problems finding resources that fall within DLESE's scope and quality guidelines that are not yet in the collection. And users still encounter null results searches (i.e., no resource matches their search request) more frequently than they would desire [27].

How will you know if you are building a collection that matches users' needs?

A digital library must build its constituency through a spiraling, iterative process. A first-cut collection is assembled that attracts early-adopter users, who convey their desires about what they would like to see in the library, and consequently the library broadens and deepens in those directions, which in turn attracts new users with interests in those areas, who in turn feed back their desires about what they would like to see in the library, and so on.

The scope and balance of the DLESE Collection has been monitored by an ongoing collections assessment program. "Collections assessment" is a "systematic comparison between an actual collection of materials in a library and the collection of materials desired by the users, staff, funders and/or overseers of the same library as expressed in the collection policy, types of materials requested by the community, user feedback, and searches for material" [28]. The DLESE Collection is methodically assessed along three dimensions that are based on attributes recorded in the metadata: topic, grade level, and resource type. Each quarter, the percentage of all requests (searches plus browses) that fall within a given category is compared against the percentage of resources in the collection in that same category (Figure 4 top). If a category is receiving many requests but has few resources, we take that as an indicator that more collecting may be needed in that domain (Figure 4 bottom). Such analysis pointed out a need for more K-12 resources early in the collection-building effort, and more audio resources as the collection matured [29]. In addition, we have investigated what kind of searches returned no resources. After setting aside syntax errors and out-of-scope topics, this analysis showed that searches that were pollution-oriented, geotechnical, earth system-focused, or very specific (e.g., searches for a specific geographic area, fossil name, or event) had a relatively high likelihood of yielding no resources.

Figure 4a and 4b: An example of the DLESE collection assessment and gap-filling process. a) Solid bars show what percentage of the resources in the DLESE Collection were cataloged in each major category of learning resource type. Striped bars show what percentage of the searches and browses coming in through the DLESE Discovery System asked for resources in those same categories. The relative size of the two bars in the various learning resource types is an indicator of whether the scope and balance of the actual DLESE Collection matches the scope of and balance of the "desired collection" that users are asking for via search and browse. As of spring 2004, we noted an imbalance in the "audio" learning resource type: we were logging approximately 300 requests per month for audio resources, but had few such resources on offer. b) After two years of almost no growth in the audio resource category, we undertook a targeted effort to find and catalog audio resources in Fall 2004.

Challenges Remaining

Building a successful digital library collection is a balancing act that requires negotiating between conflicting priorities. DLESE is grappling with the following tensions and challenges:

• Balancing growth versus quality, and avoiding a "lumpy" library
• Identifying appropriate and effective incentives
• Coping with resource instability
• Nurturing distributed collection-building efforts
• Providing appropriate resources across a wide range of ages, abilities and knowledge-levels
• Balancing the pros and cons of increasingly information-rich metadata, and
• Finding, catalyzing, and cataloging pedagogical content knowledge resources

Each of these challenges is discussed next.

Balancing growth versus quality, and avoiding a "lumpy" library

Every time a student or teacher searches in DLESE without finding a resource that matches his or her desired criteria, DLESE risks losing a user or potential user. Every time a scientist finds an error of fact in a DLESE resource, DLESE's reputation for quality suffers. The former problem encourages us to continue to grow the collection briskly; the latter problem encourages us to scrutinize every detail of every resource before ingestion. In a world with limited time and money, there is an inherent tension between these two desires.

The ideal DLESE Collection would be well populated across all areas of DLESE's scope, with respect to topic, grade level, and learning resource type. The early DLESE collection-building efforts aimed to populate the initially empty library across its full scope. The current strategy of increased reliance on accessioning thematic collections tends in the opposite direction: towards a library that is rich in resources on topics or for audiences that attracted themed collection builders – and relatively impoverished in resources not so favored.

Identifying appropriate and effective incentives

To fulfill DLESE's vision of a library built "by the users for the users," we need to mobilize educators, learners and scientists across the country to:

• suggest resources for accessioning,
• build thematic resource collections and annotation collections,
• contribute reviews, teaching tips and comments, and
• alert the library staff of flawed resources that should be considered for deaccessioning.

We haven't yet figured out the optimum combination of peer-expectation, professional recognition, awards, rewards, community-building events, stipends and funding opportunities to lure passive users of DLESE into active participation in the collection-building process.

Coping with resource instability

Because control of each educational resource remains with the resource creator or publisher, resources can disappear or change without notice. When a resource changes, its metadata, review status or annotations may become inaccurate. DLESE has taken one step to cope with resource instability, by implementing twice-daily checks for dead-end links [30]. We have not addressed the subtler problem of resources that have deteriorated, or merely changed, behind an unchanged URL. Archiving some or all of the resources in the DLESE Collection has been discussed [31], but this raises intellectual property rights issues. A system for keeping track of changes (version control), accompanied by a re-examination of drastically altered resources, has been discussed, but no cost-effective path towards such a system has been identified. Neither archiving nor version control can cope gracefully with a web site in which change is a constant and intentional feature (for example, a site with frequently updated real-time data).

Nurturing distributed collection-building efforts

DLESE's strategy of accessioning themed collections taps into the expertise and experience of motivated groups of people with vast knowledge of, and passion for, a specific topic or audience. But by devolving responsibility to distributed collection-building groups for identifying valuable resources within a theme-domain and documenting those resources in metadata, DLESE-central inevitably gives up some control over what ends up in the library. DLESE is struggling to find the right balance of procedures and policies that respect and reward the effort and professional expertise of distributed collection-building groups, and allow collection builders to gather and document resources in a way that best serves their immediate constituency--while at the same time maintaining a requisite level of consistency and quality control over the resources and metadata labeled as "DLESE."

Providing appropriate resources across a wide range of ages, abilities and knowledge-levels

DLESE aspires to serve a very wide audience in terms of developmental stage, reading level, and prior knowledge. As a consequence, DLESE must, and does, provide multiple resources on the same topic and then permits users to search by grade level. The grade level coding is inherently imperfect, in that developmental stage, reading level, and knowledge of the Earth system don't advance in lockstep; for example, some middle school students know far more about dinosaurs than most college students. Different constituencies within DLESE pull in different directions when setting priorities for collection development: if tradeoffs have to be made, K-12 educators vote for a collection with "high quality resources but gaps and thin spots," whereas college educators and developers tend to vote for a "comprehensive library with variable quality resources" [32].

DLESE has barely begun to grapple with the problem of how to identify and provide access to resources suited for learners with abilities or background very different from that of the middle-class, college-educated, environment-loving people who typically develop DLESE resources. The "challenging audience" capability of the DLESE Community Review System aggregates user recommendations about the suitability of DLESE resources for specific audiences, such as those with limited English or hearing impairment [33].

Balancing the pros and cons of increasingly information-rich metadata

DLESE's diverse audiences each wish to be able to search for resources in the DLESE Collection according to parameters that matter to them personally. Over time, new fields have been added to the DLESE metadata framework to capture information of interest to specific audiences [34]. This trend towards increasingly information-rich metadata is good, in that it allows discovery for specific purposes of interest to users. But the same trend is also problematic, in that the cost and difficulty of cataloging increase, which means that fewer resources can be cataloged. Beyond a certain point, the complexity of information in metadata exceeds that which can be exposed in an easily navigated Discovery System.

Finding, catalyzing, and cataloging pedagogical content knowledge resources

Most of the resources in DLESE are aimed at helping learners understand the Earth system--in other words, they are resources designed to foster content knowledge about the Earth. DLESE also provides resources aimed at helping educators understand how to teach more effectively about the Earth system--in other words, resources designed to foster pedagogical content knowledge pertaining to the Earth [35]. Pedagogical content knowledge (PCK) is knowledge needed to teach effectively in a particular discipline [36], including what concept-understanding students may have before instruction, typical difficulties students tend to have in learning a given topic, and strategies to assess what students have learned about a given topic. DLESE can provide PCK either embedded within other resources (e.g., in a teachers' guide section of the instructions for a lab), as freestanding resources designed for educators (e.g., "Teaching Quantitative Skills in the Geosciences" and "The Private Universe Project" [37]), or as a portal organized around educational topics (e.g., "Starting Point" [38].) DLESE is seeking mechanisms to foster the creation of PCK-rich resources and to make it easier for educators to find relevant PCK-rich resources once they are in the library. Two useful steps may be expansion of the metadata framework for pedagogical information [39], and creating an annotation service to provide links from the Discovery System description of Earth System resources to relevant pedagogical resources [40].

Concluding Remarks

Building a digital library for education has turned out to be more complicated than we expected, but also more rewarding. We have had to develop many procedures, policies and best practices from scratch, and we hope that other library builders may benefit from our mistakes and discoveries.

There is an inherent tension between a user-centered philosophy, which values the distributed wisdom of practice in the education community, and a reform agenda, which holds that much of what is being done in science education today is ineffective or misguided. Both of these values are deeply embedded in DLESE, and both are important. Both can be held simultaneously and acted on in parallel if we recognize the complexity of the educational enterprise – but the requisite balancing act is not easy.

Acknowledgements

We appreciate the combination of breadth of vision plus attention to detail of the DLESE leadership during the era when the policies, procedures and tools described here were developed: Beth Ambos, Tom Boyd, John Butler, Karon Kelly, Cathy Manduca, Mary Marlino, Tammy Sumner, Mike Wright and the members of the Steering Committee and Collections Committee. The builders of the twenty themed collections within DLESE contributed greatly to the growth of DLESE through their expertise and effort. This work was supported by the National Science Foundation through grants DUE00-85827, DUE02-26292, DUE00-85839, DUE02-26233, DUE00-85831, DUE02-26289, DUE00-85787, DUE02-26196, EAR02-15640, EAR03-05031, and EAR03-05092. This is Lamont-Doherty Earth Observatory contribution number 6837.

Notes and References

[1] National Science Digital Library: <http://www.nsdl.org>. See also: Lagoze, C., Arms, W., Gan, S., Hillmann, D., Ingram, C., Krafft, D., Marisa, R., Phipps, J., Saylor, J., Terrizzi, C., Hoehn, W., Millman, D., Allan, J., Guzman-Lara, S., and Kalt, T. 2002. Core services in the architecture of the national science digital library (NSDL). In Proceedings of the Second ACM/IEEE-CS Joint Conference on Digital Libraries July 14 - 18, 2002 Portland, Oregon, USA. (JCDL '02). ACM Press, New York, NY, 201-209. <http://doi.acm.org/10.1145/544220.544264>.

[2] DLESE homepage: <http://www.dlese.org>.

[3] It is beyond the scope of this paper to compare and contrast DLESE's approach with the approaches used by other digital libraries; on this broader topic, we recommend NISO Framework Advisory Group, 2004, A Framework of Guidance for Building Good Digital Collections. 2nd edition. Bethesda, MD: National Information Standards Organization, available from: <http://www.niso.org/framework/framework2.html>. In keeping with DLESE's founding philosophy, DLESE offers a range of services beyond digital collections, services intended to foster interaction, collaboration and sharing among educators and learners; DLESE's non-collection-based services are also outside the scope of this article. For DLESE's founding philosophy, see Manduca, C. A. and D. W. Mogk, 2000. The Digital Library for Earth System Education: A Community Plan. Final Report from the University of Oklahoma to the NSF, Grant 99-06648., 44; <http://www.dlese.org/documents/plans/CommPlanFinal_secure.pdf>, December 12, 2001.

[4] DLESE Strategic Plan: <;http://www.dlese.org/documents/plans/stratplanver12.html>.

[5] GeoRef Database: <http://www.agiweb.org/georef/index.html>.

[6] The National Science Digital Library (http://www.nsdl.org) includes collections of original research papers, for example Applied Mathematics E-Notes (http://www.maths.soton.ac.uk/EMIS/journals/AMEN/index.html), BioMed Central (http://www.biomedcentral.com/), and the CalTech Library System archive of theses and dissertations (http://library.caltech.edu/etd/about.htm).

[7] DLESE Scope Statement: <http://www.dlese.org/documents/policy/CollectionsScope_final.html>.

[8] DLESE Educational Resources Overview: <http://www.dlese.org/resources/index.html>.

[9] The multiple pathways to the DLESE Reviewed Collection are described in Table 1 of Kastens, K.A. and S. Tahirkheli, 2005, white paper on "The State of DLESE Collections", available online at <http://www.dlese.org/documents/papers/KKST_StateofCollec_WP.pdf>. More details about the seven selection criteria can be found in Best Practices for the DLESE Reviewed Collection, at <http://www.dlese.org/Metadata/collections/drc-best-practices.htm>.

[10] Most of the Community Collection resources have been gathered through three funded gathering and cataloging efforts, which explicitly aimed to populate the library across its full scope: (1) an initial test collection at the DLESE Program Center, (2) at Montana State University, and (3) as part of a collaboration involving Foothills College and the American Geological Institute. A relatively smaller number of resources have come in from the general community through the "Suggest a Resource" Site at <http://www.dlese.org/suggestor/index.jsp>.

[11] The DLESE Accessioning and Deaccessioning Policy <http://www.dlese.org/documents/policy/collections_accession.html> explains what is expected of a DLESE themed collection. Existing DLESE themed collections are summarized in Table 2 of Kastens, K.A. and S. Tahirkheli, 2005, white paper on "The State of DLESE Collections", available online at <http://www.dlese.org/documents/papers/KKST_StateofCollec_WP.pdf>.

[12] DLESE Scope Statement for Vision Learning Collection: <http://www.visionlearning.com/dlese/scope.html>. Homepage of Vision Learning Collection: <http://www.visionlearning.com/about_us.php>.

[13] Granularity discussion on DLESE metadata website: <http://www.dlese.org/Metadata/collections/granularity.htm>.

[14] DLESE Educational Resources Overview: <http://www.dlese.org/resources/index.html>.

[15] DLESE's resource metadata framework is called ADN for ADEPT/DLESE/NASA. It was originally based on the existing IMS metadata, but the DLESE metadata group has greatly expanded and refined the framework with much iterative community input and in partnership with NASA's Joined Digital Library (JDL) and the Alexandria Digital Earth Prototype (ADEPT).

[16] DLESE metadata website: <http://www.dlese.org/Metadata/>.

[17] DLESE Collection System webpage: <http://www.dlese.org/libdev/dcs_overview.html>.

[18] DLESE Cataloging Best Practices: <http://www.dlese.org/Metadata/collections/cataloging.htm>. This links to each framework's cataloging best practices (ADN, annotation, collection etc.).

[19] DLESE Collections Management Processes at <http://www.dlese.org/Metadata/collections/management-process.htm> describes the quality assurance and link checking processes in detail.

[20] For native metadata formats of DLESE Themed Collections, see Ginger, K., 2004, "DPC Collection Builder Support," presentation to the Metadata Workshop (March 9-10, 2004), online at <http://www.dlese.org/Metadata/presentations/2004/collection-support.ppt>.

[21] DLESE Suggest a Resource site: <http://www.dlese.org/suggestor/index.jsp>.

[22] Kastens, K., 2005, "The DLESE Community Review System: Gathering, Aggregating, and Disseminating User Feedback about the Effectiveness of Web-based Educational Resources." Journal of Geoscience Education. v 53, p.37-43. On-line at <http://www.nagt.org/files/nagt/jge/abstracts/Kastens_v53n1.pdf>.

[23] Kastens, K.A., 2004, "Notes Concerning Annotations and Annotation Collections in the Context of DLESE." Online at <http://www.dlese.org/Metadata/annotation/0.1.01/docs/annotations-kastens-2004.doc>.

[24] Kastens, K., 2003, DLESE Quality Workshop: Framing the Question & History of the Discussion, presentation to DLESE Quality Workshop June 30, 2003. Online at <http://www.ldeo.columbia.edu/edu/DLESE/collections/quality/framing_question.ppt>.

[25] Kastens, K. A., 2004, Report from the Workshop on Quality of the DLESE Broad Collection, held June 30 - July 1, 2003 at Lamont Doherty Earth Observatory, Columbia University. Report online at <http://www.dlese.org/documents/reports/collections/quality_wkshop_full.html>.

[26] DLESE Accession and Deaccessioning Policy: <http://www.dlese.org/documents/policy/collections_accession.html>.

[27] Apedoe, X., 2004a, Preliminary report on undergraduate student use and perception of DLESE, unpublished report from University of Georgia, <http://cybele.colorado.edu/docs/ApedoeReportQualityListserv.pdf>. Apedoe, X, 2004b, Preliminary report on a geology professor's use and perception of DLESE, unpublished report from the University of Georgia, <http://cybele.colorado.edu/docs/ApedoeFacPerceptionReport.pdf>. DeFelice, Barbara, 2003, Collections Assessment of DLESE Version 1.0: Methods and Results as of August 2003, online at: <http://www.dlese.org/documents/reports/collections/DLESE.Coll.Assess.Report.8.03.pdf>.

[28] DeFelice, B., 2003, op. cit.

[29] Kastens, K. A., DeFelice, B., DiLeonardo, C., and Tahirkheli, S., 2004, Identifying and Finding Resources and Resource Types that Work for DLESE's Various Audiences. Geological Society of America Annual Meeting and Exposition Abstracts with Programs. Paper 189-1.

[30] DLESE's entire metadata collection is checked twice a day for broken links, and duplication of URL's or content. If links remains broken, the creator and/or collection builder is contacted; if the problem can't be resolved, the item is removed from the Discovery System. Less than 1% of the DLESE Collection is "broken" in this way at any time.

[31] Summary of DLESE Steering Committee meeting, Nov 30 - Dec 1, 2002, Lamont-Doherty Earth Observatory of Columbia University: <http://www.dlese.org/documents/reports/meeting/Dec_00/Dec2000SC.html>.

[32] Buhr, S., 2004, DLESE 2003 Annual Meeting "Broadening DLESE": Participant Feedback on Quality, unpublished report from the Cooperative Institute for Research in Environmental Sciences.

[33] Kastens, K., 2005, op. cit.

[34] For discussion of potential directions in which to expand or enhance DLESE metadata, see Tahirkheli, S. and participants in DLESE Metadata workshop, 2004, Report on DLESE Metadata Workshop, March 9-10, 2004, online at <http://www.dlese.org/documents/reports/DLESE Metadata Workshop Report-Final.pdf>.

[35] Kastens, K., 2004, "Making DLESE into the Source for Pedagogical Content Knowledge pertaining to the Earth & Environment," online at <http://swiki.dlese.org/quality/uploads/1/Geo_PCK_source.pdf>.

[36] Hassard, J., 2005, The Art of Teaching Science, Oxford University Press, 476 pp. Shulman, L. S., 1986, Those Who Understand: A Conception of Teacher Knowledge. American Educator: 9-15+. Shulman, L. S., 1987, "Knowledge and Teaching: Foundations of the New Reform." Harvard Educational Review 57: 1-22.

[37] Teaching Quantitative Skills in the Geosciences: <http://serc.carleton.edu/quantskills>. The Private Universe Project: <http://www.learner.org/teacherslab/pup/index.html>.

[38] Starting Point: Teaching Entry Level Geoscience. <http://serc.carleton.edu/introgeo/>. See also: Fox, S., Manduca, C., and Iverson, E., 2005. Building Educational Portals on Top of Digital Libraries, D-Lib Magazine, v.11, no.1: <doi:10.1045/january2005-fox>.

[39] Concerning the development of more detailed metadata for the pedagogical attributes of DLESE resources, see "Examining Educational Metadata and Vocabularies for DLESE" Working Group 5 report, DLESE Quality Swiki: <http://swiki.dlese.org/quality/uploads/1/WG5_whitepaper_final.doc>.

[40] Kastens, K., 2004, "Making DLESE into the Source for Pedagogical Content Knowledge...," op. cit.

[41] Further detail on DLESE metadata at <http://www.dlese.org/Metadata/adn-item/0.6.50/index.htm>.

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | First Workshop Report
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/november2005-kastens