D-Lib Magazine
July/August 2004

Volume 10 Number 7/8

ISSN 1082-9873

If You Build It, Will They Come? Participant Involvement in Digital Libraries

Sarah Giersch - iLumina Digital Library
Eugene A. Klotz - The Math Forum @ Drexel
Flora McMartin - MERLOT
Brandon Muramatsu - SMETE.ORG, University of California, Berkeley
K. Ann Renninger - Swarthmore College
Wesley Shumar - Drexel University
Stephen A. Weimar - The Math Forum @ Drexel

Point of Contact: Sarah Giersch <sgiersch@bellsouth.net>

Introduction

A two-day workshop on Participant Involvement in Digital Libraries (PIDL), sponsored by the National Science Foundation (NSF) and hosted by The Math Forum at Drexel University [1] was held in early February 2004. The workshop brought together 35 experienced practitioners and evaluators from projects funded by NSF's National Science Digital Library (NSDL) program [2] and from the area of online community development to describe and discuss the nature of participant involvement in NSDL [3] and NSDL-funded projects, as well as methods that provide feedback about participants to these projects.

The PIDL workshop had the twin goals of 1) exploring the practical issues of building participant involvement in digital libraries (DLs) as well as bringing together the research to date on participant involvement in NSDL and 2) focusing directions for future workshops. The goals of the PIDL workshop included sharing information about work already completed and outlining the challenges involved in establishing educational DLs [4] as active communities of practice. Workshop attendees developed strategies that will help NSDL projects, and DLs in general, incorporate involvement of participants from the communities the DLs serve; this "participant involvement" in turn will help grow and support the DLs. Workshop attendees formulated recommendations that anticipate involving participants both as users of the DL and as producers of new knowledge for the community. The full PIDL workshop report can be found on the workshop website [5].

The concise report presented here details the relevance of participant involvement to DLs; provides a summary of issues and action items identified during the workshop; and offers recommendations for the direction of future research on participant involvement and community-building. While the PIDL workshop focused specifically on NSF's NSDL program, the ideas developed in the workshop have broader application to the larger DL community.

The Case for Participant Involvement

Educational digital libraries are faced with several challenges, among them: to be widely adopted; to contribute meaningfully to educational improvements; and to thrive following start-up funding. They also have the potential to be more than simply a collection of resources. Educational DLs have the potential be the junction for active, engaged communities.

While there are several public and private sector models for building and maintaining DLs, a model that involves participants who have varying strengths, needs, experiences, and interests should make a significant difference in the design, development, and utility of educational DLs. Development of a model or models that describe participant involvement should enhance the likelihood that multiple audiences and constituencies can be effectively involved in all facets of educational DLs. How DLs choose to invest in involving participants is affected by:

1. The activities and needs of current and potential participant populations
2. The knowledge and skills that exist within the participant community
3. The types of content and services offered by a DL
4. The stage of development and comparative resources of the DL

One challenge in developing a model for participant involvement that applies across all DLs is that efforts to foster interaction often evolve organically and remain localized to a particular event (e.g., resource contribution or site design) or to an individual DL. One way to overcome this challenge is to support the development of communities of practice that bring together people with various proficiencies to effectively and efficiently solve problems and develop tools, benefiting all constituents. Further, such communities can mobilize expertise, train the next generation of experts, and meet the needs of other less skilled participants. Adopting a strategic, centrally organized, participant involvement model could result in the growth of active communities of practice (e.g., around NSDL).

Participant Involvement in Practice

Workshop discussion focused on types of activities that comprise "building" digital libraries. This section identifies types of participation and activities for involving participants as well as identifies associated benefits, challenges, and recommendations related to participant involvement.

Creating, contributing and describing content

Since its inception in FY2000, the NSDL program has had a funding track to create collections of digital resources. This support has resulted in several models of how DLs could grow their collections through participant interaction. Two common models have emerged. They are:

1. Individuals (educators and learners) generate and contribute original content (e.g., the Chemistry Collective [6], GROW [7]); and,
2. Content is developed by subject matter experts in response to recommendations from peers or advisory panels (e.g., GSDL [8] in K12 and ComPADRE [9] in Physics).

One challenge related to individuals creating and contributing resources is that DLs need to demonstrate to participants that their contributions are valued.

• Recommendation: Have more human interaction behind the technology.
• Recommendation: Have interaction opportunities in addition to technology through related face-to-face interactions, such as academies, workshops, and professional meetings.

Often, the line between creating information about a resource and creating the new resource itself is blurred, as resources may be modified to meet one-time needs [10] or separate resources might be combined to create completely new content [11]. While the technology supporting these models of contributing to DL collections or describing resources is still in the early developmental phase, one challenge will be that we need to remain focused on the primary goal of using technology to support participant involvement in creating, contributing, and describing content.

Involvement in human-moderated services

Human-moderated services rely primarily on direct human intervention to fulfill users' needs. In a distributed digital environment, many human-moderated services are delivered via technology, and at some point, may overlap with technology-mediated services. Human-moderated services can support interaction between DL staff and participants and can foster interaction between expert and novice participants of communities of practice.

• Recommendation: When developing human-moderated services, recognize that 1) participants might play multiple roles, such as service producer or consumer, and 2) participants engage with services at individual stages of need, growth, and ability.

Involvement in technology-mediated services

Technology-mediated services can include recommender systems [12], aggregation and harvesting services [13], federated search [14, 15] and hosting content [16, 17]. One point that human-moderated and technology-mediated services share is that technology should supplement, rather than substitute for, human interactions. While it is relatively easy to create technologies that support participant interaction, the challenge lies in supporting sustained interaction across complex social and cultural networks. Another ongoing challenge is that often technology drives the development process [18] without understanding the users' ultimate needs.

• Recommendation: Recognize that technology-mediated services have to provide value in return for the effort of using them.
• Recommendation: Design technology-mediated services to meet the needs, not just of early-adopters but also of participants at all levels of expertise

Sustaining and expanding involvement

"Sustaining" [19] a DL has often been equated with "maintaining" the DL. This could be a limiting idea because "sustaining" can also connote nourishment and growth. From that perspective, the DL becomes but one link within an ecosystem—both providing and requiring "sustenance". Participants who contribute to a DL's sustenance in turn receive nourishment. To be sustained and sustaining, DLs should develop business strategies or models that enable them to endure beyond research funding. The following recommendations for sustaining DLs emerged from the PIDL workshop:

• Recommendation: Create pre-conditions for sustainability among participants. Some suggestions on how to do this are 1) place the digital library in the context of users' needs; 2) make the digital library/service part of users' everyday activity; and, 3) minimize the cost of sustainability by using local resources.
• Recommendation: Cultivate both a top-down and a bottom-up approach, using forms of participant involvement to bring together the interests and commitments of the key stakeholders, such as the boards of professional societies, funders, and communities of potential users who have been underserved.
• Recommendation: Focus on demonstrating value at an institutional level, particularly to institutions that can make municipal or national investments.

Summary of Participant Involvement Issues

This section describes issues and challenges associated with involving participants in DLs. Highly developed DLs such as DLESE, ENC, Math Forum, MERLOT, and others demonstrate the importance of participant involvement in the growth and development of their sites. Following are operating principles shared by DLs that involve their "users" as participants:

• DLs/sites can grow organically. There are a number of online educational resource centers and programs that were begun well before the NSDL initiative and that continue to spring forth outside this context. These centers and programs offer many opportunities for DL development and collaboration.
• Builders can be users, and users can be builders. DLs with a substantial participant base have staff who actively use their DL and who support their users to take on various leadership and development roles.
• Communities can develop where participants are. Rather than take a "build it and they will come attitude" DLs can encourage learning communities to grow out of the existing interests and activities of the participant base.

However, as workshop attendees examined, word by word, the question: "If we build it, will they come?" attendees identified assumptions that limit effective action to involve participants (see Figure 1).

Involving Participants Requires an Investment

Ideally, participants should be involved from the earliest stages of building a DL. The short-term benefits of early and continuous involvement are that the infrastructure, content, and services meet the immediate strengths, needs, experiences, and interests of participants. Additionally, the DL can become responsible for sustaining its own growth at an earlier stage. In the long-term, continued involvement can create a sense of trust in participants, leading to long-term commitments. In order to achieve this continued involvement, DLs invest time, money and human capital. At the same time, they also need to acknowledge and reward participants' investment of time and effort (discussed in more detail below). Two key challenges include: being responsive to participants' needs while progressing steadily through the DL building process and scaling human-intensive development processes or services.

Involving Participants in Projects

Involving participants in NSDL projects may be optimal for participant growth; the question is how to make this experience optimal for projects. Working with participants takes time and does not necessarily yield immediate results that may become criteria for evaluating DL effectiveness (e.g., papers, site resources, etc.). Some issues that surfaced in discussions include the following:

• Furthering research while applying research to practice. How can DLs meet participants' needs while also advancing research agendas and creating spaces for real innovation? How can a research agenda lead to and support participant involvement? How can participant involvement support research, and what is the impact of their involvement?
• Supporting a DL through sales. When is it necessary to move a tool or service from practice to market? How will this process alter research-based ideas? Who should be involved in this process?
• Integrating online and face-to-face activities. What are sustainable paths for conducting labor-intensive, highly generative and community-building workshops, and other human mediated programs, while developing high quality online resources and scalable DL services? How can participants be involved in the flow of existing DL building activity?
• Positioning between communities. An opportunity exists to pursue funding in both the education and technology communities. How do DLs, and NSDL in particular, build bridges between these communities?
• Developing communities of practice. How can participants be encouraged to change their current practices and include working with online communities as part of their everyday activity? How can experts be encouraged to take leadership roles, support novice participants, and create new content for DLs? What does it take to sustain the dynamic quality of a learning community once it is initiated?

Participant Incentives

Participants' time and effort need to be recognized and valued. This is more complicated than just providing rewards for participation, contribution, and so forth. DLs must first understand participants' strengths, needs, experiences, and motivating interests: (e.g., a participant might ask, "How will participating in the DL enhance my educational or professional life?"). If participants are involved in every phase of DL building, the DL is more likely to reflect and match the strengths, needs, experiences, and interests of participants. At the same time, DLs need to build in rewards for participating that are meaningful and valued. Experiences of workshop participants suggest that:

• A one-time reward does not assure long-term involvement.
• Rewards do not necessarily have to be monetary.
• Rewards may have the most impact when they are supplements to or extensions of existing incentive structures, such as professional development credits for teachers or opportunities for peer recognition, [20 21] for university faculty.
• Rewards should not be one-size-fits-all. They should be structured according to local practices and within the context of peer group standards.
• Rewards should come to participants "where they are." They should happen in a local, familiar environment and not require extra effort, such as going to a different physical or virtual space, for participants to receive them.
• Rewards should be appropriate to participants' roles and location.

It is a Challenge to Support the Evolution of Participation

DLs operate at various stages of development. Similarly, participants engage DLs at various stages of individual growth and need. Over time participants will shift roles or will adopt additional roles as their needs evolve; technology will mature beyond the cycle of test and development; and communities will coalesce around areas of interest, practice or need. The challenge is to recognize that DLs evolve and that participation in DLs varies and requires appropriate human and technical scaffolds to accommodate these transitions. This evolution is influenced by 1) the activities and needs of current and potential participants, 2) the knowledge and skills that exist within the participant community, 3) the types of content and services offered by a DL, and 4) the stage of development, resources and services of the DL.

Participant and DL Collaboration

Collaborations can take many forms (e.g., distributed / centralized, formal / informal). Results of collaboration can vary based on the depth of involvement from DL staff and participants, with more intensive collaborations requiring a larger investment of time and resources but potentially yielding more enduring results. A number of successful efforts at localized collaboration among specific projects, audiences, disciplines or geographic areas were identified during the workshop.

Collaborations typically occur via both relatively simple technology tools and face-to-face. Two factors that motivate DL staff and participants to collaborate are 1) that they share common needs and 2) that they value meeting their needs together. Obvious benefits for DLs include (a) leaders working and worrying together about similar issues and (b) leveraging the experiences of other DLs and participants. On a larger scale, collaboration is beneficial for the diffusion of innovation [22]. Collaboration also includes challenges:

• Miscommunication between collaborators. Specifically, there may be 1) misunderstandings about the type of collaborative work and the resources required (e.g., two developers exchanging email vs. whole teams regularly meeting with one another); 2) assumptions that partners won't face local limitations or challenges to work; and/or 3) underestimation of the need to build trust and social networks between collaborative partners.
• Poorly structured collaborations. Some collaborations do not include the necessary people or resources to complete tasks, do not meet an actual need and/or do not have realistic outcomes.

Evaluation

DLs are an important new tool that can have a significant effect on people's behavior and opportunities to learn. So, it is important to discover who is using DLs, to understand the constraints that affect use and to identify what would help individuals change their existing patterns of interaction and allow these individuals to become regular participants.

In undertaking evaluation, DLs need to know who their participants are, what the social context is for their participation in DLs and what scaffolds different types of participants need to aid their learning. As these questions are answered, DL staff can identify evaluation goals for the DL and work with researchers and evaluators to identify whether or not significant social change is occurring. This level of exploratory research is critical for all DLs, and particularly for NSDL at this time, because such research will provide an understanding of how DLs fit into participants' lives and how creative participants are using the DL to further educational and learning goals.

As it is better understood who DL participants are and why they come to a particular DL, it becomes possible to do larger, targeted and detailed descriptive studies of participants' activities and of what those activities mean to participants. This work provides a detailed understanding of what the DL means to different participants. For NSDL, broadly speaking, this level of research is expected to come after substantial exploratory work is conducted in NSDL and other DLs and would require some coordination of research efforts across different DLs. For these reasons, descriptive work may not be part of the evaluation plans of a single DL but rather the coordinated effort of evaluators across several mature DLs.

It is hoped that after exploratory and descriptive evaluation there will be large-scale explanatory studies that follow an experimental model, which would allow for assessment of educational impact. These studies could adhere to the tradition of double-blind experimental research by establishing research hypotheses, testing hypotheses with a large representative sample and explaining the results. Explanatory research with reliable results is critical for NSDL in that it is necessary to know what impact new educational communities have on teacher and student learning and practice [23]. While explanatory research should be a goal for NSDL, like descriptive research, explanatory research is often larger in scale and requires a high degree of coordination. It also requires a much greater knowledge than currently exists of what is happening in DLs. NSDL should develop plans to support coordinated explanatory studies to be undertaken by teams of researchers across different mature DLs. However, explanatory research is not likely to be part of the regular evaluation plan of individual DLs.

Because research on DL participants is in an early phase, evaluators often are not aware of what is known about social activity in DLs. In addition, research into the patterns of activity of participants in DLs and how they incorporate DLs into their daily lives is complex—it involves investigating the new social worlds created by the overlap of virtual and physical spaces. Evaluators and researchers can take advantage of new information technologies, and by studying DLs, close this gap between spaces. It is suggested that the results of evaluation be shared across NSDL so that researchers can see what others have learned and consider its application to their own DL.

Action Items

Workshop attendees concluded that in order to better inform themselves and others about involving participants in DLs, the following actions need to be taken:

• Develop a literature review about building online communities that extends current work [24] (short-term)
• Develop examples of participant involvement, in addition to the examples started in Types of Participant Involvement: A Working List of Examples of Participant Involvement in Building Digital Libraries [25] (short-term)
• Compile information about existing and needed methods for studying participation (short-term)
• Identify indicators for studying change in participant activity and learning in work with DLs (mid-term)
• Develop case studies or exemplars of participant interaction [26] (mid-term)
• Coordinate study of participation across DLs (mid- / long-term)
• Foster and encourage discussion and sharing between developers, and among PIs, developers, teachers, and evaluators (ongoing)

Acknowledgements

This work was sponsored by The National Science Foundation under grant no. DUE-0226284. Opinions expressed in this piece are not necessarily endorsed by the National Science Foundation or Drexel University.

Notes and References

[1] The Math Forum home page, <http://www.mathforum.org>.

[2] National Science Digital Library (NSDL) program, <http://www.ehr.nsf.gov/ehr/DUE/programs/nsdl/>.

[3] National Science Digital Library (NSDL), <http://www.nsdl.org>.

[4] In this report, digital library (DL) is an inclusive term, encompassing NSDL, NSDL-funded projects and other online sites of communities.

[5] PIDL workshop website, <http://pidlworkshop.comm.nsdl.org>.

[6] User-contributed content is a goal of the Chemistry Collective, <http://www.chemcollective.org/>. It provides authoring tools for both virtual labs and scenario-based learning, and is starting the project with a base of six authors who have contributed roughly one third of the activities.

[7] GROW, <http://www.grow.arizona.edu>, relies on content contributed by undergraduate engineering student employees, high school interns, professors, and other users of the collection.

[8] The Gender Science Digital Library (GSDL), <http://www.gsdl.org/>, has spent time developing sample lesson plans out of the items in the collection, based on the focus group feedback of K-12 teachers. This has been the most used and most commented on section of the site—immediately usable content with a clear context for use.

[9] Each ComPADRE, <http://www.compadre.org/>, collection has an Advisory Panel of recognized experts/practitioners in the field covered by the collection. Their job is to recommend authors and projects that can provide high quality content.

[10] At Unidata, the Internet Data Distribution (IDD), <http://my.unidata.ucar.edu/content/software/idd/index.html>, system enables hundreds of universities nationwide to receive vast quantities of real-time data (of many types in many forms) and to store custom subsets that reflect specific departmental and faculty interests. The stored data subsequently may be shared via conventional Web images or direct-access methods such as are being developed (in part with NSDL funding) under the THREDDS project, <http://my.unidata.ucar.edu/content/projects/THREDDS/index.html>).

[11] Instructional Architect, <http://ia.usu.edu/>, is a program designed to help participants use existing instructional resources from digital libraries in order to create engaging and interactive educational web pages.

[12] SMETE.ORG Recommender Service, <http://www.smete.org/smete/?path=/public/about_smete/activities/
technology/recommender/index.jhtml&xsl=1>.

[13] SMETE.ORG Technology, <http://www.smete.org:80/smete/?path=/public/about_smete/activities/
technology/index.jhtml&>, supports harvesting services.

[14] The search engine found at ENCdl.org allows single or multiple ENC and ENC collaborative collections to be searched. It is not quite a federated search, but it is a model for how multiple collections that have been described using a variety of metadata can be displayed via a common access point.

[15] MERLOT's Federated Search <http://fedsearch.merlot.org/main/search.jsp>, currently hosts federated search with SMETE.ORG and EdNA Online - Education Network Australia. This model allows users to search all three digital libraries, seamlessly.

[16] Hosting materials submitted by users is a vexing issue. The Michigan Teacher Network (MTN; <http://mtn.merit.edu/>) hosted the results of the Michigan Department of Education lesson plan contest. Teachers were paid $100 for each lesson plan submitted. The process did not involve a consistent template or any review process. Quality is essential when dealing with submitted material.

[17] The Learning Matrix, <http://thelearningmatrix.enc.org/>, has the ability to accept participant submitted materials. The participants are asked to use the ENC cataloging tool to complete metadata fields.

[18] This was not the case with ComPADRE: On the ComPADRE student site, <http://www.compadre.org/student/>, the national board of the Society for Physics Students (a group of both students and SPS chapter advisors) was consulted on the design and features for the site during construction. Working groups were selected to generate ideas about various pieces of the site.

[19] Two previous NSDL workshops have addressed sustainability topics: 1) Establishing Relationships between Educational Publishers and the NSDL (October 2002), <http://publishers.comm.nsdl.org/deliverables/>, McArthur, D., Giersch, S., Wittenberg, K. and M. Luby. NSDL Educational Publishers Workshop: Workshop Report) and 2) Exploring Business Options for NSDL identified potential business models to sustain NSDL and NSDL-funded projects (November 2003), <http://nsdlbizmodel.comm.nsdl.org/docs/>, McArthur, D. NSDL Report: A Review of Business Options for the National Science Digital Library)

[20] NEEDS Premier Award, <http://www.needs.org/premier/>.

[21] The MERLOT awards program, <http://taste.merlot.org/awards/>, is an annual award given out for 'best in discipline' (Classics Award) and 'best overall' (Editors' Choice Award)—selected by editorial boards. Materials that rate highly in the peer review process are considered for the award. The Awards program receives funding from sponsors. Editors' Choice awardees receive a small stipend and travel to the MERLOT International Conference where the award is presented. Classics award winners receive free registration to the MERLOT International Conference. All Awardees who attend the conference showcase their materials, and they are highlighted on the MERLOT site.

[22] See work by: Malcom Gladwell, The Tipping Point; Hagel and Armstrong, NetGain; Geoffrey Moore, Crossing the Chasm; Sami Paavola, <http://newmedia.colorado.edu/cscl/228.html>. Also, <http://inquiry.uiuc.edu/cil/documents.php?cilid=99&folderid=227>, contains a list of papers about collaboration and diffusion of innovation.

[23] The Math Forum <http://www.mathforum.org>, has been studying the impact of site services on students' mathematical thinking and has developed rubrics for identifying problem difficulty that allow weighted comparison of student work on one problem with work on another problem. Study has also been conducted on mentor-participant exchanges in the Ask Dr. Math Service, <http://www.mathforum.org/dr.math/>. Studies of this type have been undertaken in response to staff members' questions, and findings are used to further develop site services. These studies serve a dual purpose in that they provide needed information to the site and provide summative studies of service impact. These studies follow earlier descriptive study of the same services.

[24] K.A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace. New York, NY: Cambridge University Press.

[25] See PIDL workshop for latest version <http://pidlworkshop.comm.nsdl.org>.

[26] See, Renninger, K.A. & Shumar, W. (2002). Community building with and for teachers: The Math Forum as a resource for teacher professional development. In K.A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace (pp 60-95). New York, NY: Cambridge University Press.

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Conference Report | In Brief
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

doi:10.1045/july2004-giersch