D-Lib Magazine
November 2002

Volume 8 Number 11

ISSN 1082-9873

The Design and Evaluation of Interactivities in a Digital Library

Muniram Budhu
Professor, Department of Civil Engineering and Engineering Mechanics
Anita Coleman
Assistant Professor, School of Information Resources & Library Science
University of Arizona
<asc@u.arizona.edu>

Abstract

The US National Science Foundation has established a program to create a National Science, Mathematics, Engineering and Technology Education Digital Library (NSDL). One of the subsidiary NSDL libraries under development is the National Civil Engineering Educational Resources Library (NCERL). The first phase of NCERL is the creation and collection of digital resources in three areas of civil engineering—geotechnical (soil), rock, and water engineering (GROW). The concept of interactivities guides the design, development, and evaluation efforts of the GROW digital collection. This article describes the salient features of GROW, defines and discusses interactivities as an emerging, integral part of teaching and learning in civil engineering education. Interactivities take place at three distinct levels: the information resource, the collection, and the context. Very simply, the concept of interactivities can be defined as the emphasis on structured representations of interactive multimedia resources. Additionally, resources are designed with rich learning tasks and organized in pedagogical collections supplemented with contextual information. Preliminary evaluation of GROW-NCERL using interactivities is briefly described.

Background

The Web has become the primary system for publication and dissemination of information anytime-anyplace. The growth of information on the Web has been phenomenal, but this growth has come at a price. It has become time consuming and often frustrating to try to find quality information on specific topics of interest. It is equally clear that most electronic resources mimic the print medium and do not take advantage of the fluency, permeability, and interactive nature of digital media to enhance learning. One way to alleviate information seeker frustration and to facilitate learning is to develop digital libraries that let users discover and interact with quality digital resources on a 24/7 basis. One such digital library under development is the National Science, Mathematics, Engineering and Technology Education Digital Library (NSDL), sponsored by the US National Science Foundation (NSF). NSDL is envisioned as a network of learning environments and digital resources; these are being systematically developed as subsidiary libraries to enhance the quality of science, mathematics, engineering, and technology education. NSDL will provide integrated services while subsidiary libraries will provide both content and services.

A subsidiary NSDL library currently under development at the University of Arizona is the National Civil Engineering Education Resources Library (NCERL). Phase 1 of NCERL is the creation, collection and dissemination of quality educational resources in three sub-disciplines of Civil Engineering: Geotechnical, Rock, and Water engineering (GROW) [Note 1]. An interdisciplinary team of educators, resource creators, librarians, computer and assessment specialists from different units within the University of Arizona has been assembled to design and evaluate GROW-NCERL. These units are as follows:

• College of Engineering and Mines,
• University Library,
• School of Information Resources and Library Science,
• Computer and Communication Information Technologies, and
• Assessment and Enrollment Research.

Potential Benefits of GROW-NCERL

GROW-NCERL is intended to be a digital library collection of quality civil engineering information and resources. It will provide a knowledge base for citizens to become more informed about their built-environment [Note 2]. It will facilitate rapid access to cutting-edge civil engineering technologies and high quality education resources.

Civil engineers plan, build, construct, maintain and improve systems such as water, sewer, buildings, hydropower and transportation on which we depend every day. The water we drink, the roads we drive on, the bridges we cross, and the recreational facilities we enjoy all result from civil engineering activities. Most people take these systems for granted, only paying attention when, for example, the water doesn't taste good or the roads have potholes. People are stunned by natural disasters from earthquakes, floods and hurricanes and, when such disasters occur, mourn the loss of human lives and destruction to the environment. But, rarely do they understand and appreciate the technology that helps to predict and prepare for the occurrences of these tragic events. The American Society for Civil Engineers (ASCE) describes civil engineers thus:

"They build dams able to withstand the crushing pressure of a lake full of water. They build bridges able to resist the forces of wind and traffic. They develop environmentally friendly materials and methods, and they build things to last. So skilled is their work that we rarely stop to wonder how they design the mammoth skyscrapers we work in, the tunnels we drive in and the stadium domes we sit beneath." [ASCE, 2002a]

Large sums of money are spent on civil engineering activities every year. The US alone spends billions of dollars each year on civil engineering projects [Note 3]. Nevertheless, ASCE recently assigned a grade of D+ for this nation's infrastructure, and ASCE estimates that the US needs to spend $1.3 trillion over the next five years on infrastructures. ASCE also called "for a renewed partnership between citizens, local, state and federal governments, and the private sector" [ASCE, 2001]. An effective partnership can only be built if community leaders and stakeholder citizens become well-informed on issues regarding their built-environment. Similarly, cutting edge technologies in civil engineering involving alternative materials and methods that can potentially save time and construction costs can only be deployed if they become well-known to practicing engineers and other professionals.

Citizens are called upon to make decisions about the environment for which few have the necessary scientific, economic and technological understandings. "More and more, public policy issues are being swayed by arguments for which the scientific basis is questionable, at best" [Lane, 1994]. One reason for this apparent lack of knowledge is that technology education is not woven into our cultural and intellectual tradition, but it needs to be. We must prepare students for life in an increasingly complex, high technology world, where we must sustain the ecosystems—both naturally found and man-made. Although there are many students who will not take formal science, engineering or technical courses, these students must still develop an understanding and appreciation for the application and influence of scientific principles and engineering technologies on their daily lives.

Information and communication technologies, including the Web, have the potential to dramatically improve the understanding of science and engineering concepts by embedding them in interactive learning environments. Just as a poem, a painting, or a piece of music can be made accessible on the Web, so can the science and engineering technology of everyday systems be made accessible.

Interactive Learning Environments and Information Behaviors

The design and development of GROW-NCERL started with the following questions:

1. How can we provide an understanding of the base materials and resources pertaining to civil engineering?

2. Using modern information and communication technologies, how can we enhance learning about the structural systems that surround people in their daily lives?

3. How can we facilitate access and understanding of cutting edge materials technologies and engineering advances?

4. How can we create and organize educational resources with the explicit goal of helping to build a savvy workforce?

In addressing these questions, it became apparent that we need to enable GROW-NCERL to accommodate information behaviors that accompany and enhance learning [Note 4].

Typically, digital library design initiatives include some aspect of scenario planning or user modeling to precede the software requirements and library development process. However, digital library development is complicated by many factors, and since there is no such thing as a typical library user, the task of user modeling for a digital library is complex. The features and functions of a digital library are also harder to specify since the different communities who may access the digital library have diverse needs. Therefore, modeling information behaviors and generating requirements based on those models is more effective. For our modeling, we distinguish between three types of information behaviors: information seeking, information searching, and information use.

Information seeking behavior is "purposive seeking for information" while information searching is the "'micro-level' of behavior employed by the searcher in interacting with systems of all kinds." Information use behavior consists of "the physical and mental acts involved in incorporating the information found into the person's existing knowledge base. Therefore, it may involve physical acts such as marking sections in a text to note their importance or significance, as well as mental acts involving, for example, comparison of new information with existing knowledge" [Wilson, T.D., 2000]. Information use behaviors for learning can be identified using any number of educational theories. For example, using Bloom's Taxonomy of Educational Objectives, information use behaviors may include: memorization, classification, categorization, comprehension, construction, calculation, diagramming, and interpretation. [Bloom, 1956].

Libraries, whether traditional or digital, generally focus on providing solutions to the problems of information searching/discovery by managing collections and developing services to support searches and uses. Digital libraries, however, are expensive propositions and must be designed to be widely usable; that is, information uses beyond mere information retrieval must be accommodated. Educational digital libraries that support learning are quickly becoming important [Zia, 2001]. This is not an easy transition of library purpose, and many still question the importance given to the learning uses of a library over its more accepted scholarship/research (information discovery) functions. Nevertheless, the broad thrust and goals for developing the GROW digital library include:

1. Cross-disciplinary Collections: GROW-NCERL will organize fundamental and high quality civil engineering resources about three naturally found materials on earth: soil, rock, and water.

2. Resources for Lifelong Learners: GROW-NCERL will be a one-stop shop on the Internet for engineering information promoting lifelong learning. Accommodating lifelong learning means recognizing that there is a shift from teaching to independent learning. This shift is irrespective of whether the learners are disciplinary experts or novices. Learning objects in GROW will serve educational levels categorized as: K-12 students, K-12 teachers, higher education (college students and teachers), practitioners (working engineers and technicians), and the general public (for example, homeowners).

3. Round-the-Clock Access: The Internet is a 24/7 medium, i.e., available 24 hours a day, 7 days a week. Access to content will not be constrained by time limits.

4. Interactive Learning Objects: Internally created GROW resources are interactive, multimedia learning objects that incorporate instructional design principles and theories.

5. Information to Be Used for Learning: GROW will support specific information behaviors that underpin instruction and learning such as: information seeking, browsing, encountering, foraging, sharing, gathering, filtering, and using. William Arms rightly noted that information overload will be a major problem in digital libraries [Arms, W., 2000], and from the outset GROW-NCERL seeks to avoid this problem.

We wanted to keep these goals at the forefront of our development efforts and also embed concepts for empirical user-centered evaluation into the design of GROW-NCERL. Hence, we searched for a single unifying idea to tie the disparate activities of design, development, and evaluation together—an idea that would capture the imagination and bind project participants from different disciplinary backgrounds with divergent, assorted vocabularies and terminologies.

John Dewey's "learning-by-doing" is often used to provide the pedagogical theory behind the use of computers for designing interactive learning environments [Dewey, 1916], and K. Wilson defined interactive learning environments as:

"...environments that allow for the electronically integrated display and user control of a variety of media formats and information types, including motion video and film, still photographs, text, graphics, animation, sound, numbers and data. The resulting interactive experience for the user is a multidimensional, multisensory interweave of self-directed reading, viewing, listening, and interacting, through activities such as exploring, searching, manipulating, writing, linking, creating, juxtaposing, and editing." [Wilson, K., 1992].

Interactivities emerged as our unifying idea and binding glue, and we use it to improve the definition and description of information behaviors and tasks in digital libraries with interactive learning environments [Note 5].

Interactivities

Interactivities: The GROW Conceptual Framework

A digital library is made up of many components: collections, services, tools, and users are often mentioned explicitly, while the interfaces to these components are invisibly subsumed. There are many architectures and frameworks for design, evaluation and interaction described in the literature [Note 6]. Recognizing that it is beyond the scope of GROW-NCERL to experiment with these architectures or develop varied interfaces, we focused on defining interactivities as our key design and evaluation concept by starting with interaction and the properties of interactivity.

The Oxford English Dictionary (OED) defines interaction as "reciprocal action; action or influence of persons or things on each other" and includes the Human Computer Interaction (HCI) definition limiting the scope to information processing and flow of information between computer interfaces and people. (OED Online). Information processing is a fundamental cognitive activity underlying information uses such as learning. Caroline Arms, for example, notes that: "libraries have always supported interactions with the fund of knowledge, interactions that come in many shapes and sizes...Interacting with knowledge is what life-long learning is all about" [Arms, C., 2000]. Thus, when considering that our evaluation goal is user rather than system centered, interactivities rather than interfaces or interactive learning environments emerged as a measurable way to design and evaluate. Interactivities are simply defined as interactivity that enhances learning. Interactivities are consciously designed and developed based on the pedagogical, technical, discipline, and social dimensions of interaction between users and systems.

Interactivities have the potential to operate at several levels: the user and the digital object and/or resource, the user and the managed collection, the user and services, the user and the library as a single entity, and user to user. For initial efforts in GROW-NCERL, we have confined interactivities to operate at three distinct levels: the information resource (user to resource), the collection (user to collection), and the context (contextual) and further defined it as the emphasis on structured object representations, graphics and interactive animations of rich learning tasks in pedagogically organized collections that are supplemented with contextual information. The following are key attributes of interactivities:

• Reciprocity - there must be a reciprocal action when the user does something.

• Feedback - the amount and type of feedback provided to the user must be just right.

• Immediacy - both reciprocity and feedback must be immediate and immediacy differs on task, context, and user preferences.

• Relevancy - relevance is calculated based on task and context.

• Synchronicity - the degree to which users consider their input into system and system response is felt to be simultaneous.

• Choice - the user can always choose among alternatives.

• Immersion - experiences are immersive.

• Play - a sense of play is used to stimulate and motivate learning.

• Flow - this is related to the user's cognitive flow and locus of attention.

• Multi-dimensionality - the sensory experience has more than one dimension.

• Control - the voluntary and instrumental action that users have over the outcome, or rate, sequence, and type of feedback.

These characteristics, from areas such as HCI and instructional design, and online consumer behavior [Liu, 2002] are used to drive the design, development and evaluation of interactivities in the GROW-NCERL components, including the resources, objects, collections, interfaces, and metadata..

Interactivities in GROW Collections: The Waterfall of Knowledge

Philosophers, economists, and cognitive scientists have classified knowledge in many different ways. The emerging consensus is that data, information, knowledge, and wisdom all exist in continua. The collections in GROW are focused on interactive, multimedia, educational resources "story-booked" to emphasize active learning and provide learning experiences rather than merely providing snippets of information. Each GROW collection consists of a hierarchy of learning objects to meet the needs of users from K-12 to continuing education for professionals. There is also another underlying hypothesis in keeping with the development of learning objects: namely, that base resources can be repackaged to provide quality information at the appropriate learning level. GROW-NCERL collections consist of four groups of learning objects: elements, learning units, modules and themes (Figure 1). In the context of GROW-NCERL, a learning object is "any digital resource that can be reused to support learning" [Wiley, 2000]. Each learning object within a group has properties that allow it to be reusable, self-contained, aggregated and tagged with metadata. General users interact with themes and modules, and developers/creators interact with all four groups of learning objects. For example, a developer/creator may create learning units and/or modules from elements using available authoring software such as Macromedia's Flash or Authorware, and may then submit them to be considered for inclusion under a new or existing theme. The submitted materials are peer reviewed before placement in the GROW collection.

Figure 1: Learning objects groups in GROW-NCERL.

Element: An element is the most basic type of learning object. An element may be an image, text, or data file, etc. For example, Figure 2 shows three elements (each of which is an image): an electronic scale, an oven and a metal cup. These elements can be used to make learning units.

Figure 2: Elements—images of scale cup and an oven.

Learning unit: A learning unit is the smallest self-contained lesson that can be made from elements and that has at least one learning outcome. Learning units within GROW-NCERL are created using interactive multimedia technologies. For example, a learning unit constructed using the elements shown in Figure 2 provides a lesson on the determination of the weight of an object. The images of the scale and cup were imported into Macromedia Flash 5.0 and scripted to enable the user to drag the cup, place it on the scale, and view the weight, (as illustrated in Figure 3). The learning outcome here is the determination of the weight of an object. Images of other objects can also be imported and used to illustrate the weights of different objects. Other examples of learning units can be found within GROW-NCERL [GROW 2002].

Figure 3: Dragging a virtual cup to weigh it on a virtual scale.

Module: A module is a collection of learning units with one or more learning outcomes. The learning units in a module can be sequenced to accomplish different learning objectives. For example, a module constructed of learning units provides a virtual laboratory experiment where a user performs a series of interactivities to determine the water content of different classes of soils (e.g., clays, silts, or sands). The module uses learning units in turn built from the elements shown in Figure 2 as well as other elements. Figures 4a, 4b and 4c illustrate a few of the steps (not in sequence) a user follows to determine the water content of a soil. These steps are contained within a Macromedia Flash 5.0 interactive movie created from learning units. The user conducts the virtual experiment as if she/he were in a real laboratory. Other examples of modules can be found at GROW-NCERL [GROW 2002].

Figure 4: Three of several steps in a module on the determination of the water content of a soil.

Theme: A theme is a collection of modules addressing a global topic within the three broad collection areas. For example, one of the themes in GROW-NCERL is 'Effects of Water on Soils'. The current modules associated with this theme are listed below:

Theme: Effects of Water on Soils.

In the theme "Effects of Water on Soils", the user can learn from a series of hyperlinked modules (indicated by ). The modules provide learning experiences for a variety of related topics on the influence of water on soils, from K-12 level to graduate and professional levels.

There are three key characteristics of themes. First, the hierarchy of knowledge in the themes provides learning flexibility and guided choices that allow a user to learn as much as he/she wants using the various modules in GROW-NCERL. Second, the modular structure allows users—in particular educators—to select modules from within a theme (or from several themes) to build his/her own library or course. Third, the themes and modules serve as gateways to additional links on similar topics on the Web external to the GROW-NCERL reviewed and ranked collections. The theme organization provides resource supplements capable of bridging knowledge gaps between what users want to know and what they need to know for learning to take place.

Interactivities in GROW Resources

Interactivities are consciously designed in GROW-NCERL resources for resource-user interaction; for example, we can use heuristics to govern how interactivity can be increased or lowered for learning. In that instance, interactivity is based on user factors such as cognitive learning styles, gender, and cultural styles. In the first stage, however, we focus on cognitive learning styles that have been shown to be important in developing good engineers.

Current GROW-NCERL resources use graphics and flow of information primarily through storytelling, to build the interactivities. The goal is to appeal to many learning styles and to enhance learning. Why the emphases on graphics and storytelling? Engineers, particularly civil engineers, practice the adage "a picture is worth a thousand words." Engineering projects require engineers to conceptualize, visualize and realize. This is done first mentally and then physically through engineering drawings. Civil engineering activities are also hands-on activities and include a sequence of interactive events for success.

Consider the design and construction of a bridge. A civil engineer will create a mental picture and will then draw sketches of the bridge before proceeding to calculations. When calculations are completed, the results are then transferred to the drawings used for construction. After construction, the public sees the bridge and then forms opinions based on its look and, in some cases, its functionality. To enhance learning, our goal is to capture the functional environment of engineers as well as the way the public perceives engineering works. By using animated graphics, we capitalize on the information-rich environment provided by computer technologies to develop the visual information processing capabilities of learners.

In general, engineering curricula are rigidly structured to meet accreditation requirements. Engineering students, especially at the undergraduate level, rarely use conventional library resources [Garfield, 2000]. In practice, engineers consult design and construction manuals, codes of practice, and reports. These are obtained not from conventional libraries but from their employers' libraries, government agencies and commercial entities. Using the powerful features of computer technologies to enable learning-by-doing in the form of interactivities, the resources in GROW-NCERL thereby provide an alternative way of learning to conventional civil engineering learning. Table 1 lists some important differences between lecture based, current approaches to learning and the approach using interactivities.

With GROW-NCERL resources, users actively participate in the learning process rather than passively accumulate information. The modules provide self-paced learning activities with immediate feedback. Modules utilize a variety of media types including: interactive animation, video, sound, images and text. Text is kept to the minimum required for a good understanding of the information presented. Both audio and text are provided in many modules, and the user can choose audio only, text only or both. Many of the modules may be modified by the user for local conditions, and users can change objectives if necessary or desirable. In general, the modules offer the following:

1. Introduction and objectives to provide the context of the intended activities and the expected learning outcomes

2. Rich learning tasks to produce the learning outcomes

3. Reinforcement to alert the user to essentials or main points

4. Summarization of the information within the module

5. Exploration of information seeking to further learning (by provision of links to sites with related information)

6. Self-evaluation through a quiz or similar assessment tool.

A variety of programming tools are used to develop the interactivities of learning resources in GROW-NCERL. These include commercial animation and authoring tools from Macromedia such as Flash 5, Authorware 6, Director 8.5 and Dreamweaver ultra dev.

Interactivities Example in a GROW-NCERL Module

One of the modules in the theme "Virtual Geotechnical Laboratory" is a virtual laboratory test called the "Virtual Consolidation Test", which is used to determine settlement of soils. The consolidation test is a common laboratory test in undergraduate civil engineering curricula and is generally one of about ten required laboratory tests conducted over one semester for a course value of one credit hour. This test is very time-consuming and can last from two to seven days. Unfortunately, due to time constraints students often only have time to actually carry out a three-hour part of the test. Below, a brief description of this consolidation test is given to illustrate how interactivities have been built into a virtual lab test module to enrich student-learning experiences at the user-resource, user-instructor, and user-interface levels of interaction. More details about the virtual laboratories are available in Budhu [2000a, 2000b, 1999a, 1999b, and 1999c] as well as at GROW-NCERL [GROW 2002].

Because sample preparation requires at least 3 hours, in physical consolidation tests sample preparation is not normally done by students but rather by teaching assistants, instructors or technicians. In the virtual consolidation test, all the procedures—including sample preparation—are simulated in a 3D laboratory environment and can be performed by the students.

The instructional methods and steps followed in developing the virtual consolidation test are those proposed by Gagné, Clark, and Felder and Silverman [Gagné, 1995, Clark, 1989, and Felder and Silverman, 1988]. These include: gaining attention, informing students of the objective, stimulating recall of prior knowledge, presenting the stimulus, providing learning guidance, eliciting performance, providing feedback, assessing performance, enhancing retention and transferring learning. Interactivities related to gaining attention and presenting the stimulus are described in the following:

• Gaining a user's attention. In the virtual consolidation test, the leaning tower of Pisa is used to capture a user's attention as shown in a screen shot in Figure 5. This figure also shows the main menu items that the student can select prior to conducting a test.

Figure 5: A screen shot of a frame that is intended to gain a user's attention by relating the test to a popular situation.

• Presenting the stimulus. Guided interactivities are used to simulate all the actions a technician would take to conduct a consolidation test. For example, the user is guided through an interactive sequence to prepare a soil sample for testing by extracting a specimen of soil from a sample tube. The user places the tube in the jack and extrudes a small part of the sample by activating the jack handle. Using a cutter, he/she then cuts out the top part of the soil (Figure 6) and then continues with further extrusion and sample preparation.

Figure 6: The user is prompted to move a cutter to cut out part of a soil sample during sample preparation.

• Guiding the user with rich learning tasks. Once the sample is prepared, the user is guided to assemble the testing apparatus, set up the soil specimen in the apparatus and conduct the test. Two screen shots of the interactive sequence of setting up the apparatus and the soil specimen are shown in Figures 7a, and 7b. The user applies the desired loads, observes the test results and is guided to interpret the results and apply them to a practical situation. Users can explore various loading or initial soil conditions to address "what-if" situations. During each stage of interpreting the test results, the user is prompted to insert calculated values, and these values are checked automatically. If after three tries the user is unable to get the correct answer, the detailed solution is displayed. Quizzes are interspersed within the virtual laboratory. A record is kept of the score and user performance.

Figure 7: A screen shot of an interactive sequence in setting up (a) a soil specimen in a cell and (b) the consolidation apparatus.

Physical laboratories in engineering programs require significant investments in real estate, and equipment costs have also been steadily rising. Because of this, some universities are finding it difficult to maintain and update existing laboratories. In universities with large undergraduate enrollment, it is practically impossible to have enough equipment and space for each student to fully carry out experiments.

Contextual Interactivities

Contextual interactivities in GROW-NCERL are defined as the ability to discover definitions and visualize integrated maps of scientific classifications and nomenclature, along with layman's terms and novice language, for any concept or phrase used in a resource. These interactivities generally operate at the user-collection level interface and are provided by developing two tools, a glossary and a thesaurus, and creating a new form of interactive resource, the concept map:

• Glossary: A glossary of terms that defines technical words used in all the resources is available to the user at all times.

• Thesaurus: A thesaurus of terms and phrases maps the subject concepts and relationships for soil engineering. Entry terms, non-preferred terms and preferred terms serve as access points. For example, in-situ stress is a concept that can be studied as part of a soils laboratory test, or rock engineering fundamentals. For the novice trying to understand the context, it will be useful to show disciplinary relations through the thesaurus structures and relationships of in-situ stress (broad terms, narrow terms, scope notes). Other types of affiliations that can be shown include: measurement/instruments, materials/objects, and process/phenomenon.

• Concept Map: Concept maps can be automatically extracted from both the user performance interaction logs and from learning resources. They can be presented subsequently to users to help them make sense of their own learning and progress. A draft concept map of one learner's view of the concept of stress and relation to the over-consolidation ratio is shown below in Figure 8.

Figure 8: Example of a concept map of one user's performance on interactivities in virtual consolidation.

• Other contextual interactivities: Information seeking and searching behaviors, both purposive searching and browsing, are enabled by a search interface that will allow searching on every piece of metadata/text for each resource and browsing through the use of subject theme topics, in the three collections areas.

Evaluation

Like design, evaluation of a digital library is a complex matter. Educational evaluation carries expectations of transformations achieved in terms of learning or competencies gained. Furthermore, empirical evaluation of digital libraries is needed to support the findings that 'library' features (what we have defined as contextual interactivities and distinguished from resource interactivities) do promote learning. The NSDL Evaluation Workgroup, individual projects, and the Association of Research Libraries (ARL) are identifying some of these metrics [Note 7]. For example, usage data can be collected regarding how many resources and objects are in the library, as well as how they are used and how often they are used. Less frequently, data is collected that reveals the nature of the digital resources stored, objects presented and used, or the kinds and nature of interactions users have with digital objects [Note 8]. As an NSDL subsidiary library, GROW is constrained by design, development, and evaluation choices that have already been made for subsidiaries (for example, the decision to use Dublin Core (DC) metadata elements). Hence, our initial evaluation efforts are strictly grounded in seeing how our definition of interactivities needs to be refined in measures that can be developed for evaluation of learning and systematic understanding of information behaviors along the different user-interaction levels. To do this, we have undertaken some pilot studies and tests. These are briefly reported below and summarized in Appendix 1 as digital library components, their evaluation, design goals, and interactivities.

Evaluation of Interactivities in GROW Resources

Currently, our concept of interactivities has reached its fullest potential in the virtual laboratories. Many of the GROW virtual laboratories have been, and continue to be, used at several universities worldwide. Previous evaluations have been conducted at the University of Arizona and elsewhere to determine whether these virtual laboratories enhance learning and retention. Preliminary assessment data from these evaluations are encouraging. For example, evaluation at the University of Arizona showed that students who used the GROW modules were subsequently able to better perform tests in the physical labs and have better understanding of the concepts learned than did students who did not use these virtual labs before they used the physical labs. However, conclusive assessment of the effectiveness of the virtual labs should not be made until comprehensive evaluations at different institutions over several years are completed. User testing of resources based on the interactivity properties identified is currently in the planning stages.

In the spring of 2002, we compared the use of one of the virtual GROW labs, the Consolidation lab, with its physical counterpart lab. Data from this study shows that contextual interactivities in the form of entry vocabularies and categorizations may facilitate yet another level of user-collection interaction. Novice vocabulary needs to be linked with the expert or scientific terms commonly found in thesauri and manuals. We plan to extend the study to other virtual labs in GROW.

Evaluation of Interactivities in GROW Resource Selection

Resource selection is an important digital library task for those who maintain libraries. Following the NSDL model, GROW-NCERL resource selection is also a distributed activity, whereby the continuing collection development of the library rests in the hands of an expert and volunteer community. Therefore, we wanted to make resource selection meaningful as well as easy, and we explored how the process of initial interactive resource selection might be at least partially automated. Our current resource selection process manually collects freely accessible resources on the Internet using the following criteria for resource selection:

1. Is the resource relevant to education in Civil Engineering? (The ASCE Keyword List to the CEDB [ASCE, 2002b] provides the definitive list of topics.)

2. Does the resource function relatively well? Is it bug-free?

3. Has the URL been accessible at least twice in two different weeks?

4. Does the URL begin with http:// ? (GROW-NCERL does not want resources that begin with ftp://, etc.)

5. Is the resource available in full-text?

6. Is the resource available at no charge?

7. Is the resource from an authoritative source? (Authority is defined loosely at this point as domain expert.)

8. Is the resource attractive, interesting, current and informative? (Currency is defined as an update/revision within the last full year.)

9. Is the resource interactive? (Does the resource require the user to do something else besides just scrolling pages or clicking on hyperlinks?)

10. Does the resource include at least two different formats? (For example, does it provide text and images, text and datasets, text and movie, or text and audio?)

11. Is the resource archived or stored on the server from which it is currently available? (If it is only mirrored or linked, try to find the current and primary site where the resource is located and use that URL.)

12. Is there a contact name and address for the resource?

13. Is the resource fully in English? (GROW currently does not include materials in any language other than English.)

Our feasibility test with these criteria has just ended. Twenty-seven selectors used the above criteria to find resources on specified topics such as, seepage of water through soils, determination of hydraulic conductivity, stress-strain behavior of soils, and more. While over 400 resources were selected over a 30-day period, a survey of the resources shows that less than half of these resources can be considered truly interactive. A preliminary definition of interactive resources includes only criteria # 9 and # 10 above. These two properties can be also be thought of as structural properties in resources and identified automatically.

Evaluation of Description of Interactivities in GROW Metadata

GROW-NCERL is using the Dublin Core (DC) metadata [DCMI, 2002]. We supplement the standard 15 DC elements (title, identifier, creator, publisher, contributor, date of publication, subject, description, format, type of resource, relation, coverage, source, language, and rights) with four elements of description from the IEEE LOM protocol [IEEE, 2002]. Note that IEEE LOM is implemented as the IMS Metadata for Learning Resources [IMS, 2002]. The four IEEE LOM elements include: type of interactivity, audience, duration, and level of interactivity. While we have yet to formally evaluate the usefulness of these elements and values, which are taken directly from the standard, preliminary feedback suggests that these elements and values need to be supplemented with much more robust definitions and vocabularies for interactivities. In other words, an in-depth checklist to help metadata creators identify 'interactivities' in resources is needed. We are in the process of creating such a checklist using the properties of interactivities such as flow, locus of control, etc. The difficulty is in coining standard and objective values to describe these features such that metadata creators can use them or automatic metadata generation tools can extract them.

Evaluation of Interactivities in GROW Interface

Schneiderman defines universal usability as having more than "90% of all households as successful users of information and communications services at least once a week" [Schneiderman, 2000]. Three challenges to universal usability are technology variety, diversity of users, and the gaps between what people know and what they don't know. Universal usability will help libraries continue their long tradition of upholding universal access to information and we are interested in incorporating this into our evaluation efforts of the GROW-NCERL interface. Our study participants will include all our end user groups and the metadata creators, resource selectors, and peer reviewers. We hope to start such an evaluation next year using refined definitions of interactivities.

Conclusion

GROW-NCERL seeks to fill a niche in digital library research by integrating design and evaluation efforts using interactivities. We have a preliminary definition of interactivities and we have used the concept to define the nature and characteristics of interactive resources, the usable interface properties needed by at least two different categories of users (learners and community library developers) to successfully interact with them, and the contextual interactivities that enable interactive information use to enhance scientific and content learning.

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. DUE-0121691. The help of Dr. Lee Zia, NSF program manager, is gratefully acknowledged. Some of the resources on the GROW-NCERL web site were also developed from NSF grant #DUE-9950906. Colleagues Professor John Kemeny, Professor William Rasmussen, Ms. Maliaca Oxnam, Mr. Jeremy Frumkin, Mr. Wayne Brent, Mr. Paul Bracke, Mr. Ronan Dempsey and Dr. Elena Berman are involved in developing GROW-NCERL and helped formulate the interactivities concepts described in this paper. Thanks also to Dr. Stuart Sutton and Dr. Tamara Sumner for their helpful comments on an earlier draft of this paper.

Notes

[Note 1] Geotechnical engineering involves the use of fundamental laws of physics to understand the mechanical behavior of soils for the design of earth systems such as foundations, dams, tunnels and slopes. Rock engineering is similar to geotechnical engineering except the material is rock rather than soils. Water engineering involves works relating to the design of reservoirs, rivers, dams, drainage and irrigation schemes.

[Note 2] Built-environment may be defined as man-made environmental infrastructures.

[Note 3] For example, in 1999, $59.4 billion was spent on roads and highways while another $5 billion was spent for bridges.

[Note 4] Learning is defined rather broadly to mean "the alteration of behaviour as a result of individual experience. When an organism can perceive and change its behaviour, it is said to learn" (from Encyclopædia Britannica). The term "information behaviors" is also defined broadly as "a state in which one may or may not act on the information received." [Chatman, 2000].

[Note 5] See the MIT OpenCourseWare (http://ocw.mit.edu/index.html) and Open Knowledge Initiatives (http://web.mit.edu/oki/) for interactive learning environments. There is also a Journal of Interactive Learning Research, (http://www.aace.org/pubs/jilr/default.htm>) and many conferences on this topic. It is, however, not a new pedagogical approach as the Socratic art of question and answer mode of instruction captures the essence of interactivity.

[Note 6] For example, the Kahn-Wilensky framework defines the design of digital objects for services and focuses on mechanisms that assist in the discovery and retrieval of those objects [Kahn 1995] and Choudhury et al. have proposed a framework for evaluating digital library services [Choudhury et al., 2002]. However, Hansen notes that information access tasks in digital libraries pose special interaction and interactivity challenges for user interfaces in digital libraries that transcend information retrieval research [Hansen, 1998]. Digital library applications require a wide variety of interfaces; these interfaces may control simple tasks such as presenting digital objects for learning or more advanced tasks such as controlling the level and amount of interactivity. Therefore, when considering information behaviors and interactive learning environments, usable interfaces become the first bridge for design and evaluation efforts. A similar effort is the Interaction Framework that has been used at the New Zealand Digital Library project to tie user studies and design efforts [Bryan-Kinns et al, 2000].

Many different types of user interfaces exist for digital library tasks that underlie reading, including searching, collecting and manipulating, and these have been discussed in the literature. For example, Belkin reports on research about interactive information retrieval [Belkin, 1993], and Schneiderman discusses how textual manipulation tasks include highlighting, searching, cutting and pasting, and hyperlinking [Schneiderman, 1998]. Visual interfaces for digital libraries are also emerging as viable alternatives [Weiss-Lijn, 2001]. Winograd has proposed a high-level human-centered interaction theory that is moving us away from the desktop and towards immersive information environments accommodating three important properties of human interaction: object-based perception, individual-dependent interpretation, and action-perception coupling [Winograd, 2001].

[Note 7] See ARL New Measures Initiative [ARL, 2000] and NSDL Evaluation Workgroup [NSDL, 2002].

[Note 8] B. Wilson notes the link between usability and evaluation [Wilson, B., 2002] and Morse summarizes evaluation protocols for complex information management systems such as digital libraries [Morse, 2002].

[Note 9] Usability is always an evaluation and design goal for a DL and can be adequately evaluated when data on three variables is collected with about three to five users performing a goal-directed task: 1) the time the task takes, 2) the error rate, and 3)the users' subjective satisfaction. In the initial phases of design, such formative usability testing is informally conducted. Re-use of "elements" is also another design goal but is not included here.

References

[ARL] ARL (2000). New Measures Initiative. URL: <http://www.arl.org/stats/newmeas/newmeas.html>.

[Arms, C.] Arms, Caroline R. (2000). Some Observations on Metadata and Digital Libraries. In Bicentennial Conference on Bibliographic Control for the New Millennium: Confronting the Challenges of Networked Resources and the Web. Sponsored by the Library of Congress Directorate, 15 -17 November 2000, Washington D.C. URL: <http://lcweb.loc.gov/catdir/bibcontrol/arms_paper.html>.

[Arms, W.] Arms, William Y. (2000). Digital Libraries. Cambridge, Mass.: MIT Press.

[ASCE 2002a] ASCE (2002), American Society of Civil Engineers. URL: <http://www.asce.org/>.

[ASCE 2002b] ASCE (2002). ASCE Keyword List to the CEDB. URL: <http://www.pubs.asce.org/subjlst.html>.

[ASCE 2001] ASCE (2001). Report Card for America's Infrastructure, <http://www.asce.org/reportcard/index.cfm?reaction=factsheet>.

[Belkin] Belkin, N. (1993). Interaction with Texts: Information Retrieval as Information-Seeking Behavior. URL: <http://citeseer.nj.nec.com/cache/papers/cs/5355/ftp:zSzzSzscils.rutgers.eduzSzpubzSz
belkinzSzpaperszSzgi_ir93.pdf/belkin93interaction.pdf>.

[Bloom] Bloom, Benjamin S. and Krathwohl, David R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals, by a committee of college and university examiners. Handbook I: Cognitive Domain. New York, Longmans, Green.

[Bryan-Kinns] Bryan-Kinns, N., Blandford, A., and Thimbleby, H. (2000). Interaction Modelling for Digital Libraries. In Proceedings of Evaluation of Information Management Systems, 2000. URL: <http://www.uclic.ucl.ac.uk/annb/eims.pdf>.

[Budhu 2000a] Budhu, M. (2000a). Interactive Soil Mechanics and Foundations, John Wiley & Sons, NY.

[Budhu 2000b] Budhu, M. (2000b). Soil Mechanics and Foundations, John Wiley & Sons, NY.

[Budhu 1999a] Budhu, M. (1999). A Simulated Soils Laboratory Test, Proceedings of the International Conference on Simulation and Multimedia in Engineering Education, Society for Computer Simulation, San Francisco, Paper No. I505 (on CD-ROM), Jan. 17-20, 1999.

[Budhu 1999b] Budhu, M. (1999). A Multimedia Geotechnical Laboratory Test Courseware, Proceedings of the 1999 American Society for Engineering Education Annual Conference, Charlotte, NC, USA, Paper No. 1315 (on CD-ROM), June 20-23, 1999.

[Budhu 1999c] Budhu, M. (1999). A Virtual Triaxial Test Courseware, in Special Technical Publications 109, Educational Issues in Geotechnical Engineering, 2000, pp. 60 - 70.

[Chatman] Chatman, Elfreda A. (2000). Framing Social Life in Theory and Research. New Review of Information Behavior Research: Studies of Information Seeking In Context, v.1, pp. 3-17.

[Choudhury] Choudhury, S., Hobbs, B., and Lowrie, M. (2002) A Framework for Evaluating Digital Library Services. D-Lib Magazine, 8 (7/8), July. URL: <http://www.dlib.org/dlib/july02/choudhury/07choudhury.html>.

[Clark] Clark, R. C. (1989). Developing Technical Training, Buzzards Bay Press, Phoenix, AZ.

[DCMI] DCMI (2002). Dublin Core Metadata Element Set, Version 1.1: Reference Description. URL: <http://dublincore.org/documents/dces/>.

[Dewey] Dewey, J. (1916). Democracy and Education: An Introduction to the Philosophy of Education. New York, Macmillan. Full-text available, URL: <http://140.211.62.101/dewey/contents.html>.

[Felder] Felder, R. M. and Silverman, L. K. (1988). Learning and Teaching Styles, Engineering Education, April Issue, pp. 674-681

[Gagné] Gagné, R. (1995) The Conditions of Learning, Holt, Rinehart and Wilson, New York.

[Garfield] Garfield, E. (2000). Why s the Engineer So Different From The Scientist? The Scientist, 14 [6]: 4, Mar. 20. <http://www.garfield.library.upenn.edu/commentaries/tsv14(06)p04y20000320.pdf>.

[GROW] GROW. (2002). GROW NCERL. URL: <http://www.grow.arizona.edu>. [to be released: December 2002]

[Hansen] Hansen, P. and Karlgren, J. (1998). Interaction and Interactivity: User Interfaces for Digital Libraries. A Challenge Paper for the 8th DELOS Workshop, Stockholm, Sweden, 21-23 October, 1998. URL: <http://www.ercim.org/publication/ws-proceedings/DELOS8/hansen.pdf>.

[IEEE LOM] IEEE (2002). IEEE Learning Object Metadata. URL: <http://ltsc.ieee.org/wg12/>.

[IMS] IMS (2002). IMS Learning Resource Meta-data Specification. URL: <http://www.imsglobal.org/metadata/index.cfm>.

[Kahn] Kahn, R. and Wilensky, R. (1995). A Framework for Distributed Digital Object Services. URL: <http://www.cnri.reston.va.us/home/cstr/arch/k-w.html>.

[Lane] Lane, F. L. (1994). Keynote Address, Project Impact: Disseminating Innovation in Undergraduate Education, NSF 95-69, Virginia, pp.6-9.

[Liu] Liu, X. (2002). Interactivity and its Implications for Consumer Behavior. Ph.D. Dissertation. Newark, New Jersey, Rutgers University.

[Morse] Morse, E. L. (2002). Evaluation Methodologies for Information Management Systems. D-Lib Magazine, 8 (9), September. URL: <http://www.dlib.org/dlib/september02/morse/09morse.html>

[NSDL] NSDL Evaluation Workgroup. (2002). Educational Impact Standing Committee Home. URL: <http://eduimpact.comm.nsdlib.org/>

[Schneiderman 1998] Schneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human Computer Interaction. 3rd ed. Addison-Wesley, Reading, Mass.

[Schneiderman 2000] Schneiderman, B. (2000). Universal Usability. Communications of the ACM, 43 (5), May, pp. 84-91.

[Weiss-Lijn] Weiss-Lijn, M., McDonnel, J. and James, L. (2001). Supporting document use through interactive visualization of metadata. URL: <http://vw.indiana.edu/visual01/weiss-lijn-et-al.pdf>.

[Wiley] Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects: Online Version, <http://reusability.org/read/chapters/wiley.doc>.

[Wilson, B.]Wilson, B. (2002). Usability and Evaluation. Editorial. D-Lib Magazine, 8 (10), October. URL: <http://www.dlib.org/dlib/october02/10editorial.html>.

[Wilson, K.] Wilson, K. (1992). Discussion on two multimedia R & D projects: The Plaenque Project and the Interactive Video Project of the Museum Education Consortium. In M. Giardina (Ed.), Interactive Multimedia Learning Environments. Berlin: Springer-Verlag, pp.186-196.

[Wilson, T.D.] Wilson, T.D. (2000). Human Information Behavior. Informing Science. URL: <http://citeseer.nj.nec.com/cache/papers/cs/19561/http:zSzzSzinform.nuzSzArticle
szSzVol3zSzv3n2p49-56.pdf/wilson00human.pdf>.

[Winograd] Winograd, Terry. (1999). Towards a Human-Centered Interaction Architecture. Unpublished draft. URL: <http://graphics.stanford.edu/projects/iwork/papers/humcent/index.html>.

[Zia] Zia, L. (2001). Growing a National Learning Environments and Resources Network for Science, Mathematics, Engineering, and Technology Education: Current Issues and Opportunities for the NSDL Program. D-Lib Magazine, 7(3), March. URL: <http://www.dlib.org/dlib/march01/zia/03zia.html>.

Appendix 1

Design and Evaluation Goals for Interactivities in GROW (Phase 1, January 2002-September 2003)

See Note 9 in the Notes section of this article.

Top | Contents
Search | Author Index | Title Index | Back Issues
Previous Article | Conference Report
Home | E-mail the Editor

D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/november2002-coleman