1.0 Introduction
The rapid growth of the Internet and global network systems for the development, presentation and transfer of digital information is causing educators to rethink the method in which they provide information. The advent of low cost, widely accessible computers and high bandwidth networks has opened the door for new methods to enhance and transform the traditional education process. Throughout the education community, the quantity of material available digitally is increasing daily as the "publishers" of this new era are making many of their materials available on the World Wide Web. Digital courseware, computer-based tools, large datasets, and electronic supplements to learning are becoming widely accessible by faculty, students and learners in general.
NEEDS — The National Engineering Education Delivery System (see www.needs.org) — is the distributed architecture developed by Synthesis: A National Engineering Education Coalition (see www.synthesis.org) to enable new pedagogical models based on Internet-mediated learning environments [1, 2]. The emergence of the World Wide Web (WWW) in the early 1990's as a viable means for national and international sharing and re-use of education materials fundamentally changed our view of the way education and learning can be delivered. Internet-mediated learning environments provide mechanisms for the learner to be anyone, anywhere, at anytime. NEEDS has expanded its scope and currently catalogs courseware and other instructional software developed nationally and internationally to provide a resource where both instructors and learners can search, access, and download educational materials over the World Wide Web. In addition, NEEDS supports a multi-tier courseware evaluation system including a national award competition — the Premier Award for Excellence in Engineering Education Courseware [3].
NEEDS' vision of what a digital library for undergraduate engineering education should be is more than just a traditional academic library in digital form. The digital library of the future will be a community of learners — encompassing faculty, students, and life-long learners. With our current system, NEEDS has taken the first steps to developing this "new" digital library; we are driven by recent advances in research on education and student learning. We distinguish ourselves from commercial, Web-based search engines by providing focused, value-added services to our community. NEEDS is in the process of supplementing our current services by adding features such as user reviews and attachments, user registration, and discussion tools.
This article will provide a description of the NEEDS project. Following a brief history, we will examine the current NEEDS system. We will then outline our current thrusts in engineering and outline the new features we are deploying. Throughout the article we outline some of the challenges in developing a digital library of educational materials and describe how some of our current thrusts address these challenges.
2.0 Brief History
Synthesis: A National Engineering Education Coalition, with funding from the National Science Foundation and industrial partners, was established in 1989 to reform undergraduate engineering education. This broad, nationally distributed coalition of eight diverse universities (California Polytechnic State University, San Luis Obispo; Cornell University; Hampton University; Iowa State University; Southern University; Stanford University; Tuskegee University; and the University of California, Berkeley) embraced instructional technologies as a means of assisting our curricular goals in the reform of undergraduate engineering education. Courseware, whether it is stand-alone and CD-ROM based, or networked and accessible over the Internet, provides methods to enhance the traditional learning process and address multiple learning styles. Key in the early stages of the Coalition was a vision to enable the sharing and reuse of materials developed within the Coalition.
Synthesis developed NEEDS as the mechanism to support the development, use and more importantly re-use of these instructional technologies. The nationally distributed nature of the Coalition and our interest in cross/multi-disciplinary education proved to be extremely important. Our experiences in dealing with a wide variety of institutions, technologies, and disciplinary and multidisciplinary content provided us with a broad view of how a national resource can be used, who can be its target audience, what instructional technologies can be made available through this resource, and what's needed to describe them.
2.1 NEEDS Infrastructure
NEEDS has developed a robust infrastructure built around a distributed server architecture. We maintain strong ties to the best of state-of-the art research in databases, information retrieval, and digital libraries. These ties to research, coupled with a production focus, have allowed us to evolve NEEDS in sync with the rapid changes occurring with information technologies in order to meet the needs of our users. NEEDS debuted in the early 1990's as a "text-based search engine over a
centralized library catalog with pointers to digital course material"
[1] accessible via telnet and transitioned to a Web-based on-line database
in 1994. Through its current design, NEEDS has undergone three major
updates in response to users' needs collected through surveys of faculty
and students in engineering education. NEEDS is ultimately about providing a service to our user community; we apply the appropriate technology to meet our users' needs. Along the way, we have developed and implemented key elements in a reliable, distributed, scalable system architecture, including: redundant Web servers; an integrated database with WWW-based searching and downloading, utilizing state-of-the-art database and information technologies; and an indirection system to support multiple, nationally-distributed archive servers for courseware download.
NEEDS continues to grow and evolve based upon international standards for data description (USMARC and emerging metadata descriptors) and data access (HTTP and the World Wide Web), because they allow access to all users [4, 5]. NEEDS uses a robust structured query language (SQL) relational database to store and index courseware records. The underlying framework for these courseware records is based upon a standard library format for indexing and storing documents (USMARC) [4, 5]. The USMARC standard lets us apply a wealth of cataloging experience gained in the library community. Key to continued success is evolution, which currently includes adoption of the metadata descriptors developed by the Educause Instructional Management Systems (IMS) project [6].
Materials indexed and searchable through NEEDS are diverse — content ranges from single topics that can be covered in a few minutes to fully integrated, term-long courses. One of the most powerful concepts supported by NEEDS is courseware modularity, in some cases NEEDS catalogs courseware as well as the individual elements (e.g., images, videos, and text) that comprise the courseware. Courseware elements provide a vehicle for continued re-use of content material beyond the lifespan of any particular courseware module. These elements can be used as is, or distilled from multiple sources and joined together to create new, customized courseware. The modularity supported within NEEDS is seen as a major enabling technology for fostering educational material adaptation and re-use.
Courseware cataloged in NEEDS is acquired through a Web-based interface. Authors and other submitters are able to add materials to NEEDS in the same fashion that learners can access those materials, anytime, anywhere. Consistent with library cataloging standards, we perform a basic review of courseware as it is cataloged. We provide this review to insure that the courseware record has certain required fields — e.g., a title, author, publisher and platform (e.g., PC, Mac, or WWW); to perform a standard check for viruses; and to verify that the program can be operated on the intended platform(s). We use this basic functionality check in place of forced editorial oversight before courseware archiving; we have learned that it is counterproductive to introduce too many barriers to the cataloging process. Instead we have developed review systems to evaluate courseware once archived in NEEDS.
Figure 3 - Bibliographic Record
Click on Figure for Larger Image
|
|
Bill reads the bibliographic record and takes a look at the Virtual Disk Drive Design Studio website and determines that this courseware is just what he needs to introduce the mechatronics concept to his students. He recalls from the NEEDS Home Page that the Virtual Disk Drive Design Studio has won the Premier Award, so he knows that his peers have reviewed this courseware for engineering content and instructional design. The courseware provides an entertaining, yet technically challenging introduction to the disk drive — a common mechatronics device.
|
| |
In Figure 3 we see the bibliographic record. The bibliographic record describes the pertinent information about the courseware, in the same manner that traditional on-line public access catalogs provide information on books (i.e., title, author, publisher, subject heading, keywords, etc.). The user can then view the bibliographic record that describes the courseware and provides hyperlinks to download the courseware for different platform(s). Other hyperlinks on the courseware record provide access to related information indexed along multiple axes (i.e., author, publisher, subject heading, and courseware series).
|
3.0 Quality Review of Courseware
Courseware on NEEDS, as with any scholarly effort, requires review to assure quality and provide recognition [3]. The Quality Review of Courseware effort was developed to advance the field of technology-enhanced learning review as well as continually raise-the-bar for excellent courseware. The first step in developing a review procedure was to examine existing schemes of software, courseware and paper review and adopt those aspects that were deemed most appropriate for a rapidly changing environment of courseware creation, in particular, and multimedia technology, in general.
3.1 Evaluation Criteria
Since 1994, NEEDS has worked with numerous experts nationwide including students, engineering educators, instructional designers, cognitive scientists, and learning theory experts [7, 8], to develop and implement a quality review procedure for engineering education courseware. This process, led by Professor Pamela A. Eibeck, Chair and Professor of Mechanical Engineering at Northern Arizona University, has resulted in evaluation criteria and a multi-tiered review system for courseware. Based on our initial study [8], NEEDS has developed and has developed a comprehensive, tiered peer review including: non-reviewed, endorsed, and premier courseware.
The key ingredient to the peer review process was the development of evaluation criteria for engineering education courseware. In developing the criteria, NEEDS has taken a learner-centric outlook on the educational experience. Many of our beliefs about education are interwoven into the criteria we have chosen as well as how we view courseware that meet the criteria. The criteria builds upon some of the latest thinking about higher education and value constructivist learning. We value courseware that includes learning goals and instructor's guides that prescribe pedagogical applications of the courseware.
There are a seemingly infinite number of ways to set forth criteria for evaluating courseware. In fact, in the process of developing the set presented below, we did keep rearranging the individual criteria. However, in the end we assigned the criteria to three categories — Engineering Content, Software Design, and Instructional Design — because we felt that this division mirrored the evolution of courseware design and development by our target audience, engineering professors. (See http://www.needs.org/premier/1999/criteria.html for additional information about the criteria.)
Table 1 — Evaluation Criteria for Engineering Education Courseware
|
Instructional Design:
- Interactivity
- Cognition/ conceptual change
- Content
- Multimedia use
- Instructional use/adaptability
|
Software Design:
- Engagement
- Learner interface and navigation
- Technical reliability
|
Engineering Content:
- Accuracy of content
- Organization of content
- Consistency with learning objectives
|
3.2 Premier Award for Excellence in Engineering Education Courseware
Our initial focus has been on developing the highest level of scholarly review described above, the Premier Award for Excellence in Engineering Education Courseware. NEEDS, Synthesis, and John Wiley & Sons, Inc. (see www.wiley.com) developed the Premier Award to "recognize high-quality, non-commercial courseware designed to enhance engineering education." The Premier Award competition results in courseware being named Premier Courseware. The winning instructional courseware packages are chosen through a rigorous application and review process. They represent excellence in the use of computer-based instruction to enhance engineering education. It is important to recognize that the Premier Award recognizes not just educational software but rather the entire learning experience. The Premier Courseware have distinctive strengths, representing a breadth of styles, sophistication, pedagogies, and use of multimedia that should encourage incorporation of the courseware by others into their learning environments. Through the first two years of this national competition, NEEDS has developed the structure necessary to ensure the Premier Award becomes an established and respected aspect of the engineering community.
NEEDS solicits courseware from engineering educators via our website, conference and workshop presentations, ads in ASEE Prism magazine, and by word of mouth. The submission procedure (see http://www.needs.org/premier/1999/submit.html) is designed to help the judges better evaluate the courseware. The procedure requires submitters to address the review criteria directly and to respond to the following:
- Describe the impact of this courseware.
For example, what topic areas are covered in the courseware? How and where has the courseware been used? In a lab or lecture section? At a different institution? In different departments? How many learners have been impacted by the courseware? A single class? An entire department?
- Describe how the courseware is used by a learner
. Include pedagogical objectives, learning goals, lesson plans, instructor's and user's guides, etc.
- Describe the evaluation and assessment performed to assess improved student learning through use of the courseware.
For example, how was student learning improved? How was student learning measured? Was some process/product (e.g., report writing, test scores, etc.) measurably improved?
- Provide Letters/Statements of Reference.
These letters/statements should be by instructors other than the author(s) that have either adopted or adapted the courseware for use in their own classroom. Ideally one or more of these instructors will be at a different institution than the author(s).
NEEDS has conducted two Premier Award competitions with the Inaugural competition held in 1997. In the recently completed 1998 competition, the following three outstanding, high-quality courseware were named the Premier Courseware of 1998:
The Della Steam Plant Case Study by P. K. Raju and Chetan S. Sankar at Auburn University — The Della Steam Plant Case Study presents an engaging, real-life engineering case study requiring active participation and teaming by engineering students. Through the case study and associated classwork, students will develop higher-level, cognitive skills such as problem identification, critical thinking, and problem solving.
MDSolids by Timothy A. Philpot at Murray State University — MDSolids offers students numerical, descriptive, and visual results and details that illustrate and explain many types of problems in introductory mechanics of solids courses. In conjunction with traditional instruction techniques, MDSolids supports a learner's development of basic concepts and problem-solving skills through self-study.
Structural Engineering Visual Encyclopedia — University of New Hampshire by Robert Henry at the University of New Hampshire — SEVE-UNH offers students a quick visual reference to common engineering terms and technical drawings related to the design and construction of buildings. SEVE-UNH provides a rich and valuable encyclopedia of terminology, animations, drawings and photographs to address a gap in many civil/structural/construction engineering courses.
As part of the Premier Award, NEEDS and Wiley have committed to widely publicizing and distributing the Premier Courseware. NEEDS distributed 1,300 CD-ROM's of the Premier Courseware of 1998 at the 1998 Frontiers in Education Annual Conference (FIE) and to the Deans of Engineering. Over the last two years, NEEDS has also distributed approximately 2,300 3–CD-ROM sets of the Premier Courseware of 1997 at FIE 1997 and 1998, at the American Society of Engineering Education Annual Conference (ASEE), at the International Conference on Engineering Education 1998, and to the Deans of Engineering. The Premier Award and Premier Courseware have been highlighted in NEEDS and Wiley exhibit booths at FIE and ASEE.
4.0 Current Thrusts — Expanding our Engineering Core
NEEDS has been awarded a three year grant from the NSF Engineering Action Agenda program to provide services to the engineering education community [9]. Our goals are:
- Maintain NEEDS as a service to the engineering education community, expanding cataloging efforts and viewership to all U.S. Colleges of Engineering.
- Continue to grow and evolve NEEDS as the foundation for an on-line engineering education community — adding user reviews, feedback to authors on dissemination and use, discussion tools, and improved indexing for engineering education.
- Expand review and evaluation services to improve the quality of engineering education courseware.
- Serve as a bridge towards the development of a broader program to support the Science, Mathematics, Engineering, and Technology Education (SMETE) community as a whole.
The success of NEEDS as a service depends not only on the development of a critical mass of content, but also on the development of a critical mass of viewership. The development of an on-line community — consisting of developers, adapters, adopters, interested parties and learners of the content made available through NEEDS — provides the means of sustaining NEEDS as a resource. The content continually draws the user back, and stimulates discussion among community members regarding adoption and adaptation of existing courseware, leading to new courseware development and courseware acquisitions.
The community can address many of the concerns and challenges in cataloging courseware (e.g., inadequate courseware description or lack of support material) by becoming a user-based support and evaluation system. Many of the new features we plan to add to NEEDS are focused on strengthening our present methods of review for all courseware within the system in order to help our user community evaluate whether to adopt or adapt a particular piece of courseware.
For the last four years we have relied on standard usage data generated by Web servers (e.g., browser-type and where the browser is located) to guide system development and features offered. However, Web usage logs do not allow us to easily couple information about what queries (searches) are performed with the results of that query (i.e., which courseware was presented), which courseware was downloaded, whether it was successfully downloaded, who downloaded it and why it was downloaded. To do this, we are instituting a transaction-based system to track queries and their results. This Extended Usage Tracking system will allow us to provide authors with attempted download statistics and better understand our users' search patterns.
We also instituting a User Registration system that allows us to map "who" uses courseware to "where" and "how" they are using it. This in turn facilitates longitudinal tracking and dissemination efforts for authors, and provides them with greater user feedback. We can track "successful" downloads with a follow-up survey to ascertain the adoption or adaptation of courseware; find similar or related courseware; and receive feedback for continuous system refinement. In addition, we will allow registered users to develop User Profiles through which we will proactively identify newly cataloged courseware of interest to the user. NEEDS thus becomes demand-driven by carefully targeting acquisition efforts toward identified content needs.
We are expanding our concept of bibliographic record to allow User Reviews and supplementary Support Materials. User Reviews will provide a wealth of community-based support for courseware. They may include information on a user's experience with the courseware and how a user employed the courseware — potentially expanding upon the author's original intent. A good example of User Reviews can be found at Amazon.com where book buyers can read reviews, written by other consumers, to better judge a book before making a purchase decision. Support Materials extend the catalog record by attaching user-provided links to related information.
We are well into the third year of the Premier Award for Excellence in Engineering Education Courseware. As we do so, we expect to raise the expectation of what represents quality in courseware. We have issued a "Call for Submissions" and expect to see a wide range of high-quality courseware. Submissions for the 1999 Premier Award are due July 31, 1999 (see http://www.needs.org/premier/1999/submit.html).
In addition to a traditional Peer Review system, we are developing Feature Reviews similar to book reviews found in journals and newspapers. Finally, we are adding "off the shelf" threaded Discussion tools to further transform the static courseware record into a dynamic, living entity. These services extend the courseware record by providing a forum for users to discuss the courseware and its application.
5.0 Lessons Learned and Future Directions
The NEEDS project has had close to a decade of experience running a digital library of courseware for engineering education. We have greatly benefited from continuous feedback from our users, through focus groups and questionnaires, and from our advisory groups. Our ties to the latest digital library research have allowed us to periodically update our system to meet the growing needs of our user community and quickly deploy advanced information technologies. The decision to use hardened relational database technology as a back-end has allowed us to efficiently manage large datasets and quickly update the user interface. As an example, we were able to move from a X-windows interface to an HTML interface with a single day of coding in 1994.
We believe the decision to build on evolving library standards in archiving and indexing is a great strength of our system. Our original use of MARC records for cataloging has allowed us to easily move to IMS standards. The library standard of maximizing access led us to create interfaces that are accessible via the WWW, across all platforms and that require minimal levels of technical expertise. In addition to providing a minimum level of presentation standards, such as title and author information, we perform virus checking and verify platform requirements before archiving courseware on NEEDS. In order to allow greater author control and to scale the number of records we could catalog, we moved away from employing a full time library cataloger to a system of author-assisted cataloging. The author provides the initial information for his or her courseware through a Web-based cataloging form. Nationally-distributed Internet catalogers will then review the records, perform a minimal level of acceptance testing and communicate with the author if changes or more information is needed.
We are expanding our collection development to the interface between engineering and science and math. In doing so, we hope to break down the artificial barriers imposed by a strict disciplinary view of education. This expansion is a natural evolution of our work because physics, chemistry and math are the prerequisite lower division classes that can ultimately culminate in an engineering degree. The NEEDS infrastructure is robust enough to allow us to catalog, archive, and search for courseware across the boundaries between engineering, physics, chemistry, and math.
The frenetic pace of change in information technologies has exacerbated the age old problem of finding information. With everyone a "publisher" in this new information age, the volume of materials is drastically increasing. Even if we "just" limit ourselves to engineering education, the volume is enormous. NEEDS is developing a new vision of a digital library to serve the engineering education "community." By forming a community of learners, we can leverage the personal expertise of its participants to help the community-at-large organize, describe and locate the appropriate resources. The addition of user reviews, support materials and discussion tools add a strong, community-based support system to further promote the adoption and adaptation of courseware.
Acknowledgements
NEEDS is a team effort and we wish to acknowledge the ongoing contributions of: William H. Wood III, Assistant Professor of Mechanical Engineering at the University of Maryland, Baltimore County; Gregory Paschall, NEEDS Lead Programmer, University of California at Berkeley; Flora McMartin, NEEDS Evaluation Director, University of California at Berkeley; George Toye, President of WITHInc and Consulting Associate Director of the Stanford Learning Laboratory; Pamela A. Eibeck, founding NEEDS Editor and Chair and Professor of Mechanical Engineering at Northern Arizona University; and Joseph G. Tront, NEEDS Editor and Assistant Dean of Engineering Computing at Virginia Tech. We also wish to acknowledge those that contributed to the NEEDS foundation: Steven Bradley, University of California at Berkeley; Arvide Eide, Iowa State University; Larry Genalo, Iowa State University; Jim Harris, Cal Poly San Luis Obispo; Jeff Huston, Iowa State University; Jaya Krishnagopalan, Tuskegee University; Dave Martin, Iowa State University; Adebisi Oladipupo, Hampton University; John Saylor, Cornell University; Robert Thomas, Cornell University; the members of the NEEDS Quality Review of Courseware Committee; and Standards Study Group.
NEEDS has been supported by the National Science Foundation (EEC-9872570, IIS-9817406, EEC-9625456, EEC-9053807) and by our industrial partners including Apple, IBM, John Wiley & Sons, and Sun Microsystems.
[1] Agogino, A. M. and W. H. Wood III, "The Synthesis Coalition: Information Technologies Enabling a Paradigm Shift in Engineering Education," Keynote talk, in Hyper-Media in Vaasa '94: Proceedings of the Conference on Computers and Hypermedia in Engineering Education (Ed., M. Linna and P. Ruotsala, Vaasa Institute of Technology) pp. 3-10.
[2] Agogino, A. M. and Muramatsu, B., "The National Engineering Education Delivery System (NEEDS): A Multimedia Digital Library of Courseware," International Journal of Engineering Education, Vol. 13, No. 5, pp. 333-340, 1997.
[3] Eibeck, P., "Criteria for Peer-Review of Engineering Courseware on the NEEDS Database," IEEE Transactions on Education, Special Issue on the Application of Information Technologies to Engineering and Science Education, Volume 39, Number 3, pp. 381-387, August 1996.
[4] Synthesis Coalition, "Standard Study Report 1.0," Ithaca, NY, March 1993.
[5] Wood III, W. H. and Agogino, A. M., "Engineering Courseware Content and Delivery: the NEEDS Infrastructure for Distance-Independent Education," Journal of the American Society for Information Science, Vol. 47, No. 11, pp. 863-869, 1996.
[7] Muramatsu, B., Eibeck, P. A., Stern, J. L., and Agogino, A. M., "Effective Processes to Give Engineering Educators Easy Access to Quality-Reviewed Electronic Courseware," Invited Presentation, NSF Engineering Education Innovators' Conference, Washington, D.C., April 8, 1997.
[8] Synthesis Coalition, "Quality Workshop," Synthesis Coalition: San Luis Obispo, CA: May 1995.
[9] Agogino, A. M., "Expanding the National Engineering Education Delivery System as the Foundation for an On-Line Engineering Education Community," NSF Grant No. EEC-9872570, 1998.
Top | Contents
