D-Lib Magazine
June 2000

Volume 6 Number 6

ISSN 1082-9873

Carol Ann Hughes
Director, Collections Management
Questia Media, Inc.
Formerly, Interim Director, Information and Research Services
The University of Iowa Libraries
chughes@questia.com

Introduction

One of the goals of the competition was to encourage U.S. libraries, archives, museums, and historical societies to assist in the development of the American Memory electronic collections. But in making the awards, the LC/Ameritech program also offered many institutions that would not otherwise have had sufficient institutional resources the opportunity to explore by "getting their hands dirty" the sometimes confounding issues involved in planning and carrying out a complex digitization project.

The Redpath Chautauqua Collection

Comprising some 648 linear feet of materials dating between 1890 and 1940, the Redpath Collection is considered to be the most extensive holding of circuit Chautauqua materials in existence. The talent portion is the largest series (about 139 linear feet) in the collection. It consists of nearly 8,000 flyers, averaging four pages in length, promoting the talents of over 4,500 performers. As a record of the "business of culture" at the turn of the century, it is a natural resource to select for presentation to the widest possible audience via the web.

The spirited language and vivid artwork used in the promotional flyers reflect the emotions and ideals of the Chautauqua movement. Presentation of the flyers in digital form makes the possibly dry topic of U.S. cultural history come alive to the casual and serious researcher alike. For instance, young students can discover that the public's fascination with pet tricks did not begin with David Letterman. Rather, they would be able to see that Pamahasika's 50 Highly Educated Pets were a popular source of entertainment in the early part of the century. The general public of today would be just as curious as their great-grandparents about tales of hardship and misfortune, such as the experiences of Mrs. Florence E. Maybrick. Unjustly convicted of poisoning her husband, she was sentenced by a judge who went insane shortly thereafter. Yet she served 14 years of her punishment and then went on the circuit to lecture on prison life and the need for judicial and prison reform.

Initial Plan for Workflow

The flyers are irreplaceable. No consideration was given to outsourcing the imaging process. Iowa City is not near enough to a major urban area to allow a vendor to come on-site to perform outsourced tasks. Therefore, the budget and workflow plan had to accommodate library staff for the imaging workflow, which meant that full-time staff would be needed to train, supervise, and provide quality control for the bulk of the work that would be done by student workers. There would be no possibility of also funding full-time staff to supervise students who might be digitizing/SGML encoding the text, especially considering the timeframe of the grant. Therefore the task of keying and "light" SGML encoding, based on a TEI-Lite compatible DTD, was outsourced to a vendor through a bid-process managed by the university's purchasing department.

Over the years, researchers and staff have not been uniformly attentive to the need to maintain the materials in the original order. It was anticipated that we would need to update our description of the collection's size and shape as part of the grant development process. The number of flyers in the collection and the number of unique performers was estimated by a random sampling of one folder in each of the 256 boxes of folders comprising the series. This process indicated that there were 9,600 flyers representing 7,600 different performers. However, as we launched into the workflow it was discovered that there were fewer unique performers (about 4,500) represented by about 8,000 unique flyers. Lesson: Study carefully the dimensions of the project you are undertaking and then add in sufficient resources to accommodate the unexpected, especially if one is working in a special collection of substantial size and variety.

Text Keying/Encoding

The Library of Congress encourages SGML encoding of textual materials in the American Memory project in support of full-text searching and to serve as the basis for more sophisticated search and discovery strategies that might be available in the future. Although the talent flyers are highly graphical and the nature of a promotional flyer might not seem to warrant full text searching, the quality of the text and the richness it provides as context to the images led the project team to concur with a plan for encoding the flyers. However, the team also decided that deep encoding was not necessary and only nineteen elements were identified for the vendor's tagging work. The elements selected were primarily structural, like <figure> and <cell>. Other elements focused on the types of text that might be likely to provide special insight into contemporary culture and style, like citations and quotes from verse. It was reasoned that if use of the collection warranted more encoding, it could be inserted at a later date when more money was available for such activities.

Initial estimates of 28 million characters were based on counting the number of characters (including spaces) in a relatively small sample of flyers. Unfortunately, the sample proved to be skewed and the number of characters required for keying and encoding was underestimated by about 100 percent, even considering the fact that there were fewer flyers than was first assumed. Part of the under-estimation was due to inexperience in estimating the number of keystrokes that would be involved in the insertion of the SGML tagging. This miscalculation was compounded with the discovery that there were quite a few flyers that had substantial textual material. Thus, the team severely underestimated the number of characters to be keyed, which led in turn to an underestimation in the budget required to support this dimension of the project. Lesson: Take the time to do a substantial sampling of the textual materials to ensure that one has a solid sense of the amount of text to be keyed, and then add at least 25% for tagging character strokes.

Fortunately, the commitment of the University Libraries administrative team to this project was strong, and the commitment of the vendor to the project was equally deep. Negotiation of both the fee and the logistics (with both sides accommodating some of the extra work) made it possible to finish the keying. Lesson: Find a vendor who considers the venture to be a partnership and with whom you can communicate frequently. And prepare the library administration for the unexpected throughout the project.

One other unanticipated wrinkle in the workflow occurred during the keying process. The quality of the photocopies to be used by the vendor in the keying and encoding process was often difficult to decipher. The flyers make great use of shading for background effect, and the text is artistically displayed in a variety of fonts around and across multiple images. This is one reason that OCR was not considered for the keying process; another is the poor quality of affordable OCR software, which makes necessary a great deal of proofreading and correction.

Photocopies of text printed in black font on a gray or colored background were in many instances illegible to the keying staff. It is especially important to have completely legible photocopies for keying staff who may not be proficient enough in the language of the text to decipher a poor reproduction. Lesson: The quality control procedures were sufficient for the preservation photocopying, but we did not employ the same procedures for photocopies to be used by the keying vendor. Now the team needs to devise a method for filling in the <gap> and <illegible> tags.

Intellectual Access

The Redpath Collection has been cataloged at the collection level for quite a while, but cataloging at the flyer level was a requirement of the grant. Workstation-based packages for simple MARC cataloging did not suit our needs. We wanted to catalog the item once and then automatically export a MARC file for LC as well as a TEI header with the bibliographic data inserted in the correct elements. So we developed a form-based entry system that enters data into a File-Maker Pro database using Tango for the web interface.

After cataloging each flyer, staff have an additional task that provides the lynch-pin for the entire process: assigning the file naming structure to be used as the unique identifier for linking together the TEI headers with the text files and the page images of the flyer. The basic directories are based first upon file type: "sgm" for the text, "gif" for the thumbnail image, "jpg" for the JPEG images. The second level of the directory path indicates the performer, sequential number of unique flyer in the collection by that performer, and the individual page image. Therefore:

indicates the Traveling Culture collection within the University Library's Scholarly Digital Resources Center site (SDRC), the talent flyer series (chau1), the image database, the performer named Alonso Stagg, the third unique talent flyer in the box, and the gif image of page 5 of that flyer.

Requiring that the file naming structure reflect a particular performer has both advantages and drawbacks. One advantage is that it reflects the traditional structure of the collection, which has always been based upon performer name. The naming procedure makes sense to human beings. This assists in the construction of the unique identifier for the files and may assist humans in quality control when linking the page image, text, and header files together.

However, as more flyers by the same performer are found or as more flyers by performers with similar name stems are cataloged, the system begins to be subject to transcription/construction errors. No change in the file naming process is planned, but these considerations will inform our choices in the next project. Lesson: Plan for as much extensibility in naming structures as possible, even when dealing with collections that are no longer growing in size. It may be the case that machine-assigned/random file names provide the most flexibility in the long run for large text collections, especially those that may expand unpredictably in coverage.

Imaging

The general procedure is as follows: the original flyers arrive from the catalogers with a photocopy of the title page indicating the unique identifier/file name stem for the flyer. Students scan the flyer pages creating 600dpi 32-bit color TIFF images (a time-consuming process) that are saved to Jaz disks. The Jaz disks are used to burn archival CDs. Each CD is labeled sequentially with a note indicating which performer's flyers/pages are included. The uncompressed color TIFF files are so large that each CD averages only 6 images.

An automated program creates a 300dpi 32-bit color file on the workstation desktop from the 600dpi TIFF image after it is saved to the Jaz disk. The 300dpi color file is again downsampled into one 300dpi 1-bit black/white PDF file, one 150dpi color JPEG, one 72dpi color GIF image and one 6dpi color GIF thumbnail. All images for a flyer are saved in one folder on the server. The 300dpi color file is then discarded.

We originally tried to use Debabelizer for the downsampling, but that software could not rename the files according to our naming scheme as they were downsampled. So we use a semi-automated procedure based on Photoscripter. Quality control on the imaging initially identified a "waffle-effect" in the GIFs that was caused by a script error so we have had to repeat the downsample process, but that is an automated batch process from the 150dpi color JPEGs residing on the image server.

GIF images at 6dpi are used as thumbnail navigational aids in the search interface; 72dpi GIFs are the default display images. The JPEG image is an option that may be selected by the user who wishes to view or download a higher quality image. PDF images are provided for ease of printing.

Many talent flyers contain images that span two pages, or even three. The project team made a strategic decision that images that span pages should be presented to the public as coherent whole images. This requires that the image be "stitched together" from the individual page images after they are downsampled, a job that requires patience, time, and a keen artistic eye. This is a task for one highly skilled team member. Lesson: Consider carefully whether one can create automated routines to prevent the need for extensive training and lots of "hand-work." The budget impact of the need for special handling can be substantial.

Putting it All Together

General Lessons Learned: This project was accomplished largely through the efforts of student staff, especially undergraduate student staff. All of the imaging, and much of the cataloging, was done by students with hourly appointments. Although hiring, training, and scheduling student staff is time consuming, the quality of the work and their facility with the technology has been quite satisfactory. The final budget report for the grant indicates that we came within $20,000 of what we anticipated spending for the cost-sharing portion of the work. This would not have been possible without the use of hourly staff. We would not hesitate to employ students in the next digitization project.

Technical and work-schedule "hiccups" have occasionally delayed work, but the people and equipment have generally worked well. We could use more students, more equipment, and more hours in the day, but the experience gained in this project has been invaluable as a library-wide development and enrichment activity. Nothing replaces the opportunity to "get one's hands dirty."

In August 1999, the project staff gave a presentation to the Libraries about the project. Our colleagues were impressed with the complexity and scope of the work and were glad to learn more about exactly what steps were involved in the process. But what impressed us was their equal conviction that the Redpath Chautauqua Collection is indeed a treasure that should be available to the world. We share that conviction and look forward to offering "Traveling Culture" to people around the world.

URLS of Interest

Copyright © 2000 Carol Ann Hughes

D-Lib Magazine Access Terms and Conditions

DOI: 10.1045/june2000-hughes