Digitising History
CHAPTER 3 : FROM SOURCE TO DATABASE

Anyone involved in a project (or potential project) that involves the creation of a digital resource has to think in terms of a finished product. Given that few projects in the UK are funded solely on the basis of resource creation, those involved in the project have to balance the requirements of providing academic research results with the need to produce a database that is both of value to them and potentially to others. It is this balancing act, and the implications it has for the resultant data, that this section discusses.

3.2.1 General project objectives

All historical research of this kind involves the subtle synthesis of traditional research approaches with computer-based methodologies. Ultimately, the computer is a research tool and is used to explore themes, issues or important questions. Perhaps the best diagrammatic summary can be found in Harvey and Press (1996, 3). Traditionally the focus of most historical research projects has been the publication of findings. The gathering, management and analysis of documentary evidence is part of the necessary process of historical discourse, but usually the source material does not have a tangible place in the final objective of a project. Unless the work involves a scholarly edition of some kind, the sources are relegated to the background whilst the research findings take centre stage. Funding is provided for historians on the basis that new insights into the field are produced, rather than the publication of existing historical documents. Coping with the pressure to produce results is at the heart of all research programmes.

However, for those whose project envisages the creation of digital data, this traditional model of practice soon becomes of limited use. This guide strongly encourages researchers in this field to consider carefully their 'general project objectives' in this case. In other words, researchers working on historical projects that include resource creation should think about their sources, the questions to be asked, the database design phase, the database creation phase and the database documentation phase. Some of these may not be linear stages (for example documentation is best tackled on-the-fly: see Chapter 5), yet careful planning and preparation will be essential to the project's success. For an excellent summary commentary, see Harvey and Press (1996, 76-81). It would not be an over-simplification to suggest that the inclusion of database creation genuinely requires an innovative approach to research projects. Yet perhaps 'Is database technology appropriate for my research?' might be the very first question any project team or individual might ask. If it is established that such technology is appropriate, a range of issues and considerations to be tackled emerge. There are a number of ways in which the resource creation element of a projectmay have an impact on the project as a whole. The following questions need to be considered:

• Are all the elements of the source suitable for transcription to tabular database form? If not, what is the strategy for working with problematic elements?
• Are the research goals of the project feasible, especially given the resource creation element of the project?
• Are enough resources allocated to meet the research and data objectives (time especially)?
• Does the project recognise the wider value of the source material? Would the digital source be of benefit to future researchers?
• Has project planning provided an acceptable balance between resource creation and data analysis?
• Is the project aware of outstanding copyright issues involved when replicating original source documents?

Creators should consider the above as a checklist. Notice that traditional research projects do not have to grapple with such questions.

3.2.2 Data objectives

One of the central issues that historians need to know of any database is 'what questions can be asked of it?' Of course, all databases, whether statistical or textual in character, have a finite analysis compass, all are limited in some way, but lack of forethought can lead to a resource that has an extremely limited potential audience and which, therefore, however technically perfect, will be of only very limited wider use. Take the fictitious example of a project that is exploring nineteenth-century migration patterns from census enumerators books (CEBs). The project team has decided that their database will contain only five fields, namely 'Surname', 'Forename', 'Age', 'Place of Enumeration' and 'Place of Birth'. Although this is a completely legitimate project decision, it is clear that such a resource can only be used to study migration, and therefore has only a niche market, and one in which the project team itself may have already explored the issues in sufficient detail. Given that CEBs contain information that goes beyond merely the study of migration, the project team might consider the option that they could in fact produce a more complete database, transcribing more fields than they need, and so provide wider benefits to the scholarly community in general. Taking this view would almost certainly add extra importance to their project and ensure its citation across historical discourse, despite project-focused limited research objectives. (Consider the Mormon transcription of the entire 1881 Census (GSU and FFHS 1997). Their extracted indices of names and birthplaces are of use only to genealogists. However, their original work which transcribed the entire CEBs has ensured that this project not only has use beyond genealogy, but also happened to create the most important digital historical resource of the twentieth century.)

The point is that resource creators should carefully consider the extent to which they can separate the database resource itself from the project objectives. Creators should recognise at an early stage the value of their original source documents in a wider sense. Thus, if the resources exist, creators should design the database not solely on the variables they are themselves interested in, but should consider conversion of the material as a whole. Traditionally, the creation of research databases has been solely driven by the need to extract results within the project time frame. As a consequence, a whole swathe of digital databases have been created (and archived) that cannot be used to answer questions beyond those the original researchers were asking. If more valuable resources are to exist, then projects need to factor in the necessary requirements for considering their database as a 'structured edition' of the original source in as much as this is feasible. This, in a sense, is a different slant to the seemingly 'all or nothing' approach of the old 'Source Oriented vs. Model Oriented' argument (Denley 1994) as it is something of a compromise between the two.

The 'data objectives' of any project of this nature need to be considered carefully. All too often the data materials are left as the afterthought of a project, which are offered for deposit with a data archive out of obligation rather than in a genuine attempt to provide a new and interesting resource. Yet many data collections held by the HDS are thoroughly commendable. What distinguishes these key resources is that the creators have planned carefully and given the creation and maintenance of their database a high level of priority. These are projects that have precisely defined their data-related objectives and have employed significant resources in ensuring that such objectives are met. It is also extremely important to note that the capacity to create a valuable and meaningful resource need not impinge on the project's ability to produce satisfactory academic results. Indeed, it can provide significant academic benefits for the original researcher/s in the ability to reflect on research and re-visit their complete database over time.

As a data-focused good practice checklist, consider the following:

• Is the project aware of its data needs?
• Has the project a clear idea of intended data volume?
• Are the resources set-aside for data entry, validation, management and documentation sufficient?
• Is there a disaster recovery strategy at the outset (e.g. data back-ups)?
• Does the project, overall, have sufficient technical expertise at hand?
• Has the project settled on and tested the target software application?

All of the above points are pure common sense. For smaller, one-person based, projects some of the above may of course not be applicable. Yet even for the smallest data gathering endeavour, issues of data back-up and careful technical planning remain important considerations. The over-riding guideline here is to consider the possible wider interest of the source material, and also to ensure that the new digital database is of high quality and usability. Achieving this may require extra planning, but the end result would reflect on those involved very highly indeed. The purpose of defining aims and objectives carefully is that it allows those involved in the project to plan ahead and to base research around database creation.