Search D-Lib:

D-Lib Magazine

March/April 2016
Volume 22, Number 3/4
Table of Contents

A New Approach to Configuration Management for Private LOCKSS Networks

Tobin M. Cataldo
Birmingham Public Library, Birmingham, Alabama
tcataldo@bham.lib.al.us

DOI: 10.1045/march2016-cataldo

Printer-friendly Version

Abstract

The node-based configuration management model is an alternative approach to configuration management for Private LOCKSS Networks that reduces external dependencies and subsequent vulnerabilities by flattening the hierarchical vendor-consumer model into a preservation node-based service. The node-based configuration management model also describes approaches for leveraging the LOCKSS preservation protocols to distribute and preserve the configuration data, and maintain continuous service regardless of network membership changes or infrastructural failure.

1 Introduction

The Alabama Digital Preservation Network (ADPNet) is a Private LOCKSS Network (PLN) providing digital preservation services for memory organizations throughout Alabama. Specifically, ADPNet is a collaborative network of libraries and archives in Alabama working in a peer-based organizational framework. In this peer-based framework, there is no true administrative hierarchy, and management tasks rotate among network members. As a PLN, LOCKSS preservation software forms a core component to the identity of ADPNet.

Lacking the intra-network capacity to create and operate a PLN independently, founding members of ADPNet leaned heavily on the expertise of LOCKSS to design and instantiate the preservation network. As a result, network administration, configuration, and software support for ADPNet were organized in the hierarchical vendor-consumer administrative model that mimicked the established Global LOCKSS Network (GLN). In the vendor-consumer administrative model, LOCKSS controls the single authoritative source, or configuration server, for all of the ADPNet preservation nodes. While LOCKSS is generally referred to in this document as the "vendor" in the vendor-consumer model, particularly for networks directly administered by LOCKSS, the vendor role could be filled by any entity or institution maintaining singular operational control of the preservation network. For a PLN not directly administered by LOCKSS, the organization providing or reselling the preservation network would be classified as the vendor. The hierarchical vendor-consumer model provides a number of benefits, but also carries with it several, perhaps unintended, consequences. The most serious consequence is that a disruption in the link between the ADPNet preservation nodes and the LOCKSS configuration server has the potential to negatively impact the ability of ADPNet to offer continuous preservation services.

2 Private LOCKSS Networks — In Brief

In general terms, a PLN can be described as a known set of peer preservation nodes operating independently, but participating cooperatively, in a defined group for the discovery of content, the polling of preserved data, and consequently self-repair, or self-alignment to the group majority [1] [2]. In a nominally operating PLN, any individual preservation node could fail without significant consequence to the veracity of the preserved data or the soundness of the network, provided sufficient redundancy exists. This data protection and node fault tolerance are key benefits brought about due to the general design of the LOCKSS preservation network. Independent nodes maintain full replicas of preservation data and self-align to the majority through the process of polling [3]. The result is several, fully retrievable replicas of preservation data in the network.

PLNs are an extension of the original electronic journal-centric GLN devised by leveraging the LOCKSS preservation software to collect and preserve a wider variety of data types. While the GLN, by necessity, is strongly hierarchical, PLNs are more freeform and can take various organizational models, from centrally controlled to flattened, collaborative models. The hierarchical vendor-consumer model is most prominent, perhaps, on account of factors ranging from financial control of the network to the less troublesome nature of centralized technical control. No particular organizational model is software-defined mandatory for a PLN. Many organizational models are possible for PLNs due to the collaboratively-independent nature of the preservation nodes. In a LOCKSS preservation network, nodes participate in preservation groups, and the group boundary is defined on a common authoritative configuration server.

3 Configuration Server

The preservation nodes query the configuration server to discover network peers, archival unit title lists, shared plugin definitions, and software operating parameters. Configuration data is typically maintained in a set of XML documents. The most vulnerable period in the execution of LOCKSS software is during startup. The absence of the expected configuration documents will impede the ability of LOCKSS to operate nominally. It should be noted, however, that even with the inability of LOCKSS software to execute, the locally cached replicas of the preservation data are still accessible through file system interaction. LOCKSS-driven preservation activities, namely voting and polling, for the data are suspended until the LOCKSS software is executing normally with accurate configuration data. In the hierarchical vendor-consumer model, LOCKSS maintains independent control of the configuration server, and in numerous, perhaps most, scenarios, vendor control of the configuration server may be preferable due to the vendor's unparalleled expertise in LOCKSS software configuration and performance tuning. The configuration server is a critical component for the successful execution of LOCKSS software. When operating nominally LOCKSS software creates a robust, independent, and redundant network for preserved data.

LOCKSS and other reputable PLN vendors include backup and fail-over procedures for the configuration server [4]. The vendor-consumer administrative model is not a description of the network topology within the administrative control of the vendor. To the consumer, in the hierarchical vendor-consumer administrative model, there is little difference between the primary vendor-controlled configuration server and the backup.

4 Vendor-Consumer Administrative Model

In the vendor-consumer administrative model, from the node perspective, the described redundancy and retrievability of the preservation data does not extend to the configuration server data. In fact, in the most common scenarios, there is little redundancy of production-ready configuration data within the preservation network.

In normal LOCKSS operations, plugins are cached on the preservation node, and while the configuration data could be retrieved and cached on a preservation node, the process to do so it is not regularly and universally done. For LOCKSS-administered PLNs, LOCKSS maintains the title list on version control [5]. During recovery, the title list could be rebuilt from the version control sources, provided the entire title list is represented. The ability to recover the title list may not extend to alternative PLN vendors. The LOCKSS configuration document could be reconstructed, to some extent, with artefactual hints represented in the LOCKSS software logging mechanism. However, for a PLN without LOCKSS software operational knowledge, moving from recovery to production would be a challenging if not monumental task. In the vendor-consumer model, LOCKSS operational knowledge is not prerequisite, and PLNs may have little understanding on how to recover and reconstruct the necessary configuration documents. Dependency on the vendor for operational control of the preservation network does not create an environment emphasizing the necessity of obtaining operational knowledge. As a result, an unrecoverable disruption in the link between the preservation nodes and the LOCKSS configuration server could leave ADPNet without the means to define the preservation group or control network operations until a new configuration service is brought online. For a PLN with little to no understanding on how to reconstruct the configuration data or with no access to title list information through alternative sources, an unrecoverable disruption in the link with the LOCKSS configuration server could be serious, perhaps fatal.

Despite its benefits, the vendor-consumer administrative model creates a network dependency, in various degrees depending on vendor, that is not fault tolerant or redundant and is ultimately a vulnerability to a preservation network. The raison d'etre of a preservation network is to offer continuous and reliable data protection regardless of mishaps, events, and institutional changes. Accordingly, the preservation network should work to identify all such dependencies and vulnerabilities and create an action plan to mitigate the negative effects of those dependencies. While it is true that every PLN should develop preparedness for network contingencies, actual implementation should be consistent with the PLN's intra-network capacity to operate independently.

5 Node-based Configuration Management

According to the vendor-consumer model, risks associated with network dependency primarily relate to the control of the configuration server and the lack of redundant production-ready configuration documents outside of the control of the vendor. Additionally, in the vendor-consumer administrative model, dependency on the vendor for operational management of the preservation network does not facilitate growth of the intra-network capacity of a PLN to operate independently. One method of reducing the risks associated with the vendor-consumer model is by reducing dependency on the vendor. The first inclination would be to reproduce the configuration server within direct control of a network member; that is, bring up configuration software services on a discrete server within the virtual walls of a member institution. One could argue, however, that this is simply replacing one set of dependencies with another without addressing the vulnerability that could ultimately affect continuous preservation service. Redeploying the configuration server within the direct control of a single institution in a software services environment that is not reproducible across the network creates a network dependency on an institution that is not tolerant to membership changes. Rather than redeploying network-critical configuration services on an isolated server, PLNs should define a configuration model that is redundant, both in technical and knowledge-based infrastructure, is easily brought up, and is easily administered from a common, shared platform — the preservation node.

In node-based configuration management, a preservation node is promoted to the role of configuration services provider. The promoted node provides the configuration data for the network and exposes the configuration data as an archival unit for collection by the network peers. When the configuration data is replicated as an archival unit across all network nodes, every preservation node has the necessary tools to assume the role of configuration services provider. By providing and replicating configuration data across all preservation nodes, LOCKSS software operational management is exposed to all members of the PLN.

5.1 Persistent URL

In LOCKSS preservation networks, the preservation nodes reference the configuration server by URL. In the node-based configuration management model, a promoted preservation node is the configuration server. Rather than hard-coding the direct URL to the preservation node acting as the configuration services provider into the local configuration file on each network node, a persistent configuration URL is defined and registered as a host answer record for the registered domain in DNS. The persistent configuration URL is the static, standard address reference for use by the preservation nodes. The PLN utilizes the DNS record to specify the IP of the promoted node for the persistent configuration URL, and the IP endpoint can change as needed without modification to the URL.

Example Patterns

Persistent Configuration URL: configuration.my-lockss-pln.org
Standard LOCKSS Configuration Reference: http://configuration.my-lockss-pln.org/lockss.xml
DNS Answer Record: configuration.my-lockss-pln.org   A   [NODE STATIC IP]

5.2 Configuration Data Archival Unit

In addition to providing the core LOCKSS configuration data through the known persistent configuration URLs, the node promoted to configuration services provider should also expose real-time software configuration data and back-end sources to the network peers. By exposing sources and configuration data, the PLN can define the exposed data as an archival unit and utilize the LOCKSS preservation software to create redundancy and retrievability of the configuration data across all of the network peers. The base URL for the configuration data archival unit definition is the persistent configuration URL. Collection and preservation of the configuration data archival unit require no modification to the base operation of LOCKSS software.

5.3 Services Transfer

In the event of a service disruption on the promoted node, an alternate preservation node can be promoted to configuration services provider by loading the locally cached configuration data and identifying the new IP endpoint in the DNS record for the persistent configuration URL. If the locally cached configuration data is current, the transfer of configuration services to a new node would be a seamless process. By defining the configuration data archival unit with a base URL of the persistent configuration URL, the established configuration data archival unit requires no modification when configuration services are transferred to a new node [6].

5.4 Network Topology Considerations

It should be emphasized that excising the vendor from the preservation network topology is not a goal of the node-based configuration model. The model is concerned with creating redundant, reproducible fail-safe systems that can ensure preservation services in a continuous manner regardless of events. Redundant systems should include technical and knowledge-based considerations. Through the process of creating an inventory of dependent services and developing an action plan to mitigate those dependencies, the PLN can position itself to move more seamlessly from recovery to production. Vendor cooperation is important for all PLNs and is particularly important for PLNs without the capacity to operate independently.

6 Node-based Configuration Management at ADPNet

In the node-based configuration management model, a preservation node is promoted to the role of configuration services provider. Depending on implementation, a standard set of ancillary software may be added to the promoted preservation node in order to accomplish the software tasks needed to provide the configuration data. In the case of ADPNet's implementation of node-based configuration management, the framework for provisioning the title list was reworked to be database-driven and preservation node-specific [7]. For ADPNet, the ancillary software needed to provide configuration services reduced to a Web server, a database and a programming language executable within the context of the Web server. The only relevant preconditions for software selection were wide availability, easy customization, transportable configuration files, and execution within a relatively small footprint. In order to maximize network reproducibility and minimize down time, each preservation node should, ideally, be pre-equipped, even if latent, with the software infrastructure to assume the role of configuration services provider. The configuration data archival unit exposes the software configuration files, software bring up walk-throughs, and database data needed to reproduce the executing configuration services environment and re-expose configuration sources and data on the newly promoted preservation node. After updating the DNS record to the new IP endpoint, configuration services can continue, and the LOCKSS preservation system can proceed with regular ingest of the configuration data archival unit. The result is a preservation network that can offer continuous service.

7 Security Implications

The current LOCKSS software ingest process for archival units utilizes Web protocols for data transfer. In the node-based configuration management model, system and software operational parameters are exposed over Web protocols. The resulting implications and challenges to security can be mitigated by limiting communication ports to the known peer preservation nodes and establishing good security practices regarding local access on each individual preservation node. Ultimately, the PLN will need to find a balance between data security and systems reproducibility across heterogeneous peers.

8 Conclusion

Utilizing the LOCKSS preservation software to create redundancy and retrievability of configuration data is a good method of creating a fault tolerant preservation network. By configuring the preservation node to provide the configuration data, no additional hardware is needed, and the role can be assumed by any of the network peers. By exposing production-ready configuration data, the network encourages the growth of its own intra-network capacity to operate independently. The goal is long-term, sustainable digital preservation by providing readily available infrastructure and readily available knowledge.

References

[1]	LOCKSS: Basic Concepts.
[2]	Rosenthal, David S. H., Daniel L. Vargas, Tom A. Lipkis and Claire T. Griffin. "Enhancing the LOCKSS Digital Preservation Technology." D-Lib Magazine. 21, no. 9/10, (September-October 2015). http://doi.org/10.1045/september2015-rosenthal
[3]	LOCKSS: Polling and Repair Protocol.
[4]	LOCKSS: Property Server Operations.
[5]	LOCKSS Daemon — TDB.
[6]	LOCKSS Software: File System Deep Dive.
[7]	Cataldo, Tobin M. "Data Partitioning for Private LOCKSS Networks." Presentation at the LOCKSS PLN Community Meeting, Montgomery, AL, October 24-25, 2013.

About the Author

Tobin M. Cataldo is the Information Systems Manager at Birmingham Public Library in Birmingham, Alabama and has been involved with the Alabama Digital Preservation Network since 2011.