Guidelines on the Production and Preservation of Digital Audio Objects (web edition)

printed edition | web edition

TC04 IASA

This is the English language web edition of IASA-TC 04 (Second Edition, 2009), an accepted authority on digital audio preservation in the sound archiving field. It is available in Paper and PDF copies and also available online in other languages.

Recommended citation style:
IASA Technical Committee, Guidelines on the Production and Preservation of Digital Audio Objects, ed. by Kevin Bradley. Second edition 2009. (= Standards, Recommended Practices and Strategies, IASA-TC 04). www.iasa-web.org/tc04/audio-preservation

 

Table of contents

Publication information

IASA Technical Committee

Standards, Recommended Practices and Strategies

Guidelines on the Production and Preservation of Digital Audio Objects

IASA-TC04, Second Edition

 

Edited by Kevin Bradley
 

Contributing authors

Kevin Bradley, National Library of Australia, President IASA and Vice Chair IASA TC; Mike Casey, Indiana University; Stefano S. Cavaglieri, Fonoteca Nazionale Svizzera; Chris Clark, British Library (BL); Matthew Davies, National Film and Sound Archive (NFSA); Jouni Frilander, Finnish Broadcasting Company; Lars Gaustad, National Library of Norway and Chair IASA TC; Ian Gilmour, NFSA; Albrecht Hefner, Sudwestrudfunk, Germany; Franz Lechleitner, Phonogrammarchiv of the Austrian Academy of Sciences (OAW); Guy Marechal, PROSIP; Michel Merten, Memnon; Greg Moss, NFSA; Will Prentice, BL; Dietrich Schuller, OAW; Lloyd Stickells and Nadia Wallaszkovits, OAW.

 

Reviewed by the IASA Technical Committee which included at the time (in addition to those above)
Tommy Sjoberg, Folkmusikens Hus, Sweden; Bruce Gordon, Harvard University; Bronwyn Officer, National Library of New Zealand; Stig L. Molneryd,The National Archive of Recorded Sound and Moving Images, Sweden; George Boston; Drago Kunej, Slovenian Academy of Sciences and Arts; Nigel Bewley, BL; Jean-Marc Fontaine, Laboratoire d’Acoustique Musicale; Chris Lacinak; Gilles St. Laurent, Library and Archives, Canada; and Xavier Sene, Bibliotheque Nationale de France.

 

Published by the International Association of Sound and Audio Visual Archives
c/o Secretary-General:
Ilse Assmann
Media Libraries
South African Broadcasting Corporation
PO Box 931, 2006 Auckland Park
South Africa

1st Edition Published 2004
2nd Edition Published 2009
IASA-TC04 Guidelines in the Production and Preservation of Digital Audio Objects: standards, recommended practices, and strategies: 2nd edition/ edited by Kevin Bradley

This publication provides guidance to audiovisual archivists on a professional approach to the production and preservation of digital audio objects

Includes bibliographical references and index
ISBN 978-91-976192-2-6

Copyright: International Association of Sound and Audio Visual Archives (IASA) 2009

Translation is not permitted without the consent of the IASA Executive Board and may only be undertaken in accordance with the Guidelines & Policy Statement,Translation of Publications Guidelines, Guidelines for the Translation of IASA Publications & Workflow for Translations
www.iasa-web.org/guidelines-translating-iasa-publications

This publication is approved by the Sub-Committee on Technology of the Memory of the World Programme of UNESCO

 

Translation sponsor:

 

 

 

Gold sponsor:

 

 

 

Silver sponsor:

 

 

 

Bronze sponsors:

 

Preface to the Second Edition

The process of debating the principles which underpin the work of sound preservation, and then discussing, codifying and documenting the practices that we as professional sound archivists use and recommend, is to necessarily identify the strengths and weaknesses in our everyday work.When the first version/edition of TC-04 Guidelines in the Production and Preservation of Digital Audio Objects was completed and printed in 2004 there was, in spite of our pride in that previous publication, little doubt amongst the IASA Technical Committee that a second edition would be necessary to address those areas that we knew we would need to be working on. In the intervening four years we as a committee have grown, extending our knowledge and expertise in many areas, and helped to develop the standards and systems which implement sustainable work and preservation practices. This second edition is the beneficiary of that work, and it contains much that is vital in the evolving field of sustainable sound preservation by digital means.

Though we have incorporated much new information, and refined some of the fundamental chapters, the advice provided in the second edition does not stand in opposition to that provided in the first. The IASA-TC 04 Guidelines in the Production and Preservation of Digital Audio Objects is informed by IASA-TC 03 The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy. A revised version of TC 03 was published in 2006 which took account of new developments in digital audio archiving, and of the much more practical role of TC 04. TC 03 2006 concentrates on the principles and supersedes the earlier versions, and these guidelines are the practical embodiment of the principles.

The major amendments in this second edition of TC 04 can be found in the chapters dealing with the digital repositories and architectures. Chapter 3, Metadata, has been extensively enlarged and provides significant and detailed advice on approaches to the management of data and metadata for the purposes of preservation, reformatting, analysis, discovery and use. The chapter ranges widely across the subject area from schemas through to structures to manage and exchange the content and considers the major building blocks of data dictionaries, schemas, ontologies, and encodings. The sibling to this is Chapter 4, Unique and Persistent identifiers, and it provides guidance on naming and numbering of files and digital works.

The new section included as Chapter 6 Preservation Target Formats and Systems, is structured around the functional categories identified in the Reference Model for an Open Archival Information System (OAIS): Ingest,Archival Storage, Preservation Planning,Administration and Data Management, and Access, and each of the subsequent Sections deals with each subject area. The use of this conceptual model in organising the book has two important consequences: Firstly, it uses the same functional categories as the architectural design of the major repository and data management systems, which means it has real world relevance. Secondly, identifying separate and abstracted components of a digital preservation strategy allows the archivist to make decisions about various parts of the process, rather than trying to solve and implement the monolithic whole. Chapter 9, Partnerships, Project Planning and Resources, is an entirely new chapter, and provides advice on the issues to consider if a collection manager decides to outsource all or part of the processes involved in the preservation of the audio collections.

Chapter 7, Small Scale Approaches to Digital Storage Systems, considers how to build a low cost digital management system which, while limited in scope, still adheres to the principles and quality measures identified in chapter 6.

Chapter 8 revisits the risks associated with optical disc storage and makes recommendations as to their management, while suggesting the advice in chapters 6 and 7 may be more useful in the long term management of digital content.

Chapter 5, Signal Extraction from Original Carriers, was one of the most practical and informed components of the first edition, and it remains a source of practical knowledge, and information on standards and advice. As part of the review process the chapters on signal extraction have been refined, and extra useful advice has been added. An extra section, 5.7 Field Recording Technologies and Archival Approaches, has been added, and it addresses the question of how to create a sound recording in the field for which the content is intended for long term archival storage.

Chapter 2, Key Digital Principles, adheres to the same standards expressed in the earlier edition. There is, however, more explanatory detail, and technical information, particularly regarding the digital conversion processes, has been provided in more precise and standard language.

TC 04 represents a considerable effort and commitment from the IASA Technical Committee, not only those who created the original text, but also those who reviewed and analysed the chapters until we reached a satisfactory explanation. To my friends and colleagues in the TC goes my respect for the detailed knowledge and gratitude for their generosity in sharing it. The quality of this new edition is a testament to their expertise.

Kevin Bradley
Editor
November 2008

Introduction to the First Edition

Digital audio has, over the past few years, reached a level of development that makes it both effective and affordable for use in the preservation of audio collections of every magnitude. The integration of audio into data systems, the development of appropriate standards, and the wide acceptance of digital audio delivery mechanisms have replaced all other media to such an extent that there is little choice for sound preservation except digital storage approaches. Digital technology offers the potential to provide an approach that addresses many of the concerns of the archiving community through lossless cloning of audio data through time. However, the processes of converting analogue audio to digital, transferring to storage systems, of managing and maintaining the audio data, providing access and ensuring the integrity of the stored information, present a new range of risks that must be managed to ensure that the benefits of digital preservation and archiving are realised. Failure to manage these risks appropriately may result in significant loss of data, value and even audio content.

This publication of the Technical Committee of the International Association of Sound and Audiovisual Archives (IASA) "Guidelines on the Production and Preservation of Digital Audio Objects" is intended to provide guidance to audiovisual archivists on a professional approach to the production and preservation of digital audio objects. It is the practical outcome of the previous IASA Technical Committee paper, IASA-TC 03 "The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy, Version 2, September 2001". The Guidelines addresses the production of digital copies from analogue originals for the purposes of preservation, the transfer of digital originals to storage systems, as well as the recording of original material in digital form intended for long-term archival storage. Any process of digitisation is selective, the audio content itself supplies more information to potential users than is contained in the intended signal and the standards of analogue to digital conversion fix the limits of the resolution of the audio permanently and, unless carefully undertaken, partially.

There are three main parts to the content of the Guidelines:

  • Standards, Principles and Metadata
  • Signal Extraction from Originals
  • Target Formats

Standards, Principles and Metadata: Of the four basic tasks that are performed by all archives - acquisition, documentation, access, preservation, the primary task is to preserve the information placed in the care of the collection (IASA-TC 03, 2001). However, if the tasks of acquisition and documentation are undertaken in combination with a well planned digital preservation strategy that adheres to adequate standards, the task of providing access is facilitated by the process. Long term access is a product of appropriate preservation.

Adherence to widely used and accepted standards suitable for the preservation of digital audio is a fundamental necessity of audio preservation. The IASA Guidelines recommend linear PCM (pulse code modulation) (interleaved for stereo) in a .wav or preferably BWF .wav file (EBU Tech 3285) for all two track audio. The use of any perceptual coding (“lossy compression") is strongly discouraged. It is recommended that all audio be digitised at 48 kHz or higher, and with a bit depth of at least 24 bit. Analogue to Digital (A/D) conversion is a precision process, and low cost converters integrated into computer sound cards cannot meet the demands of archival preservation programs.

Once encoded as a data file, the preservation of the audio faces many of the same issues as those of all digital data, and foremost in managing these is the assigning of a unique Persistent Identifier (PI) and providing appropriate metadata. Metadata is not just the descriptive information that allows the user or archive to identify the content, but also includes the technical information that enables the recognition and replaying of the audio, and the preservation metadata that retains information about the processes that went to generate the audio file. It is only thus that the integrity of the audio content can be guaranteed. The digital archive will depend on comprehensive metadata to maintain its collection. A well planned digital archive will automate the production of much of the metadata and should include the original carrier, its format and state of preservation, replay equipment and parameters, the digital resolution, format, all equipment used, the operators involved in the process and any processes or procedures undertaken.

Signal Extraction from Originals: “Sound archives have to ensure that, in the replay process, the recorded signals can be retrieved to the same, or a better, fidelity standard as was possible when they were recorded... (also) carriers are the bearers of information: desired or primary information, consisting of the intended sonic content, and ancillary or secondary information which may take manifold forms. Both primary and secondary information form part of the Audio Heritage.” (IASA-TC 03, 2001).

To take full advantage of the potential offered by digital audio it is necessary to adhere to the above principles and ensure that the replay of the audio original is made with a full awareness of all the possible issues. This requires knowledge of the historic audio technologies and a technical awareness of the advances in replay technology.Where appropriate, the Guidelines provide advice on the replay of historical mechanical and other obsolete formats, including, cylinders and coarse groove discs, steel wire and office dictation systems, vinyl LP records, analogue magnetic tape, cassette and reel, magnetic digital carriers such as DAT and its video tape based predecessors, and optical disk media such as CD and DVD. For each of the formats there is advice on selection of best copy, cleaning, carrier restoration, replay equipment, speed and replay equalisation, corrections for errors caused by misaligned recording equipment and removal of storage related signal artefacts and the time required to undertake a transfer to digital. All of these are important topics which are addressed in the Guidelines with a consideration of the associated ethical issues, though the latter issue is particularly significant as many digitisation plans fail to budget for the considerable time constraints of an audio transfer process.

All the parameters mentioned above must be determined objectively, and appropriate records kept of every process. Digital storage and associated technologies and standards enable an ethical approach to sound archiving by enabling the production of documentation and its storage in linked, related metadata fields.

Target Formats: Data can be stored in many ways and on many carriers and the appropriate type of technology will be dependant on the circumstances of the institution and its collection. The Guidelines provide advice and information on various suitable approaches and technologies including Digital Mass Storage Systems (DMSS) and small scale manual approaches to digital storage systems, data tape, hard disks, optical disks including CD and DVD recordable and magneto optical disks (MO).

No target format is a permanent solution to the issue of digital audio preservation, and no technological development will ever provide the ultimate solution; rather they are a step in a process whereby institutions will be responsible for maintaining data through technological changes and developments, migrating data from the current system to next for as long as the data remains valuable. The DMSS with suitable management software are the most appropriate for the long term maintenance of audio data.”Such systems permit automatic checking of data integrity, refreshment, and, finally, migration with a minimum use of human resources” (IASA-TC 03, 2001). These systems can be scaled to suit smaller archives, though this will most often result in an increased responsibility for manual data checking. Discrete storage formats such as CD and DVD recordable, and magneto optical disks (MO) are inherently less reliable. The Guidelines suggest standards and approaches to maintaining the data on these carriers, while recommending that the more reliable solutions found in integrated storage systems are to be substantially preferred.

1: Background

1.1 Audiovisual archives hold a responsibility for the preservation of cultural heritage covering all spheres of musical, artistic, sacred, scientific, linguistic and communications activity, reflecting public and private life, and the natural environment, held as published and un-published recorded sound and image.

1.2 The aim of preservation is to provide our successors and their clients with as much of the information contained in our holdings as it is possible to achieve in our professional working environment. It is the responsibility of an archive to assess the needs of its users, both current and future, and to balance those needs against the conditions and resources of the archive. The ultimate purpose of preservation is to ensure that access to the audio content of collection is available to approved users, current and future, without undue threat or damage to the audio item.

1.3 As the lifespan of all audio carriers is limited by their physical and chemical stability, as well as the availability of the reproduction technology and, as the reproduction technology itself may be a potential source of damage for many audio carriers, audio preservation has always required the production of copies that can stand for the original as preservation duplicates, which in the parlance of digital archiving have come to be known as “preservation surrogates”. The need to migrate content to another storage system applies to carriers of digital audio originals perhaps even more so as they may be endangered by the ever shorter lifetimes of highly sophisticated hardware and related software in the market, which, sometimes only a few years after their market introduction, will lead to the total obsolescence of replay equipment. However, the same constraints that apply to the original item will wholly, or in part, apply to the preservation target format, requiring continued reduplication. If preservation had continued by serial duplication in the analogue domain this would have produced a degradation of the audio signal with each subsequent generation.

1.4 The potential offered by the production of digital surrogates for the purpose of preservation seems to provide an answer to linked issues of preservation and access. However, the decisions made about digital formats, resolutions, carriers and technology systems will impose limits on the effectiveness of digital preservation that cannot be reversed, as will the quality of audio being encoded. Optimal signal extraction from original carriers is the indispensable starting point of each digitisation process. As recording media very often requires very specific replay technology, timely organisation of copying into the digital domain must take place, before obsolescence of hardware becomes critical.

1.5 The ability to recopy the captured digital copy without further loss or degradation has often led enthusiastic archivists to describe it as “eternal preservation”. The easy production of low bit-rate distribution copies broadens the ability of archives to provide access to their collections without endangering the original item. However, far from being eternal, poorly managed digital archiving practices may lead to a reduction in the effective lifespan and integrity of audio content, whereas a well managed digital conversion and preservation strategy will facilitate the realisation of the benefits promised by digital technology. Similarly, a poorly planned system requiring manual intervention may present a management task of considerable dimension that could be beyond the capabilities of the collection managers and curators and so endanger the collection. A well planned system should enable automation of the processes and so preservation can proceed in a timely manner. No system for preserving sound will provide a one-off solution; any preservation solution will require future transfers and migrations that must be planned for when the material is first digitised and stored.

1.6 The Guidelines address audio carriers such as cylinders and coarse groove discs, steel wire and office dictation systems, vinyl LP records, analogue magnetic tape, cassette and reel, magnetic digital carriers such as DAT and its video tape based predecessors, and optical disk media such as CD and DVD. Though many of the principles contained herein will be applicable, sound for film is not specifically addressed. This document does not consider piano rolls, MIDI files or other systems which are player directions rather than encoded audio. The following principles outline the areas in which critical decisions must be made in the transfer to and management of digital audio materials.

2: Key Digital Principles

2.1 Standards: It is integral to the preservation of audio that the formats, resolutions, carrier and technology systems selected adhere to internationally agreed standards appropriate to the intended archival purposes. Non-standard formats, resolutions and versions may not in the future be included in the preservation pathways that will enable long term access and future format migration.

2.2 Sampling Rate: The sampling rate fixes the maximum limit on frequency response.When producing digital copies of analogue material IASA recommends a minimum sampling rate of 48 kHz for any material. However, higher sampling rates are readily available and may be advantageous for many content types. Although the higher sampling rates encode audio outside of the human hearing range, the net effect of higher sampling rate and conversion technology improves the audio quality within the ideal range of human hearing. The unintended and undesirable artefacts in a recording are also part of the sound document, whether they were inherent in the manufacture of the recording or have been subsequently added to the original signal by wear, mishandling or poor storage. Both must be preserved with utmost accuracy. For certain signals and some types of noise, sampling rates in excess of 48 kHz may be advantageous. IASA recommends 96 kHz as a higher sampling rate, though this is intended only as a guide, not an upper limit; however, for most general audio materials the sampling rates described should be adequate. For audio digital-original items, the sampling rate of the storage technology should equal that of the original item.

2.3 Bit Depth: The bit depth fixes the dynamic range of an encoded audio event or item. 24 bit audio theoretically encodes a dynamic range that approaches physical limits of listening, though in reality the technical limits of the system is slightly less. 16 bit audio, the CD standard, may be inadequate to capture the dynamic range of many types of material, especially where high level transients are encoded such as the transfer of damaged discs. IASA recommends an encoding rate of at least 24 bit to capture all analogue materials. For audio digital-original items, the bit depth of the storage technology should at least equal that of the original item. It is important that care is taken in recording to ensure that the transfer process takes advantage of the full dynamic range.

2.4 Analogue to Digital Converters (A/D)

2.4.1 In converting analogue audio to a digital data stream, the A/D should not colour the audio or add any extra noise. It is the most critical component in the digital preservation pathway. In practice, the A/D converter incorporated in a computer’s sound card can not meet the specifications required due to low cost circuitry and the inherent electrical noise in a computer. IASA recommends the use of discrete (stand alone) A/D converters connected via an AES/EBU or S/PDIF interface, IEEE1394 bus-connected (firewire) discrete A/D converters or USB serial interface-connected discrete A/D converters that will convert audio from analogue to digital in accordance with the following specification. All specifications are measured at the digital output of the A/D converter, and are in accordance with Audio Engineering Society standard AES 17-1998 (r2004), IEC 61606-3, and associated standards as identified.

2.4.1.1 Total Harmonic Distortion + Noise (THD+N)
With signal 997 Hz at -1 dB FS, the A/D converter THD+N will be less than -105 dB unweighted, -107 dB A-weighted, 20 Hz to 20 kHz bandwidth limited.
With signal 997 Hz at -20 dB FS, the A/D converter THD+N will be less than -95 dB unweighted, -97 dB A-weighted, 20 Hz to 20 kHz bandwidth limited.

2.4.1.2. Dynamic Range (Signal to Noise)
The A/D converter will have a dynamic range of not less than 115 dB unweighted, 117 dB A-weighted. (Measured as THD+N relative to 0 dB FS, bandwidth limited 20 Hz to 20 kHz, stimulus signal 997 Hz at -60 dB FS).

2.4.1.3. Frequency Response
For an A/D sampling frequency of 48 kHz, the measured frequency response will be better than ± 0.1 dB for the range 20 Hz to 20 kHz.
For an A/D sampling frequency of 96 kHz, the measured frequency response will be better than ± 0.1 dB for the range 20Hz to 20 kHz, and ± 0.3 dB for the range 20 kHz to 40 kHz.
For an A/D sampling frequency of 192 kHz, the frequency response will be better than ± 0.1 dB for the range 20Hz to 20 kHz, and ± 0.3 dB from 20 kHz to 50 kHz (reference audio signal = 997 Hz, amplitude -20 dB FS).

2.4.1.4 Intermodulation Distortion IMD (SMPTE/DIN/AES17)
The A/D converter IMD will not exceed -90 dB. (AES17/SMPTE/DIN twin-tone test sequences, combined tones equivalent to a single sine wave at full scale amplitude).

2.4.1.5 Amplitude Linearity
The A/D converter will exhibit amplitude gain linearity of ± 0.5 dB within the range -120 dB FS to 0 dB FS. (997 Hz sinusoidal stimuli).

2.4.1.6 Spurious Aharmonic Signals
Better than -130 dB FS with stimulus signal 997 Hz at -1 dBFS

2.4.1.7 Internal Sample Clock Accuracy
For an A/D converter synchronised to its internal sample clock, frequency accuracy of the clock measured at the digital stream output will be better than ±25 ppm.

2.4.1.8 Jitter
Interface jitter measured at A/D output <5ns.

2.4.1.9 External Synchronisation
Where the A/D converter sample clock will be synchronised to an external reference signal, the A/D converter must react transparently to incoming sample rate variations ± 0.2% of the nominal sample rate. The external synchronistation circuit must reject incoming jitter so that the synchronised sample rate clock is free from artefacts and disturbances.

2.4.2 IEE1394 and USB Audio Interfaces. Many A/D converters now provide the facilities to directly interface to a host computer via the high speed IEEE1394 (firewire) and USB 2.0 serial interfaces. Both systems are successfully implemented as audio transmission interfaces across the major personal computer platforms, and can reduce the requirement to install a specialised, high-quality soundcard interface in the computer chassis. Audio quality is generally independent of the bus technology in use.

2.4.3 Selection of A/D Converters: The A/D converter is the most critical piece of technology in the digital preservation pathway.When choosing a convertor, and before any further evaluation is undertaken, IASA recommends that all specifications are tested against the reference standards described above. Any converter which does not meet the basic IASA technical specifications will produce less than accurate conversions. In conjunction with technical evaluation, statistically valid blind listening tests should be carried out on short listed converters to determine overall suitability and performance. All the specifications and testing described above are stringent and complex, and these specifications are highly important in selecting and evaluating analogue to digital convertors. The published specifications from the equipment manufacturers are sometimes challenging to compare, often incomplete and occasionally difficult to reconcile with the performance of the device they purport to represent. It may suit certain communities or groups to undertake group or panel testing to maximise resources. Certain institutions, such as state archives, libraries or academic science departments may be in a position to assist with testing.

2.5 Sound Cards: The sound card used in a computer for the purposes of audio preservation should have a reliable digital input with a high quality digital audio stream synchronisation mechanism, and pass a digital audio data stream without change or alteration. As a discrete (stand alone) A/D converter must be used, the primary purpose of a sound card in audio preservation is in passing a digital signal to the computer data bus, though it may also be used for returning the converted signal to analogue for monitoring purposes. Care should be taken in choosing a card that accepts the appropriate sampling and bit rates, and does not inject noise or other extraneous artefacts. IASA recommends the use of a high quality sound card that meets the following specification:

2.5.1 Sample rate support: 32 kHz to 192 kHz +/- 5%.

2.5.2 Digital audio quantisation: 16-24 bits.

2.5.3 Varispeed: automatic by incoming audio or wordclock.

2.5.4 Synchronisation: internal clock, wordclock, digital audio input.

2.5.5 Audio interface: high speed AES/EBU conforming to AES3 specifications.

2.5.6 Jitter acceptance and signal recovery on inputs up to 100ns without error.

2.5.7 Digital audio subcode pass-through.

2.5.8 Optional timecode inputs.

2.6 Computer Based Systems and Processing Software: Recent generations of computers have sufficient power to manipulate large audio files. Once in the digital domain, the integrity of the audio files should be maintained. As noted above the critical points in the preservation process are converting the analogue audio to digital (which relies on the A/D converter), and entering the data into the system, either through the sound card or other data port. However, some systems truncate the word length of an item in order to process it, resulting in a lower effective bit rate and others may only process compressed file formats, such as MP3, neither of which is acceptable. IASA recommends that a professional audio computer based system be used whose processing word length exceeds that of the file (i.e. greater than 24 bit) and which does not alter the file format.

2.7 Data Reduction: It has become generally accepted in audio archiving that when selecting a digital target format, formats employing data reduction (frequently mistakenly called data “compression”) based on perceptual coding (lossy codecs) must not be used. Transfers employing such data reduction mean that parts of the primary information are irretrievably lost. The results of such data reduction may sound identical or very similar to the unreduced (linear) signal, at least for the first generation, but the further use of the data reduced signal will be severely restricted and its archival integrity has been compromised.

2.8 File Formats

2.8.1 There are a number of linear audio file formats that may be used to encode audio, however, the wider the acceptance and use of the format in a professional audio environment, the greater the likelihood of long term acceptance of the format, and the greater the probability of professional tools being developed to migrate the format to future file formats when that becomes necessary. Because of the simplicity and ubiquity of linear Pulse Code Modulation (PCM) [interleaved for stereo] IASA recommends the use of WAVE, (file extension .wav) developed by Microsoft and IBM as an extension from the Resource Interchange File Format (RIFF). Wave files are widely used in the professional audio industry.

2.8.2 BWF .wav files [EBU Tech 3285] are an extension of .wav and are supported by most recent audio technology. The benefit of BWF for both archiving and production uses is that metadata can be incorporated into the headers which are part of the file. In most basic exchange and archiving scenarios this is advantageous; however, the fixed nature of the embedded information may become a liability in large and sophisticated data management systems (see discussion chapter 3 Metadata and Ch 7 Small Scale Approaches to Digital Storage Systems). This, and other limitations with BWF, can be managed by using only a minimal set of data within BWF and maintaining other data with external data management systems. AES31-2-2006, the AES standard on “Network and file transfer of audio - Audio-file transfer and exchange - File format for transferring digital audio data between systems of different type and manufacture” is largely compatible with the standard set in BWF, and its is expected that future development in the area will continue to make the format viable. The BWF format is widely accepted by the archiving community and with the limitations described in mind IASA recommends the use of BWF .wav files [EBU Tech 3285] for archival purposes.

2.8.3 Multitrack audio and film or video soundtracks, or large audio files, may use RF64 [EBU Tech 3306], which is compatible with BWF,AES-31 or as a wav file in an Media Exchange Format (MXF) wrapper. As these are all still under development, one pragmatic approach may be to create multiple time coherent mono BWF files wrapped in the tar (tape archive) format.

2.9 Audio Path: The combination of reproduction equipment, signal cables, mixers and other audio processing equipment should have specifications that equal or exceed that of digital audio at the specified sampling rate and bit depth. The replay equipment, audio path, target format and standards must exceed that of the original carrier.

3: Metadata

3.1 Introduction

 3.1.1 Metadata is structured data that provides intelligence in support of more efficient operations on resources, such as preservation, reformatting, analysis, discovery and use. It operates at its best in a networked environment, but is still a necessity in any digital storage and preservation environment. Metadata instructs end-users (people and computerised programmes) about how the data are to be interpreted. Metadata is vital to the understanding, coherence and successful functioning of each and every encounter with the archived object at any point in its lifecycle and with any objects associated with or derived from it.

3.1.2 It will be helpful to think about metadata in functional terms as “schematized statements about resources: schematized because machine understandable, [as well as human readable]; statements because they involve a claim about the resource by a particular agent; resource because any identifiable object may have metadata associated with it” (Dempsey 2005). Such schematized (or encoded) statements (also referred to as metadata ‘instances’) may be very simple, a single Uniform Resource Identifier (URI), within a single pair of angle brackets < > as a container or wrapper and a namespace. Typically they may become highly evolved and modular, comprising many containers within containers, wrappers within wrappers, each drawing on a range of namespace schemas, and assembled at different stages of a workflow and over an extended period of time. It would be most unusual for one person to create in one session a definitive, complete metadata instance for any given digital object that stands for all time.

3.1.3 Regardless of how many versions of an audio file may be created over time, all significant properties of the file that has archival status must remain unchanged. This same principle applies to any metadata embedded in the object (see section 3.1.4 below). However, data about any object are changeable over time: new information is discovered, opinions and terminology change, contributors die and rights expire or are re-negotiated. It is therefore often advisable to keep audio files and all or some metadata files separate, establish appropriate links between them, and update the metadata as information and resources become available. Editing the metadata within a file is possible, though cumbersome, and does not scale up as an appropriate approach for larger collections. Consequently, the extent to which data is embedded in the files as well as in separate data management system will be determined by the size of the collection, the sophistication of the particular data management system, and the capabilities of the archive personnel.

3.1.4 Metadata may be integrated with the audio files and is in fact suggested as an acceptable solution for a small scale approach to digital storage systems (see section 7.4 Basic Metadata). The Broadcast Wave Format (BWF) standardized by the European Broadcasting Union (EBU), is an example of such audio metadata integration, which allows the storage of a limited number of descriptive data within the .wav file (see section 2.8 File Formats). One advantage of storing the metadata within the file is that it removes the risk of losing the link between metadata and the digital audio. The BWF format supports the acquisition of process metadata and many of the tools associated with that format can acquire the data and populate the appropriate part of the BEXT (broadcast extension) chunk.The metadata might therefore include coding history, which is loosely defined in the BWF standard, and allows the documentation of the processes that lead to the creation of the digital audio object. This shares similarities with the event entity in PREMIS (see 3.5.2 ,3.7.3 and Fig.1 ).When digitizing from analogue sources the BEXT chunk can also be used to store qualitative information about the audio content. When creating a digital object from a digital source, such as DAT or CD, the BEXT chunk can be used to record errors that might have occurred in the encoding process.

A=<ANALOGUE> Information about the analogue sound signal path
A=<PCM> Information about the digital sound signal path
F=<48000, 441000, etc.> Sampling frequency [Hz]
W=<16, 18, 20, 22, 24, etc.> Word length [bits]
M=<mono, stereo, 2-channel> Mode
T=<free ASCII-text string> Text for comments
Coding History Field: BWF (http://www.ebu.ch/CMSimages/en/tec_text_r98-1999_tcm6-4709.pdf) ,>

A=ANALOGUE, M=Stereo,T=Studer A820;SN1345;19.05;Reel;AMPEX 406
A=PCM, F=48000,W=24, M=Stereo,T=Apogee PSX-100;SN1516;RME DIGI96/8 Pro
A=PCM, F=48000,W=24, M=Stereo,T=WAV A=PCM, F=48000,W=24, M=stereo,T=2006-02-20 File Parser brand name
A=PCM, F=48000,W=24, M=stereo,T=File Converter brand name 2006 -02-20; 08:10:02

Fig. 1 National Library of Australia’s interpretation of the coding history of an original reel converted to BWF using database and automated systems.

3.1.5 The Library of Congress has been working on formalising and expanding the various data chunks in the BWF file. Embedded Metadata and Identifiers for Digital Audio Files and Objects: Recommendations for WAVE and BWF Files Today is their latest draft available for comment at http://home.comcast.net/~cfle/AVdocs/Embed_Audio_081031.doc. AES X098C is another development in the documentation of process and digital provenance metadata.

3.1.6 There are however, many advantages to maintaining metadata and content separately, by employing, for instance a framework standard such as METS (Metadata Encoding and Transmission Standard see section 3.8 Structural Metadata – METS). Updating, maintaining and correcting metadata is much simpler in a separate metadata repository. Expanding the metadata fields so as to incorporate new requirements or information is only possible in an extensible, and separate, system, and creating a variety of new ways of sharing the information requires a separate repository to create metadata the can be used by those systems. For larger collections the burden of maintaining metadata only in the headers of the BWF file would be unsustainable. MPEG-7 requires that audio content and descriptive metadata are separated, though descriptions can be multiplexed with the content as alternating data segments.

3.1.7 It is of course possible to wrap a BWF file with a much more informed metadata, and providing the information kept in BWF is fixed and limited, this approach has the advantage of both approaches. Another example of integration is the tag metadata that needs to be present in access files so that a user may verify that the object downloaded or being streamed is the object that was sought and selected. ID3, the tag used in MP3 files to describe content information which is readily interpreted by most players, allows a minimum set of descriptive metadata. And METS itself has been investigated as a possible container for packaging metadata and content together, though the potential size of such documents suggests this may not be a viable option to pursue.

3.1.8 A general solution for separating the metadata from the contents (possibly with redundancy if the contents includes some metadata) is emerging from work being undertaken in several universities in liaison with major industrial suppliers such as SUN Microsystems, Hewlett-Packard and IBM. The concept is to always store the representation of one resource as two bundled files: one including the ‘contents’ and the other including the metadata associated to that content. The second file includes:

3.1.8.1 The list of identifiers according to all the involved rationales. It is in fact a series of “aliases” pertaining to the URN and the local representation of the resource (URL).

3.1.8.2 The technical metadata (bits per sample / sampling rate; accurate format definition; possibly the associated ontology).

3.1.8.3 The factual metadata (GPS coordinates / Universal time code / Serial number of the equipment / Operator / ...).

3.1.8.4 The semantic metadata.

3.1.9 In summary, most systems will adopt a practical approach that allows metadata to be both embedded within files and maintained separately, establishing priorities (i.e. which is the primary source of information) and protocols (rules for maintaining the data) to ensure the integrity of the resource.

3.2 Production

3.2.1 The rest of this chapter assumes that in most cases the audio files and the metadata files will be created and managed separately. In which case, metadata production involves logistics – moving information, materials and services through a network cost-effectively. However, a small scale collection, or an archive in earlier stages of development, may find advantages in embedding metadata in BWF and selectively populating a subset of the information described below. If done carefully, and with due understanding of the standards and schemas discussed in this chapter, such an approach is sustainable and will be migrate- able to a fully implemented system as described below. Though a decision can be made by an archive to embed all or some metadata within the file headers, or to manage only some data separately, the information within this chapter will still inform this approach. (See also Chapter 7 Small Scale Approaches to Digital Storage Systems).

3.2.2 Until recently the producers of information about recordings either worked in a cataloguing team or in a technical team and their outputs seldom converged. Networked spaces blur historic demarcations. Needless to say, the embodiment of logistics in a successful workflow also requires the involvement of people who understand the workings and connectivity of networked spaces. Metadata production therefore involves close collaboration between audio technicians, Information Technology (IT) and subject specialists. It also requires attentive management working to a clearly stated strategy that can ensure workflows are sustainable and adaptable to the fast-evolving technologies and applications associated with metadata production.

3.2.3 Metadata is like interest - it accrues over time. If thorough, consistent metadata has been created, it is possible to predict this asset being used in an almost infinite number of new ways to meet the needs of many types of user, for multi-versioning, and for data mining. But the resources and intellectual and technical design issues involved in metadata development and management are not trivial. For example, some key issues that must be addressed by managers of any metadata system include:

3.2.3.1 Identifying which metadata schema or extension schemas should be applied in order to best meet the needs of the production teams, the repository itself and the users;
3.2.3.2 Deciding which aspects of metadata are essential for what they wish to achieve, and how granular they need each type of metadata to be. As metadata is produced for the long-term there will likely always be a trade-off between the costs of developing and managing metadata to meet current needs, and creating sufficient metadata that will serve future, perhaps unanticipated demands;
3.2.3.3 Ensuring that the metadata schemas being applied are the most current versions.
3.2.3.4 Interoperability is another factor; in the digital age, no archive is an island. In order to send content to another archive or agency successfully, there will need to be commonality of structure and syntax. This is the principle behind METS and BWF.

3.2.4 A measure of complexity is to be expected in a networked environment where responsibility for the successful management of data files is shared. Such complexity is only unmanageable, however, if we cling to old ways of working that evolved in the early days of computers in libraries and archives –before the Web and XML. As Richard Feynman said of his own discipline, physics, 'you cannot expect old designs to work in new circumstances'. A new general set of system requirements and a measure of cultural change are needed. These in turn will permit viable metadata infrastructures to evolve for audiovisual archives.

3.3 Infrastructure

 3.3.1 We do not need a ‘discographic’ metadata standard: a domain-specific solution will be an unworkable constraint.We need a metadata infrastructure that has a number of core components shared with other domains, each of which may allow local variations (e.g. in the form of extension schema) that are applicable to the work of any particular audiovisual archive. Here are some of the essential qualities that will help to define the structural and functional requirements:

3.3.1.1 Versatility: For the metadata itself, the system must be capable of ingesting, merging, indexing, enhancing, and presenting to the user, metadata from a variety of sources describing a variety of objects, It must also be able to define logical and physical structures, where the logical structure represents intellectual entities, such as collections and works, while the physical structure represents the physical media (or carriers) which constitute the source for the digitized objects. The system must not be tied to one particular metadata schema: it must be possible to mix schema in application profiles (see 3.9.8) suited to the archive’s particular needs though without compromising interoperability. The challenge is to build a system that can accommodate such diversity without needless complication for low threshold users, nor prevent more complex activities for those requiring more room for manoeuvre.

3.3.1.2 Extensibility: Able to accommodate a broad range of subjects, document types (e.g. image and text files) and business entities (e.g. user authentication, usage licenses, acquisition policies, etc.). Allow for extensions to be developed and applied or ignored altogether without breaking the whole, in other words be hospitable to experimentation: implementing metadata solutions remains an immature science.

3.3.1.3 Sustainability: Capable of migration, cost-effective to maintain, usable, relevant and fit for purpose over time.

3.3.1.4 Modularity: The systems used to create or ingest metadata, and merge, index and export it should be modular in nature so that it is possible to replace a component that performs a specific function with a different component, without breaking the whole.

3.3.1.5 Granularity: Metadata must be of a sufficient granularity to support all intended uses. Metadata can easily be insufficiently granular, while it would be the rare case where metadata would be too granular to support a given purpose.

3.3.1.6 Liquidity: Write once, use many times. Liquidity will make digital objects and representations of those objects self-documenting across time, the metadata will work harder for the archive in many networked spaces and provide high returns for the original investments of time and money.

3.3.1.7 Openness and transparency: Supports interoperability with other systems. To facilitate requirements such as extensibility, the standards, protocols, and software incorporated should be as open and transparent as possible.

3.3.1.8 Relational (hierarchy/sequence/provenance): Must express parent- child relationships, correct sequencing, e.g. the scenes of a dramatic performance, and derivation. For digitized items, be able to support accurate mappings and instantiations of original carriers and their intellectual content to files. This helps ensure the authenticity of the archived object (Tennant 2004).

3.3.2 This recipe for diversity is itself a form of openness. If an open W3C (World Wide Web Consortium) standard, such as Extensible Markup Language (XML), a widely adopted mark-up language, is selected then this will not prevent particular implementations from including a mixture of standards such as Material Exchange Format (MXF) and Microsoft’s Advanced Authoring Format (AAF) interchange formats.

3.3.3 Although MXF is an open standard, in practice the inclusion of metadata in the MXF is commonly made in a proprietary way. MXF has further advantages for the broadcast industry because it can be used to professionally stream content whereas other wrappers only support downloading the complete file. The use of MXF for wrapping contents and metadata would only be acceptable for archiving after the replacement of any metadata represented in proprietary formats by open metadata formats.

3.3.4 So much has been written and said about XML that it would be easy to regard it as a panacea. XML is not a solution in itself but a way of approaching content organisation and re-use, its immense power harnessed through combining it with an impressive array of associated tools and technologies that continue to be developed in the interests of economical re-use and repurposing of data. As such, XML has become the de-facto standard for representing metadata descriptions of resources on the Internet. A decade of euphoria about XML is now matched by the means to handle it thanks to the development of many open source and commercial XML editing tools (See 3.6.2).

3.3.5 Although reference is made in this chapter to specific metadata formats that are in use today, or that promise to be useful in the future, these are not meant to be prescriptive. By observing those key qualities in section 3.3.1 and maintaining explicit, comprehensive and discrete records of all technical details, data creation and policy changes, including dates and responsibility, future migrations and translations will not require substantial changes to the underlying infrastructure. A robust metadata infrastructure should be able to accommodate new metadata formats by creating or applying tools specific to that format, such as crosswalks, or algorithms for translating metadata from one encoding scheme to another in an effective and accurate manner. A number of crosswalks already exist for formats such as MARC, MODS, MPEG-7 Path, SMPTE and Dublin Core. Besides using crosswalks to move metadata from one format to another, they can also be used to merge two or more different metadata formats into a third, or into a set of searchable indexes. Given an appropriate container/transfer format, such as METS, virtually any metadata format such as MARC-XML, Dublin Core, MODS, SMPTE (etc), can be accommodated. Moreover, this open infrastructure will enable archives to absorb catalogue records from their legacy systems in part or in whole while offering new services based on them, such as making the metadata available for harvesting – see OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting).

3.4 Design - Ontologies

3.4.1     Having satisfied those top-level requirements, a viable metadata design, in all its detail, will take its shape from an information model or ontology1. Several ontologies may be relevant depending on the number of operations to be undertaken. CIDOC’s CRM (Conceptual Reference Model http://cidoc.ics.forth.gr/) is recommended for the cultural heritage sector (museums, libraries and archives); FRBR (Functional Requirements for Bibliographic Records http://www.loc.gov/cds/downloads/FRBR.PDF) will be appropriate for an archive consisting mainly of recorded performances of musical or literary works, its influence enhanced by close association with RDA (Resource Description and Access) and DCMI (Dublin Core Metadata Initiative). COA (Contextual Ontology Architecture http://www.rightscom.com/Portals/0/Formal_Ontology_for_Media_Rights_Tran...) will be fit for purpose if rights management is paramount, as will the Motion Picture Experts Group rights management standard, MPEG-21.RDF (Resource Description Framework http://www.w3.org/RDF/), a versatile and relatively light-weight specification, should be a component especially where Web resources are being created from the archival repository: this in turn admits popular applications such as RSS (Really Simple Syndication) for information feeds (syndication). Other suitable candidates that improve the machine handling and interpretation of the metadata may be found in the emerging ‘family’ of ontologies created using OWL (Web Ontology Language). The definition of ontologies and the reading of ontologies expressed in OWL can easily be made using “Protégé”, an open tool of the Stanford University: http://protege.stanford.edu/. OWL can be used from a simple definition of terms up to a complex object oriented modelling. 


1 W3C definition: An ontology defines the terms used to describe and represent an area of knowledge. Ontologies are used by people, databases, and applications that need to share domain information (a domain is just a specific subject area or area of knowledge, like medicine, tool manufacturing, real estate, automobile repair, financial management, etc.). Ontologies include computer-usable definitions of basic concepts in the domain and the relationships among them (note that here and throughout this document, definition is not used in the technical sense understood by logicians). They encode knowledge in a domain and also knowledge that spans domains. In this way, they make that knowledge reusable.

3.5 Design – Element sets

3.5.1     A metadata element set comes next in the overall design. Here three main categories or groupings of metadata are commonly described:

3.5.1.1  Descriptive Metadata, which is used in the discovery and identification of an object.
3.5.1.2  Structural Metadata, which is used to display and navigate a particular object for a user and includes the information on the internal organization of that object, such as the intended sequence of events and relationships with other objects, such as images or interview transcripts.
3.5.1.3 Administrative Metadata, which represents the management information for the object (such as the namespaces that authorise the metadata itself), dates on which the object was created or modified, technical metadata (its validated content file format, duration, sampling rate, etc.), rights and licensing information. This category includes data essential to preservation.

3.5.2     All three categories, descriptive, structural and administrative, must be present regardless of the operation to be supported, though different sub-sets of the data may exist in any file or instantiation. So, if the metadata supports preservation – “information that supports and documents the digital preservation process” (PREMIS) – then it will be rich in data about the provenance of the object, its authenticity and the actions performed on it. If it supports discovery then some or all of the preservation metadata will be useful to the end user (i.e. as a guarantor of authenticity) though it will be more important to elaborate and emphasise the descriptive, structural and licensing data and provide the means for transforming the raw metadata into intuitive displays or in readiness for harvesting or interaction by networked external users. Needless to say, an item that cannot be found can neither be preserved nor listened to so the more inclusive the metadata, with regard to these operations, the better.

3.5.3     Each of those three groupings of metadata may be compiled separately: administrative (technical) metadata as a by-product of mass-digitization; descriptive metadata derived from a legacy database export; rights metadata as clearances are completed and licenses signed. However, the results of these various compilations need to be brought together and maintained in a single metadata instance or set of linked files together with the appropriate statements relating to preservation. It will be essential to relate all these pieces of metadata to a schema or DTD (Document Type Definition) otherwise the metadata will remain just a ‘blob’, an accumulation of data that is legible for humans but unintelligible for machines.

3.6 Design – Encoding and Schemas

3.6.1     In the same way that audio signals are encoded to a WAV file, which has a published specification, the element set will need to be encoded: XML, perhaps combined with RDF, is the recommendation stated above. This specification will be declared in the first line of any metadata instance <?xml version=“1.0” encoding=“UTF-8” ?>. This by itself provides little intelligence: it is like telling the listener that the page of the CD booklet they are reading is made of paper and is to be held in a certain way. What comes next will provide intelligence (remember, to machines as well as people) about the predictable patterns and semantics of data to be encountered in the rest of the file. The rest of the metadata file header consists typically of a sequence of namespaces for other standards and schema (usually referred to as ‘extension schema’) invoked by the design.

<mets:mets xmlns:mets=“http://www.loc.gov/standards/mets/ ” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance ” xmlns:dc=“http://dublincore.org/documents/dces/ ” xmlns:xlink=“http://www.w3.org/TR/xlink/ ” xmlns:dcterms=“http://dublincore.org/documents/dcmi-terms/ ” xmlns:dcmitype=“http://purl.org/dc/dcmitype ” xmlns:tel=“http://www.theeuropeanlibrary.org/metadatahandbook/telterms.html ” xmlns:mods=“http://www.loc.gov/standards/mods/ ” xmlns:cld=“http://www.ukoln.ac.uk/metadata/rslp/schema/ ” xmlns:blap=“http://labs.bl.uk/metadata/blap/terms.html ” xmlns:marcrel=“http://id.loc.gov/vocabulary/relators.html ” xmlns:rdf=“http://www.w3.org/1999/02/22-rdf-syntax-ns#type ” xmlns:blapsi=“http://sounds.bl.uk/blapsi.xml ” xmlns:namespace-prefix=“blapsi”>

Fig 2: Set of namespaces employed in the British Library METS profile for sound recordings

3.6.2     Such intelligent specifications, in XML, are called XML schema, the successor to DTD. DTDs are still commonly encountered on account of the relative ease of their compilation. The schema will reside in a file with the extension .xsd (XML Schema Definition) and will have its own namespace to which other operations and implementations can refer. Schemas require expertise to compile. Fortunately open source tools are available that enable a computer to infer a schema from a well-formed XML file. Tools are also available to convert xml into other formats, such as .pdf or .rtf (Word) documents into XML. The schema may also incorporate the idealised means for displaying the data as an XSLT file. Schema (and namespaces) for descriptive metadata will be covered in more detail in 3.9 Descriptive Metadata – Application Profiles, Dublin Core (DC) below.

3.6.3     To summarise the above relationships, an XML Schema or DTD describes an XML structure that marks up textual content in the format of an XML encoded file. The file (or instance) will contain one or more namespaces representing the extensionr schema that further qualify the XML structure to be deployed.

3.7 Administrative Metadata – Preservation Metadata

3.7.1     The information described in this section is part of the administrative metadata grouping. It resembles the header information in the audio file and encodes the necessary operating information. In this way the computer system recognises the file and how it is to be used by first associating the file extension with a particular type of software, and reading the coded information in the file header. This information must also be referenced in a separate file to facilitate management and aid in future access because file extensions are at best ambiguous indicators of the functionality of the file. The fields which describe this explicit information, including type and version, can be automatically acquired from the headers of the file and used to populate the fields of the metadata management system. If an operating system, now or in the future, does not include the ability to play a .wav file or read an .xml instance for example, then the software will be unable to recognise the file extension and will not be able to access the file or determine its type. By making this information explicit in a metadata record, we make it possible for future users to use the preservation management data and decode the information data. The standards being developed in AES-X098B which will be released by the Audio Engineering Society as AES57 “AES standard for audio metadata – audio object structures for preservation and restoration” codify this aspiration.

3.7.2     Format registries now exist, though are still under development, that will help to categorise and validate file formats as a pre-ingest task: PRONOM (online technical registry, including file formats, maintained by TNA (The National Archives, UK), which can be used in conjunction with another TNA tool DROID (Digital Record Object Identification – that performs automated batch identification of file formats and outputs metadata). From the U.S, Harvard University GDFR (Global Digital Format Registry) and JHOVE (JSTOR/Harvard Object Validation Environment identification, validation, and characterization of digital objects) offer comparable services in support of preservation metadata compilation. Accurate information about the file format is the key to successful long-term preservation.

3.7.3     Most important is that all aspects of preservation and transfer relating to audio files, including all technical parameters are carefully assessed and kept. This includes all subsequent measures carried out to safeguard the audio document in the course of its lifetime. Though much of the metadata discussed here can be safely populated at a later date the record of the creation of the digital audio file, and any changes to its content, must be created at the time the event occurs. This history metadata tracks the integrity of the audio item and, if using the BWF format, can be recorded as part of the file as coding history in the BEXT chunk. This information is a vital part of the PREMIS preservation metadata recommendations. Experience shows that computers are capable of producing copious amounts of technical data from the digitization process. This may need to be distilled in the metadata that is to be kept. Useful element sets are proposed in the interim set AudioMD (http://www.loc.gov/rr/mopic/avprot/audioMD_v8.xsd), an extension schema developed by Library of Congress, or the AES audioObject XML schema which at the time of writing is under review as a standard.

3.7.4     If digitising from legacy collections, these schemas are useful not only for describing the digital file, but also the physical original. Care needs to be taken to avoid ambiguity about which object is being described in the metadata: it will be necessary to describe the work, its original manifestation and subsequent digital versions but it is critical to be able to distinguish what is being described in each instance. PREMIS distinguishes the various components in the sequence of change by associating them with events, and linking the resultant metadata through time.

3.8 Structural Metadata – METS

3.8.1     Time-based media are very often multimedia and complex. A field recording may consist of a sequence of events (songs, dances, rituals) accompanied by images and field notes. A lengthy oral history interview occupying more than one .wav file may also be accompanied by photographs of the speakers and written transcripts or linguistic analysis. Structural metadata provides an inventory of all relevant files and intelligence about external and internal relationships including preferred sequencing, e.g. the acts and scenes of an operatic recording. METS (Metadata Encoding and Transmission Standard, current version is 1.7) with its structural map (structMap) and file group (fileGrp) sections has a recent but proven track record of successful applications in audiovisual contexts (see fig. 3).

METS components

Fig 3: components of a METS instance and one possible set of relationships among them


3.8.2     The components of a METS instance are:

3.8.2.1     A header describes the METS object itself, such as who created this object, when, for what purpose. The header information supports management of the METS file proper.
3.8.2.2     The descriptive metadata section contains information describing the information resource represented by the digital object and enables it to be discovered.
3.8.2.3     The structural map, represented by the individual leaves and details, orders the digital files of the object into a browsable hierarchy.
3.8.2.4     The content file section, represented by images one through five, declares which digital files constitute the object. Files may be either embedded in the object or referenced.
3.8.2.5     The administrative metadata section contains information about the digital files declared in the content file section. This section subdivides into:
3.8.2.5.1 technical metadata, specifying the technical characteristics of a file
3.8.2.5.2 source metadata, specifying the source of capture (e.g.,direct capture or reformatted 4 x 5 transparency)
3.8.2.5.3 digital provenance metadata, specifying the changes a file has undergone since its birth
3.8.2.5.4 rights metadata, specifying the conditions of legal access.
3.8.2.6     The sections on technical metadata, source metadiata, and digital provenance metadata carry the information pertinent to digital preservation.
3.8.2.7     For the sake of completeness, the behaviour section, not shown above in Fig. 2, associates executables with a METS object. For example, a METS object may rely on a certain piece of code to instantiate for viewing, and the behavior section could reference that code.

3.8.3     Structural metadata may need to represent additional business objects:

3.8.3.1 user information (authentication)
3.8.3.2 rights and licenses (how an object may be used)
3.8.3.3 policies (how an object was selected by the archive)
3.8.3.4 services (copying and rights clearance)
3.8.3.5 organizations (collaborations, stakeholders, sources of funding).

3.8.4     These may be represented by files referenced to a specific address or URL. Explanatory annotations may be provided in the metadata for human readers.

3.9 Descriptive Metadata – Application Profiles, Dublin Core (DC)

3.9.1 Much of the effort devoted to metadata in the heritage sector has focussed on descriptive metadata as an offshoot of traditional cataloguing. However, it is clear that too much attention in this area (e.g. localised refinements of descriptive tags and controlled vocabularies) at the expense of other considerations described above will result in system shortcomings overall. Figure 4 demonstrates the various inter-dependencies that need to be in place, descriptive metadata tags being just one sub-set of all the elements in play.

sample descriptive metadata

Fig 4: simple descriptive metadata (courtesy Dempsey, CLIR/DLF primer, 2005)

3.9.2 Interoperability must be a key component of any metadata strategy: elaborate systems devised independently for one archival repository by a dedicated team will be a recipe for low productivity, high costs and minimal impact. The result will be a metadata cottage industry incapable of expansion. Descriptive metadata is indeed a classic case of Richard Gabriel’s maxim ‘Worse is better’. Comparing two programme languages, one elegant but complex, the other awkward but simple, Gabriel predicted, correctly, that the language that was simpler would spread faster, and as a result, more people would come to care about improving the simple language than improving the complex one. This is demonstrated by the widespread adoption and success of Dublin Core (DC), initially regarded as an unlikely solution by the professionals on account of its rigorous simplicity.

3.9.3 The mission of DCMI (DC Metadata Initiative) has been to make it easier to find resources using the Internet through developing metadata standards for discovery across domains, defining frameworks for the interoperation of metadata sets and facilitating the development of community- or discipline-specific metadata sets that are consistent with these aims. It is a vocabulary of just fifteen elements for use in resource description and provides economically for all three categories of metadata. None of the elements is mandatory: all are repeatable, although implementers may specify otherwise in application profiles – see section 3.9.8 below. The name “Dublin” is due to its origin at a 1995 invitational workshop in Dublin, Ohio;”core” because its elements are broad and generic, usable for describing a wide range of resources. DC has been in widespread use for more than a decade and the fifteen element descriptions have been formally endorsed in the following standards: ISO Standard 15836-2003 of February 2003 [ISO15836 http://dublincore.org/documents/dces/#ISO15836 ] NISO Standard Z39.85-2007 of May 2007 [NISOZ3985 http://dublincore.org/documents/dces/#NISOZ3985 ] and IETF RFC 5013 of August 2007 [RFC5013 http://dublincore.org/documents/dces/#RFC5013 ].

Table 1 (below) lists the fifteen DC elements with their (shortened) official definitions and suggested interpretations for audiovisual contexts.

DC element DC definition Audiovisual interpretation
Title A name given to the resource The main title associated with the recording
Subject The topic of the resource Main topics covered
Description An account of the resource Explanatory notes, interview summaries, descriptions of environmental or cultural context, list of contents
Creator An entity primarily responsible for making the resource Not authors or composers of the recorded works but the name of the archive
Publisher An entity responsible for making the resource available Not the publisher of the original document that has been digitized. Typically the publisher will be the same as the Creator
Contributor An entity responsible for making contributions to the resource Any named person or sound source.Will need suitable qualifier, such as role (e.g. performer, recordist)
Date A point or period of time associated with an event in the lifecycle of the resource Not the recording or (P) date of the original but a date relating to the resource itself
Type The nature or genre of the resource The domain of the resource, not the genre of the music. So Sound, not Jazz
Format The file format, physical medium, or dimensions of the resource The file format, not the original physical carrier
Identifier An unambiguous reference to the resource within a given context Likely to be the URI of the audio file
Source A related resource from which the described resource is derived A reference to a resource from which the present resource is derived
Language A language of the resource A language of the resource
Relation A related resource Reference to related objects
Coverage The spatial or temporal topic of the resource, the spatial applicability of the resource, or the jurisdiction under which the resource is relevant What the recording exemplifies, e.g. a cultural feature such as traditional songs or a dialect
Rights Information about rights held in and over the resource Information about rights held in and over the resoure

Table 1: The DC 15 elements

 

3.9.4 The elements of DC have been expanded to include further properties. These are referred to as DC Terms. A number of these additional elements (‘terms’) will be useful for describing time-based media:

DC Term DC definition Audiovisual interpretation
Alternative Any form of the title used as a substitute or alternative to the formal title of the resource An alternative title, e.g. a translated title, a pseudonym, an alternative ordering of elements in a generic title
Extent The size or duration of the resource File size and duration
extentOriginal The physical or digital manifestation of the resource The size or duration of the original source recording(s)
Spatial Spatial characteristics of the intellectual content of the resource Recording location, including topographical co-ordinates to support map interfaces
Temporal Temporal characteristics of the intellectual content of the resource Occasion on which recording was made
Created Date of creation of the resource Recording date and any other significant date in the lifecycle of the recording

Table 2: DC Terms (a selection)

 

3.9.5 Implementers of DC may choose to use the fifteen elements either in their legacy dc: variant (e.g., http://purl.org/dc/elements/1.1/creator) or in the dcterms: variant (e.g., http://purl.org/dc/terms/creator) depending on application requirements. Over time, however, and especially if RDF is part of the metadata strategy, implementers are expected (and encouraged by DCMI) to use the semantically more precise dcterms: properties, as they more fully comply with best practice for machine-processable metadata.

3.9.6 Even in this expanded form, DC may lack the fine granularity required in a specialised audiovisual archive. The Contributor element, for example, will typically need to mention the role of the Contributor in the recording to avoid, for instance, confusing performers with composers or actors with dramatists. A list of common roles (or ‘relators’) for human agents has been devised (MARC relators) by the Library of Congress. Here are two examples of how they can be implemented.

<dcterms:contributor>
<marcrel:CMP>Beethoven, Ludwig van, 1770-1827</marcrel:CMP>
<marcrel:PRF>Quatuor Pascal</marcrel:PRF>
</dcterms:contributor>

<dcterms:contributor>
<marcrel:SPK>Greer, Germaine, 1939- (female)</marcrel:SPK>
<marcrel:SPK>McCulloch, Joseph, 1908-1990 (male)</marcrel:SPK>
</dcterms:contributor>

The first example tags ‘Beethoven’ as the composer (CMP) and ‘Quatuor Pascal’ as the performer (PRF). The second tags both contributors, Greer and McCulloch, as speakers (SPK) though does not go as far as determining who is the interviewer and who is the interviewee. That information would need to be conveyed elsewhere in the metadata, e.g. in Description or Title.

3.9.7 In this respect, other schema may be preferable, or could be included as additional extension schema (as illustrated in Fig. 2). MODS (Metadata Object Description Schema http://www.loc.gov/standards/mods/), for instance allows for more granularity in names and linkage with authority files, a reflection of its derivation from the MARC standard:

name
 Subelements:
   namePart
     Attribute: type (date, family, given, termsOfAddress)
   displayForm
   affiliation
   role
      roleTerm
         Attributes: type (code, text); authority
         (see: http://www.loc.gov/standards/sourcelist/)
      description
   Attributes: ID; xlink; lang; xml:lang; script; transliteration
   type (enumerated: personal, corporate, conference)
authority (see: http://www.loc.gov/standards/sourcelist/)

3.9.8 Using METS it would be admissible to include more than one set of descriptive metadata suited to different purposes, for example a Dublin Core set (for OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) compliance) and a more sophisticated MODS set for compliance with other initiatives, particularly exchange of records with MARC encoded systems. This ability to incorporate other standard approaches is one of the advantages of METS.

3.9.9 DC, under the governance of the Dublin Core Metadata Initiative (DCMI), continues to develop. On the one hand its value for networking resources is strengthened through closer association with semantic web tools such as RDF (see Nilsson et al, DCMI 2008) while on the other it aims to increase its relevance to the heritage sector through a formal association with RDA (Resource Description &Access http://www.collectionscanada.gc.ca/jsc/rda.html) due to be released in 2009. As RDA is seen as a timely successor to the Anglo America Cataloguing Rules this particular development may have major strategic implications for audiovisual archives that are part of national and university libraries. For broadcasting archives other developments based on DCMI are noteworthy At the time of writing the EBU (European Broadcast Union) is completing the development of the EBU Core Metadata Set, which is based on and compatible with Dublin Core.

3.9.10 The archive may wish to modify (expand, adapt) the core element set. Such modified sets, drawing on one or more existing namespace schemas (e.g. MODS and/or IEEE LOM as well as DC) are known as application profiles. All elements in an application profile are drawn from elsewhere, from distinct namespace schemas. If implementers wish to create ‘new’ elements that are not schematized elsewhere, for instance contributor roles unavailable in the MARC relators set (e.g. non-human agents such as species, machines, environments), then they must create their own namespace schema, and take responsibility for ‘declaring’ and maintaining that schema.

3.9.11 Application profiles include a list of the governing namespaces together with their current URL (preferably PURL – permanent URL). These are replicated in each metadata instance. There then follows a list of each data element together with permitted values and style of content. This may refer to in-house or additional rules and controlled vocabularies, e.g. thesauri of instrument names and genres, authority files of personal names and subjects. The profile will also specify mandatory schemes for particular elements such as dates (YYYY-MM-DD) and geographical co-ordinates and such standardised representations of location and time will be able to support map and timeline displays as non-textual retrieval devices.

 

Name of Term Title
Term URI http://purl.org/dc/elements/1.1/title
Label Title
Defined By http://dublincore.org/documents/dcmi-terms/
Source Definition A name given to a resource
BLAP-S Definition The title of the work or work component
Source Comments Typically, a Title will be a name by which the source is formally known
BLAP-S Comments If no title is available construct one that is derived from the resource or supply [no title]. Follow normal cataloguing practice for recording title in other languages using the ‘Alternative’ refinement.Where data are derived from the Sound Archive catalogue, this will equate to one of the following title fields in the following hierarchical order:Work title (1), Item title (2), Collection title (3), Product title (4), Original species (5) Broadcast title (6), Short title (7), Published series (8), Unpublished series (9)
Type of term Element
Refines  
Refined by Alternative
Has encoding scheme  
Obligation Mandatory
Occurrence Not repeatable

Fig 5: Part of the British Library’s application profile of DC for sound (BLAP-S):

Namespaces used in this Application Profile
DCMI Metadata Terms http://dublincore.org/documents/dcmi-terms/
RDF http://www.w3.org/RDF/
MODS elements http://www.loc.gov/standards/mods/
TEL terms http://www.theeuropeanlibrary.org/metadatahandbook/telterms.html
BL Terms http://labs.bl.uk/metadata/blap/terms.html
MARCREL http://id.loc.gov/vocabulary/relators.html

3.9.12 The application profile therefore incorporates or draws on a data dictionary (a file defining the basic organisation of a database down to its individual fields and field types) or several data dictionaries, that may be maintained by an individual archive or shared with a community of archives. The PREMIS data dictionary (http://www.loc.gov/standards/premis/v2/premis-2-0.pdf currently version 2) relating exclusively to preservation is expected to be drawn on substantially. Its numerous elements are known as ‘Semantic units’. Preservation metadata provides intelligence about provenance, preservation activity, technical features, and aids in verifying the authenticity of a digital object. The PREMIS Working Group released its Data Dictionary for Preservation metadata in June 2005 and recommends its use in all preservation repositories regardless of the type of materials archived and the preservation strategies employed.

3.9.13 By defining application profiles and, most importantly by declaring them, implementers can share information about their schemas in order to collaborate widely on universal tasks such as long-term preservation

3.10 Sources of Metadata

3.10.1     Archives should not expect to create all descriptive metadata by themselves from scratch (the old way). Indeed, given the in-built lifecycle relationship between resources and metadata such a notion will be unworkable. There are several sources of metadata, especially the descriptive category that should be exploited to reduce costs and provide enrichment through extending the means of input. There are three main sources: professional, contributed and intentional (Dempsey:2007): they may be deployed alongside each other.

3.10.2     Professional sources means drawing on the locked-in value of legacy databases, authority files and controlled vocabularies which are valuable for published or replicated materials. It includes industry databases, as well as archive catalogues. Such sources, especially archive catalogues, are notoriously incomplete and incapable of interoperation without sophisticated conversion programmes and complex protocols. There are almost as many data standards in operation in the recording and broadcasting industries and the audiovisual heritage sector as there are separate databases. The lack of a universal resolver for AV, such as ISBN for print, is a continuing hindrance and after decades of discographical endeavour there is still disagreement about what constitutes a catalogue record: is it an individual track or is it a sequence of tracks that make up an intellectual unit such as a multi-sectioned musical or literary work? Is it the sum total of tracks on a single carrier or set of carriers, in other words, is the physical carrier the catalogue unit? Evidently, an agency that has chosen one of the more granular definitions will find it much easier to export its legacy data successfully into a metadata infrastructure. Belt and braces approaches to data export based on Z39.50 (http://www.loc.gov/z3950/agency/ protocol for information retrieval) and SRW/SRU (a protocol for search and retrieve via standardized URL’s with a standardized XML response) will continue to provide a degree of success, as will the ability of computers to harvest metadata from a central resource. However, more effective investment should be made in the shared production of resources which identify and describe names, subjects, places, time periods, and works.

3.10.3     Contributed sources means user generated content. A major phenomenon of recent years has been the emergence of many sites which invite, aggregate and mine data contributed by users, and mobilize that data to rank, recommend and relate resources. These include, for example, YouTube and LastFM. These sites have value in that they reveal relations between people and between people and resources as well as information about the resources themselves. Libraries have begun to experiment with these approaches and there are real advantages to be gained by allowing users to augment professionally sourced metadata. So-called Web 2.0 features that support user contribution and syndication are becoming commonplace features of available content management systems.

3.10.4     Intentional means data collected about use and usage that can enhance resource discovery. The concept is borrowed from the commercial sector, Amazon recommendations, for instance, that are based on aggregate purchase choices. Similar algorithms could be used to rank objects in a resource. This type of data has emerged as a central factor in successful websites, providing useful paths through intimidating amounts of complex information.

3.11 Future Development Needs

3.11.1     For all the recent work and developments, metadata remains an immature science, though this chapter will have demonstrated that a number of substantial building blocks (data dictionaries, schemas, ontologies, and encodings) are now in place to begin to match the appetite of researchers for more easily accessible AV content and the long-held ambition of our profession to safeguard its persistence. To achieve faster progress it will be necessary to find common ground between public and commercial sectors and between the different categories of audiovisual archives, each of which has been busy devising its own tools and standards.

3.11.2     Some success has been achieved with automatic derivation of metadata from resources.We need to do more, especially as existing manual processes do not scale very well. Moreover, metadata production does not look sustainable unless more cost is taken out of the process.”We should not be adding cost and complexity, which is what tends to happen when development is through multiple consensus-making channels which respond to the imperatives of a part only of the service environment” (Dempsey:2005).

3.11.3     The problem of the reconciliation of databases, i.e. the capacity of the system to understand that items are semantically identical although they may be represented in different ways, remains an open issue. There is significant research being undertaken to resolve this issue,but a widely suitable general solution has yet to emerge. This issue is also very important for the management of the persistence in the OAIS as the following example demonstrates. The semantic expression that Wolfgang Amadeus Mozart is the composer of most of the parts of the Requiem (K.626) is represented in a totally different way in FRBR modelling when compared to a list of simple DCMI statements. In CDMI ‘Composer’ is a refinement of ‘contributor’ and ‘Mozart’ is its property; while in FRBR modelling, ‘composer’ is a relation between a physical person and an opus. The use of controlled vocabularies is also a way of ensuring that W.A. Mozart represents the same person as Mozart.

4: Unique and Persistent Identifiers

4.1 Introduction

4.1.1     A digital sound recording, whether stored on a mass storage system or on discrete carriers, must be able to be identified and retrieved. An item cannot be considered preserved if it cannot be located, nor linked to the catalogue and metadata record that gives it meaning. There is a need for every digital item to be unambiguously and uniquely named. In ensuring that the digital object is unambiguously and uniquely named the first step in the identification is to determine what is being named, and at what level.

4.1.2     All computer records by their very nature have some sort of system identifier that enables them to be stored without conflict. This identifier may be an acceptable public identifier, but more often than not such identifiers are system oriented and subject to change based on system requirements. There is a subsequent need for a persistent public identifier to maintain an item’s accessibility, to ensure that it can be located and displayed by those who wish to use it so that citations and links made to it continue to provide access to it. There is also a requirement for that identifier to resolve to the item to which it refers regardless of where it has been stored or what its system identifier may have become.

4.1.3     The Resource Description Framework (RDF) standard is an important reference for the identification of digital objects (http://www.w3.org/RDF/ ). RDF is based on the concept of identifying things using Web identifiers called URIs (Universal Resource Identifiers). The identification systems are based on two basic mechanisms. The first is the naming of an item by creating an identifier based on semantics or other rules of labelling such that the identifier will remain attached to the item. In the RDF standard, such identifiers are called URNs (Universal Resource Names). The second is the locator, which is organising a location system so that the item intended to be identified could be found from the locator. In the RDF standard, such identifiers are called URLs (Universal Resource Locator).

4.1.4     There have been many proposed schemes for naming a digital object, some specifically for audio or audiovisual objects, amongst them the EBU Technical Recommendation R99-1999 ‘Unique’ Source Identifier (USID) for use in the <OriginatorReference> field of the Broadcast Wave Format (BWF). Such schemes are intended to provide a unique number within a particular community. Such schemes have not been successful in obtaining universal acceptance.

4.2 Persistent Identifiers

4.2.1     Even before the issue of digitisation made it critical, libraries, archives and audio collections generally have tended to develop systems with varying degrees of sophistication, which allow them to access their materials. These numbering systems, which tend to be unique within their own domain, can be incorporated into more universal naming schemes with the addition of a unique name for the domain or institution. This kind of structure allows maximum flexibility to an organisation in the local identification of its resources, whilst allowing the identifiers to be incorporated into a global system with the addition of an appropriate naming authority component. These persistent identifiers are for the user of the content to be able to identify a work (as opposed to a file) which remains constant through time as a reference for that work regardless of how the file naming conventions have changed.

4.2.2     A Persistent Identifier (PID) is an identifier constructed and implemented such that the identified resource will remain the same independently of the location of its representation and independent of the fact that several copies are available at various locations. It means that the PIDs are URNs.

4.3 File Naming Conventions and Unique Identifiers

4.3.1     Care should be taken when discussing this subject to maintain the distinction between the persistent identifier used to refer to a work, and the file naming conventions. In many practical system there may well be links between the two. This section makes recommendations about file naming conventions. Data files managed in any given repository may include several types of data, not just audio. A Unique Identifier (UID) uniquely identifies a resource. This means that the identifier may change for the particular embodiment of the resource and each copy of the resource has its own ID. It consequently means that the UID are URL’s. For the purposes of this discussion, file names will also be referred to as unique identifiers.

4.3.2     For linkages within and external to any system the unique identifier is the primary key to managing audio data and all of its associated files, e.g. the master copies, playback copies, compressed versions of playback copies, metadata files, edit lists, accompanying texts, images, versions of any one of those master files or derivatives. Therefore, unless the archive is using a system-assigned ‘dumb’ identifiers, it is vitally important that the unique identifier’s structure is logically determined, clearly understood by those who have to apply it, and able to be read by people and machines. It is also important to reveal the connections between ‘families’ of data files: one commentator likens this connectivity to “the persistent ‘thread’ that enables resources to be re-tagged or re-stitched on the Web”. Talking in terms of ‘resources’ rather than collections is an important underlying concept in these guidelines.

4.3.3     One of the most powerful ways of constructing an identification system that reveals those connections is to base it on the concept of Root ID (RID). The RID is the identifier of entity. All the files and folders involved in the representation of the entity will be derived from the RID by addition of prefixes and suffixes such as the creation of unique identifiers.

4.3.4     Regardless of whether identifiers have embedded intelligence or not, it is normal for computer-generated and computer-readable identifiers to have fixed length codes as the primary key. This offers the following advantages:

4.3.4.1     They enable rules to be established for creating new unique identifiers.

4.3.4.2     They guarantee unambiguous recognition in the system (and for users who know the rules).

4.3.4.3     They permit validation of the code or components of the code.

4.3.4.4     They support searching, sorting and reporting.

4.3.5     There has been a prolonged debate about the relative merits of dumb and intelligent or expressive unique identifiers. Most systems allocate a dumb identifier the moment data are saved. They are quickly applied, require no human intervention and their uniqueness is guaranteed. However, their randomness and arbitrariness means that other ways have to be found to show how the different files generated in the life-cycle of a digital resource connect. A better way to do this is by use of intelligent, expressive identifiers.

4.4 Identifier Characteristics

4.4.1     The following characteristics should be considered when developing a naming scheme:

4.4.1.1     Uniqueness, the naming scheme must be unique within the context of the organisation’s digital resources and, if necessary, globally unique.
4.4.1.2     There should be a commitment to persistence; an organisation must have a commitment to maintain the association of the current location of the resource with the persistent identifier.
4.4.1.3     An identifier system will be more effective if it is able to accommodate the special requirements of different types of material or collections.
4.4.1.4     Although not absolutely critical, and not essential for machine generated persistent identifiers, a system will generally be more successful if it is easy to understand and apply, and if it lends itself to short and easy to use citations.
4.4.1.5     The identifier should be capable of distinguishing parts of an item, as well as versions and roles that a digital item might have. Relying on the file extension to distinguish a distribution copy from an archival copy is not advisable as the format may change over time, though the role remains the same (Dack 1999).
4.4.1.6     The identifier should permit batch renaming for ingestion into different content management systems.

5: Signal Extraction from Original Carriers

5.1 Introduction

5.1.1     The first, and most significant part of the digitisation process is the optimisation of signal retrieval from the original carriers. As a general principle, the originals should always be kept for possible future re-consultation. However, for two simple, practical reasons any transfer should attempt to extract the optimal signal from the original. Firstly, the original carrier may deteriorate, and future replay may not achieve the same quality, or may in fact become impossible, and secondly, signal extraction is such a time consuming effort that financial considerations call for an optimisation at the first attempt.

5.2 Reproduction of Historical and Obsolete Mechanical Formats

5.2.1 Introduction

5.2.1.1 The first audio recordings made were mechanical recordings, and this approach remained almost the only viable method for capturing sound until developments in electrical circuitry began to create a market for magnetic recordings during and after the 1930s. Mechanical recordings are recognised by the presence of a continuous groove in the surface of the carrier into which the signal is encoded. The encoding of monophonic audio is achieved either by modulating the bottom of the groove up and down with respect to the surface (vertical or hill-and-dale recordings), or from side to side (lateral recordings). All cylinder recording are vertical recordings, as are Edison Diamond Discs, some early shellacs and discs recorded by Pathé up until about 1927, when they began to record laterally cut discs. For a time, some radio transcription discs were also vertically cut recordings, primarily in the US. Lateral cut recordings are the more common form, and most coarse groove recordings (sometimes called 78s), transcription, and instantaneous discs are lateral, as are monophonic Long Play (LP) microgroove records. Microgroove discs are discussed separately in section 5.3.

5.2.1.2 Mechanical sound recording formats are analogue, so called because groove wall is modulated in a continuous representation of the wave form of the original audio. Almost all of the mechanical recordings discussed are now obsolete,in that the industry which once created these artefacts no longer supports them.Early mechanical recordings were acoustic,as the sound waves acted directly on a lightweight diaphragm which drove the cutter directly into the recording surface. Later mechanical recordings were “electrical recordings” as they used a microphone and amplifier to drive an electrical cutting head. From 1925 onwards almost all recording studios began to make electrical recordings.

5.2.1.3 As the early mechanical recordings were all made when the industry was developing, there were few standards. Those that existed were poorly observed as the technology was constantly evolving, and many of the manufacturers would keep their latest techniques secret in order to gain a market advantage. One legacy of this period is the immense degree of variation in most aspects of their work, not least in the size and shape of the recorded groove (see 5.2.4), recording speed (5.2.5) and equalisation required (5.2.6). Consequently, there is a need for those working with the recordings to have specific knowledge about the historical and technical circumstances under which these recording were created. For obscure or non standard recordings, it is advisable to seek advice from specialists, and even for the more common types of recording, caution should be exercised.

5.2.2 Selection of Best Copy

5.2.2.1 Mechanical recording may be either instantaneous or replicated. The former are mostly unique items, single recordings created of a particular event. These include wax cylinders1,  lacquer (also known as acetate) discs and recordings created by office dictation machines (see 5.2.9). Replicated recordings, on the other hand, are pressed or moulded reproductions of an original master, and are almost always manufactured in multiples. Instantaneous recordings should be identified and treated separately and carefully.

5.2.2.2 Instantaneous cylinders may be distinguished by their waxy appearance and feel, and were generally made of a soft metallic soap. Their colour typically can vary from a light butterscotch to a dark chocolate brown, or very rarely, black. Replicated cylinders were made of a much harder metallic soap, or alternatively of a celluloid sleeve over a plaster core. These were manufactured in a variety of colours, though black and blue were the more common, and usually bear some content information embossed into a flattened end.

5.2.2.3 The first disc format capable of instant replay appeared around 1929. The discs were made of an uncoated soft metal (usually aluminium, possibly copper or zinc) into which a lateral groove was embossed rather than cut, and are easily distinguished from replicated shellac discs. Like the subsequent lacquer discs, the embossed metal format was designed to allow the discs to be replayed on standard gramophones of the time, and so recordings can be loosely categorised as coarse groove and 78 rpm, but the transfer engineer should expect some variation, not least in the groove profile.

5.2.2.4 Lacquer or acetate discs, introduced in 1934, are most frequently described as laminated, although that is not their method of manufacture, or as acetates, which is not the nature of their recording surface. They most commonly consist of a strong and stiff base (aluminium or glass, occasionally zinc) covered with a layer of cellulose nitrate lacquer, coloured dark to improve observation of the cutting process. Rarer are discs which have a cardboard base. The cutting properties are controlled by the addition of plasticisers (softening agents), such as castor oil or camphor.

5.2.2.5 Lacquer discs can appear similar to shellac or more typically vinyl, but they can be distinguished in several ways. The base material can often be seen between the outer lacquer layers, either within the centre hole or at the disc edge.Where the disc has a paper label the content information will often be typed or handwritten rather than printed. On discs without paper labels one or more additional off-centre drive holes may be seen near the centre hole. Though cellulose nitrate lacquer discs on metal or glass base are the most common instantaneous disc, in practice a great variety of other materials were used, such as cardboard as the base media, or gelatine as the recording surface, or as a homogenous recording disc.

5.2.2.6 Due to their inherent instability lacquer discs should be transferred with a high priority.

5.2.2.7 The selection of the best copy, in those circumstances where multiple copies on instantaneous discs exist, is usually a process of determining the most original intact copy of an item. In the case of mass produced mechanical recordings, where the existence of multiple copies is the normal situation, the following guide to selection of best copy applies.

5.2.2.8 Selection of the best copy of replicated mechanical media draws on knowledge of the production of the recording, and the ability to visually recognise wear and damage which would have an audible effect on the signal. The recording industry uses numbers and codes, generally located in the space between the run-out groove and the label in a disc recording, to identify the nature of the recording. This will help the technician determine which recordings are in fact identical, or alternate recordings of the same material.Visual signs of wear or damage are best seen in the way a recording reflects light. To best show the effect an incandescent light is a necessity, generally aimed at the recording from behind the technician’s shoulder, so that they are looking down the beam of light. Fluorescent tubes, or energy saving compact fluorescent lights do not provide the necessary coherent light source to reveal wear and should not be used. A stereoscopic microscope is helpful in assessing groove shape and size, and in examining wear caused by previous replay, which helps selection of the correct replay stylus. A more objective approach involves using a stereo-microscope with a built-in reticule which enables more accurate selection of styli (Casey and Gordon 2007).


1 The earliest commercial wax cylinders were replicated acoustically, one from another, and performers would often do multiple sessions to create batches of similar recordings. They should all be regarded as unique items.

5.2.3 Cleaning and Carrier Restoration

5.2.3.1 Grooved media may be adversely affected by past use, or through natural decomposition of the constituent materials, hastened to a greater or lesser degree by environmental storage conditions. Debris including dust and other airborne material can accumulate within the grooves, and fungal growth may be present where climatic conditions have allowed. This is particularly common with instantaneous cylinders. In addition, lacquer discs may experience exudation of the plasticisers from the lacquer itself. This typically has a white or gray mould-like appearance, but is distinguished by a greasy consistency. Mould, on the other hand, is typified by feathery or thread-like white or gray growth. Each of these conditions will compromise the ability of the replay stylus to follow the groove pattern accurately, and so appropriate cleaning of the carrier is necessary.

5.2.3.2 The most appropriate cleaning method will depend on the specific medium and its condition. In many cases a wet solution will produce the best results, but the choice of solution must be made carefully, and in certain cases it may be best to avoid the use of any liquids. Record cleaning solutions which do not disclose their chemical composition should not be used. All decisions about the use of solvents and other cleaning solutions should only be made by the archivist in consultation with the appropriate technical advice by qualified plastics conservators or chemists. It can however be stated that lacquer and shellac discs, and all types of cylinder, should never be exposed to alcohol, which may have an immediate corrosive effect. Shellac discs frequently contain absorbent fillers which can expand on sustained contact with moisture, and so should be dried immediately after cleaning with any wet solution. Any wet cleaning process should avoid contact with paper disc labels.

5.2.3.3 Castor oil has commonly been used as a plasticiser in the production of cellulose nitrate lacquer discs, which, as it exudes from the disc surface typically breaks down into palmitic and stearic acids. The loss of plasticiser causes the coating to shrink and consequently crack and peel away from the base. This is known as delamination. Several solutions have been employed successfully in removing the exuded acids (see in particular Paton et al 1977; Casey and Gordon 2007, p27).It has been observed however that after cleaning, lacquer discs may continue to degrade at an accelerated rate. It is sensible therefore to create digital copies of the material held on cleaned lacquer discs as soon as possible after cleaning. It must again be stressed that the effect of all solvents should be tested before use. Some early lacquer discs have a gelatine rather than cellulose nitrate playing surface for example, which is soluble and would instantly suffer irreversible damage if treated with any liquid solution.

5.2.3.4 Certain other media may not be appropriate for wet cleaning, including shellac and lacquer discs which were manufactured with paper or card layers beneath the playing surface. Similarly, lacquer discs displaying cracking or peeling surfaces must be treated with great care, and instantaneous cylinders should be cleaned with a soft dry brush only, applied along the groove path. However, where mould spores are thought to be present, the utmost care should be taken to minimise cross contamination. Special care should be taken when cleaning moulds and spores as these may cause serious health problems. Operators are strongly advised to obtain professional advice before commencing work on such infected materials.

5.2.3.5 In cases where wet cleaning is deemed appropriate, it should be carried out with both the solution and carrier at room temperature, to avoid any damage to the carrier caused by thermal shock.

5.2.3.6 Often the most effective and efficient method of wet cleaning is to use a record cleaning machine employing a vacuum to remove the waste liquid from within the groove, such as those made by Keith Monks, Loricraft or Nitty Gritty.

5.2.3.7 Particularly dirty carriers, or those with stubborn marks such as dried-on paper deposits, may be more appropriately cleaned using an ultrasonic bath, into which the carrier (or portion of the carrier) is placed. The process works by vibrating a solution around the item, loosening dirt deposits.

5.2.3.8 In cases where it is not possible or appropriate to employ such equipment, hand washing may be carried out using an appropriate short bristled brush. Clean tap water may be used in the washing process, but should always be followed by a thorough rinse in demineralised water to remove any consequent contamination.

5.2.3.9 In addition to cleaning, some further form of restoration may be required. Shellac discs and cylinders of all types are brittle and liable to break if mishandled, and shellac will melt and warp at high temperatures. The exudation of plasticiser from lacquer discs causes the lacquer layer to contract upon a stable metal or glass base, creating stresses between the layers and resulting in cracking and peeling of the lacquer playing surface. Reconstruction of broken discs and cylinders is ideally done without resorting to glues or adhesives, as these inevitably form a barrier between the parts being joined which, however small, will be audible. Such processes are also generally irreversible, allowing for no second chances. The manufacturing processes used in replicating both shellac discs and cylinders will often result in a degree of internal stress in the carrier. If broken, the divergent stresses in the constituent pieces may cause them to contort somewhat. To minimise the effect of this, broken carriers should be reconstructed and transferred as soon as possible after the breakage occurs. The individual parts of broken carriers should be stored without touching. Storing them unsecured in their reconstructed form may encourage the finely detailed broken edges to rub together, causing further damage.

5.2.3.10 Shellac discs are usually best reconstructed on a turntable, upon a flat platter larger than the disc itself (another, disposable or non-archival disc is often ideal). The pieces are placed upon it in their correct positions and held in place around the centre spindle with re-usable pressure sensitive adhesive putty such as Blu-Tack, U-Tack, or similar around the outside of the disc.Where discs are thinner around the edge than in the middle, the putty may be used to raise the edges to the correct height. Take note of the direction through the groove that the stylus will travel: where the pieces cannot be perfectly vertically aligned, it is better for both the stylus and the resulting transfer that the stylus be obliged to drop down onto a lower fragment rather than be pushed up abruptly onto a higher one.

5.2.3.11 Cylinders which have suffered a neat break can often be reconstructed around the playback mandrel using 1/4 inch splicing tape as a form of bandage. More complex breakages will require specialist help.

5.2.3.12 Flakes from peeling lacquer disc surfaces may be temporarily fixed to allow the disc to be played, using tiny amounts of petroleum jelly between the flake and disc base. The long term effects of this procedure are likely to be deleterious, and it is used to attempt replay of discs which are judged to be unplayable by any other currently practicable means.

5.2.3.13 Where it is possible to play a warped or bent disc without flattening it, this should be the preferred option, as the risks associated with disc flattening described below will attest. The ability to play a warped disc can often improve when the rotational speed of the disc is reduced (see 5.2.5.4).

5.2.3.14 Shellac discs may be flattened in a laboratory (i.e., non-domestic) fan-assisted oven. The disc should be placed on a sheet of pre-heated toughened glass, and it is imperative that both disc and glass be clean, to prevent dirt fusing with the disc surface. There is a danger that in curing vertical warpage, some lateral warpage may occur. The disc should therefore not be heated any more than it has to be, and a temperature of around 42C is often sufficient (Copeland 2008 Appendix 1).

5.2.3.15 Flattening discs is a useful process because it can make unplayable discs playable. However, current research into the procedure of flattening discs with heat shows that it causes a measurable rise in subsonic frequencies, even in the low audible frequency range (Enke 2007). Though the research is not conclusive the point should be noted in determining whether to flatten a particular disc. The analysis of the affect of flattening was carried out on vinyl discs and whether it applies to shellac is yet to be determined, though the lower temperatures associated with treating shellac make it much less of a risk. Nonetheless, the possibility of such damage has to be weighed against enabling the playing of the disc.

5.2.3.16 Though it is strongly advised not to attempt to permanently flatten an instantaneous disc (and any attempt is likely to be unsuccessful and damage the disc surface), in some instances the warpage may be temporarily reduced by clamping or otherwise fixing the disc edges to the turntable. Great care must be taken, especially with lacquer discs whose surface may be damaged if placed under stress. Laminated flexible discs with a warp may have been rendered flat by placing the disc on the vacuum platter of a disc cutting lathe and carefully bringing the disc flat. All physical treatment should be undertaken with great care to avoid damage.

5.2.3.17 Some replicated discs have been produced with a non-centric spindle hole. It is preferable to play such discs on a turntable with a removable spindle or to raise the height of the disc above the spindle using, for example, waste discs and rubber interleave. In the latter case the height of the pickup arm should be raised at the supporting column by the same amount. It is possible to re-centre the hole using a reamer or drill, but such invasive approaches should be undertaken cautiously and never with unique or single copies. Altering the original artefact may well result in loss of secondary information.

5.2.4 Replay Equipment

5.2.4.1 Grooved recordings were made to be replayed with a stylus and pickup. Though optical technology has some special advantages which are discussed below (see section 5.2.4.14), and though advances in optical replay are bringing closer the likelihood of a practical system which does not require physical contact, currently the best and most cost effective approach to retrieving the audio content from such a recording is with the correct stylus. For lateral recordings a set of styli with different radii in the range of 38 µm (1.5 mil2), to 102 µm (4 mil), with an additional focus on 76 µm (3 mil) and 65 µm (2.6 mil) for early and late electrics respectively, is essential. The correct stylus for the particular groove will ensure best possible replay by fitting properly into the replay area, and avoiding worn or damaged sections of the groove wall. Records in good condition will reproduce with greater accuracy and reduced surface noise with elliptical tips; records in visually poor condition may be better suited to conical tips.Wear from previous use may well be to a particular region of the groove wall leaving some undamaged areas. Choosing an appropriate tip size and shape will allow these undamaged sections to be reproduced without including distortions caused by the damaged sections. A truncated stylus of either shape will better avoid any damaged areas in the bottom of the groove. Care should be taken in the replay of Pathé lateral discs as they typically have a larger groove width, and may require larger tip radii to avoid damage to the groove bottom.

5.2.4.2 Mono pickups are available, but it is more common to use stereo pickups as these allow separate capture of each groove wall. Moving coil pickups are often highly regarded because of their enhanced impulse response which aids in improving the separation of groove noise from audio signal. However, the range of various tip sizes for moving coil pickups is not as wide as that for moving magnet, are integral to the pickup, and those that can be ordered are around four time more expensive. Moving magnet pick ups are more common, more robust, lower cost, and generally more than adequate for the task.When replaying shellac discs a tracking force in the range of 30-50mN (3-5 grams) is often appropriate. It is recommended that a lesser tracking force be applied to lacquer discs. An advantage in using a stereo pickup is that this allows the two resultant channels to be stored separately, enabling future selection or processing of the separate channels. For listening the two channels may be combined in phase for a lateral recording, and out of phase (with respect to the pickup) for a vertical recording.

5.2.4.3 Selection of a suitable stylus in vertical recordings is governed by different criteria to lateral recordings. Rather than choosing a stylus to sit in a particular space on the side of a groove wall, playback of cylinders and other vertical cut recordings requires that a stylus be chosen that is a best match for the bottom of the groove. This is critical with instantaneous cylinders, where even very light tracking forces will cause damage if the incorrect stylus is chosen. A spherical stylus is generally preferred especially if the surface is damaged, though an elliptical stylus may well avoid frequency dependent tracking error. Typical sizes are between 230 (9 mil) and 300 µm (11.8 mil) for standard cylinders (100 grooves/inch) and between 115 (4.5 mil) and 150 µm (5.9 mil) for 200 grooves/inch cylinders. Cylinders should be replayed with a stylus whose tip has a radius a little smaller than the bottom radius of the groove. A truncated stylus will damage the groove because tracking will take place at the edge rather than the tip, resulting in increased pressure to that part of the groove.

5.2.4.4 When it comes to making decisions about what equipment to acquire, knowledge of the content of a particular collection will be the primary guide to determining the type of equipment required. Different types of carriers will obviously require different types of replay equipment, but even within similar carriers some specialist needs may arise.

5.2.4.5 Generally, historical equipment should not be used, mainly because of its poor rumble performance and in the case of cylinder players, greatly increased tracking force compared with equivalent modern replay equipment. Some problematic cylinders may not be playable on this type of equipment as modern cylinder players normally track the grooves with auto- controlled feed retrieved from the motion of the needle.When using this set up it is virtually impossible to properly track locked grooves, or scratches nearly parallel to the groove. This problem can be solved by using a modern player with fixed feed, or a modified historical cylinder player.

5.2.4.6 Radio transcription discs commonly have a diameter of 16 inches. If such discs are held in a collection, it will be necessary to procure a turntable, arm and pickup for discs of this size. For standard discs up to 12 inch records generally a modern precision turntable, modified to allow varispeed in a wide range, is required.

5.2.4.7 Negative metal stampers manufactured for mass replication of discs can themselves be replayed if an appropriate bi-point or stirrup stylus is available. This type of stylus sits astride the ridge (which is a negative impression of a disc groove) and needs to be placed carefully so as to avoid falling between adjacent ridges. As the stamper holds an inverse spiral to the discs it was designed to replicate, it should revolve anticlockwise, that is, in the opposite direction to a replicated disc, in order to be played from start to finish. To do this correctly would require a fully reverse-mounted tone arm. Much simpler and just as effective would be to play the stamper from finish to start on a standard clockwise turntable, and reverse the resulting digital transfer, using any current high quality audio editing software.

5.2.4.8 Bi-point styli are now extremely difficult to obtain, and fall into two categories, namely low- and high-compliance. The former are designed to repair manufacturing defects in metal stampers and as such are not ideally suited to archival transfer work. The latter, employing a significantly lighter tracking force are designed for audible replay rather than physical modification of the stamper, and so can be considered more suitable.

5.2.4.9 Turntables and cylinder phonographs for archival transfer purposes need to be precision mechanical devices in order to produce the minimum transmission of spurious vibrations to the record surface, which acts as a receiving diaphragm for the pickup. Low frequency vibrations are called rumble, and these vibrations frequently have a considerable vertical component. To reduce rumble generated by external vibrations,the replay apparatus must be placed on a stable foundation that is not likely to transmit structural vibrations. The replay machine should have a speed accuracy of at least 0.1 per cent; wow and flutter (DIN 45 507 weighted) better than 0.01 per cent; and an unweighted rumble of better than 50 dB. The turntable will be either belt or direct drive; friction drive wheel machines are not recommended as suitable speed accuracy and low rumble is not possible with these devices.

5.2.4.10 Any power supply wiring and the electric motor must be shielded to prevent injection of electrical noises into the pickup circuit. If required, additional Mu-Metal plates may be used to shield the motor from the pickup. The connecting cable to the pre-amplifier must be within the specifications regarding the loading impedance for the pickup. The installation should follow best analogue practice and adequate grounding procedures must be adhered to in order to ensure noise is not added to the audio signal. All of the above suggestions and specifications should be quantified, by analysing the output from test discs (see 5.2.8).

5.2.4.11 Both turntables and cylinder phonographs should be capable of variable replay speed, with the possibility of half-speed replay being particularly desirable (see 5.2.5.4), and feature a speed readout to allow documentation, possibly as a signal suitable for automatic logging for metadata. The pickup arm must sit on a base that can be adjusted, not only as regards distance from the turntable centre, but also in elevation.

5.2.4.12 In order to evaluate and decide on the most appropriate equipment and settings, comparisons must be made between the different options. This is best achieved through simultaneous, or A/B comparison, and audio editing software should be chosen which allows multiple audio files to be compared simultaneously. Transferring portions of a recording with different parameters and aligning the different resulting audio files in the editor for listening purposes, allows repeated direct comparison and reduces the inherent subjectivity of the process to a minimum.

5.2.4.13 A decision will need to be made as to the application of an equalisation curve prior to digitisation (see 5.2.6 Replay Equalisation).Where this is desirable, an appropriate preamplifier will be required, adjustable to recreate all necessary settings.

5.2.4.14 As an alternative to contact pickups the entire surface of a disc or a cylinder can be scanned or photographed at high resolution then converted to sound.Various projects have been developed up to a (quasi-) commercial level (ELP LaserTurntable; IRENE by Carl Haber,Vitaliy Fadeyev et al; VisualAudio by Ottar Johnsen, Stefano S. Cavaglieri, et al, Sound Archive Project, P. J. Boltryk, J.W.McBride, M. Hill, A. J. Nasce, Z. Zhao, and C. Maul). However, all of the techniques investigated so far present some limits (optical resolution, image processing, etc.), resulting in poor sound quality, if compared to using standard mechanical devices. A typical application for optical retrieval technology is for records in very bad condition, where mechanical replay devices would fail, or where the recordings are so fragile that the replay process would cause unacceptable damage.


2 1 mil is .001" (1,000th of an inch)

5.2.5 Speed

5.2.5.1 Despite being referred to as “78s”, it was very often the case that coarse groove shellac discs were not recorded at precisely 78rpm, and this is especially the case with recordings made prior to the mid-1920s. At different times certain recording companies would set different official speeds, and even these were varied by recording engineers, on occasion during recording sessions. There is insufficient space here to discuss specific settings, though they are covered elsewhere in detail (see for example Copeland 2008, Chapter 5).

5.2.5.2 It is imperative that the disc be replayed for transfer as close to the original recording speed as is possible, in order to recapture the sound event originally recorded as faithfully and objectively as possible. However, subjective decisions often have to be made, and to this end knowledge of the recorded content or context in which the recording was made can be useful. The chosen replay speed should be documented in accompanying metadata. This is particularly important where any doubt remains as to the actual recording speed.

5.2.5.3 Recording speeds of commercial replicated cylinders standardised at 160rpm around 1902, although prior to that Edison, at least, applied several short-lived speed standards (all lower than 160 rpm; see Copeland 2008, Chapter 5). Instantaneous cylinders, while often recorded around 160 rpm or so, have been found with recording speeds ranging from below 50 rpm to over 300 rpm. In the absence of a recorded reference pitch (as provided occasionally by some early recordists) these will need to be set by ear, and documented accordingly.

5.2.5.4 Replaying a disc or cylinder at reduced speed may improve the ability to accurately track damaged carriers. There are many ways that this can be attempted depending on the equipment available, but attention should always be paid to the effect this will have on the sample rate of the digital file when adjusted to compensate for the change, and an appropriate sample rate should be chosen accordingly. Half-speed replay may be the simplest to employ, as it can be coupled with a doubled sample rate to produce corrected-speed transfers with a minimum of distortion caused by sample rate conversion. It should be noted that reduced speed playback is just one of many techniques that may be used to solve tracking problems. It is useful to try other procedures first such as adjusting the anti-skate to counter-balance the direction that the stylus jumps from a skip or using more or less tracking force to keep the stylus in the grooves.

5.2.5.5 Although playback with reduced speed may deliver increased surface noise compared with original speed, it is also the case that the action of filtering equipment, digital or otherwise, may be more effective. Playing at reduced speed means that the high frequency signal is halved in frequency, while the rise time of the unwanted impulse noise caused by surface damage remains the same and can be more easily distinguished from each other. However, some sophisticated predictive filtering equipment may be less effective at non- original speeds. Low speed copies must be flat transfers, without applied equalisation which can be introduced later (see 5.2.6).

5.2.6 Replay equalisation

5.2.6.1 Equalisation became a possibility with the introduction of electrical recordings; it also became a necessity. Equalisation in recording is the application of a frequency dependant boost or cut to the signal before it is recorded, and the inverse cut or boost on replay. This became a possibility with electrical recordings because the recording and replay systems now included electrical circuitry which enabled a process which could not have been applied in the acoustic recording process. It became a necessity because the way sound is represented on a disc would not allow the dynamic range or frequency response that the electrical technology enabled, to be recorded otherwise.

5.2.6.2 Sound can be recorded on a disc in two different ways;”constant velocity” or “constant amplitude”. Constant velocity on a disc is when the transverse speed of the stylus remains constant regardless of the frequency. An ideal acoustic disc recording would display constant velocity characteristics throughout its recordable range. One of the implications of constant velocity is that the peak amplitude of the signal is inversely proportional to the frequency of that signal, which means that high frequencies are recorded with small amplitudes, and low frequencies are recorded with comparatively large amplitudes. The difference in amplitude can be very marked; across 8 octaves, for example, the ratio in amplitude between the lowest and highest frequency is 256:1. At low frequencies, constant velocity is unsuitable as the excursion of the groove becomes excessive, reducing the amount of available recording space, or causing cross over between tracks.

5.2.6.3 Constant amplitude, on the other hand, is when the amplitude remains constant regardless of the frequency. Constant amplitude, while most suitable for low frequencies, is unsuitable for higher frequencies as the transverse velocity of the recording or replay stylus could become so excessive as to cause distortion. To overcome the dilemma caused by both these approaches, disc manufacturers recorded electrical discs with more or less constant amplitude at the lower frequencies and constant velocity at the higher frequencies. The point of change between the two is described as the low frequency turnover (see table 5.2).

5.2.6.4 As the recording technology improved and increasingly higher frequencies could be captured, these higher frequencies resulted in correspondingly smaller amplitudes on the disc. A consequence of the very small amplitude of these high frequency components is that the ratio of the signal to the irregularity in the surface of the disc approaches equivalence. This would mean that the very high frequencies would be comparable in amplitude to the unwanted surface noise, otherwise known as a poor signal to noise ratio. To overcome this, the disc manufacturers began to boost the higher frequency signals so that these very high frequencies were often, though not always, constant amplitude recordings. The point at which the higher frequencies are switched from constant velocity to constant amplitude is called HF Roll-off Turnover (see table 5.2). The function of this higher frequency equalisation is improvement in the signal to noise ratio, and it is commonly termed pre-emphasis in recording and de-emphasis in replay.

5.2.6.5 The commonly used dynamic or magnetic pick-ups are velocity transducers, and their output can be directly fed into a standard preamplifier, if that is desired. Piezo-electrical and optical replay systems are amplitude transducers. In these cases a general 6dB/octave slope equalisation must be applied as the difference between a constant velocity and constant amplitude recording is 6dB per octave.

5.2.6.6 Acoustically recorded discs have no intentionally applied equalisation in recording (though engineers were known to adjust parts of the recording path). As a consequence of the recording process, the spectra of an acoustic disc would display resonant peaks in amplitude and related lows. Applying a standard equalisation to compensate for the acoustic recording process is not possible as resonances in the recording horn and the stylus diaphragm, not to mention other mechanical damping effects, can vary between recordings, even recordings from the same session. In such cases the recordings should be replayed flat, i.e. without equalisation, and equalisation should be applied after the transfer has been made.

5.2.6.7 With electrical recordings it is necessary to decide whether to apply an equalisation curve on replay, or to transfer flat.Where the curve is accurately known equalisation may be applied either at the preamplifier prior to making the copy, or applied digitally after making a flat copy.Where doubt remains as to the correct equalisation curve, a flat transfer should be made. Subsequent digital versions may employ whichever curve seems most appropriate, so long as the process is fully documented, and the flat transfer retained as the archival master file.Whether or not equalisation is applied during the initial transfer, it is imperative that noise and distortion from the analogue signal chain (everything between the stylus and analogue-to-digital converter) is kept to an absolute minimum.

5.2.6.8 It is worth noting that a flat transfer will require around 20dB more headroom than one where an equalisation curve has been applied. However, as the potential dynamic range of a 24 bit digital to analogue convertor exceeds that of the original recording, the extra 20dB headroom can be accommodated.

5.2.6.9 Apart from the dynamic range limitations mentioned above, a drawback with transferring electrically recorded discs without de-emphasis is that stylus selection is primarily made through aural assessment of the effectiveness of each styli, and it is more difficult, though not impossible, to make reasonable assessment of the effect of different styli while listening to unequalised audio. An approach taken by some archives is to apply a standard, or house, curve to all recordings of a particular type in order to make stylus selection and other adjustments, and subsequently produce a simultaneous flat and equalised digital copy of the audio. As the exact equalisation is not always known, a flat1 copy has the advantage of allowing future users to apply equalisation as required, and is the preferred approach.

5.2.6.10 There is some debate as to whether noise reduction tools for the removal of audible clicks, hiss etc are more effective when used before an equalisation curve is applied rather than afterwards. The answer very likely varies according to the specific choice of tool and the nature of the job to which it is applied, and in any event will be subject to change as tools continue to evolve. The most important point in this regard is that noise reduction equipment, even fully automated tools with no user-definable parameters, ultimately employs subjective and irreversible processes, and so should not be used in the creation of archival master files.

5.2.6.11 A complete record of all decisions made, including choice of equipment, stylus, arm, and equalisation curve (or its absence) must be recorded and maintained in metadata.

5.2.6.12 The main equalisation curves for replay are listed below.

Equalisation Chart for Electrically
recorded coarse groove
(78 rpm) Discs
LF Turnover2 HF Roll-off
Turnover (-6 dB/octave,
except where marked)
Roll-off @ 10 kHz
Acoustics 0   0 dB
Brunswick 500 Hz (NAB)   0 dB
Capitol (1942) 400 Hz (AES) 2500 Hz -12 dB
Columbia (1925) 200 Hz (250) †5500 Hz (5200) -7 dB (-8.5)
Columbia (1938) 300 Hz (250) 1590 Hz -16 dB
Columbia (Eng.) 250 Hz   0 dB
Decca (1934) 400 Hz (AES) 2500 Hz -12 dB
Decca FFRR (1949) 250 Hz 3000 Hz* -5 dB
early 78s (mid-’30s) 500 Hz (NAB)   0 dB
EMI (1931) 250 Hz   0 dB
HMV (1931) 250 Hz   0 dB
London FFRR (1949) 250 Hz 3000 Hz* -5 dB
Mercury 400 Hz (AES) 2500 Hz -12 dB
MGM 500 Hz (RIAA) 2500 Hz -12 dB
Parlophone 500 Hz (NAB)   0 dB
Victor (1925) 200–500 Hz †5500 Hz (5200) -7 dB (-8.5)
Victor (1938–47) 500 Hz (NAB) †5500 Hz (5200) -7 dB (-8.5)
Victor (1947–52) 500 Hz (NAB) 2120 Hz -12 dB

Table 1 Section 5.2 Equalisation Chart for Electrically Recorded Coarse Groove (78 rpm) Discs3.

* 3 dB/octave slope. N.B.A 6 dB/octave slope should not be used on these marked frequencies because though it may be adjusted to give the correct reading at 10kHz, rolloff would commence at the wrong frequency (6800 Hz) and be incorrect at all other frequencies.

This only a recommended roll-off in order to achieve a more natural sound. The pronounced HF content is probably due to resonant peaks of the microphone and not due to the recording characteristic.


1 Flat is generally taken to mean the unequalised output from a velocity type pickup

2   See Table 2, Section 5.3, footnote 5 for definitions of “Turnover” and “Rolloff”.
3   Ref: Heinz O. Graumann: Schallplatten-Schneidkennlinien und ihre Entzerrung, (Gramophone Disc-Recording Characteristics and their Equalizations) Funkschau 1958/Heft 15/705-707. The table does not include every curve ever used, and other reputable sources vary slightly in their description of some of those listed. Research in this area is ongoing, and readers may wish to compare with other findings, such as Powell & Stehle 1993 or Copeland 2008, Chapter 6 etc.

5.2.7 Corrections for Errors Caused by Misaligned Recording Equipment

5.2.7.1 Any misalignment in the cutting stylus should ideally be replicated in the alignment of the replay stylus, in order to follow the cutter movement as closely as possible, and so capture as much information from the groove as accurately as possible. There are several ways in which a cutter may have been misaligned, most of which are difficult to identify, quantify and correct. However the most common misalignment is somewhat easier to identify and deal with. This occurs when a flat cutter has been mounted off its major axis, resulting in a recording which, when played with an on-axis elliptical stylus, reproduces a delay between channels. If the elliptical stylus cannot be rotated to match the cutter angle, (by appropriately mounting the pick up), replay using a conical stylus may ameliorate the problem to some extent, though with a possible compromise in high frequency response. Otherwise the delay may be fixed later in the digital domain, subsequent to the initial archival transfer.

5.2.8 Calibration Discs

5.2.8.1 Calibrating an audio system involves applying a defined input and measuring the corresponding output over a range of frequencies. A pre- amplifier/equaliser may be calibrated by supplying the input with a constant signal of variable frequency while loaded with the correct impedance, and the measurement consists in plotting (or data-logging) the output against frequency. Automatic apparatus exists for this. In use the input comes from a pickup cartridge, a transducer that converts a mechanical input to electrical output, and for this we need a mechanical calibrating signal.When mechanical recordings were commercially available test discs were produced for this purpose. The Audio Engineering Society (AES), via its Standardisation Committee, runs an ongoing and active project of developing and publishing a series of simple test discs, both for coarse groove work and for microgroove. The AES 78 rpm Calibration Disc Set: ”Calibration Disc Set for 78 rpm Coarse-Groove Reproducers. AES Cat. No. AES -S001-064” is available from the AES website. http://www.aes.org/standards/data/x064-content.cfm

5.2.8.2 If the calibration by means of a test disc has been performed with sufficient resolution, the plotted curve may be regarded as a plot of the transfer function of the pickup or the pickup-preamplifier¡equalizer combination. Apart from the fact that visual inspection of the curve will tell the operator of gross deficiencies, it may actually form the basis of a digital filter that may filter the digitised signal from the mechanical record, so that it becomes independent of the actual pickup (and preamplifier and equaliser) used. All it takes is to be certain that no adjustment has been changed between using the test disc and the mechanical record to be transferred (and ideally that the record materials for those two inputs behave the same way). (For further discussion see Brock-Nannestad 2000).

5.2.9 Office Dictation Systems

5.2.9.1 Sound recording technology has been marketed and used as a business tool virtually since its inception. Three broad categories of mechanical dictation formats can be defined, namely cylinders, discs and belts (see 5.4.15 for magnetic dictation formats).

5.2.9.2 Early cylinders and recording equipment sold for office use were generally the same as those used for other purposes, the resultant recordings being on standard 105 mm (4 1/8”) length cylinders (see 5.2.4.3). However cylinder formats designed specifically for office use were made for many years by both Columbia (later Dictaphone) and Edison, both producing cylinders approximately 155 mm (6 1/8 inch) long with 160 and 150 grooves/inch respectively (Klinger 2002). Some later cylinder dictation machines recorded electrically rather than acoustically, but little if anything is known today about pre- emphasis applied.

5.2.9.3 Various grooved disc formats were launched, mostly after World War II, including the Edison Voicewriter and the Gray Audograph.While many such formats require specialist replay equipment, seven inch flexible Edison Voicewriter discs may be replayed on a standard turntable employing a US-type spindle adaptor and microgroove stylus. Recording speeds for these were generally below 33 1/3 rpm.

5.2.9.4 Beginning in the 1940s, several belt recording formats appeared. These were essentially flexible plastic cylinders, fitted over a twin drum assembly for recording and playback. Perhaps the best known of these is the Dictaphone Dictabelt. Their flexibility allowed them to be flattened for storage and delivery much like other office stationery, but this often resulted in their becoming permanently creased, creating challenges for the replay engineer. Carefully and gently raising the temperature of the belt and replay equipment has been known to be effective in this regard, though how appropriate this is will depend on, among other things, the particular plastic used in the belt. Specialist replay equipment will be required to replay belt formats.

5.2.10 Time Factor

5.2.10.1 A complex transfer may easily take 20 hours for 3 minutes of sound (a ratio of 400:1). An average transfer may take 45 minutes for 3 minutes of sound (a ratio of 15:1), which represents time spent on finding the correct settings for the equipment and choice of stylus, based on an analysis of the recording as it relates to others of its time and storage history. Some experienced archives suggest that, for the transfer of unbroken cylinders in average condition, two technical staff, (one expert and one assistant) can transfer 100 cylinders per week (a ratio of about 16:1). Obviously experience will improve both the ratio and the ability to estimate time required.

5.2.10.2 Digitisation can seem expensive and labour intensive, requiring a great deal of equipment, expertise and man-hours to transfer audio and create all necessary metadata. However this initial front-loading of effort and resources will be offset by the long-term benefits and savings of retaining a well-managed digital mass storage repository, greatly reducing future costs of access, duplication and migration. Note that a crucial factor here is the maintenance of the repository, discussed in detail in chapter 6 and elsewhere. The extraction of the optimum signal from the original carrier, as defined in this chapter, is a vital component of this strategy.

5.3 Reproduction of Microgroove LP Records

5.3.1 Introduction

5.3.1.1 Long Play (LP) microgroove1 records first made their appearance around 1948, pressed in flexible vinyl2 and hailed as ‘unbreakable’ in comparison to the preceding commercial records pressed from a rigid (and easily broken) shellac base.

5.3.1.2 By the time the vinyl disc was developed there was a greater industry agreement on standards. Grooves were cut at 300-400 to the inch as opposed to the 100 or so grooves per inch that was characteristic of the shellac pressings, and with a standard sized and shaped stylus on a cutting lathe that revolved at a speed of 33 1/3 rpm. 7” vinyl records, both singles and ‘Extended Play’ (EP), were made to be replayed at 45 rpm and in some cases 33 1/3 rpm. Larger diameter discs were on rare occasions produced for replay at 16 2/3 rpm for speech, where up to 60 mins could be recorded on one side. Equalisation characteristics still varied between companies, (see Table 2 Section 5.3 Equalisation Chart for Pre-1955 LP Records) however, many preamps catered for these variations. Eventually agreement was reached and the RIAA (Record Industry Association of America) curve became standardised throughout the industry.

5.3.1.3 Stereo records were commercially available from around 1958, and initially many records were produced in both mono and stereo versions. The groove walls are at right angles to each other and inclined by 45º to the vertical. The inner wall of the groove contains the left channel information, and the outer groove the right channel information recorded perpendicular to the respective groove wall. This has remained the standard, although at the time of its introduction a small number of stereo discs were made with a combination of lateral and vertical technology, an approach that was soon discontinued. Stereo pick-ups may be used to play mono records, but playing a stereo record with a mono pick-up will cause severe groove damage.


1 As some late generation coarse groove recording were pressed in vinyl the use of the term “microgroove” is preferred to using “vinyl” as a collective description.

2 “Vinyl” is a colloquial term for the material of the discs which basically consists of a polyvinyl chloride / polyvinyl acetate co-polymer (PVC/PVA)

5.3.2 Selection of Best Copy

5.3.2.1 As with historical mechanical and other obsolete formats (see Section 5.2.2 Selection of best copy) selection is primarily made visually, for speed and to prevent wear. Staff should be well versed in the codes and identifiers used by the various record companies and usually placed just outside the label. This may reveal alternative or later takes, remasterings, or pressings. In selecting the best copies for digitisation, co-operation with other collections should be considered.

5.3.2.2 The working space must make parallel, oblique light available as overhead fluorescent lighting may obscure evidence of wear. The quality of light must be such that it is very clear what constitutes merely heavy modulation and what constitutes wear. If two copies only exist, and they display different wear characteristics, then retain both and transfer both.

5.3.3 Cleaning and Carrier Restoration

5.3.3.1 LPs should be handled very carefully, never allowing fingers to touch the groove area of any vinyl disc. Sweat and other skin borne deposits may in themselves cause replay noise, however they will also attract and adhere dust to the surface and enable the growth of moulds and fungi increasing replay noise. Cotton gloves should be worn when handling discs. If appropriate gloves are not practical, discs should be withdrawn from (and replaced in) their sleeves in a manner that ensures the finger tips are placed on the label area and the base of the thumb at the edge, leaving the groove area untouched.

5.3.3.2 Dust, the enemy of all sound recordings, is a major problem with LPs for two reasons. The finer groove means dust particles are comparable in size with the stylus and cause clicks and pops. The electrostatic nature of vinyl increases the attraction of dust to the surface of the disc.Various commercial devices have been developed in an attempt to neutralise these static charges, from carbon-fibre brushes to piezo-electric ‘guns’ that ‘fire’ a neutralizing charge at the record surface, all of which are effective to varying degrees.

5.3.3.3 The most effective way of cleaning records is to wash them. Cleaning machines, such as the well known Keith Monks machine, coat the surface with a cleansing fluid which is then removed by a tracking suction device which moves across the surface to suck up both the fluid and any dust or dirt in the grooves. A simpler method is washing, avoiding the label area, with demineralised water and a mild detergent or non-ionic wetting agent such as diluted (1 per cent) Cetrimide (n-cetyl pyridinium chloride) which has anti-fungal and anti- bacterial properties. The disc may then be brushed in a circular motion with a soft camel hair paint brush, again avoiding the label area, and rinsing off, once more using distilled water. Greasy deposits on vinyl discs may be removed with isopropyl alcohol. As non-vinyl discs may be affected by alcohol, care should be taken to ensure that the solvent does not cause damage to the disc.

5.3.3.4 Record cleaning solutions which do not disclose their chemical composition should not be used. All decisions about the use of solvents and other cleaning solutions should only be made by the archivist in consultation with the appropriate technical advice by qualified plastics conservators or chemists.

5.3.3.5 As with historical mechanical and other obsolete formats (see 5.2.3 Cleaning and Carrier Restoration), ultrasonic cleaning may be effective. Care should be taken in the selection of solvent, though a 1 per cent solution of Cetrimide in distilled water is an appropriate cleaning solution. The label should be kept clear of the fluid, and the disc rotated slowly until the whole groove area has been wetted.

5.3.3.6 Perhaps the most effective method of reducing the effects of dirt, dust, and static charge is to play the records wet. This may be achieved by simply covering the disc with a Cetrimide solution, or by tracking a soft wet brush ahead of the stylus.Wetting the record can dramatically reduce the incidence of clicks and pops, however, it has the effect of increasing surface noise in all subsequent ‘dry’ plays. Wet playing using liquids containing alcohol is not recommended as the polymer bearings of cantilevers may chemically react with negative results.

5.3.3.7 The most frequently needed restoration of a disc recording is flattening. The following approach applies whether the disc is dish-shaped or bent. A thermostatic oven (a laboratory style oven is mandatory, a domestic oven is not appropriate) is required at a setting usually not exceeding 55º C and provided with two very clean sheets of hardened and polished glass, thickness 7 mm, 350 mm square. After hand cleaning and drying the record it is placed on the pre-heated bottom sheet in the oven and the top sheet is suspended in the oven. After ca. half an hour the record is inspected and may well have sunk to a flat position. If not, the elasticity is tested as an indication of softening, and experience will tell if placing the hot top plate on the record might have the desired effect. The sandwich is left for half an hour, and the top sheet is lifted using gloves. If the record is perfectly flat, the complete sandwich is removed from the oven and left to cool on an insulating support. If flattening has not been obtained, the temperature is raised in 5º C intervals and the procedure repeated. Never apply the flattening force unless the softening is sufficient.

5.3.3.8 Flattening discs is a useful process because it can make unplayable discs playable; however, current research into the procedure of flattening discs with heat shows that it causes a measurable rise in subsonic frequencies, and even in the low audible frequency range (Enke 2007). Though the research is not conclusive the point should be noted in determining whether to flatten a particular disc. The analysis of the affect of flattening was carried out on vinyl discs but the range of tests were not extensive and further research is required. The possibility of such damage should be weighed against the benefit of enabling the playing of the disc.

5.3.4 Replay Equipment

5.3.4.1 Optical replay is available for LPs and should be investigated before selecting any transfer equipment, however contact transducers, or styli, are presently more common, perceived as less complicated and preferred by most technicians.When using contact transducers there are so many variables in the reproduction chain that exact repeatability of any particular replay is not possible. Pick-up arm, cartridge, stylus, tracking force, previous groove deformation or wear all contribute to the variability in replay. Even temperature can affect the replay characteristics of a cartridge/stylus combination to some degree. However, if LPs are to be captured for digitisation high quality components in the playback chain from stylus to recording equipment will ensure the most accurate audio capture.

5.3.4.2 Perhaps the most important part of the replay chain is the cartridge/stylus combination. Moving coil pickups, considered by some to be the most sensitive, tend to have a price tag and lack of robustness that precludes their use for anything but very careful domestic use. A good,high compliance,low tracking force (less than 15 mN, commonly quoted as 1.5 grams) variable reluctance (moving magnet) cartridge with a bi-radial (“elliptical”) stylus will be the most practical choice.Replay styli should include a range from 25 µm (1 mil), commonly used on early mono LPs, to 15 µm (0.6 mil), including conical, elliptical and truncated styli depending on the age and condition of discs to be played.

5.3.4.3 Attention should be given to the adjustment of vertical tracking angle (VTA) of the pickup system, which ideally should match the VTA produced in the recording process. The recommended playback VTA during the 1960s was 15±5º, which changed by 1972 to 20º±5º. It is impossible, however, to check the VTA of a given record (unless with test records which permit the evaluation of the intermodulation distortion of a vertical signal). As a basic adjustment, however, attention should be given to the horizontal position of tone arm, parallel to the surface of the record, under the appropriate tracking force. This should ensure the VTA intended by the pick-up system manufacturer. Any deviation from there can be achieved by lifting or lowering the tone arm.

5.3.4.4 Another angle to be adjusted is the tangential tracking angle (TTA).With tangential tone arms it must be insured that the system is mounted to lead the stylus exactly along the radius of the disc. With conventional (pivoted) tone arms a compromise must be made by adjusting the position of the stylus (= effective tone arm length) with the help of gauge, generally supplied by precision equipment manufacturers.

5.3.4.5 A high quality, low noise preamp capable of reproducing the standard RIAA curve as well as reproducing a flat transfer of the audio will be required. If pre-1955 records are being transferred, then a preamp capable of coping with the equalisation variations listed in Table 2 Section 5.3 Equalisation Chart for Pre-1955 LP Records, may be necessary. Multiple setting preamplifiers are not readily available, and it may be preferable to modify the equalisation after the normal preamp output, or applying custom equalisation to a flat transfer in the digital domain.

5.3.4.6 Vital to calibrating the replay chain is a test record cut with the recording characteristics of the records being transferred, and adjusting the frequency band of a graphic or parametric equaliser to achieve the proper output. An accurate RIAA test disc can be used to calibrate the system for non RIAA equalisation providing the characteristics of the replay curve are known. Finding an appropriate test record may prove difficult and even if available, older test records can suffer from wear and no longer give an accurate response, especially at the higher frequencies.

5.3.4.7 The vast range of playback components available in the 1960s and 1970s is no longer offered, and whilst not as difficult to locate as replay equipment for 78s, a much more limited range is now available. Though relatively impervious to damage and decay, LPs can become inaccessible if suitable replay equipment becomes unavailable. Although a good stock of spares and consumables is recommended for medium term access, it is important to note that styli and assemblies do not have an infinite shelf life.

5.3.5 Speed

5.3.5.1 Adherence by the recording companies to the standards reduced concern regarding speed setting that was common with earlier formats. A turntable equipped with strobe measurement and manual adjustment of speed is recommended as a minimum to ensure replay equipment complies with standards. The use of a crystal oscillator drive is preferable.

5.3.6 Replay Equalisation

5.3.6.1 The need for equalisation and the manner in which it was developed is explained in Section 5.2.6. Equalisation is also applied to microgroove recordings and primarily involves reducing the level of frequencies below about 500 Hz which is the LF turnover below which the recording is constant amplitude, and boosting those above about 2 kHz. Between 500 Hz and 2 kHz the recording is characterised by constant velocity (see 5.2.6). The application of equalisation in the recording process has to be compensated for in the replay chain. Many companies had their own, usually minor, variations on this theme, and for accurate reproduction, exact replay equalisation needs to be applied (see Table 1 Section 5.3 below).

5.3.6.2 Records made after about 1955 complied with what is now known as the RIAA (Record Industry Association of America) curve which became a well observed standard throughout the industry. RIAA replay characteristics are defined by a replay cut of 6 dB/octave from 20 Hz to 500 Hz, a flat shelf between 500 Hz and 2.12 kHz (318 µs and 75 µs respectively) and a 6 dB/Octave treble cut from 2.12 kHz. The flat shelf is approximately 19.3 dB below zero.

5.3.6.3 The Equalisation curves for replay are listed below.

 

Equalisation Curves
by Name
LF Roll-off LF Turnover HF Roll-off Turnover
(-6 dB/octave, except
where marked)
Roll-off @ 10 kHz
AES 50 Hz 400 Hz (375) 2500 Hz -12 dB
FFRR (1949) 40 Hz 250 Hz 3000 Hz* -5 dB
FFRR (1951)   300 Hz (250) 2120 Hz -14 dB
FFRR (1953) 100 Hz 450 Hz (500) 3180 Hz (5200) -11 dB (-8.5)
LP/COL 100 Hz 500 Hz3 1590 Hz -16 dB
NAB   500 Hz 1590 Hz -16 dB
Orthophonic (RCA) 50 Hz 500 Hz 3180 Hz (5200) -11 dB (-8.5)
629   629 Hz (750)    
RIAA 50 Hz 500 Hz4 2500 Hz -13.7 dB

Table 1 Section 5.3 Equalisation Curves by Name

 

Equalisation Chart for
Pre-1955 LP Records5
LF Roll-off LF Turnover HF Roll-off Turnover
(-6 dB/octave, except
where marked)
Roll-off @ 10 kHz
Audio Fidelity   500 Hz (NAB) 1590 Hz -16 dB
Capitol   400 Hz (AES) 2500 Hz -12 dB
Capitol-Cetra   400 Hz (AES) 2500 Hz -12 dB
Columbia   500 Hz (COL) 1590 Hz -16 dB
Decca   400 Hz (AES) 2500 -12 dB
Decca (until 11/55) 100 Hz 500 Hz (COL) 1590 Hz (1600) -16 dB
Decca FFRR (1951) 3dB slope   300 Hz (250) 2120 Hz -14 dB
Decca FFRR (1953) 3dB slope   450 Hz (500) 2800 Hz -11 dB(-8.5)
Ducretet-Thomson   450 Hz (500) 2800 Hz -11 dB(-8.5)
EMS   375 Hz 2500 Hz -12 dB
Epic (until 1954)   500 Hz (COL) 1590 Hz -16 dB
Esoteric   400 Hz (AES) 2500 Hz -12 dB
Folkways   500 Hz (COL) 1590 Hz -16 dB
HMV   500 Hz (COL) 1590 Hz -16 dB
London (up to LL-846) 100 Hz 450 Hz (500) 2800 Hz -11 dB(-8.5)
London International 100 Hz 450 Hz (500) 2800 Hz -11 dB(-8.5)
Mercury (until 10/54)   400 Hz (AES) 2800 Hz -11 dB
MGM   500 Hz (NAB) 2800 Hz -11 dB
RCA Victor (until 8/52) 50 Hz 500 Hz (NAB) 2120 Hz -12 dB
Vox (until 1954)   500 Hz (COL) 1590 Hz -16 dB
Westminster (pre-1956)
or
  500 Hz (NAB)
400 Hz (AES)
1590 Hz
2800 Hz
-16 dB
-11 dB

Table 2 Section 5.3 Equalisation Chart for Pre-1955 LP Records


3. modified from NAB: less bass below 150 Hz, requiring about 3 dB boost.

4. RIAA and NAB are very similar.

5. This information is taken from several sources: the “DialYour Discs” chart which appeared in High Fidelity magazine during the early 1950s, the chart compiled by James R. Powell, Jr. and published in the ARSC Journal, and the jackets of various early LPs. “Turnover” (col. 2) is the frequency below which the record manufacturer diminished the bass when mastering the disc, requiring a corresponding boost during playback. In the chart, turnover is stated using the name of the recording curve, as given on most older pre-amps; a list of these curves and their turnover frequencies is at the end of the chart. ”Roll-off”(col.3) is the amount of treble cut at 10kHz required during playback to compensate for pre-emphasis added during disc mastering. In the chart, roll-off is stated in dB.

5.4 Reproduction of Analogue Magnetic Tapes

5.4.1 Introduction

5.4.1.1 Analogue magnetic tape recording technology has permeated every area of the recording industry since its mass distribution and popularisation in the post WWII era. Technological advancements made tape the primary recording format for professional recording studios, and manufacturing developments made the reel recorder affordable for the domestic market. The introduction of the Philips Compact Cassette in 1963 put a recording device within the grasp of many people and it became possible and practical for people to record whatever seemed important to them.Virtually every sound archive and library holds analogue magnetic tape recordings, and PRESTO (Wright and Williams 2001) estimates there are over 100 million hours of analogue tape recordings in collections throughout the world, a figure in no way contradicted by the IASA survey of endangered carriers (Boston 2003). Since the 1970s sound archivists recommended quarter inch analogue reel tape as the preferred archival carrier, and in spite of inherent noise and impending chemical decay, some still stand by them today as a stable carrier. Nonetheless, the imminent demise of the analogue tape industry and the consequent and almost total cessation of the production the replay equipment demand that immediate steps be taken to transfer this vast store of recorded cultural history to a more viable system of management.

5.4.1.2 Magnetic tape was first made commercially available in Germany in 1935, but it was the commercialisation of the American market after 1947 that drove its popularity and eventual standardisation. The first tapes were manufactured on a cellulose acetate backing and this continued until the introduction of polyester (polyethylene terephthalate PET, commercially known as Mylar). Tape manufacturers produced both acetate and PET tapes with an acetate binder, which was gradually, and most commonly, replaced from the late 1960s by a polyester urethane binder. BASF manufactured tapes on PVC from the mid 1940s until 1972, though it gradually introduced its own range of polyester from the late 1950s onward. Though PVC was primarily the province of the German manufacturer BASF, 3M also produced a PVC tape from around 1960; Scotch 311. Rarer are paper backed magnetic tapes, which date from the late 1940s to the early 1950s. Cassette tapes have always been manufactured on polyester. In 1939 the magnetic pigment used was γFe2O3, often called the oxide, and although subsequent improvements in particulate size, shape and doping increased performance and reduced noise, this formulation has remained virtually the same for almost all analogue reels and type I cassettes. Type II cassettes are CrO2 or cobalt doped Fe3O4, III (rarely encountered) are dual layered with both γFe2O3 and CrO2 and IV are metal (pure iron).

5.4.1.3 The materials that bind the magnetic particles to the tape substrate, called binders, are often identified as that part of the tape most susceptible to chemical breakdown. This is especially so with polyester urethane binder tapes which most commonly use a PET substrate from the 1970s, though AGFA and BASF and their subsequent owners, Emtec, used a PVC based binder on many of their studio and broadcast tapes, notably 468.

5.4.2 Selection of Best Copy

5.4.2.1 Recordable media such as magnetic tape tend not to have multiple copies of the same generation. With the exception of cassette, audio on tape was only infrequently mass replicated and so the sound archivist must choose between generational duplicates. As a rule, the most original copy is the best copy to select for the purposes of preservation. However, the original tape may have suffered some form of physical or chemical degradation, such as hydrolysis, whereby a duplicate made in accordance with proper procedure prior to that decay might be better. Tape rarely shows visible signs of decay or damage so, where multiple copies of an item exist, the best approach is to carefully spool through, and then audition the tape to determine the best copy.

5.4.2.2 Curatorial decisions must also be made to ensure that the most appropriate or complete duplicate is selected. This is primarily an issue where the tapes have been produced as a result of a sequential production process such as audio mastering or in the production of sound for film or video.

5.4.3 Cleaning and Carrier Restoration

5.4.3.1 Tape Cleaning: Dirty or contaminated tapes should be cleaned of dust and debris with a soft brush and low vacuum before spooling. Deformed reels may seriously damage tapes, especially in the fast winding mode, and must be replaced before any further steps are carried out. The tape should be carefully spooled guiding the tape so as not to cause damage. The tape may then, if necessary, be spooled on a tape-cleaning machine that has a soft cloth or other lint free material cleaning surface. This may also be beneficial after treatment for hydrolysis (see below). Some tape cleaning or restoration machines pass the tape across a sharp surface or blade, which removed the top layer of oxide. Such machines were developed for the re-use of recorded tapes and are not recommended for archival tapes. Special attention should be paid to dirty cassette tapes as some reputable double capstan machines may damage dirty tapes during replay.Without adequate tape tension control a loop may develop between the capstans.

5.4.3.2 Leader Tapes and Tape Splices: Many tapes have splices either through editing or the addition leader tapes. Such splices are likely to have failed, either through dry failure of the adhesive, or bleeding of the adhesive layer. The former must be replaced. Bleeding splices constitute a more serious problem. The adhesive may spread from the splice to the adjacent layers which may have encouraged the dissolution of the binder. It may also cause the layers to adhere to each other and increase speed fluctuations. Old adhesive must be removed using a solvent that does not damage the binder. Highly purified light fuel is an appropriate solvent and may be applied using a Q-tip or lint free cloth. It is advisable to keep the amount applied to the tape to the minimum required, and no more than would be applied with a Q-tip. As with all solvents, a small amount should be tested on an unused portion of the tape. The tape should be left unwound for a few minutes to ensure full evaporation. Evaporation may be accelerated by an air stream. It is sometimes necessary to replace or add leader tape to enable the complete recording on the tape to be played.

5.4.3.3 Hydrolysis (Sticky Shed Syndrome): When replayed, many of the tapes manufactured since the 1970s show the artefacts of a chemical breakdown of the binder. Often described as sticky shed syndrome, the main component of the reaction is hydrolysis1 , by which term it is often described. It is typified by a sticky brown or milky deposit on tape heads and fixed guides, often accompanied by an audible squeal and reduction in audio quality.

5.4.3.4 The following treatments represent various approaches to the treatment of binder degradation:

5.4.3.4.1 Room Temperature, Low Humidity: Hydrolysis involves the splitting of a chemical bond through the introduction of water, and providing that an irreversible recombination has not subsequently occurred, hydrolytic reactions should be reversible through the simple process of removing all water. This can be achieved by placing the tapes in a chamber approaching 0% relative humidity (RH) for extended periods of time, up to several weeks. Slightly elevating the temperature increases the reaction time. Tests have shown that this treatment, while successful in some cases does not always completely reverse all the artefacts of a degraded tape (Bradley 1995).

5.4.3.4.2 Heated Respooling: Sometimes very degraded tapes may bind one layer upon another and uncontrolled spooling may cause damage. In such cases, if baking is not being undertaken, it may be possible to apply warm dry air directly to the point in the tape pack where the tape is sticking, and then commence to unspool the tape at a controlled rate of 10-50 mm per minute.

5.4.3.4.3 Elevated Temperature, Low Humidity: An approach commonly used in the treatment of hydrolysed tapes is heating the tape in a chamber at a stable temperature approaching 50 ºC and 0% RH for period of around 8- 12 hours. The temperature of 50 ºC probably equals or exceeds the glass transition temperature2 of the tape binder, however, whether that has a long term effect on the physical characteristics of the tape when returned to room temperature is unclear. It does, however, have a positive short term electro-acoustic effect by returning the replay characteristics to original condition. Interleaving with new tape may be of benefit in reducing the level of print activity, which can be activated by temperature increases. Tapes should be rewound a number of times to reduce the effects of print through caused by elevated temperatures (see 5.4.13.3).

5.4.3.4.4 This latter procedure has a high success rate, but should not be carried out in a domestic oven. Domestic ovens have poor temperature control, which may exceed safe thresholds. Additionally the thermostat control of such ovens cycles back and forward across of range of temperatures and this action may damage the tape. A microwave oven should never be used as it heats small parts of the tape to very high temperature and may damage the tape and its magnetic characteristics. A laboratory oven is preferred, or other stable low temperature device. Higher temperatures should never be allowed as these may cause deformation of the tape.

5.4.3.5 Exposing tapes to controlled, elevated temperatures as described above should be undertaken very carefully and only where absolutely necessary.

5.4.3.6 Restoration may be only temporary, yet should enable replay for transfer. Anecdotal evidence is that hydrolysed tapes which require longer treatment are becoming more prevalent.


1. Hydrolysis: A chemical decomposition by addition of water, or a chemical reaction in which water reacts with a compound to produce other compounds

 2. Glass Transition Temperature;That temperature at which an adhesive loses its flexibility and becomes hard, inflexible, and “glasslike.”

5.4.4 Replay equipment: Professional Reel Machines

5.4.4.1 As analogue reel tape has been the mainstay of the sound recording and archiving community for decades the virtual cessation of the manufacture of reel player/recorders is a major crisis in the sound archiving community. Very few new professional tape machines are currently available from manufacturers, possibly only from Otari who continue to make a single machine, which may be described as the third generation of their mid-range model when compared to their earlier range, and Nagra Kudelski, who still list two portable field recording analogue tape machines as available. Not all machines meet the necessary replay specification (below) and archives must check for compliance before making a purchase. The alternative is to purchase and restore second hand machines, and the market in high end analogue reel machines is quite strong. It is recommended that only widely used machines should be purchased as this will facilitate the acquisition of parts and maintenance. The characteristics of a suitable archival reel machine include the following:

5.4.4.2 Reel Replay Speeds: The standard tape speeds are as follows: 30 ips (76.2 cm/s), 15ips (38.1 cm/s), 7.5 ips (19.05 cm/s), 3¾ ips (9.525 cm/s), 1 78 ips (4.76 cm/s) and 1516 ips (2.38 cm/s). The need to replay all these speeds will depend on the makeup of the individual collections. No single machine will play all 6 speeds, but it is possible to cover all speeds with two machines.

5.4.4.3 Mono and stereo 1/4 inch recording equipment come in 3 basic track configurations; full track, 1/2 track and 1/4 track. There are variations in the actual track width according to the particular standard. A tape replayed with a head with less replay width than the actual recorded track width will exhibit an altered low frequency response known as the fringe effect, and show poorer signal to noise than optimum. So a recorded track width of 2.775mm replayed with a 2mm stereo head will result in a loss of signal to noise ratio of approximately 2dB. The fringe effect is of the order of about +1dB at 63 Hz at 19.05 cm/s (7.5 ips) (McKnight 2001). A tape replayed with a head with a wider replay width than the actual recorded track width will exhibit slightly worse signal to noise and may pick up unwanted hiss or signal from adjacent tracks.”It amounts to the ratio of 1.9 mm to 2.1 mm, corresponding to a 1 dB level shift for these head widths; or 1.9 mm to 2.8 mm, corresponding to 3.3 dB for these widths.” (McKnight 2001) In practice these compromises are often accepted for small variation in track width in replay provided no unwanted signal is included (note that the unrecorded portion of previously erased tape may exhibit higher noise levels). Though some machines may include half track and 1/4 track replay heads, it may be necessary to have more than one machine to deal with these standards.

full track head configuration

  A B
IEC1 94-1
(pre 1985)
6.3 mm,
(0.248 in)
6.3 mm,
(0.248 in)
NAB 1965 6.3 mm,
(0.248 in)
6.05 mm
(0.238 in)
IEC 94-6
1985
6.3 mm
(0.248 in)
5.9 mm
(0.232 in)

Fig 1. section 5.4 full track head configuration and dimensions.


two track and half track head configurationtwo track and half track head configuration

  A maximum
recording
width1
B C
Ampex 6.3 mm,
(0.248 in)
6.05 mm
(0.238 in)
1.9 mm
(0.075 in)
2.14 mm
(0.084 in)
IEC 94-6
1985 2 track
6.3 mm,
(0.248 in)
5.9 mm
(0.232 in)
1.95 mm
(0.077 in)
2.00 mm
(0.079 in)
IEC home stereo (pre 1985) 6.3 mm,
(0.248 in)
6.3 mm,
(0.248 in)
2.0 mm
(0.079 in)
2.25 mm
(0.089 in)
NAB 1965 6.3 mm,
(0.248 in)
6.05 mm
(0.238 in)
2.1 mm
(0.082 in)
1.85 mm
(0.073 in)
IEC-1 Time code
DIN mono half track
6.3 mm,
(0.248 in)
6.3 mm,
(0.248 in)
2.3 mm
(0.091 in)
1.65 mm
(0.065 in)
IEC 94-6
1985 Stereo
6.3 mm,
(0.248 in)
5.9 mm
(0.232 in)
2.58 mm
(0.102 in)
0.75 mm
(0.03 in)
IEC-1 Stereo (pre 1985)
Mono half track
6.3 mm,
(0.248 in)
6.3 mm,
(0.248 in)
2.775 mm
(0.108 in)
0.75 mm
(0.03 in)
IEC
½ inch
12.6 mm
(0.496 in)
  5.0 mm
(0.197 in)
2.5 mm
(0.098 in)

Fig 2. section 5.4 two track and half track head configuration and dimensions.


 

quarter track head configuration

  A B C
IEC1
NAB
6.3 mm,
(0.248 in)
1 mm
(0.043 in)
0.75 mm
(0.43 in)

Fig 3 section 5.4 quarter track head configuration and dimensions.


Stereo Cassette head configuration

  A B C
IEC
Philips
3.81 mm,
(0.15 in)
0.6 mm
(0.02 in)
0.3 mm
(0.012 in)


Fig 4 Section 5.4 Stereo Cassette head configuration and dimensions.


Mono Cassette head configuration

  A B
ANSI
Philips
3.81 mm,
(0.15 in)
1.5 mm
(0.06 in)

Fig 5 Section 5.4 Mono Cassette head configuration and dimensions.


 

5.4.4.4 Head dimensions are specified in different ways in the European and US standards. Initially, the International Electrotechnical Commission (IEC), predominately referred to by European manufacturers, specified the tape with regard to the centre of the tape and the distance between the tracks, while the American based standards referred to the size of the recording track width defined diagrammatically with respect to one side. The size of the tape itself, has changed over time, initially a quarter of an inch, it was defined as 0.246 ± 0.002 inch (6.25 ± 0.05 mm) and later as 0.248 ± 0.002 in (6.3 ± 0.05 mm)”. IEC defines recording width in a full track recording in the following manner,”A single track shall extend over the whole width of the tape.” (IEC 94 1968:11), whereas the American based standards define the size of the recorded track to slightly less than the width of a 0.246 inch tape at 0.238 +0.010 -0.004 inch track size (this is a pragmatic solution to the problem of “grooves” in head wear and extends to all track dimensions). IEC later changed their full track width to 5.9 mm (0.232 inches). The number of standard track widths specified in figs.1 to 5 suggests that there is very little standardisation. (Eargle 1995, Benson 1988, IEC 94-1 1968, 1981, IEC 94-6 1985, NAB 1965 McKnight 2001, Hess 2001).

5.4.4.5 The net effect of replaying tapes on mismatched head widths is discussed in 5.4.2.2 above. It is important to attempt to assess the correct head width with which the original tapes were recorded and to then replay them on the most appropriate machine available. half” and 1” two track recordings are generally made in half track configuration only, and with specialised professional recording equipment with the intention of providing very high quality analogue audio. The same type and standard of equipment is required for replay, and an even closer attention to the detail of the record/replay standards.

5.4.4.6 Multitrack recordings range from domestic 1/4” standards to professional 2” and care must be taken to ensure the replay of those tapes is accurate. If time code has been recorded as part of the recording it must be captured and encoded in such a way that it may be used for later synchronisation (see 2.8 for file formats).

5.4.4.7 Tape machines should be capable of replaying signals with a frequency response of 30 Hz to 10 kHz ±1 dB, and 10 kHz to 20 kHz +1, -2 dB.

5.4.4.8 The equalisation on a reel replay machine should be capable of being aligned for replaying NAB or IEC equalisation, preferably being able to switch between them without re-alignment.

5.4.4.9 Wow and flutter unweighted better than 0.05% at 15 ips, 0.08% at 7.5 ips, and average variation from true speed better than 0.1%.

5.4.4.10 A professional archival reel machine should also have gentle tape handling characteristics so that it does not damage the tape during replay. Many of the early and middle generation studio machines depended on the robust characteristics of the modern tape carrier for their successful operation. These machines may cause damage to older tapes, or to long play tapes or thin tapes used for field recording.


1. Maximum recording width refers to the width measured from the outside edge of the outer tracks (see section 5.4.4.4)

5.4.5 Replay equipment: Professional Cassette Machines

5.4.5.1 Professional cassette replay machines are unavailable new. Also, the second hand market for professional cassette machines is not as strong as that for reel machines making it difficult to locate appropriate equipment. This represents a critical problem for sound archives, many of whose collections hold large numbers of recorded cassette tapes. Thus it should be a matter of priority for any collection with cassette tapes to seek out and acquire professional cassette replay machines. The characteristics that distinguish a professional machine from a domestic machine, apart from the replay specification, include solid mechanical construction, the ability to adjust replay characteristics and head azimuth, and the provision of balanced audio outputs. Many high quality audiophile machines provide some of the above characteristics. The characteristics of a suitable archival cassette replay machine include the following:

5.4.5.2 Replay speeds 17/8 ips (4.76 cm/s) (note that speeds of 15/16 ips and 3 ? ips may also be required for replay of specially recorded cassettes).

5.4.5.3 Variation from speed better than 0.3%.Wow and flutter weighted better than 0.1%.

5.4.5.4 Replay frequency response of 30 Hz to 20 kHz +2, -3 dB.

5.4.5.5 Ability to replay Type I, II, and IV cassettes (as required).

5.4.5.6 Most cassette machines will automatically select the correct replay equalisation by reading the holes or notches on the top of the cassette housing or shell to determine the tape type. A few machines do not read the notches but have a switch that the operator uses to select the appropriate equalisation. Type III cassettes may be problematic as they are enclosed in shells identical to Type I cassettes, while requiring the same replay equalisation curve as Type II cassettes.Where no explicit option to replay Type III has been provided by the playback machine, it may be necessary to use a deck with adjustable equalisation or to rehouse the tape in a Type II shell (see Section 5.4.12.5 Cassette Enclosures).

5.4.6 Maintenance

5.4.6.1 All equipment will require regular maintenance to keep it in working order. However, as analogue replay equipment is going out of production, it is necessary to make plans for spare parts as manufacturers will only maintain spare parts for a finite, and possibly short, period of time.

5.4.7 Alignment (equalisation below)

5.4.7.1 Analogue equipment requires regular alignment to ensure that it continues to operate within specification. It is recommended that heads and tape path be thoroughly cleaned every 4 hours of operation, or more frequently if required, using a suitable cleaning fluid such as isopropyl alcohol on all metal parts. Rubber pinch rollers should be cleaned with dry cotton buds or with cotton buds dampened with water as necessary. The older, original rubber pinch rollers can gradually become brittle if cleaned with alcohol, increasing wow and flutter. The new generation of polyurethane pinch rollers, generally coloured dark green, may dissolve if cleaned with alcohol. Heads and tape path to be demagnetised every 8 hours of operation, tape path and replay characteristics checked for alignment every 30 hours of use and equipment should receive a total alignment and check every 6 months.

5.4.7.2 In the same way that machines and tape are going out of production, suitable test tapes are likewise becoming difficult to obtain, and some are now unobtainable. It behoves the archivist to acquire enough open reel and cassette test tapes to manage the transfer of their collection.

5.4.8 Speed

5.4.8.1 Although speed correction is also possible in the digital domain, it is better to avoid such later digital correction and to carefully choose replay speed in the first transfer process, and to document chosen speed and justification. Tape recorders are very likely to have exhibited inaccurate speed characteristics due to fault, poor alignment, or in some cases, unstable power supply. Consequently no tape speed should be taken for granted.

5.4.9 Capstan-less Machines and Non-linear Speeds

5.4.9.1 Some early generation reel recording machines were designed to run without the control of the capstan and pinch roller, and consequently exhibit steadily increasing speed. If these tapes are played at a standard, unchanging speed, the resultant signal would decrease in pitch as the tape was replayed. To play the tape correctly the replay speed must change in the same manner as the recording speed. Some of the more recent replay machines, such as those made by Nagra or Lyrec, have incorporated a voltage driven external speed control which allows the operator to design a simple circuit with a curve that matches the speed of the original. Some of the last generation replay machines, such as the Studer A800 series, incorporated microprocessor control allowing for programmable manipulation of the speed, and others like the Lyrec Frida allowed the speed to be manipulated in the MIDI environment. However, care should be taken in assuming that the speed increase is linear. The early capstan-less machines were made cheaply and the speed varied according to the load on the reel, the speed increase is often less at the beginning or end of the tape where one or the other of the reels is full making a graph of the replay speed over time far from linear.

5.4.10 Replay Equalisation

5.4.10.1  The signal representation in most analogue audio formats is deliberately not linear in terms of frequency response. Correct replay, therefore, requires appropriate equalisation of the frequency response.

5.4.10.2  The most common of the equalisation standards for audio replay of analogue tape are as set out below (Table 1 Section 5.4). It should be noted that equalisations have developed over time. The current standards are given in bold type, together with their date of introduction. Earlier recordings must be replayed by applying the respective historical standards and simple additional circuits may be utilised. The overlapping of old and new standards should be taken into account when decisions are to be made for tapes recorded in times of transition. Prior to that there were a number of manufacturers’ standards.

30 ips, 76 cm/s IEC2
AES
(1981) current standard 17.5 μs
30 ips, 76 cm/s CCIR
IEC1
DIN
(1953–1966)
(1968)
(1962)
35 μs
15 ips. 38 cm/s IEC1
CCIR
DIN
BS
(1968) current standard
(1953)
(1962)
35 μs
15 ips. 38 cm/s NAB
EIA
(1953) current standard
1963
3180 μs 50 μs
7½ ips, 19 cm/s IEC1
DIN(studio)
CCIR
(1968) current standard
1965
1966
70 μs
7½ ips, 19 cm/s IEC 2
NAB
DIN(home)
EIA
RIAA
(1965) current Standard
(1966)
(1963)
(1968)
3180 μs 50 μs
7½ ips, 19 cm/s Ampex (home)
EIA (proposed)
(1967) 50 μs
7½ ips, 19 cm/s CCIR
IEC
DIN
BS
(up to 1966)
(up to 1968)
(up to 1965)
100 μs
3¾ ips 9.5 cm/s IEC2
NAB
RIAA
(1968) current standard
(1965)
(1968)
3180 μs 90 μs
3¾ ips 9.5 cm/s DIN (1962) 3180 μs 120 μs
3¾ ips 9.5 cm/s DIN (1955–1961) 200 μs
3¾ ips 9.5 cm/s Ampex (home)
EIA (proposed)
(1967) 100 μs
3¾ ips 9.5 cm/s IEC (1962–1968) 3180 μs 140 μs
3¾ ips 9.5 cm/s Ampex (1953–1958) 3180 μs 200 μs
17/8 ips 4.75 cm/s IEC
DIN
(1971) current standard
(1971)
3180 μs 120 μs
17/8 ips 4.75 cm/s IEC
DIN
RIAA
(1968–1971)
(1966–1971)
(1968)
1590 μs 120 μs
17/8 ips 4.75 cm/s
cassette
IEC Type I 1974 current standard 3180 μs 120 μs
17/8 ips 4.75 cm/s
cassette
DIN Type I (1968–1974) 1590 μs 120 μs
17/8 ips 4.75 cm/s
cassette
Type II and IV (1970) Current standard 3180 μs 70 μs
15/16 ips 2.38 cm/s undefined      

Table 1 Section 5.4 Common Equalisation Standards for Audio Replay of Analogue Tape4

 

5.4.10.3  At 15 ips and 7.5 ips there is a choice in replay equalisation for reel tapes even for tapes which were recently recorded according to the current standards. However, these are the two most common recording speeds, and care must be taken when choosing a replay equalisation to ensure that it corresponds with the record equalisation. Apart from the standards mentioned in table 1 section 5.4 there are a small number of more current standards intended to achieve better performance but which are different from the commonly accepted standards. At 15 ips Nagra tape recorders have the option to use a special equalisation called NagraMaster. The US version of NagraMaster had time constants ∞ and 13.5 µs, the European version of NagraMaster had time constants 8 and 13µs. Ampex used “Ampex Master Equalization” (AME), also at 15 ips but officially only on particular 1/2 inch mastering recorders introduced in 1958 and sold for several years following (MRL 2001). Logging machines and some popular semi-professional portable equipment were able to record at the very slow speed of 1516 ips (2.38 cm/s). However, it appears that there is no agreed exchange standard for these tapes and any equalisation would have adhered to proprietary conventions.

5.4.10.4  Sometimes any lack of documentation may require the operator to make replay equalisation decisions aurally. Cassette replay equalisation corresponds to the tape type, and care must be taken to ensure that the correct replay equalisation is used. Many tape recordings, specifically private recordings and those of cultural or research institutions that lacked technical support, have been made on un-aligned tape recorders. Unless there is objective evidence that would allow alternate settings, with regard to equalisation, tapes must be treated as properly aligned.


4. Note, IEC refers to IEC Pub 60094-1 4th edition, 1981, NAB to the NAB reel to reel standard 1965 (IEC2), or cassette standard 1973, DIN refers to DIN 45 513-3 or 45 513-4 and AES to AES-1971, and BS to the British Standard BS 1568). Thanks to Friedrich Engel, Richard L. Hess and Jay McKnight for generously supplying information on tape equalisation.

5.4.11 Noise Reduction

5.4.11.1 The signal recorded onto a tape may have been encoded in such a way as to mask the inherent noise of the carrier. This is known as noise reduction. If the tape has been encoded while recording, it must be decoded using the same type of decoder appropriately aligned. The most common noise reduction systems include Dolby A, and Dolby SR (professional), Dolby B and Dolby C (domestic), dbx types I (professional) and II (domestic)although rarely used and TelCom.

5.4.11.2 The alignment of the record and replay characteristics of the tape machine are critical to the adequate operation of noise reduction systems and characteristic line up tones are often included on professionally recorded tapes. The output level, as well as the frequency response can alter the response of the decoding system and it is also important to note that noise reduction may be applied to either IEC or NAB equalisation and must be replayed correctly. Dolby B and Dolby C have routinely been included in most professional cassette decks of recent years and generally do not have line up tones and have a less obvious effect on the signal than the professional systems.

5.4.11.3 Though it is possible to transfer the audio from an encoded tape for decoding at a later time, the multiple variables in alignment can compound the errors and make it difficult to decode accurately once the tape has been transferred. Decoding is better undertaken at the time of transfer.

5.4.11.4 Unless documented, it is difficult to assess whether compact cassettes have encoded with a noise reduction system. As with equalisation, the lack of documentation may require the operator to make such decisions aurally. The right replay is generally characterised by an even level of background hiss, while the fluctuation of this level indicates a wrong playback setting. A spectrum analysis tool can be helpful. If it cannot be determined, copies of cassettes should be made flat.

5.4.12 Corrections for Errors Caused by Misaligned Recording Equipment

5.4.12.1 Misalignment of recording equipment leads to recording imperfections, which can take manifold form.While many of them are not, or hardly correctable, some of these faults can objectively be detected and compensated for. It is imperative to take compensation measures in the replay process of the original documents incurred, as no such correction will be possible once the signal has been transferred to another carrier.

5.4.12.2 Azimuth and Tape Path Alignment: Inaccurate alignment of the record head of the original recording machine means that at replay, the signal retrieved will exhibit a reduced high frequency response, and, in the case of two or more track replay, an altered phase relationship between the two channels. Adjustment of the angle of the replay head such that the relationship of the head is in the same plane as the magnetised field on the tape is termed the azimuth adjustment and this simple adjustment can markedly improve the quality and intelligibility of the retrieved signal. There is no difficulty in training staff in this task, and good binaural hearing is all the measuring technology required. An accurate phase meter or oscilloscope will aid in the adjustment of mono and properly recorded tapes, they may, however, be misleading on tapes recorded on cheap, domestic equipment. In such cases aural judgement of the high frequencies should be relied on. Additionally or alternatively, a software programme providing a real time-spectrogram function can be used. Azimuth adjustment should be a routine part of all magnetic tape transfers.

5.4.12.3 Digital systems may correct the phase relationship of the signal (often described as azimuth correction), however such procedures cannot retrieve the high frequency information that is lost. Azimuth adjustments must be made on the original tape before transfer commences.

5.4.12.4 The vertical alignment of the heads on the original recording machine may present an obstacle to the appropriate reproduction of the signal. This is particularly the case with recordings made on amateur or consumer-grade equipment. In order to obtain a visual representation of the alignment of the tracks on the tape of a recording the following procedure should be followed: Recorded portions of tapes should be protected by a very thin transparent sheet of Mylar or similar transparent material. A powder or suspension of ferromagnetic material, particle size less than 3 µm, is sprayed over the transparent sheet. The magnetic properties of the recorded portion of the tape then make the tracks visible. A carefully marked series of measurement lines on the sheet will aid in detecting misalignment. These tape path adjustments are less frequently required than azimuth adjustment, but if they must be undertaken the replay equipment should be recalibrated by a qualified technician. Every care should be taken to ensure no iron particles remain in contact with the tape as these may damage the replay heads.

5.4.12.5 Cassette Enclosures: The enclosures in which low cost cassette tapes are housed may cause the tape to jam or replay with increased wow and flutter. In such cases it is often beneficial to replace the tape in a high quality screwed enclosure being sure to include the rollers, pressure pad and lubricating sheets.

5.4.12.6 Wow, Flutter and Periodic Tape Speed Variations: There is little that can be done to effectively improve periodic variations in the recorded signal. It is therefore imperative that the replay equipment is thoroughly and carefully checked, aligned and maintained to ensure that no speed related artefacts are introduced.With the availability of high resolution A/D converters and components, it seems possible to retrieve the high frequency (HF) bias signal from analogue magnetic tapes during transfer, which may enable the correction of wow and flutter. There are, however, many significant barriers to realising this, including a lack of available hardware to extract signals of such high frequencies and the inherent unreliability of the bias signal itself. As the procedure is generally time-consuming and complex, and substantial improvements concerning this matter are not to be expected, implementation is unlikely, and even then, only feasible for a limited group of tapes produced under specific circumstances.

5.4.13 Removal of Storage Related Signal Artefacts

5.4.13.1 It is preferable in most cases to minimise the storage related signal artefacts before undertaking digitisation. In linear analogue magnetic recording, for example, print-through is a well-known and disturbing phenomenon. The reduction of this unwanted signal can only be undertaken on the original tape.

5.4.13.2 Print-Through: Print-through is the unintentional transfer of magnetic fields from one layer of analogue tape to another layer on the tape reel. It reveals itself as the pre and post echoes to the main signal. The intensity of print through signal is a function of the wavelength, tape coating thickness, but primarily the spread of the coercivity5 of the particles in the magnetic layer. Almost all print through occurs soon after the tape is recorded and wound onto the pack. The increase in print-through after this reduces over time. Further significant increases in print-through occur only as a consequence of changes in temperature. When the tape is stored with the oxide facing in to the hub, the most common standard, the print on the layer outside of the intended signal is stronger then the print signal on the layer towards the hub of the spool. Consequently it has been frequently recommended that tapes be stored “tail out”, in which case the post echoes are louder than the pre echoes and less obvious. German broadcast standards specified that tapes be wound with the oxide out, in which case the reverse applies, and tapes should be stored “head out”.

5.4.13.3 Printed signals are reduced by the act of rewinding the tape prior to playing, by a process termed “magnetostrictive action”. Systematic tests have shown, however, that it is wise to rewind a tape at least three times to sufficiently diminish print through (Ref Schuller 1980). If the printed signal is very high and it does not respond adequately to rewinding, some tape machines allow the application of a low level bias6 signal to the tape during playback. This selectively erases lower coercivity particles and hence reduces print-through, though it may also have an effect of the signal, especially if over-applied, and should only be used as a last resort and then very carefully.

5.4.13.4 Though print-through can be reduced on the original tape the same level of restoration is not achievable afterwards. Once copied to another format the printed signal becomes a permanent part of the wanted signal.

5.4.13.5 Vinegar Syndrome and Brittle Acetate Tape: Acetate tape becomes brittle with age which may make it difficult to play a tape without breaking. The brittleness occurs as a result of a process of chemical degradation which occurs when the molecular bonds of the acetate compound break down to release acetic acid giving off the characteristic smell of vinegar. Broken acetate tape can be spliced without any signal loss or deterioration, because, as a result of its brittleness, no elongation of the tape occurs. Brittle tapes, however, are also subject to a variety of deformations which hinder the necessary tape-to-head contact for optimal signal retrieval. Though a process of re-plastification would be advantageous,such processes do not exist as yet. Archivists are warned against the chemical processes sometimes suggested as these may not only jeopardise the further survival of the tape,but also contaminate replay equipment and, indirectly, other tapes replayed on such machines. Instead, it is recommended that such tapes be replayed using a recent machine that permits to lower tape tension. This will enable an acceptable compromise between care of the fragile tape and the application of enough tape tension to permit the best possible tape-to-head contact.

5.4.13.6 Physical Tape Memory: Poorly stored and spooled polyester and PVC tapes may also suffer from deformation of the tape. The tape will often retain a memory of that deformation and so make poor tape to head contact, which reduces the signal quality. Repeated respooling and resting may reduce some of this effect.


5. Coercivity; A measure of the intensity of the magnetic field needed to reduce the magnetization of a ferromagnetic material to zero after it has reached saturation

6. Bias; A high frequency signal mixed with the audio during recording to help reduce tape based noise. Devised by Weber in 1940

5.4.14 Wire Recordings

5.4.14.1 Though the principles of wire recordings were demonstrated at the very end of the 19th century, and various dictation machine manufacturers produced working models in the 1920s and 1930s (see 5.4.15 below), it was not until around 1947 that the wire recorder was successfully marketed to the general public.

5.4.14.2 The speed of wire recorders was not standard and varied between manufacturers and even, on occasion, from model to model. After 1947, however, manufacturers mostly adhered to a standard speed of 24 ips and a reel size of 2¾ inches. Wire recorders did not have capstans, and so the speed would change as the take up reels became full. The size of the take up spool was integral to the correct replay of the wire, and very often related to a particular machine or manufacturer. The take-up spool is generally a fixed part of the machine. The height of popularity of the wire recorder was in the years from the mid 1940s till the early 1950s, a period which coincided with the development and introduction of the technically superior tape recorder and the wire was soon considered obsolete. Even in its heyday, the wire recorder was primarily used as a domestic recorder, though some were used for commercial purposes.

5.4.14.3 Though the wire fell quickly from favour, wires were available in speciality outlets until the 1960s. Early reel sizes were large in comparison to the 2¾ inch reels which become the most commonly used reel. Some wires, mostly early in the history of the wire recorder, were made from plated or coated carbon steel, and these may now be corroded and difficult to play. Many wires, however, are in excellent condition being made from Stainless Steel with 18% chromium and 8% nickel, and have not corroded.

5.4.14.4 The principle of wire recorders is comparatively simple, so that the construction of a replay machine is possible. However, the complexity associated with successfully spooling and playing the fine wire without tangles or breakages suggest that the best approach to replay is to use an original machine, though it is worth noting that some experts have modified tape machines to replay wires.When using original machines it is recommended that the audio electronics be overhauled to ensure best performance or, preferably, replaced with audio circuitry using modern components (Morton 1998, King: n.d.)

5.4.15 Magnetic office dictation formats

5.4.15.1 In the decades following World War II a wide variety of magnetically recorded office dictation formats appeared. That the needs of the office differ from other audio recording environments is reflected in their design: reduced size and weight, ease of operation and variable speed were prioritised, usually at the expense of audio quality. Magnetic dictation systems may be broadly divided into tape and non-tape-based formats.

5.4.15.2 Tape in this context includes various forms of wire (see 5.4.14 above), reel and cassette. Some formats may be playable using standard equipment (non-standard cassette formats may sometimes be rehoused and replayed in standard cassette shells for instance) while others may only be played on dedicated format-specific players. Where a choice is available, a decision needs to be made between the two approaches. One entails the use of high- specification, relatively easy to maintain standard equipment, potentially coupled with poor compatibility in tape width, head configuration, replay speed, equalisation, noise reduction etc. The other offers higher compatibility between carrier and player, but very likely at the cost of the lower specification and esoteric maintenance needs of the original format-specific equipment. Tape-based formats can be subdivided into linear and non¡linear speed. The former will present fewer problems if replayed on conventional equipment; the latter may also be playable in this way, but will require speed adjustment (see 5.4.9).

5.4.15.3 Non-tape formats include a bewildering array of discs, belts, rolls and sheets, all featuring magnetically coated surfaces, recorded onto and replayed using heads similar in principle to conventional tape heads. Given sufficient expertise, time and money therefore, it may be possible to build replay devices for some of these formats, incorporating components from more common tape replay equipment. In many cases however locating an original replay machine might be more feasible, and it may be possible to contract a specialist equipped to carry out the work.

5.4.16 Time Factor

5.4.16.1 The time needed for copying contents of audio material varies greatly, and is highly dependent on the nature and status of the original carrier. The step of actually playing the carrier is only one part of the process, which includes respooling, assessment, adjustment and documentation. Even a well documented, good quality analogue tape of 1 hour’s duration will take, on average, twice the time of the length of its recording to properly transfer to a digital carrier. In the mid-1990s the Archivarbeitsgruppe of ARD (Arbeitsgemeinschaft der Rundfunkanstalten Deutschlands) would regard this as optimistic as they postulated a transfer factor of 3 (1 operator: 3 hours of work for 1 hour of material) for the transfer of typical archival holdings of their radio stations. Tapes that exhibit any faults, which require repair or restoration, or need documentation or the assessment and addition of metadata, will take much longer to conserve, transfer and preserve.

5.4.17 Signal Auto detection, auto upload (failings and benefits)

5.4.17.1 It is recommended that all tapes be actively listened to while preservation transfers are being undertaken. However, in response to the sheer quantity of the material to be transferred and preserved, manufacturers of digital archiving systems have been developing ways of automatically monitoring and detecting signal faults allowing for the possibility of unattended transfers. The savings in time are obvious, as an individual operator may undertake multiple transfers simultaneously. The systems themselves seem to achieve their greatest benefit on largely homogenous collection material that is well recorded on stable carriers that can be treated identically. This is evident in that the most successful mass upload systems have been undertaken or implemented by broadcast archives where the content is largely of similar quality, the collection size is large, and the resources are available to build, manage and run such systems. For material that requires individual treatment, and this is typified in most research and heritage collections, the benefits of an automated system are not as great.

5.5 Reproduction of Digital Magnetic Carriers

5.5.1 Introduction

5.5.1.1 Under optimum conditions digital tapes can produce an unaltered copy of the recorded signal, however any uncorrected errors in the replay process will be permanently recorded in the new copy or sometimes, unnecessary interpolations will be incorporated into the archived data, neither of which is desirable. Optimisation of the transfer process will ensure that the data transferred most closely equates to the information on the original carrier. As a general principle, the originals should always be kept for possible future re-consultation however, for two simple practical reasons any transfer should extract the optimal signal from the best source copy. Firstly, the original carrier may deteriorate, and future replay may not achieve the same quality, or may in fact become impossible, and secondly, signal extraction is such a time consuming effort that financial considerations call for an optimisation at the first attempt.

5.5.1.2 Magnetic tape carriers of digital information have been used in the data industry since the 1960s, however, their use as an audio carrier did not become common until the early 1980s. Systems reliant on encoding audio data and recording onto video tapes were first used for two track recording or as master tapes in the production of Compact Discs (CD). Many of these carriers are old in technical terms and in critical need of being transferred to more stable storage systems.

5.5.1.3 A crucial recommendation of all transfers of digital audio data is to carry out the entire process in the digital domain without recourse to conversion to analogue. This is relatively straightforward with later technologies which incorporate standardised interfaces for exchanging audio data, such as AES/ EBU or S/PDIF standards. Earlier technologies may require modification to achieve this ideal.

5.5.2 Selection of Best Copy

5.5.2.1 Unlike copying analogue sound recordings, which results in inevitable loss of quality due to generational loss, different copying processes for digital recordings can have results ranging from degraded copies due to re-sampling or standards conversion, to identical “clones” which can be considered even better (due to error correction) than the original. In choosing the best source copy, consideration must be given to audio standards such as sampling and quantisation rate and other specifications including any embedded metadata. Also,data quality of stored copies may have degraded over time and may have to be confirmed by objective measurements. As a general rule a source copy should be chosen which results in successful replay without errors, or with the least errors possible.

5.5.2.2 Unique Recordings: Original source materials such as multi-track sessions, field recordings, logging tapes, home recordings, sound for film or video, or master tapes, may include unique content in whole or in part. Un-edited material may be less or more useful than the edited final product, depending on the purpose of the archived material. Curatorial decisions must be made to ensure that the most appropriate or complete duplicate is selected. Truly unique recordings do not present any choice to the archivist. In the case where content is uniquely held on a single copy within a collection it is worth considering whether alternative copies might exist elsewhere. It may be possible to save both time and trouble if other copies exist which are in better condition, or on a more convenient format.

5.5.2.3 Recordings with Multiple Copies: Preservation principles indicate that copies of digital tape should ideally be a perfect record of the media content and any associated metadata as recorded on the original digital document. Any digital copy meeting this standard is a valid source for migration of the essence to new digital preservation systems.

5.5.2.4 In reality, effects of standards conversion, re-sampling or error concealment or interpolation1 may result in data loss or distortion in copies, and deterioration over time degrades the quality of original recordings and subsequent copies. As a result, copying outcomes may differ depending on the choice of source material. Cost can also vary depending on the physical format or condition of the source material.

5.5.2.5 Determining the best source copy requires consideration of recording standards used to create copies, quality of equipment and processes used, and the current physical condition and data quality of available copies. Ideally this information is documented and readily available. If this is not so then decisions must be based on understanding of the purpose and history of various copies.

5.5.2.6 Duplicates on Similar Media: Best source material in this case will be that copy with the best data quality. First choice will usually be the most recently made unaltered digital copy. Earlier generations of unaltered digital copies may represent an alternative if the newer copies are inadequate due to deterioration or improper copying.

5.5.2.7 Copies Differing in Media or Standard: Production or preservation processes may result in availability of multiple copies on differing digital tape formats. The best source material should be identical to the original in standard, have the best available data quality, and be recorded on the most convenient format for reproduction. Judgment is called for if any of these conditions cannot be met.

5.5.2.8 If the digital recordings are only duplicates of analogue recordings, and where the analogue originals still exist, re-digitisation is an option to consider if those digital copies are inferior in standard, quality or condition.


1. Error concealment or interpolation is an estimation of the original signal when data corruption prevents accurate re-construction of the signal.

5.5.3 Cleaning, Carrier Restoration

5.5.3.1 Magnetic digital tapes are similar in materials and construction to other magnetic tapes, and suffer from similar physical and chemical problems. Digital tapes achieve high data densities through the use of thin tapes, small magnetic tracks and ongoing reductions in the size of the magnetised domains which can be written and read. Consequently even minor damage or contamination can have major impacts on signal retrievability. All tape degradation,damage or contamination will appear as increased errors. Carrier restoration problems and techniques are similar for all magnetic tapes, but since base, binder and magnetic materials are subject to ongoing development any restoration processes must be tested and proven for specific media.

5.5.3.2 Commercial cleaning machines are available for open reel magnetic tapes and for most videotape formats commonly used to carry digital audio signals and are effective for moderately degraded or contaminated tapes.Vacuum or hand cleaning may be indicated for tapes with higher levels of contamination or of greater fragility, but requires conservatorial care to avoid damaging delicate tapes and intricate cassette mechanisms. Any cleaning process has potential to cause damage and should be applied with appropriate caution.

5.5.3.3 Jigs can aid in manipulating tapes and cassette housings, and are commercially available for some formats. Purpose-built jigs for other formats can be manufactured in a moderately well equipped mechanical workshop.

5.5.3.4 Digital tapes with polyester urethane binders have the potential to suffer from hydrolysis in the same way as analogue magnetic tapes. Any rejuvenation of digital magnetic tape will require tight process control, and should only be attempted in a purpose-built environmental chamber or vacuum oven2 (see Section 5.4.3 Cleaning and Carrier Restoration). This may be even more critical with digital recordings as they will often have been made on thinner based tapes housed in complex cassette mechanisms.

5.5.3.5 Deterioration of magnetic tapes can be minimised by appropriate storage conditions. Standards for long-term digital magnetic tape storage are generally more stringent than for analogue tapes, due to their greater fragility and susceptibility to data loss through relatively minor damage or contamination. Higher than recommended temperature or humidity will promote chemical deterioration. Cycling of temperature and humidity will result in expansion and contraction of the tape and may damage the tape base. Dust or other contaminants can find its way onto the tape surface resulting in data loss and possibly physical damage during replay.

5.5.3.6 After cleaning and/or repairing measures or prior to the reproduction it may be advisable to first measure the magnetic digital tape’s error rates. The organisation of the data and the type of error correction used varies according to the tape format. For DAT for example, the error correction process uses two Reed-Solomon codes arranged in a cross code system, C2 horizontally and C1 vertically. Also, each block of data has a value assigned, known as a parity byte. Counting the Block parity errors are known as CRC errors, or sometimes as the block error rate. The sub code of the DAT (Digital Audio Tape) is also subject to errors. Error measurement should include, as a minimum:

5.5.3.6.1 C2 and C1 errors.

5.5.3.6.2 CRC or Block error rate.

5.5.3.6.3 Burst Errors (derived from C1).

5.5.3.6.4 SUBC1 correction.

5.5.3.7 If any of the error measurement reveals a sample hold, interpolated or mute level error the tape should be cleaned and the tape path checked. If after cleaning and repair one or more of the error rates exceed these thresholds refer to 5.6.3 “Selection of Best Copy.” (above).

5.5.3.8 There are very few error measuring devices available for DAT or other magnetic carriers. Any transfer, however, should include a measurement of the errors produced at the error correction chip of the replay machine and this information must be recorded in the metadata of the resultant audio file.


2. Vacuum ovens reduce the air pressure in the oven chamber and so better control moisture content

5.5.4 Replay Equipment

5.5.4.1 Replay equipment must comply with all specific parameters of a given format. Digital tape formats are mostly proprietary in nature, with only one or two manufacturers of suitable equipment. Latest generation equipment is preferred, but for older or obsolete digital formats there may be no choice but to purchase second- hand equipment.

5.5.4.2 The high recording density of R-DAT(Rotary Head Digital Audio Tape) has ensured that applications other than audio-recording-only were developed. The DDS (Digital Data Storage) format, based on DAT technology, was developed by Hewlett-Packard and Sony in 1989 and was dedicated to the storage of computer data. Steady increases in data integrity of the basic system resulted in developments which allow for signal extraction from audio DAT tapes.Various types of software are available which allow the extraction of the audio as separate files based on ID’s on the tape. Dedicated data extraction software can also generate metadata files for each program, including clock, start and end ID positions, durations, file size, audio properties, etc. Additionally the DDS format allows double speed capturing of audio material.

5.5.4.3 Nevertheless, the important questions such as format incompatibilities (e.g. the different long play modes, high resolution recordings, time code extraction etc.), proper data integrity checking, pre-emphasis handling and especially all matters concerning mechanical and tracking problems are still not yet solved by such systems and therefore need individual treatment.

5.5.5 Common Systems and Characteristics: Cassette Systems

5.5.5.1 The R-DAT (commonly referred to as DAT) is the only common system to use a cassette format specifically developed for digital audio recordings. DAT tapes have been widely used in field and studio recording, broadcasting and archiving. New DAT equipment is now virtually unavailable. Second hand professional DAT machines are a solution, but present maintenance problems as parts supplies become exhausted.

5.5.5.2 Some last generation recorders operate outside the specification, allowing high resolution recording at 96 kHz and 24 bits (at double speed), others provided Timecode (SMPTE) recording, or Super Bit Mapping, a psycho-acoustic principle and critical band analysis to maximize the sound quality of 16-bit digital audio. 20-bit recordings are quantized to 16 bits using an adaptive error-feedback filter. This filter shapes the quantization error into an optimal spectrum as determined by the short-term masking and equi-loudness characteristics of the input signal. Through this technique, the perceptual quality of 20-bit sound is available on a 16-bit DAT recording. Full quality can only be reached with signals containing frequencies lower than 5-10 kHz. Super bit mapping does not require special decoding on playback.

 

Record/playback mode

Pre-recorded tape
(Playback only)
Standard Standard Option 1 Option
2
Option 3 Normal track Wide Track
Number of
Channels
2 2 2 2 4 2 2
Sampling rate
(kHz)
48 44.1 32 32 32 44.1
Number of
quantization bits
16 (linear) 16 (linear) 16 (linear) 12 (non
linear)
12 (non
linear)
16 (linear)
Linear recording
density (KBPI)
61.0 61.0 61.1
Surface recording
density (MBPI2)
114 114 76
Transmission rate
(MBPS)
2.46 2.46 2.46 1.23 2.46 2.46
Sub-code capacity
(KBPS)
273.1 273.1 273.1 136.5 273.1 273.1  
Modulation 8–10 Conversion
Correction Dual Reed Solomon
Tracking Area split ATF
Cassette size (mm) 73x54x 10.5
Recording time*
(min)
120 120 120 240 120 120 80
Tape width (mm) 3.81
Tape type Metal-particle Oxide
Tape thickness
(μm)
13±1μ
Tape speed (mm/s) 8.15 8.15 8.15 4.075 8.15 8.15 12.225
Track pitch (μm) 13.591 13.591 20.41 (wide
track)
Track angle 6°22’59”5   6°23’29”4
Standard drum Ø 30 90° Wrap    
Drum revolution speed (r.p.m.) 2000 1000 2000 2000  
Relative speed
(m/s)
3.133 1.567 3.129 3.133 3.129
Head azimuth ±20°

Table 1 Section 5.5 Specifications for various record/playback modes of DAT for both blank and pre-recorded tapes:

5.5.5.3 Phillips DCC (Digital Compact Cassette) system was (unsuccessfully) introduced as a consumer product and offered limited compatibility with analogue compact cassettes through the ability to replay analogue cassettes on DCC equipment. DCC is now considered obsolete.

Format Variants Carrier Type Audio and data tracks Digital Audio Standards supported Interface
DAT or R-DAT Timecode is not part of the R-DAT standard but may be implemented in Sub-Code. Some pre-recorded DATS use ME tape Cassette with 3.81mm metal particle tape. Stereo. Subcode includes standardised markers plus user bits for proprietary extensions 16 bit PCM @ 32, 44.1 and 48 kHz AES-422 on professional machines. SP-DIF standard
DCC   Cassette with 3.81 CrO2 Stereo, metadata standard supports minimal descriptive data PASC compressed PCM (4:1 bit rate reduction)  
Videotape based formats — see table 4          

Table 2 Section 5.5 Digital Audio Cassettes

5.5.6 Common Systems and Characteristics: Open Reel Formats

5.5.6.1 SONY and Mitsubishi have both produced open reel digital systems for the recording studio market, and NAGRA produced a four-track field recording system, the NAGRA-D.

5.5.6.2 Sony/Studer’s DASH (Digital Audio Stationary Head) system has numerous variants, based on common formats for the digital tracks on tape. DASH I provides 8 digital tracks on ¼” tape and 24 digital tracks on ½" tape. DASH-II provides 16 digital tracks on ¼” tape and 48 tracks on ½" tape. Twin DASH formats are commonly used for ¼” stereo digital recordings and utilise twice the normal number of data tracks for each audio channel to increase the systems error correction capability so that tape splicing can be used for editing. Low speed formats double recording time by sharing data for each audio channel across multiple data tracks, halving the number of audio tracks available.

5.5.6.3 Nagra still support NAGRA-D Sony DASH and Mitsubishi Pro-Digi format machines are no longer manufactured. These formats are/were intended for high-end professional use and as a result were extremely expensive to support.

Format Variants Carrier Type Audio and data tracks Digital Audio Standards supported Interface
DASH Three speeds – F (fast), M (medium) and S (slow) ¼” or ½” tape Up to 48 audio tracks plus control track 16 bit at 32 kHz, 44.1 kHz or 48 kHz AES/EBU SDIF-2 MADI interface
  DASH-I (single density) and DASH-II (double density)        
  Two tape widths Q (quarter inch) and H (half Inch)        
Mitsubishi Pro Digi Stereo ¼” tape   32 kHz, 44.1 kHz or 48 kHz. 20 bit or 16 bit (with extra redundancy to facilitate splice editing) at 15 ips. 16 bit (normal redundancy) at 7.5 ips AES/EBU or proprietary multi-channel interface
  16 track ½” tape   32 kHz, 44.1 kHz or 48 kHz. 16 bit  
  32 track 1” tape   32 kHz, 44.1 kHz or 48 kHz. 16 bit  
NAGRA-D   ¼” MP 4 audio tracks. Extensive metadata including TOC and built-in error recording 4 tracks at up to 24 bit 48 kHz 2 tracks at 24 bit 96 kHz AES/EBU

Table 3 section 5.5 Open Reel Formats

5.5.7 Common Systems and Characteristics: Video Tape Based Formats

5.5.7.1 There are two variants within this category: systems using videotape in a standard VCR to record digital audio encoded on a standard video signal, and systems using videotape as the storage medium for proprietary digital audio signal formats.

5.5.7.2 Sony has produced a range of formats using VCR systems as a high bandwidth storage device. More recently Alesis introduced the ADAT system, which used S-VHS videocassettes as high capacity storage media for their proprietary format of digital audio, and Tascam released the DTRS system using Hi8 videocassettes as the storage medium.

5.5.7.3 Formats using video recorders were based on interface devices that incorporated A-D and D-A converters, audio controls and metering, and the hardware required to encode the digital bit stream as a video waveform. Sony’s professional system specified NTSC standard (525/60) Black-and-White U-Matic VCR, and these were manufactured specifically for digital audio use. The semi-professional PCM-F1, 501 and 701 series worked best with Sony Betamax recorders, but were generally compatible with Beta and VHS. Machines in this series supported PAL, NTSC and SECAM standards.

5.5.7.4 Reproduction of VCR based recordings requires availability of a VCR of the correct standard, plus the appropriate proprietary interface. There is normally backwards compatibility within related systems, so purchase of later generation equipment should facilitate replay of the widest range of source material. As some of the video based PCM adaptors had only one A/D converter for both stereo channels, there is a time delay between the two channels.When the tapes are replayed and the audio data is extracted the signal processor delay should be corrected in the digital domain. Transfers should be made only with equipment which allows the output of a digital signal.

5.5.7.5 Early digital recorders sometimes encoded in what are now uncommon sampling rates, such as 44.056kHz (see table 4 Section 5.5). It is recommended that the resultant files be stored at the encoding levels at which they were created. Care should be taken to ensure that automatic systems do not misrecognise the actual sampling rate (eg a 44.056kHz audio stream may be recognised as 44.1kHz, which alters the pitch and speed of the original audio). Second files can be created for users in common sampling rates using appropriate sampling rate conversion software. Nonetheless, the original file should be retained.

5.5.7.6 In addition, third-party equipment for systems based on domestic VCRs can provide useful extended functionality, including better metering and error monitoring facilities and professional inputs and outputs.

5.5.7.7 VCR based systems are obsolete, and equipment will need to be sourced second-hand.

Format Variants Carrier Type Audio and data tracks Digital Audio Standards supported Interface
EIAJ Sony PCM-F1 PCM-501 and PCM-701 systems Video signal may be PAL, NTSC or SECAM Domestic VCR — normally Betamax or VHS cassette Rare examples use ½” open reel videotape Stereo Audio 14 bit standard, Sony hardware allows 16 bit sampling (with less error correction) 44.056 kHz in NTSC systems, 44.1 kHz in PAL systems Analogue line in and out standard. Digital I/O capability with third party add-ons
Sony PCM1600 PCM1610 and PCM1630   U-Matic – Black and White, 525/60 (NTSC) Stereo audio plus Compact Disc PQ codes Timecode on U-matic linear audio track 16 bit 44.1 kHz Sony proprietary system. Digital audio on separate Left and Right Channels plus word-clock
DTRS (1991)   Proprietary format on Hi8 video cassettes   16 bit 48 kHz 20 bit recording optional on some systems SP-DIF or AES/ EBU
ADAT (1993)   Proprietary system on S-VHS cassettes     SP-DIF or AES/ EBU

Table 4 section 5.5 Digital Audio on Videotape – Common Systems

5.5.8 Replay Optimisation

5.5.8.1 Precise identification of the format and detailed characteristics of the source material is essential to ensure optimum reproduction, and is complicated by the variety of formats with outwardly similar physical characteristics but different recording standards. Machines should be cleaned and regularly aligned for best signal reproduction. Any operator-controlled parameters such as de-emphasis must be set to match the original recording. For VCR based formats the video tracking may need to be adjusted for best signal, and any dropout compensation on the video signal must be switched off.

5.5.9 Corrections for Errors Caused by Misaligned Recording Equipment

5.5.9.1 Misalignment of recording equipment leads to recording imperfections, which can take manifold form.While many of them are not or hardly correctable, some of them can objectively be detected and compensated for. It is imperative to take compensation measures in the replay process of the original documents incurred, as no such correction will be possible once the signal has been transferred to another carrier.

5.5.9.2 Adjustment of magnetic digital replay equipment to match misaligned recordings requires high levels of engineering expertise and equipment. The relationship between the rotating heads and the tape path can be adjusted on most professional equipment, and for DAT recordings especially, this can lead to significant improvement in error correction or concealment, even making apparently unplayable tape audible. However, such adjustments require specialised equipment and only trained personnel should undertake them. Equipment should be returned to correct setting by trained service technicians after completing the transfer.

5.5.10 Removal of Storage Related Signal Artefacts

5.5.10.1 It is preferable in most cases to minimise the storage related signal artefacts before undertaking digital transfer. Digital tapes should be re-spooled periodically if possible, and in any case always re-spooled before replay. Re-spooling reduces mechanical tension, which can damage the tape base or decrease performance during replay. Open reel digital tapes that have been left unevenly wound for some time may exhibit deformations, particular of tape edges, which may cause reproduction errors. Such tape should be rewound slowly to reduce the aberrations in the wind and rested for some months, which may aid in reducing replay errors. Though cassette systems can be similarly affected, the ability to influence the pack through reduced wind speed is not as great with such equipment.

5.5.10.2 Magnetic fields do not decay measurably in a period of time that is likely to affect their playability. The proximity of adjacent tracks or layers will not cause self erasure on analogue tapes, and in the unlikely event that it may cause issues with older digital tapes this is rarely critical as any resulting errors are within the limits of the system. Some loss of signal may be measurable in the oldest video based tapes when used to record digital audio. In these circumstances the lower coercivity of the magnetic particles and the apparent short wavelength on the tape caused by recording digital information using a rotating head combine to create the conditions where this may occur, at least in theory. This may make it difficult for replay equipment to track the information on the tape. All but the very earliest video tape formulations have a much higher coercivity, combined with systems which have better error correction technology, which made this problem largely irrelevant. In any event, attention to the cleanliness of the heads of the replay machine and tape will maximise the possibility of replay, as will careful alignment of the tape path.

5.5.10.3 Seriously damaged tapes may be recoverable using techniques that could be characterised as “forensic” due to their dependence on high-level skills from a range of scientific and engineering disciplines (see Ross and Gow 1999). Management of digital tape collections should aim to ensure copying occurs before un-correctable errors become a problem, as options for restoration of failed digital tapes are very limited.

5.5.11 Time Factor

5.5.11.1 The time needed for copying contents of audio material varies greatly, and is highly dependent on the nature and status of the original carrier.

5.5.11.2 Preparation time will vary depending on condition of the source copy. Set-up time depends on details of facilities and formats in use. Signal transfer is generally slightly more than actual running time for each recoded segment, and time taken for management of metadata and materials management will depend on details of the archiving system in use. Most audio specific tape based digital recording formats do not allow upload of the data at greater than real time, with the exception of those mentioned above. However, capture systems that accurately measure error levels and warn operators when set levels are exceeded may allow for multiple systems to be run simultaneously.

5.6 Reproduction of Optical Disc Media (CD and DVD)

5.6.1 Introduction

5.6.1.1 Since their introduction in 1982 replicated optical disc media have become the dominant technology for distribution of published audio recordings. Recordable optical disc formats, first made available in the late 1980s1, play an increasingly significant role in distribution and storage of unpublished audio. Initially marketed as permanent, it has become clear that the usable life of the optical disc is finite and that steps will need to be taken to copy and preserve their data content. This is especially the case with recordable disc media, which are not only less reliable than their manufactured counterparts but are also more likely to contain unique material. Unless recorded and managed under specified conditions (see Section 6.6 Optical Discs: CD/DVD Recordables), recordable disc media constitute an unreasonable risk to collection material. This section of the Guidelines concerns itself with the accurate and efficient copying of CD and DVD optical disc media to more permanent storage systems. CD is the abbreviation for Compact Disc, DVD initially stood for Digital Video Disc, then Digital Versatile Disc but is now used without referring to a specific set of words.

5.6.1.2 The Audio CD family may include, in CD-DA format; CD manufactured, CD-R, CD-RW, and in this form are all characterised by 16 bit digital resolution, 44.1 kHz sampling frequency and 780nm wavelength read laser. DVD Audio includes SACD and DVD-A. Data formats such as .wav files and BWF files may be recorded as files on CD- ROM and DVD-ROM. DVD media are characterised by blue laser around 350 to 450nm for glass mastering and 635-650 nm playback, DVD+R (650 nm), DVD-R (both for authoring (635 nm laser)). Blu-Ray Discs (BD) are a high definition video and data format on the same diameter 12 cm disc as DVD and CD. Using a 405 nm blue laser BD is able to store 25 GB of data per layer.

5.6.1.3 Recordability, rewritability, erasability and accessibility:

5.6.1.3.1 CD and DVD (CD-, DVD-A, CD-ROM and DVD-ROM) discs are pre-recorded (pressed and moulded) read-only discs. They are neither recordable nor erasable.

5.6.1.3.2 CD-R, DVD-R and DVD+R discs are dye-based recordable (write once) discs, but not erasable.

5.6.1.3.3 CD-RW, DVD-RW and DVD+RW discs are phase-changed based repeatedly rewritable discs permitting erasure of earlier and recording new data in the same location on the disc.

5.6.1.3.4 DVD-RAM discs are phase-changed rewritable discs formatted for random access, much like a computer hard disc.

5.6.1.4 The table below (table 1 section 5.6) provides a listing of commercially available CD and DVD disc types.

Disc Type Storage
capacity
Laser
wavelength
write mode
Laser
wavelength
read mode
Typical use
CD-ROM, CD-A, CD-V read only 650 MB 780 nm 780 nm Commercially available
CD-R (SS) write once 650 MB 780 nm 780 nm Music recording, computer data, files, applications
CD-R (SS) write once 700 MB 780 nm 780 nm
CD-RW (SS) Rewritable 650 MB 780 nm 780 nm Computer data recording, files, applications
CD-RW (SS) Rewritable 700 MB 780 nm 780 nm
DVD-ROM, DVD-A,
DVD-V: SS/SL
SS/DL
DS/SL
DS/DL
read only
4.7 GB
8.54 GB
9.4 GB
17.08GB
650 nm 650 nm Movies, interactive games, programmes,
applications
DVD-R(G) write once 4.7 GB 650 nm 650 nm General use: One time video recording
and data archiving
DVD-R(A)
SL
DL
write once 3.95 or 4.7
GB
8.5GB
635 nm 650 nm Authoring/professional use
Video recording and editing
DVD+R SL
DL
write once 4.7 GB
8.5 GB
650 nm 650 nm General use: One time video recording
and data archiving
DVD-RW Rewritable 4.7 GB 650 nm 650 nm General use: Video recording and PC backup
DVD+RW Rewritable 4.7 GB 650 nm 650 nm General use: Video recording and editing, data storage. PC backup
DVD-RAM
SS
DS
rewritable 2.6 or 4.7
GB
5.2 or 9.4
GB
650 nm 650 nm Computer data: Storage repository for updateable computer data, back-ups
HD-DVD –R SL
DL
write once 15 GB
30 GB
405 nm 405 nm data and high-definition video
HD-DVD –R W SL
DL
rewritable 15 GB
30 GB
405 nm 405 nm data and high-definition video
BD-R SL
DL
write once 25 GB
50 GB
405 nm 405 nm data and high-definition video
BD-RE SL
DL
rewritable 25 GB
50 GB
405 nm 405 nm data and high-definition video

Table 1 Section 5.6 Commercially available CD/DVD disc types
SS= Single-sided, SL=Single layer, DL=double-sided, DL=dual layer

5.6.1.5 Under optimum conditions digital discs can produce an unaltered copy of the recorded signal, however, in the case of audio specific recordings, any un-corrected errors in the replay process will be permanently recorded in the new copy, or sometimes unnecessary interpolations will be incorporated into the archived data, neither of which is desirable. Optimisation of the transfer process will ensure that the data transferred is most closely equivalent to the information on the original carrier. As a general principle, the originals should always be kept for possible future re-consultation, however, for two simple, practical reasons, any transfer should attempt to extract the optimal signal from the best source copy. Firstly, the original carrier may deteriorate, and future replay may not achieve the same quality, or may in fact become impossible, and secondly, signal extraction is such a time consuming effort that financial considerations call for an optimisation at the first attempt


1. The first working CD-R system,Yamaha’s PDS (Programmable Disc System), was launched in 1988

5.6.2 Standards

5.6.2.1 Compact Disc Standards:The standard for CD was originally a product of the companies Philips and Sony. The standards are named after a colour, the first being the Red Book: Philips-Sony Red Book CD Digital Audio, also includes CD Graphics, CD (Extended) Graphics, CD-TEXT, CD-MIDI, CD Single (8cm), CD Maxi-single (12cm) and CDV Single (12cm).Yellow Book standard specifies the CD as a data file carrier, the Green Book describes CD-I or interactive data, Blue Book describes Enhanced (multimedia) CD, and White Book specifies CD-V (video) characteristics. Orange Book is the standard that refers to Recordable and Rewritable CDs (and is described more fully in Chapter 6). The colour book standards, subject to certain limitations, may be ordered from the Philips web site at http://www.licensing.philips.com/. They are primarily intended for manufacturers. The ISO standards which describe CDs are purchasable from International Standards Organisation (ISO) Central Secretariat http://www.iso.org/. IEC 908:1987, Compact Disc Digital Audio System (CD-DA) (n.b. IEC 908:1987 and Philips-Sony Red Book are basically equivalent.) ISO 9660:1988,Volume and File Structure (CD-ROM) (ECMA-119) and ISO/IEC 10149:1995, Read-Only 120 mm Optical Data Discs (CD-ROM) (ECMA-130).

5.6.2.2 DVD Standards: There is an extensive range of ISO standards for DVD. However, similarly to CD, there are also proprietary versions of the standards. These standards are referred to by an alphabetical appellation: DVD-ROM, the basic data standard, is specified in Book A, DVD video is described in Book B, DVD- Audio in Book C, DVD-R in Book D, and DVD-RW in Book E. The ISO standards are purchasable from International Standards Organisation (ISO) Central Secretariat http://www.iso.org/ ISO 7779:1999/Amd 1:2003 Noise measurement specification for CD/DVD¡ROM drives. ISO/IEC 16448:2002 Information technology -- 120 mm DVD -- Read-only disc and ISO/IEC 16449:2002 Information technology -- 80 mm DVD -- Read-only disc.

5.6.3 Selection of Best Copy

5.6.3.1  Unlike copying analogue sound recordings, which results in inevitable loss of quality due to generational loss, different copying processes for digital recordings can have results ranging from degraded copies due to re-sampling or standards conversion, to identical “clones” which can be considered even better (due to error correction) than the original. In choosing the best source copy, consideration must be given to audio standards such as sampling and quantisation rate and other specifications including any embedded metadata. Also, data quality of stored copies may have degraded over time and may have to be confirmed by objective measurements. If there is only one copy in poor physical condition in a collection, it may be wise to contact other sound archives to determine whether it is possible to find a better preserved copy of the same item.

5.6.3.2  As a general rule, a source copy should be chosen which results in successful replay without errors, or with the least errors possible. Replicated discs are more stable than recordable media and would normally be preferred if a choice is available. Physical condition may provide an indication of quality, however the only certain method for choosing an error free disc is to institute routine error checking and reporting as part of the transfer process.Even with error checking and reporting,the extraction of best possible signal is problematic as the lack of standards with drives means that different players may produce different results on the same disc (see 8.1.5 Optical Discs – Standards). As with all digital to digital transfers,an error status report must be made and incorporated in the administrative metadata of the digital archive file, along with a record of the drive used.

5.6.4 Playback Compatibility

5.6.4.1 The variety of standards and the manner in which they may be encoded makes selection of the correct replay equipment necessary. The domestic freestanding CD player, for instance, will most likely only play CD-Audio and its variants, whereas the CD-ROM drive in a computer will play all the formats, though it requires the appropriate software to determine the content. DVDs will not play in CD drives or players, although many DVD drives are compatible with CDs.

5.6.4.2 The tables below lay out the compatibility between certain drives and their appropriate media.

 

Disc type CD-ROM drive CD-RW or CD-R/RW drive CD-R Drive
Read Write Read Write Read Write
CD-ROM Yes No Yes No Yes No
CD-R Yes No Yes Yes Yes Yes
CD-RW Yes No Yes Yes Yes No

Table 2 Section 5.6 Read Write Compatibility; CD

 

Disc type Home
DVD
player
Play only
DVD-ROM
drive
Play only
(Computer)
DVD-R
(G) drive
Records
General -R
DVD-R (A)
drive
Records
Authoring
-R
DVD-RW
drive
Records
-RW, General
-R
DVD+
RW/+R drive
Records
+RW, +R
DVD-RAM
drive
Records
RAM
DVD-ROM No No No No No No No
DVD-R(A) No No No Yes No No No
DVD-R(G) No No Yes No Yes No No
DVD-RW No No No No Yes No No
DVD+RW No No No No No Yes No
DVD+R No No No No No Yes No
DVD-RAM No No No No No No Yes
CD-ROM No No No No No No No
CD-R No No Yes No Yes Yes No
CD-RW No No No No Yes Yes No

Table 3 Section 5.6. Compatibility; DVD (Write Mode)

 

Disc type Home
DVD
player
Play only
DVD-ROM
drive
Play only
(Computer)
DVD-R
(G) drive
Records
General -R
DVD-R (A)
drive
Records
Authoring
-R
DVD-RW
drive
Records
-RW, General
-R
DVD+
RW/+R drive
Records
+RW, +R
DVD-RAM
drive
Records
RAM
DVD-ROM Not
Usually
Yes Yes Yes Yes Yes Yes
DVD-R(A) Mostly Usually Yes Yes Yes Yes Yes
DVD-R(G) Mostly Usually Yes Yes Yes Yes Yes
DVD-RW Partly Usually No Yes Yes Usually Usually
DVD+RW Partly Usually Usually Usually Usually Yes Usually
DVD+R Partly Usually Usually Usually Usually Yes Usually
DVD-RAM Rarely Rarely No No No No Yes
CD-ROM Depends Yes Yes No Yes Yes Usually
CD-R Usually Yes Yes No Yes Yes Usually
CD-RW Usually Yes Yes No Yes Yes Usually
DVDAudio
DVDVideo
All DVD drives should play DVD-Audio or DVD-Video if the computer has DVD-Audio or DVD-Video software installed. DVD-RAM drives are questionable.

Table 4 Section 5.6. Compatibility; DVD (Read Mode).

5.6.5 Cleaning, Carrier Restoration

5.6.5.1 CDs or DVDs do not require routine cleaning if carefully handled, but any surface contamination should be removed before replay or in preparation for storage. It is important when cleaning to avoid damaging the disc surface. Particulate contamination such as dust may scratch the disc surface when cleaning, or use of harsh solvents may dissolve or affect the transparency of the polycarbonate substrate.

5.6.5.2 Use an air puffer or compressed clean air to blow off dust, or for heavier contamination the disc may be rinsed with distilled water or water based lens cleaning solutions. Care should be taken as the label dyes in many CD-Rs are water soluble. Use a soft cotton or chamois cloth for a final wipe of the disc. Never wipe the disc around the circumference, only radially from the centre to the outside of the disc - this avoids the risk of a concentric scratch damaging long sections of sequential data. Avoid using paper cleaning products or abrasive cleaners on optical discs. For severe contamination isopropyl alcohol may be used if required.

5.6.5.3 It is preferable that no repairs or polishing is undertaken on archival optical discs as these processes irreversibly alter the disc itself. However, if the disc surface (reading side) shows scratches that produce high level errors, repairs which return the disc to a playable state may be allowed for the purposes of transfer. These may include wet polishing systems providing careful testing of the effect of these restoration systems have been undertaken before being applied to important carriers. This should be undertaken by testing an expendable disc, undertaking the restoration process, and retesting to determine the effect of restoration (for further details consult ISO 18925:2002,AES 28-1997, or ANSI/NAPM IT9.21 and ISO 18927:2002/AES 38- 2000). Though some initial testing of wet polishing indicates adequate results, the removal of surface material makes sound archivists reluctant to endorse this approach. Moreover wet polishing is only effective with small scratches; discs with deep scratches deliberately inflicted with, for example a knife or scissors, will not be returned to playability by wet polishing. Damages on the label side will not benefit from any repair measures described.

5.6.5.4 Before and after cleaning and/or repairing measures and prior to the reproduction it may be advisable to first measure the CD’s or DVD’s error rates, as a minimum:

5.6.5.4.1 Frame burst errors (FBE) or Burst Error length (BERL)
5.6.5.4.2 Block error rate (BLER)
5.6.5.4.3 Correctable errors (E11, E12, E21, E22, errors before interpolation)
5.6.5.4.4 Uncorrectable errors (E32)

And preferably:
5.6.5.4.5 Radial noise and tracking error signals (RN)
5.6.5.4.6 High frequency signals (HF)
5.6.5.4.7 Dropouts (DO) 5.6.5.4.8 Focusing errors (PLAN)

5.6.5.5 There are a range of error measuring devices available for CD and DVD of varying sophistication, accuracy, and cost. A reliable tester is, however, a necessary part of a digital disc collection to determine if critical error thresholds are exceeded (cf 8.1.5 Optical Discs – Standards and 8.1.11 Testing Equipment). If after cleaning and repair one or more of the error rates exceed these thresholds refer to 5.6.3 “Selection of Best Copy”.

5.6.6 Replay Equipment

5.6.6.1 There are two fundamentally different approaches to reproduction of audio CD and DVD sources: traditional replay using format-specific reproduction equipment; or digital audio extraction (DAE) using a general purpose CD-ROM or DVD-ROM drive, commonly referred to as “ripping” or “grabbing”. The chief advantage of the data capture or ripping method is greater speed, for while traditional reproduction requires transfer in real time, data capture or “ripping” utilising high speed drives can easily transfer audio data in less than one tenth of the actual audio running time.

5.6.6.2 Digital Audio Extraction: The chief disadvantage of DAE is in error handling. The simplest “ripping” software has no error correction capability at all. More sophisticated systems make some attempt at error management but do not have the functionality to fully implement the error checking, correction and concealment that is necessary for accurate transfer, and which is built into format specific equipment. Top end professional systems promise error handling equivalent to the format specific approach, yet few have accurately implemented it.

5.6.6.3 Reproduction at rates significantly faster than real time are desirable in that this reduces the resources required to transfer audio material to the target archival system. If the DAE system can be automated, this has the added advantage of freeing staff resources for the more human resource intensive tasks of converting analogue audio to digital. Automated systems can be used appropriately if there is no loss of accuracy in the transfer process. In fact, in the better systems, there is less danger of data inconsistencies, particularly those affecting metadata but also possibly affecting the content itself.

5.6.6.4 Reproduction of digital audio data should always be accompanied by an accurate error detection and recognition system that describes and identifies exactly the kind and number of CD-specific errors and associates them with the metadata specific to that audio file. This is all the more critical where automated, faster-than-real-time processes are used to acquire the audio data.

5.6.6.5 The reproduction of an audio CD is a unique process where a somewhat subjective decision needs to be made about the success, or otherwise, of the transfer process. Unlike the transfer of audio data files, this decision can only be made by considering the error protocol. Data formats, such as .wav or BWF, can be objectively checked by bit for bit comparison between the new and old files. CD audio is not a digital file, but a coded stream of audio data, a significant difference when it comes to managing the audio integrity.

5.6.6.6 Systems which guarantee error detection and recognition including error protocol in a faster- than¡real-time mode up to a maximum of 12 times, relative to real time audio replay, are available on the market and are generally specifically aimed at the archival market.

5.6.6.7 The minimum requirement for archival use of DAE is that the DAE system must detect and alert the operator to any digital audio errors.

5.6.6.8 Format Specific Replay Approach: To transfer a CD encoded in CD-A format a stand-alone CD player may be used. The required replay equipment is a CD player with digital output, permitting ingest of the digital audio stream via a sound card with digital input. Preferred interface standard for the digital audio stream is AES/EBU. Use of the SPDIF interface can provide the same results but cable runs must be kept short. Any conversion between AES/EBU and SPDIF needs to accommodate the differences between the two standards, notably the different use of status bits that carry emphasis and copyright flags (Rumsey and Watkinson 1993). The disadvantage with this real time replay approach is that it is very time consuming, and no record of error correction is maintained in the record metadata.

5.6.6.9 Sound cards for ingest of CD audio must accommodate two channels at 16 bit 44.1 kHz. Replay equipment should be of commercial quality. Care taken in ensuring a stable vibration free mounting for the player will ensure maximum reliability of replay.

5.6.6.10 The CD player must be in good replay condition. In particular, optimum laser power is mandatory, and the laser lens should be cleaned from time to time. Devices such as disc-tuners are of no use to any replay of a CD. It is advised against using protective foils (so called CDfenders/ DVDfenders) because they may come off from the disc and damage the drive2.


2. CDs aus dem Kuhlschrank. Funkschau no. 23, 1994, p.36-39. The effect of improving the replay quality of CDs or DVDs by cooling them down in a refrigerator is so minute that though it was shown in theory (mathematically) it has never been shown in practice

5.6.7 Issues with DVD Audio (DVD-A)

5.6.7.1 DVD audio delivers 6 channels of audio at the 24 bit 96 kHz standard, and/or two channels at 24 bit 192 kHz, however digital outputs on most DVD players are limited to 16 bit 48 kHz resolution as a piracy control measure. The DVD forum has selected IEEE1394 (firewire) as the preferred digital interface for DVD Audio, using the “Audio and Music Data Transmission Protocol” (A&M protocol) (http://www.dvdforum.com/images/guideline1394V09R0_20011009c.pdf).

5.6.7.2 Decoding compressed formats such as MLP can be done by the player or at a later processing stage. Discs may include alternative versions or additional content including down mixing of surround signals to stereo, alternative tracks, accompanying video etc, requiring a policy decision as to whether all these versions are to be collected or if not which alternatives are required for the archive. It is also important that archive staff be aware that hybrid discs,such as those recorded in compliance with the Blue Book standard as Enhanced CDs, may contain other data. The extra graphical or textual data may be critical components of the work and are therefore necessary in acquiring and preserving the content.

5.6.8 Issues with Super Audio CD (SACD)

5.6.8.1 The SACD format is based on Direct Stream Digital (DSD), a 1 bit sampling technique at 2.8 MHz sampling frequency which is not directly compatible with linear PCM. At the time of writing there are limited options for ingesting this type of signal into a digital audio storage system, as most SACD players do not provide either an SACD bitstream output or a high quality PCM signal derived from the bitstream.Sony has its proprietary I-Link interface using firewire,and some third party manufacturers have marketed proprietary interfaces that can handle SACD in its native format, but there is no widely accepted digital interface standard for this format. Indications are that a suitable open standard protocol for transmission of SACD over IEEE 1394 firewire though promised, may never eventuate.

5.6.8.2 Workstations developed for SACD mastering have capabilities for input, output and processing of DSD signals (http://www.merging.com/). It should be noted that even basic processing such as gain adjustment of DSD or SACD streams requires a completely different computational approach, and therefore very different algorithms to that of PCM, consequently, the restoration and re-use of audio encoded into such formats will be limited unless converted to PCM.

5.6.9 Time factor

5.6.9.1  Time required for ingest of the audio data from optical disc in real time for conventional replay approaches a factor of two for every hour of audio. DAE approaches may reduce this by around a factor of 10, and an automated juke box system will load 60 or more CDs in a few hours without staff resources beyond the initial loading. Additional time must be allowed for selection of best copies, re-copying in the case of unacceptable errors, plus file and data management.

5.6.10 Minidisc

5.6.10.1 The original Minidisc (MiniDisk, MD) appeared in two forms: as a mass replicated disc, which works according to the principles of optical discs, and as a recordable, actually rewritable, disc, which is a magneto -optical recording medium (cf Section 8.2 Magneto Optical discs). Both sub-formats may be read by the same players. The discs are of 2.5” (64mm) diameter and housed in a cartridge. Minidisc recordings employ Adaptive Transform Acoustic Coding (ATRAC),a data reduction algorithm based on perceptual coding. Data reduced formats, although highly developed (at least in the later versions of ATRAC), not only omit data irretrievably that would otherwise be captured by a non-data reduced format, but also create artefacts in the time domain as well as in the spectral domain.Such artefacts can lead to misinterpretations of spectral components as well as of time-related components, especially when analysing the signal by means of a spectral tool. The artefacts of data reduction codecs cannot be recalculated or compensated for at a post processing stage, as they are dependant on the level, dynamics and frequency spectrum of the original signal. ATRAC is a proprietary format, with many versions and variations, and for archival purposes it is advisable to re-encode the resultant files of compressed recording formats as .wav files.

5.6.10.2 Many minidisc players have digital output which will allow the production of “pseudolinarised” data stream. The resultant file should comply with specifications laid out in chapter 2 Key digital principles and stored in accordance with that section. Metadata about the origin of such signals are imperative, as pseudolinearised signals cannot be distinguished from signals recorded without data reduction. This information would be recorded in the coding history of a BWF file, or be rendered as change history as per PREMIS recommendations (see Chapter 3 Metadata).

5.6.10.3 In 2004 the Hi-MD was marketed, and it incorporated changes to hardware which, with the new media, would record up to 1 GB of audio data.With Hi-MD it was possible to record several hours of data reduced signals, but more importantly, it was also capable of recording linear PCM signals. For archival purposes these recording should be treated like CD signals and transferred as a data stream to a suitable file storage system. Extracting audio data directly from HD-MD at higher transfer rates requires specific proprietary software, some of which is available from manufacturers’ websites. It is advisable to purchase dedicated replay equipment and software immediately as prolonged manufacturer’s support cannot be guaranteed.

5.6.10.4 The use of Minidisc as an original recording machine is not recommended (see section 5.7 Field Recording Technologies and Archival Approaches).

5.7 Field Recording Technologies and Archival Approaches

5.7.1 Introduction

5.7.1.1  Many collections are created through programs of field recording rather than, or perhaps in addition to, the acquisition and preservation transfer of historic recordings to stable digital storage formats and systems. These field recordings may be used in the creation of oral history collections, programs of traditional and other cultural performance, environment and wildlife recordings, or as part of the responsibility of broadcast collections. Regardless of the subject matter, where these recordings are destined for long term retention in archival collections it is most effective to make a decision about matters relating to their archival life at the time of planning the recording. In fact, inappropriate formats and technologies can severely limit the life and usability of the resultant audio.

5.7.1.2  Field recording may be undertaken in a variety of locations and situations, and the subject of such recording may be anything that makes a sound; from people, technology, plants or animals, to the environment itself. Recordings may be made to capture the acoustic context, i.e. in which the desired sound is recorded in an acoustic environment, or may be isolated from it, in which the recording technology may be deployed in a way which minimises the environment in which the recording is made. Recordings may be made in lounge chairs in big cities, on the verandas of remote bungalows, or where there is neither technology nor society to support it. The possibilities are virtually limitless and consequently this chapter on field recording technologies does not seek to discuss the specific discipline-related details of field recording techniques. Rather, it answers a simple question: "How do you best create a sound recording in the field in which the content is intended for long term archival storage?"

5.7.1.3  This subject of this section falls somewhat between the previous chapters on signal extraction, and the following chapters on digital storage technologies. It is included here, as it addresses the creation of digital audio content which is ingested into the digital storage systems as per the following chapters.

5.7.2 Field Recording Standards

5.7.2.1 The same technical recording standards apply to field recordings as they do to archival transfers; i.e. they should be captured and stored in a widely used, standard linear audio file format, normally .wav or BWF .wav format; they should be created with a suitable sampling rate; at least 48 kHz, but, depending on intentions, possibly higher, either 96 kHz or maybe in some circumstances 192 kHz or higher. It is advisable to make recordings at 24 bit. Lower rates will not reflect the dynamic range of the performance and the environment in which the recording is made and could well result in low level signals of very poor quality.

5.7.2.2 Whatever the recording resolution, it is advisable to record natively to a standard format. This allows direct transfer to archival storage without alteration of the format and simplifies the archiving processes. Using BWF facilitates the collection of critical metadata which is necessary to the life cycle of archival digital information.

5.7.2.3 The use of data reduced (popularly called compressed) recording formats, such as MP3 or ATRAC encoding will produce recordings which do not meet archival standards. Data reduced formats, although highly developed, not only omit data irretrievably that would otherwise be captured by a non-data reduced format, but also create artefacts in the time domain as well as in the spectral domain. Such artefacts can lead to misinterpretations of spectral components as well as of time-related components, especially when analysing the signal by means of a spectral tool. The artefacts of data reduction codecs cannot be recalculated or compensated for at a post processing stage, as they are dependant on the level, dynamics and frequency spectrum of the original signal. For archival purposes it is advisable to re-encode the resultant files of compressed recording formats as .wav files (this is also the case with Minidisc,and early technology which used lossy codecs (See 5.6.10 Minidisc). While this does not replace the missing data, it does reduce further dependence on the codecs.

5.7.3 Selection of Recording Equipment

5.7.3.1  The decision about the use of a particular piece of recording equipment depends on many matters. There are, however, a number of technical issues common to all field recording situations and these can be grouped under three headings: archival compatibility, audio quality, and reliability.

5.7.3.2  Archival compatibility

5.7.3.2.1 The choice of the recording format in the digital domain has a long, and irreversible, impact on archival life: e.g. lossy compression formats may reduce particular usages. For this reason the recording device should be chosen according to the archival compatibility of its recording format. Current technology offers the possibility of recording directly to a file based format using hard disk and solid state recorders. Such devices usually provide a choice of several linear and data reduced recording formats. The selection of .wav or BWF .wav is recommended. Raw or proprietary formats should be avoided as these often have to be transferred to .wav or BWF .wav via proprietary software for future long term archiving. In keeping with archival recommendations, data reduced recording formats should not be used.

5.7.3.2.2 An alternative to dedicated portable recorders is a suitably equipped laptop computer. With the addition of a high quality microphone pre amp and analogue to digital convertor (see Section 2.4 Analogue to Digital Converters (A/D)) sound can be directly recorded to a laptop using widely available recording software. The same recommendations regarding file format applies to laptops as well, i.e. it is generally best to record directly in the storage format. This solution is practical, but high power consumption, as well as the acoustic noise which may be generated by the laptop itself, and the conspicuousness of the computer make this suitable for only some situations.

5.7.3.2.3 The laptop, and many of the portable recording devices, can be configured to record simultaneously to an external hard disk. This additional safety strategy is outlined in 5.7.5.1 (Transfer and Backup of content in the Field).

5.7.3.3 Audio quality

5.7.3.3.1 The audio quality should be chosen according to archival recommendations in Chapter 2, Key Digital Principles. The requirement for good quality recording applies to all types of content. Contrary to widespread opinion, spoken word recordings benefit from the same high resolution as music recordings, in fact it may be argued that the dynamics of speech places more demands on recording technology than many forms of music. In addition, if detailed signal analysis (e.g. formant / transient consonant analysis etc.) is required, the higher quality is a necessity.

5.7.3.4 Microphones

5.7.3.4.1 The discussion below regarding microphones is limited to issues related to the creation of archival recordings. Much more can be said about microphones as these are, in essence, the tools used in the most creative and manipulable part of the process and it is recommended that any field recordist be familiar with the use of microphones.

5.7.3.4.2 The use of external microphones, separate from the recorder, is recommended in the majority of recording situations. This minimises the inherent system noise captured by inbuilt microphones, and avoids handling noise when operating the recorder. The quality of the microphones should be sufficient to match the needs of the recording task as well as the specifications of the recording device, noting especially the signal to noise ratio (SNR). In order to capture the full dynamic range possible, and hence record 24 bit recordings, the use of good quality external microphones with a suitable preamplifier are necessary as most of the lower quality recording devices and microphones compromise at this crucial point.

5.7.3.4.3 In some recording situations the positional characteristics associated with the event are important. To capture such information a pair of external microphones deployed in a standard array is required (see Section 5.7.4.3 below). A standardised microphone array will provide comprehensible stereo sound characteristics whereas fixed internal microphones, as provided by many devices, usually do not match any standardised microphone array and are not manipulable. Condenser microphones are the most sensitive, and generally preferred for best recording results. Condenser microphones need phantom power which is normally provided by a professional recording device, (ideally switchable) but can also be provided by an external battery or mains powered supply. Condenser microphones tend to be more likely to be damaged in poor conditions and it may be preferable to trade off sensitivity and use more robust microphones such as dynamic microphones in some situations. Condensor microphones are also quite expensive, and very good results can be achieved with some of the higher quality electret-condenser microphones which, having a permanently charged capsule, can operate for extended periods of time on a small battery. Outdoor recording, especially with condenser or electret-condenser microphones, requires adequate high quality wind shields. Incorrect and ad hoc wind shields can be detrimental to the recording characteristics and alter the polar patterns of the microphones making the recording less predictable. Users should be aware of this effect when selecting and using windshields.

5.7.3.5 Reliability

5.7.3.5.1 Unreliable equipment has the potential to lose already recorded material or fail just when it is required for a recording. To minimise the risk of failure, recording equipment should be chosen to give the best possible reliability. Low cost consumer-grade devices are in many cases, flimsy and insubstantial, and easily subject to damage, and should not be used in the field before being extensively tested. In addition to more robust construction professional devices offer more reliable circuitry and interfaces, such as balanced microphone inputs, and so allow long cable runs and more reliable professional connectors. Even though low cost equipment is more likely to be susceptible to damage and failure, cost should only be an indicator of reliability and all field equipment should be tested extensively before being used in the field.

5.7.3.6 Testing and maintenance

5.7.3.6.1 Regardless of cost or quality, all recording equipment should be regularly tested and maintained to ensure accurate and reliable functionality especially under field conditions. The integrity of the recording system should be tested, especially after equipment has been dropped or transported under irregular conditions. The frequency response of microphones should be regularly measured to ensure they are functioning adequately. Dust and humidity protection is vital in keeping equipment in good working condition. Regular checking and cleaning of the devices, including connectors and other surfaces is vital to maintaining a reliable recording device. Equipment should be allowed to acclimatise to changing environmental conditions, especially when moved from a cool dry environment, such as a plane’s cargo hold, to a hot humid environment. All test results should be kept to allow the production of a continuous report of the maintenance condition of field equipment and to foresee necessary exchange of components.

5.7.3.7 Other considerations for field recording equipment

5.7.3.7.1 Though the technical specifications and characteristics help determine the quality and reliability of a recording device, other practical issues can impact on the choice of equipment according to the envisaged recording situation.Important features include;adequate recording time when battery-supplied; a rugged and clear design; easy handling; and a small and light-weighted but robust construction. Illuminated controls are essential for recording in the dark but result in higher battery consumption. A decision should be made as to whether the recording situation makes devices with changeable media (such as Flash or SD cards) or a back up hard disk preferable to enable a suitable safety strategy (see Section 5.7.5 Transfer and Backup of content in the Field). Ideally the device should allow fast and simple data transfer and duplication, and have an inconspicuous design (the latter of which reduces the visual impact on a documentary recording, and may also minimise the risk of theft).

5.7.4 Approach to recording

5.7.4.1 The purpose of the recording and the rules of the particular discipline to which it belongs will govern many aspects of recording approaches, microphone techniques and the like. There are, however, a number of common concerns in making such a recording.

5.7.4.2 Field recordings usually record or document a given situation and under these circumstances the original dynamics of the documented action should be respected in the recording as well. Audio input levelling should orientate on the wanted signal, and not the general background noise, and continuous adjustment of the level during a recording should be done judiciously, if at all. Use of automatic gain control functions is not recommended as such features falsify original dynamics by raising low level parts (and therefore noise) and reducing the wanted signal dynamics. Likewise any limiters used in a recording should be applied cautiously. A well adjusted limiter will rescue the recording if an unexpected high level signal is captured but have absolutely no impact on the majority of the recording because it is not triggered by the level of the recording. On the other hand, a poorly adjusted limiter may simulate a perfect level on the meters of the recording device while the microphone itself may already be overloaded due to the input signal.Whenever possible, manual levelling is to be preferred and any limiter, adjusted so as it has no impact on the normal signal, only switched in after an optimum level has been achieved.

5.7.4.3  When making a recording where the signal is embedded in a noisy environment advantages are to be found in using standard stereo microphone arrays. There are many approaches but those that are briefly discussed here include near-coincident technique of which ORTF (Office de Radiodiffusion Télévision Francaise) is an example, XY crossed pair,AB parallel pair and MS (Mid-Side) techniques.

5.7.4.4 ORTF seems to be most useful where analysis and evaluation properties of the documentary recording are an important requirement. In this technique the microphone capsules are separated by 17cm at an angle of 110º. An ORTF recording, when analysed via headphones, enhance the ear and brain’s ability to trace a wanted signal within a noisy surrounding; the so called “cocktail party effect”. The head-related binaural microphone array imparts the extra information and so helps identifying wanted signals in noisy sound fields. Also, as the specification for ORTF is defined, the microphone set-up can be much more easily replicated in a standard way.

5.7.4.5  Standard XY crossed pairs are arranged so that the microphone capsules are as close together as possible, but pointing at least 90º away from each other. The intensity of the signal information is recorded, but ideally no phase difference is noted. This technique produces a recording that reproduces well on speakers, but does not have as much separation information as other techniques. AB parallel pair uses two omni-directional microphones in parallel separated by around 50cm. This technique has been favoured in very good acoustic environments but will rarely produce acceptable results in very noisy environments. It may have phase cancellation problems when summed to mono.

5.7.4.6  MS (Mid-Side) technique places a bidirectional microphone (figure 8) at right angles to the sound source, and a cardioid pick up pattern microphone (or sometime an omni directional microphone) pointing at the sound source. The two recorded signal may then be manipulated to produce mono compatible stereo recording (M+S, M-S). If recorded as MS information, the signal may also be manipulated after the event, and so gain some control over the apparent spread of microphones.

5.7.4.7  Some situations, where the exact nature of the event is unknown prior to the recording being made, can take advantage of movable directional microphones, multi-microphone techniques and multi¡track recording. Interviews may use two microphones pointed at the participating individuals, which presents very acceptable recordings. Clip microphones are, in many cases, less useful, as they pick up unwanted noise from body movements, breathing, clothing and jewellery, and record little or no information about the environment in which the recording was made, which is often an integral and necessary part of the field recording.

5.7.4.8  Microphone techniques contribute to the quality of the recorded content and this very brief consideration of them is only a guide to the possibilities. It is recommended that all those intending to make recordings in the field should become familiar with the possibilities afforded by good microphone techniques before making important recordings.

5.7.5 Transfer and Backup of content in the Field

5.7.5.1 Field recordings remain vulnerable while in the field, and unless back up copies are created, are at risk of being lost. A second copy of a field recording should be made at the time of recording or as soon as possible after the recording is completed. Different workflows and situations make for different approaches, but generally speaking, the workflow selected should offer the best possible safety strategy.

5.7.5.2 Hard disk and solid state recorders offer a file based recording technology either on hard disks or on changeable card media. The recording is generally deleted from either of these media after the wanted file is transferred to another storage environment. This is clearly an area of risk in the use of the new technology and must be managed carefully to ensure no loss of wanted material. The recording medium should be regarded as an original carrier as long as possible. It should be erased only after verifying the correct data transfer into an archival system. In the case where a long field trip requires the management of large amounts of data which cannot be immediately archived, duplicates should be created and stored in the field. In the case of flash card or SD recorders it may be useful to invest in additional storage cards which are used to store recording until recorded content is transferred to a more sustainable storage system. In the case of hard disk or laptop recording devices, portable hard disk storage devices can be used to create backup copies until the data has been successfully transferred.

5.7.5.3 In practice, some devices offer parallel use of internal hard disk and storage cards, or allow the parallel recording to hard disk. This is an advantage as it enables the automatic creation of a safety copy as part of the recording process and should be undertaken whenever possible. Alternately, safety copies can be manually created in the field, using external hard disks, laptops or at least CD/ DVD drives.

5.7.5.4 Some devices create file names automatically when a new storage medium is inserted (automated numbering starting with the same file name on each new medium), so the copy process has to be carefully managed to be sure that files named the same on different carriers can be correctly matched with the correspondent metadata/ field notes etc. In the worst case this can lead to accidental erasure of identically named files and so a careful structure and naming strategy is necessary. Renaming the files after the copy process is recommended, provided that the original file is not changed or manipulated in some other way.

5.7.6 Metadata and Collection Description

5.7.6.1 The absence of metadata describing the field recording, its context and related rights, severely limits the value of the recording. The lack of metadata (including preservation metadata) can have serious implications not only for ingestion into a repository, but also for subsequent archival management and dissemination of archival information. This data is so significant that its lack may lead an archives manager to reject the content. There is also critical technical and preservation information necessary to acquiring field recordings which should be obtained at the time of recording and included in the archival record. These include:

5.7.6.1.1 Recording device: Brand, model number, description of dynamically made adjustments during the course of the recording, recording level, recording format, encoding (not recommended but should circumstances require its use, it must be documented).

5.7.6.1.2 Microphones: microphone types/ polar pattern, information about the microphone array, distance, special approach (like clip microphones, analytic multi microphone technique etc).

5.7.6.1.3 Use of additional equipment such as windshields etc. description of room situation, etc.

5.7.6.1.4 Carrier: type, specifications of original carrier (flash card, disk etc) or hard disk.

5.7.6.1.5 Power source: batteries, 50 or 60 Hz AC, unstable or fluctuating power conditions, etc.

5.7.7 Metadata and Field Tools

5.7.7.1 Field recordings exist in relation to each other and to other events, objects and information. Developments in the research communities are leading towards integrated data and metadata acquisition tools which document and relate different objects and the times and place in which they were created.Various international projects meanwhile have created tools that meet the requirements of specific metadata schemes. Such tools offer a relatively complete metadata collection and make transfer to established database systems easier and ensure accurate data for future researchers. At the time of writing such tools and concepts are in an early stage of development, they also tend to contain data that is discipline specific and so are not discussed here, however, it is important that all the technical data described above is acquired for populating future management and access systems. All data acquired should have in mind the transfer compatibility to the final archiving system. Until standards come into being, use of UNICODE characters and XML format is recommended.

5.7.7.2 If metadata is collected manually, without using acquisition tools as mentioned above, it is recommended to use a format that can easily be transferred to usual database structures. Alternatively, institutes and archives sometimes provide their individual tools and if possible these should be used in the field.

5.7.8 Time Factor

5.7.8.1 The time required to record an important event or interview can be quite extensive. The time required to preserve a field recording can be reduced to the time it takes to ingest the data and metadata if the field recording approach has been designed properly. If the system depends on manual approaches it is quite likely that much valuable information will be lost due to human error, or lack of resources to undertake this time consuming, but important, archival task.

6: Preservation Target Formats and Systems

6.1.1 Introduction

6.1.1.1 The following information on the management, long term storage and preservation of digitally encoded audio is based on the premise that there is no ultimate, permanent storage media, nor will there be in the foreseeable future. Instead, those managing digital audio archives must plan to implement preservation management and storage systems which are designed to support processes that go with the inevitable change in format, carrier or other technologies. The rate and direction of technological change is something over which archives have no control and very little influence. The aim and emphasis in digital preservation is to build sustainable systems rather than permanent carriers.

6.1.1.2 The choice of technological storage system is dependent on many factors, of which cost is but one. Though the type of technology selected for preserving a collection may differ according to the specific circumstances of the individual institution and its circumstances, the basic principles outlined here apply to any approach to management and long term storage of digital audio.

6.1.2 Data or Audio Specific Storage

6.1.2.1  To effectively manage and maintain digital audio it is necessary to transform it to a standard data format. Data formats are the file types, such as .wav, BWF, or AIFF, which computer systems recognise. These files, unlike audio specific carriers, technologically define the limits of their own content and are generally encoded in such a way that a loss of data is recognised and remedied by the host system. IASA recommends the use of BWF as defined in Section 2.8 File Formats.

6.1.2.2  Audio specific recording formats which have been available in the past include DAT (Digital Audio Tape) and CD-DA (Compact Disc-Digital Audio). DAT, though once largely used for the remote or field recording of 16 bit, 48 kHz audio is now an obsolete recording system. IASA recommends that any significant content recorded on DAT tape be transferred to a more reliable storage system in accordance with the guidance provided in section 5.5 Reproduction of Digital Magnetic Carriers.

6.1.2.3  The recordable compact disc can be used to record audio in either audio-only (CD-A or CD-DA) or data (CD-ROM) formats. In CD-DA format the encoded digital audio resembles an audio stream and so does not have the advantages of a closed file such as might be recorded on the CD-ROM formatted disc. In the latter though, less data can be stored on the same amount of disc space. IASA does not recommend recording audio in CD-DA form as a preservation target format. There are considerable risks associated with using a recordable CD as a target format in any form and those risks are outlined in Chapter 8 Optical Disks: CD/DVD Recordable. The ever reducing prices and increasing reliability of data management and storage systems make media specific storage approaches, such as CD-R, unnecessary, or at least uneconomic.

6.1.3 Principles of Digital Preservation

6.1.3.1 Digital Mass Storage Systems (DMSS) Principles

6.1.3.2 The following information is based very closely on the practical aspects of Data Protection Strategies from the UNESCO Guidelines for the Preservation of Digital Heritage. It is modified only to reflect the possibility of systems that incorporate non-automated back up, and to reflect the single format concerns of audio digital preservation. The section is included with the kind permission of the author (Webb 2003:16.13).

6.1.4 Practical Aspects of Data Protection Strategies

6.1.4.1 There is a reasonably standard suite of strategies used to manage data in long-term storage. Most are predicated on an assumption that the data carrier itself does not need to be preserved, only the data. The following comprises, in part, those strategies.

6.1.4.2 Allocation of responsibility: Someone must be given unambiguous responsibility for managing data storage and protection. This is a technical responsibility requiring a particular set of skills and knowledge as well as management expertise. For all collections, data storage and protection require dedicated resources, an appropriate plan and must be accountable for these strategies, and even very small collections must have access to the necessary expertise and a dedicated person responsible for that task.

6.1.4.3 Appropriate technical infrastructure to do the job: Data must be stored and managed with appropriate systems and on an appropriate carrier. There are digital asset management systems or digital object storage systems available that meet the requirements of audio digital preservation programmes, some approaches to which are discussed below. Once requirements have been determined, they should be thoroughly discussed with potential suppliers. Different systems and carriers are suited to different needs and those chosen for preservation programmes must be fit for their purpose.

6.1.4.4 The overall system must have adequate capabilities including:

6.1.4.5 Sufficient storage capacity: Storage capacity can be built up over time, but the system must be able to manage the amount of data expected to be stored within its life cycle.

6.1.4.6 As a fundamental capability, the system must be able to duplicate data as required without loss, and transfer data to new or ‘refreshed’ carriers without loss.

6.1.4.7 Demonstrated reliability and technical support to deal with problems promptly.

6.1.4.8 The ability to map file names into a file-naming scheme suitable for its storage architecture. Storage systems are based around named objects. Different systems use different architectures to organise objects. This may impose constraints on how objects are named within storage; for example, disk systems may impose a hierarchical directory structure on existing file names, different from those that would be used on a tape system. The system must allow, or preferably carry out, a mapping of system-imposed file names and existing identifiers.

6.1.4.9 The ability to manage redundant storage. As digital media has a small, but significant failure rate, redundant copies of files at every stage are a necessity, especially the final storage phase.

6.1.4.10 Error checking. A level of automated error checking is normal in most computer storage. Because audio and audio-visual materials must be kept for long periods, often with very low human usage, the system must be able to detect changes or loss of data and take appropriate action. At the very least the strategies in place must alert collection managers to potential problems, with sufficient time to allow appropriate action.

6.1.4.11 Technical infrastructure must also include means of storing metadata and of reliably linking metadata to stored digital objects. Large operations often find they need to set up digital object management systems that are linked to, but separate from, their digital mass storage system, in order to cope with the range of processes involved, and to allow metadata and work interfaces to be changed without having to change the mass storage.

6.1.5 Philosophy of System Sustainability

6.1.5.1 All technology, whether it be the hardware or software, formats or standards, will eventually change as a result of market forces, performance requirements or other needs or expectations. The task of the audio archivist charged with maintaining digital and digitised audio content is to navigate a way through these technological changes such that the content of their collections are maintained for current and future users in a reliable and authentic form in as cost effective way as can be managed.

6.1.6 Long Term Planning

6.1.6.1 Long term planning for a digital audio archive involves more than just the technical standards for a data storage system. The technical issues must be carefully resolved, but the social and economic aspects of running a digital storage system are vital to ensuring the continued access to the content. Long term planning should consider the following issues.

6.1.6.2 The sustainability of the raw data: that is the retention of the byte-stream in its proper and logical order. The data in the storage system must be returned to the system without change or corruption. It is worth noting that computer systems expertise identifies a considerable risk in the maintenance and refreshment of data, and only a well managed and designed approach to IT will ensure adequate results.

6.1.6.3 Formats and ability to replay: Digital data is only useful in a sound archive if it can be rendered as audio in the future. The proper choice of file format ensures that the future sound archive can replay the content of the data files, or will be able to acquire the technology to migrate the files to a new format. Not incorporating a lossy compression algorithm in that format allows that future transformation process to occur without altering the original audio content.

6.1.6.4 Metadata, identification and long term access: All digital audio files must be identifiable and findable in order for that audio material to be used and the value of the content realised.

6.1.6.5 Economics and Sound Archives: this includes the continued viable existence of the institutions that support the data storage systems and repositories as well as those that own, manage, or gain value from, the digital audio stored therein. The cost of maintaining a digital audio collection is ongoing and their must be a plan and a budget that realistically plans for long term preservation of collections. The cost of curating and managing the audio collections is also ongoing. Digital preservation is as much an economic issue as a technical one. The requirements of ongoing sustainability demand at their base a source of reliable funding, necessary to ensure that the constant, albeit potentially low level, support for the sustainability of the digital content and its supporting repositories, technologies and systems can be maintained for as long as it is required.

6.1.6.6 Storage, management and preservation alternatives: Given that the economic and technical environment may well be volatile it is recommended that agreements be established between archives and institutions regarding the storage of data as archives of last resort. This would require some standard agreement about file formats and data organisation as well as social and technical aspects of management of the content.

6.1.6.7 Tools, Software and long term planning: Hardware, software and systems are not things in themselves to be preserved, but are merely tools to support the task of preserving the content. The repository software D-Space, for example, does not describe itself as a preservation solution, but only useful in “enabling institutions with a sustainable ability to retain information assets and offer services upon them.” (DSpace, Michael J. Bass et al. 2002). The repository software itself is a tool, as are the various components designed to aid in operation, simplify processes, and automate and validate the harvesting of metadata. Long term planning involves being able to change or upgrade systems without endangering the content.

6.1.7 Defining the Digital Object

6.1.7.1 The audio file is only one part of the information that is to be preserved. The Reference Model for an Open Archival Information System (OAIS) identifies four parts to the digital object, described by them as the information package. These are the content information and the preservation description information, which are packaged together with packaging information, and which is discoverable by virtue of the descriptive information.

Information Package concept & relationships

6.1.7.2 Though the information may be distributed across the storage system, it is well to remember that the conceptual package is the audio information, the ability to replay that audio, to know its provenance and to describe and find it. There may also be critical relationships between the one audio file and others in the collection, and these relationships are important to the use of the material and so must also be preserved.

6.1.8 The Open Archival Information System (OAIS)

6.1.8.1 The Reference Model for an Open Archival Information System (OAIS) is a widely adopted conceptual model for a digital repository and archival system. The OAIS reference model provides a common language and conceptual framework that digital library and preservation specialists now share. The framework has been adopted as an International Standard, ISO 14721:2003. Though some critics identify shortcomings in the detail of the OAIS, the concept of constructing repository architectures in a form that corresponds with the OAIS functional categories is critical to the development of modular storage systems with interoperable exchange of content. The following sections of the Guidelines adopt the major functional components of the OAIS reference model to assist in the analysis of the available software and to develop recommendations for necessary development.

6.1.8.2 There are a finite number of functions an archival digital repository must be able to perform in order for it to reliably and sustainably perform the purpose for which it is designed. These are defined in the Reference Model for an Open Archival Information System (OAIS) as Ingest, Access, Administration, Data Management, Preservation Planning and Archival Storage.

Open Archival Information System (OAIS)

6.1.8.3 The OAIS also defines the structure of the various information packages that are necessary for the management of the data, according to the place in the digital life cycle. These are the Submission Information Package (SIP), Dissemination Information Package (DIP) and Archival Information Package (AIP). A package is the conceptual parcel of the data and relevant metadata and descriptive information necessary to the particular object. This object is conceptual only in the sense that the package contents may be dispersed in the system or collapsed into a single digital object. OAIS defines an information package as the Content Information and associated Preservation Description Information which is needed to aid in the preservation of the Content Information.

6.1.8.4 The SIP is an Information Package that is delivered to the system for ingest. It contains the data to be stored and all the necessary related metadata about object. The SIP is accepted into the system and used to create an AIP.

6.1.8.5 The AIP is an Information Package which is stored and preserved within the system. It is the information package the system stores, preserves and sustains.

6.1.8.6 The DIP is the information package created to distribute the digital content. There are three roles in this system. First is access, and this DIP would be in a form that the users can use and understand. Second is exchange for the purpose of distributing risk. An archival repository may choose to share parts of its content with other similar institutions, or with an organisation whose role is archival storage. In this case the DIP would contain all the relevant metadata necessary to undertake this role. The third is for distributing content to archives as a last resort. The scenario where a particular archive or institution no longer has the resources to maintain its collection is not difficult to imagine. A standard DIP for this purpose allows other similarly architected systems to undertake the role with the minimum of manual intervention.

6.1.9 Trusted Digital Repositories (TDR) and Institutional Responsibility

6.1.9.1 The technical specification of the digital storage environment is an important part of ensuring that the digital content that is managed is still accessible to researchers in the future. It is not of its own, however, enough to ensure that this will be achieved. The institution within which the digital archive resides has to be able to ensure that the content it manages is curated and maintained responsibly. In 2002 the Research Libraries Group (RLG) and the Online Computer Library Center (OCLC) jointly published “Trusted Digital Repositories: Attributes and Responsibilities” (TDR), which articulated a framework of attributes and responsibilities for trusted, reliable, sustainable digital repositories which were “required for an archive to provide permanent or indefinite long-term preservation of digital information”.

6.1.9.2 These attributes include compliance with the OAIS reference model, organisational viability, financial sustainability, technological and procedural suitability, the security of the system and the existence of appropriate policies to ensure that the steps are taken to manage and preserve the data.

6.1.9.3 The practical instantiation of this is a document known as the “Trustworthy Repositories Audit and Certification (TRAC): Criteria and Checklist” (2007). Using this document an archival institution can establish whether the practices, approaches and technologies they have or are planning to implement are appropriate to the permanent preservation of the digital information for which they have responsibility.

6.1.9.4 The concern which the checklist addresses incorporates three main areas: organisational infrastructure; digital object management and technologies; and technical infrastructure and security.

6.1.9.5 Organisational infrastructure provides a series of checks against appropriate governance and organisational viability, organisational structure and staffing, procedural accountability and policy framework, financial sustainability and a consideration of the licenses, and liabilities. Digital object management section considers the acquisition of content, the creation of an archivable package, planning for preservation, archival storage and planning, information management and access control. The third part of this checklist audits the system infrastructure, the use of technologies appropriate to the tasks and system and institution security.

6.1.9.6 The terminology used in the “Trustworthy Repositories Audit & Certification (TRAC): Criteria and Checklist” is chosen to represent digital archives in the broadest sense of the word, and so the document’s meaning may occasionally appear opaque to an audio archivist. Nonetheless, the issues examined and tested by it are critical to the planning and management of a digital audio archive. It is strongly recommended that the digital sound archivist uses the checklist to examine the suitability of an institution to manage a digital collection, or to identify weaknesses within an existing digital preservation strategy.

6.1.10 Audio Archives and Technical Responsibility

6.1.10.1 Though a particular institution may be responsible for the management of a collection or set of audio items, it does not necessarily follow that institution will undertake the responsibility for maintaining the digital storage system. An institution may instead become a part of a distributed storage system,or may identify a third party provider to archive their content in a more standard approach.

6.1.10.2 A distributed data storage approach such as that being promoted and developed for web based material by Stanford University under the name of LOCKSS (Lots of Copies Keep Stuff Safe) replicates data in a number of places on the web. The system manages the data on the grid and risk of loss of data is reduced because the information can be found in many different places. Such a system is not appropriate for material which has access restrictions or copyright which prohibits dissemination. Such a system also requires that a development and management responsibility to be shouldered by an institution.

6.1.10.3 An institution may decide that they do not have the technical capability to undertake the development and management of a digital storage system. In this case they may establish a relationship with a third party provider. That provider may be another archive which will take the collection and store its content, or may be a commercial provider who will provide and manage the storage and content for a fee.

6.1.10.4 The information provided here is provided as though the institution is intending to take on its own preservation. However, if any of the above alternatives are considered, then this information is useful for determining if those approaches are reliable and valid.

6.1.11 Digital Repository Software, Data Management and Preservation Systems

6.1.11.1 Digital repository software is generally that software which supports storage and access to the digital content. It should incorporate indexing and metadata systems that manage information about the content, and a variety of tools to find and report on the content.

6.1.11.2 Data management is the management of the byte stream, or data, that the system is responsible for. This may include back up procedures, multiple copies and changes.

6.1.11.3 Preservation processes are those that ensure the content remain accessible in the long term, that the content is still meaningful and that the data management system’s tasks are documented and maintained. All three of these steps are necessary to achieve long term preservation to content.

6.2 Ingest

6.2.1 Submission Information Package (SIP)

6.2.1.1 The SIP is an Information Package that is delivered to the repository and digital storage system for ingest. The SIP includes the audio data to be stored and all the necessary related metadata about the object and its content. Ingest, in the OAIS model, is the process that accepts the content and all its related metadata (SIP), verifies the file, extracts the relevant data and prepares the AIP for storage, and ensures that AIPs and their supporting Descriptive Information become established within the OAIS.

6.2.1.2 A digital repository and preservation system should be able to accept and validate an audio file. Validation is a process that ensures that the files which are being accepted into the digital storage system comply with the standards. Non standard files may become difficult to use in the future when current replay systems no longer exist. Tools exist for automated validation of file formats, and some open source solutions, like JHOVE (JSTOR/Harvard Object Validation Environment), are available and being further developed.

6.2.2 Format

6.2.2.1  IASA recommends the use of .wav or preferably BWF .wav files [EBU tech 3285]. The difference between the two is that the BWF contains a set of headers which can be used to organise and manage metadata. Though BWF metadata is adequate for many purposes, in some sophisticated systems and exchange situations a more comprehensive package is required, and in these circumstances Metadata Encoding and Transmission Standard (METS) is often used. The METS schema is a standard for encoding descriptive, administrative, and structural metadata regarding objects within a digital library, expressed using XML (eXtensible Markup Language). A METS package, which consists of metadata and content, is often used as an exchange standard between digital archives.

6.2.2.2  Material eXchange Format (MXF) is a container format for professional digital video and audio media defined by a set of SMPTE standards. MXF has been mostly taken up by the video archiving community, though it is capable of managing audio. Like METS, it is primarily a set of metadata which “wraps” the content, in this case, audio. Both these are very useful formats in the exchange and management of content and information between archives and repositories.

6.2.2.3  The format of the SIP will depend on the system and the size and sophistication of the enterprise. It is quite possible to establish a viable archive using .wav files and manually entering most of the necessary metadata into the system by hand, and acquiring the necessary technical metadata at the ingest stage. This however, would only be appropriate for the smallest of collections. Large collections with remote and separate digitisation processes and large quantities of material must build sophisticated ingest and data exchange systems to ensure the content is adequately ingested into the data storage systems. Production and verification software generates much of this data as standardised XML-files that may be used for preservation purposes. The National Library of New Zealand Metadata Extractor tool, for example, is a Java-based tool that extracts preservation metadata from digital objects and outputs that metadata in a standard format (XML).

6.2.3 Preservation Metadata

6.2.3.1 The metadata needed to manage preservation processes at the ingest stage is all the information regarding the creation of the digital audio object and the changes to format that have occurred prior to ingest. In this way the technical provenance of the object is preserved, which allows a pathway between the present form of the item and original from which it was created to be traced.

6.2.3.2 BWF has a non-compulsory recommendation for BWF entitled “Format for CodingHistory field in Broadcast Wave Format” http://www.ebu.ch/CMSimages/en/tec_text_r98-1999_tcm6-4709.pdf which describes how changes to the file may be described. Local usage of the ASCII free text field allows the description of the technical equipment or software that was used in the creation of the digital audio object.

6.3 Archival Storage

6.3.1 Archival Information Package (AIP)

6.3.1.1  The definition of the term Archival Storage in OAIS includes the services and functions necessary for the storage of theArchival Information Package (AIP). Archival storage encompasses data management and includes processes such as storage media selection, transfer of AIP to storage system, data security and validity, backup and data restoration, and reproduction of AIP to new media.

6.3.1.2  AIP, as defined in OAIS reference model (CCSDS 650.0-B-1 Reference Model for an Open Archival Information System (OAIS)), is an information package that is used to transmit archival objects into a digital archival system, store the objects within the system, and transmit objects from the system. An AIP contains both metadata that describes the structure and content of an archived essence and the actual essence itself. It consists of multiple data files that hold either a logically or physically packaged entity. The implementation of AIP can vary from archive to another archive; it specifies, however, a container that contains all the necessary information to allow long term preservation and access to archival holdings. The metadata model of OAIS is based on METS specifications.

6.3.1.3  From physical point of view the AIP contains three parts; metadata, essence and packaging information, which all consists of one or more files (see 6.1.3 Defining the Digital Object). Packaging information can be thought as wrapper information and it encapsulates metadata and essence components.

6.3.2 Archival Storage basics

6.3.2.1  Archival Storage provides the means to store, preserve and provide access to archived content. In small systems the storage can stand alone and may be manually operated, but in larger systems storage is usually implemented in conjunction with cataloguing applications, asset management systems, information retrieval systems and access control systems in order to control and manage archived content and provide a controlled way to access them.

6.3.2.2  Archival Storage must be connected to equipment that ingests and creates the digital asset to be archived, and it must provide a secure and reliable interface that can be used to import assets to the storage system.

6.3.2.3  A system that is used to store archival content must be reliable in several ways: It must be available for use without any significant interruptions, and it must be able to report to the system or user who imports content whether the import was successful or not, thus enabling the importing party to delete the ingest copy of the of the archival file if appropriate. Archival Storage must also be able to preserve the content it manages for a long period of time and be able to protect the content from all kinds of failures and disasters.

6.3.2.4  An Archival Storage system should be built according to the needs of its functional owner: it must be correctly-sized to carry out the tasks that are needed, and manage the capacities that are required in every day operations. In addition, Archival Storage must provide controlled access to the content it manages for the users who have permissions or rights to access the content.

6.3.3 Digital Mass Storage Systems (DMSS)

6.3.3.1  A Digital Mass Storage System refers to an IT based system that has been planned and built to be able to store and maintain large amounts of data for a given or extended period of time. These systems come in many forms; a basic DMSS could be a personal computer which has large enough hard disk drive and some kind of catalogue that can be used to keep track of the assets the system possesses. A more complex DMSS may consist of hard disk drive and/or tape storage and group of computers that control the storage entity. A DMSS can also contain many tiers of storage with different characteristics; a fast Fibre Channel based hard disk drive tier can be used to cache assets whose access time is critical while a tier built of cheaper hard disk drives could be used to hold material whose access time is not so critical, and finally tape based storage can be used as the most cost-effective tier of storage.

6.3.3.2  When a number of different storage technologies are used in a large system to build the functional entity, a HSM (Hierarchical Storage Management) system is usually deployed in such a way that it supports the different technologies working together. Larger scale systems may also be distributed geographically in order to achieve better performance and make the system more fault tolerant.

6.3.4 Data Tape Types and Formats Introduction

6.3.4.1  The following is an outline of some of the main data tape formats and tape automation systems that may be used for storing AV content in data form. Data tapes are only used in conjunction with other components of a DMSS. It is prudent to commence a section on comparison of the various data tape formats with a reminder that no carrier is permanent and that, all things being equal, they will only be viable as long as the data systems in which they are incorporated continue to support them.

6.3.5 Data Tape Performance

6.3.5.1  Format geometry and dimensions govern performance. Data transfer speed, one aspect of performance, is a direct product of the number of tracks written and read simultaneously, as well as the tape- head speed, linear density and the channel-code. Similarly, physically smaller, lighter tape housings are faster to move in a robotic library. Data density is a product of:

6.3.5.1.1  tape length and thickness trade-offs
6.3.5.1.2  track width and pitch
6.3.5.1.3  linear density of data payload within each track

6.3.6 Tape Coatings

6.3.6.1  There are two main types of tape coatings: particulate and evaporated. The earliest coated data tapes used metal oxides similar to video tape, whereas more recent data tapes use metal particles (MP). Pure iron with inert ceramic and oxide passivation layers is dispersed in polymer binders which are applied evenly to a PET or PEN base-film or substrate which in turn provides dimensional stability and strength under tension. Some of the highest density data tapes currently on the market now use evaporated metal foil coatings of cobalt alloys and similar material to those used on hard disks. This achieves a much higher purity of magnetic material and allows thinner coatings. Most metal-evaporated (ME) tapes have a protective polymer coating similar to the binder material on MP tapes. The more recent formulations include a ceramic protective layer as well. Several of the early ME tapes failed during heavy usage due to de-lamination (Osaki 1993:11).

6.3.7 Tape Housing Design

6.3.7.1  Two basic styles of housings are used, dual-hub cassettes, which may enable faster access times and single-hub cartridges, which offer greater capacity within a given external volume.

6.3.7.2  Dual hub cassettes include:

3.81mm, principally DDS [derived from DAT]
QIC [quarter-inch cartridge] and Travan
8mm formats, including Exabyte and AIT
DTF
Storagetek 9840

6.3.7.3  Single-hub cartridges include:

IBM MTC and Magstar formats such as 3590, 3592 and TS1120
Quantum S-DLT and DLT-S4
LTO Ultrium [100, 200, 400 & 800 GB]
Storagetek 9940 and T10000
Sony S-AIT

6.3.7.4  Neither design is necessarily superior for long-term archiving, since the life is governed by a range of details specific to each format.For instance,some models of the single-ended half-inch cartridges have large-diameter guides within the housing, which ensure minimum friction and accurate tape guidance. Problems have been experienced with the leader latching mechanism on older single-ended cartridges, although more recent designs have improved reliability in this area. Some dual-hub cassettes can be positioned to park halfway along the tape to minimise the amount of spooling time to any particular file. This contradicts the traditional practice in AV archives of spooling tapes carefully to one end before storage so that only leader tape is exposed to the threading mechanism. Tapes generally don’t incorporate a hermetically sealed enclosure in the way that hard disks are protected.

6.3.8 Linearly and Helically Scanned Tapes

6.3.8.1 Data tapes may be written or read with a fixed head, generally described as linear, or with a rotating or helical head. Linear tapes typically follow a serpentine track layout, and it has been argued that this shuttling can lead to wear or a so-called shoe-shine effect. In practice, modern tapes are designed to last for large numbers of passes, however, it is still prudent to access frequently used content from hard disc. Tapes, which experience chemical decomposition from hydrolysis and other causes, will usually run better over fixed guides and components in the tape path at speeds of around1-2 m/s or greater, which are typical of fixed- head or linear formats. Rotary-head or helical formats typically have higher tape-head speeds which create a greater air-bearing effect between the tape surface and the read-write heads, but the linear tape speed over the fixed guides and heads is much slower, so this is where fouling often occurs.

6.3.9 Ancillary Storage and Access Devices

6.3.9.1 Formats such as AIT include solid-state ‘Memory in Cassette’ or MIC which stores file positional information similar to a Table of Contents (TOC) on Compact Disks for rapid location of data. DTF uses rf memory.

6.3.10 Format Obsolescence and Technology Cycles

6.3.10.1  The inherent nature of data storage is of constant progress and development, which means inevitable change, and ongoing obsolescence. Realistic long-term management of content must accept and build upon the continuing evolution and upgrading of hardware and media. Although central infrastructure such as data cabling or storage libraries may remain in operation for ten or twenty years, individual tape drives and media have a finite life much shorter than this. All of the main data tape formats have development roadmaps projecting upgrades every 18 months to 2 years. Backward compatibility for read-only access is sometimes assured over one or two generations of media within any common family. As a result,each generation of tape drives and media may be viable for 4 to 6 years, after which time it is essential to migrate the data and move on.1  Also the hardware maintenance cost of mass storage systems tend to rise notably when the system gets older than its projected life or the guarantee period ends. After this it may be difficult to obtain new spare parts for the tape libraries or tape drives, for example. A summary of projected roadmaps is presented below. Many formats have read-only compatibility with at least one previous generation.

Family 1st Generation 2nd Generation 3rd Generation 4th Generation 5th Generation 6th Generation
Quantum SDLT SDLT220 110GBytes SDLT320 160GBytes SDLT600 300GBytes DLT-S4 800GBytes    
IBM     3592 2004 300GB 40MB/s TS1120 2006 700GB 104MB/s    
Sun - Storagetek   9940B 2002 200GB 30MB/s T10000 2006 500GB 120MB/s T10000B-2008 ITB 120MB/s    
LTO LTO-1 2001 100GB 20MB/s LTO-2 2003 200GB 40MB/s LTO-3 2004 400GB 80MB/s LTO-4 2007 800GB 120MB/s LTO-5 no date (2009+) 1.6TB 180MB/s (estimated) LTO-6 no date (2011+) 3.2TB 270MB/s (estimated)
Sony S-AIT S-AIT 2003 500 GB 30MB/s S-AIT2 2006 800 GB 45MB/s        
Sony AIT     AIT-3 2003 100 GB 12MB/s AIT-4 2005 200 GB 24MB/s    

Table 1 Section 6.3: Projected Development Roadmap for Data Tapes


1. This implies a degree of waste and environmental pressure beyond the scope of our purely technological discussion, but in reality, a large-scale library of older data tapes will consume more polymers and require more petrochemicals for manufacture than a newer, high-density system with more energy-efficient drives and robotics, occupying less real-estate at the same time

6.3.11 Automated Robotics or Manual Retrieval

6.3.11.1  For small-scale operations it is possible to back up data from a single workstation onto a single data tape drive and manually load tapes for storage on traditional shelving, and even small scale networked systems will undertake manual backup of their storage (see also Chapter 7 Small Scale Approaches to Digital Storage Systems). The same guidelines for storage environments apply as for other magnetic tapes,though increased attention to minimising the presence of dust and other particulates and pollutants would be beneficial.For larger-scale operations,particularly in countries where labour costs are high, and capital equipment budgets are favourable, a degree of automation is normally desirable and more economical than purely manual systems. The degree of automation depends upon the scale and consistency of the task, type of access to the content, and the relative costs of the main resources.

6.3.11.2   Autoloaders and Robotic Tape Libraries: The next step from single drives is the small-scale auto¡loader, which usually has one drive (occasionally two), and a single row or carousel of data tapes which are fed in sequentially to support backup operations. One of the key differences between autoloaders, and large-scale robotic libraries is that the recorded tapes are not logged by the backup software in a central database which can then enable automated retrieval. The task of searching, retrieving and reloading individual files still falls to a human operator. All that autoloaders do, as the name implies, is to allow a series of tapes to be written or read sequentially to overcome the size limitations of individual data media, and to negate the requirement for a human operator’s presence to load the next tape in a long backup sequence.

6.3.11.3   By way of contrast, even the smallest robotic tape libraries are programmed to behave as a single, self -contained storage system. The location of individual files on different tapes is transparent to the user, and the library controller keeps track of addresses of files on each tape, and of the physical location of tapes within the library. If tapes are removed or reloaded, the robotic sub-system re-scans the tape slots as it initialises, to update its inventory with metadata from barcodes, rf tags, or memory chips in the tape housings.

6.3.11.4  Large tape libraries have some benefits when compared to the smaller tape libraries. They can be built to be redundant and distributed, i.e. downtime can be minimised and the read/write load can be balanced between several similar systems. Large tape library can also be used as a multi-purpose system; they can, for example, maintain a company’s normal IT backups as well as manage all archived video and audio.

6.3.11.5   Data tapes or cartridges used in a robotic system will have some system of barcoding, rf tags or other ID. These optical or electromagnetic recognition systems sometimes operate in conjunction with MIC for supplementing information about tape ID and content. Some formats have a global ID system for barcoding tapes so that a tape used in one robotic library can be recognised in another library system.

6.3.11.6   Backup and Migration Software and Schedules: Some confusion and misunderstanding exists both in IT circles, and in the wider community as to the purpose and operation of long-term data archives. There are two popular misconceptions regarding long term data archives. The first; that archiving is the process of moving infrequently used material from expensive, on-line networked disc storage, to less expensive,inaccessible offline shelving from whence it may never be retrieved and the other;that backup is a regular daily and weekly routine of making a copy of everything stored in the system.

6.3.11.7   With regard to the first misconception, the reality is that some of the most important and valuable material may not be used for months or years, but its survival must be guaranteed unequivocally. Likewise with the second, if suitable rules are established, vast amounts of material may not need to be replicated daily or weekly when only small percentages are updated. In practice, while a stringent regime of replicating data on different media in different locations is essential to minimise risks from technology failures and to ensure recovery from disasters, the particular characteristics of digital heritage material requires some procedures that differ from routine IT data management.

6.3.11.8   Conventional HSM (Hierarchical Storage Management) systems may be optimised for backing up everything on a regular basis, and moving out infrequently-used content to inaccessible locations, but the better systems can be configured to suit the business rules and practices in archives of different sizes with different levels of access. A medium-sized organisation may ingest 100 GB of audio data every week or 1TB of video. It is fairly straightforward to ensure that copies are made as soon as valuable material is ingested, and that frequently used material remains accessible.

6.3.11.9   Some of the primary tasks of storage management software are to optimise the use of resources and to manage devices in the hardware layer, while regulating traffic with minimal delays to users. HSM software offers a choice of conditions for migrating files from on-line disk to tape, such as older than a certain date, larger than a nominated size, located in particular sub-folders or when available disk space falls outside certain limits (high and low watermark).

6.3.11.10  Typically, where both high resolution files, as well as low resolution access copies are produced, the larger, high resolution files used for preservation and broadcast will be migrated to tape to free up space on the more expensive hard disk array. A balance is needed to maintain availability of material, and to optimise use of tape drives and media. If tapes are being accessed very frequently, a large number of mounts and unmounts, spooling and restore operations will degrade system performance. More sophisticated content management systems sometimes incorporate lower levels of storage management so that users are less aware of individual files and components that support the system.

6.3.12 Selection and Monitoring of Data Tape Media

6.3.12.1  As with any conventional preservation system, it is important not only to have backups and redundancy in case of failures in media or components, but it is vital to establish and to measure performance standards for key parts of the system. Software such as SCSI-Tools will allow a lower level of interrogation of individual drives and devices on a network to determine if media and hardware are performing at their optimum level. LTO tape has an interface for data monitoring, however this functionality is rarely utilised though it would be advantageous for archival systems. Some HSM systems are capable of monitoring the quality of stored assets on a regular basis. These systems monitor the error rates of tapes while users access the assets or read the assets without user intervention if a tape has not been used during a certain period of time.

6.3.13 Costs

6.3.13.1  Typically, the cost of data tape storage is spread in four areas: Tape media: procurement and replacement of primary and backup tape media every 3-5 years. Tape drives: procurement and replacement every 1-5 years, with support. Robotic Library purchase and maintenance within a 10 year life-cycle, and software purchase, integration/development and maintenance.

6.3.13.2  In a manual system, the costs for shelving will be lower, although the space requirement for staff is greater, and the labour cost for manual retrieval and checking is higher. In an automated robotic system, much of the human cost is offset by up-front expense for hardware and software. Large scale robotic tape libraries can be purchased in a modular fashion to spread the cost over several years as demand for storage grows.Within the life of a robotic tape library, individual components such as tape drives will be replaced by newer technology every three to five years. If content from an archive is accessed continuously the life time of drives can be considerably short, even only one year or less. Older tape media and drives may be kept on hand for redundancy if required. If an archive does not grow rapidly, the present and next generation of tapes and drives can co-exist in a tape library while the archive content is migrated to the next generation of media or technology. If an archive grows continuously it may be cost-effective to create a tape library of a specific size to only store the amount of content that shall be archived during the life time of the then current technology, and to then acquire a new larger tape library to store the content that shall be stored using the next generation of technology including the old content that will be migrated. The later approach is also necessary if old and new technology cannot co-exist in the same unit.

6.3.13.3  It is good business practice to keep at least one redundant copy of data off-site or geographically separate. Typically a radius of 20 to 50 km is common for natural and man-made disasters, and still allows manual retrieval within a few hours. To reduce risks further, redundant copies should be on different batches or sources of media, or even on different technologies. Some data tapes are only manufactured at a single supplier, and chances of a single point of failure are increased. Three copies of data are safer than two, and although costs for media increase, the hardware and software costs are only slightly higher than for the first copy.

6.3.14 Hard Disk Drives (HDD) Introduction

6.3.14.1  Hard Disk Drives (HDDs) have served as the primary memory and data storage in computers since IBM introduced the model 3340 disk drive in 1973. Nicknamed “the Winchester”, because it had 30MB of fixed memory and 30MB of removable and the working designation of 30/30 resembled, in name at least, the famous rifle, it pioneered head designs that made operation of the hard disk viable. Subsequent reduction in size and more recent developments in head and disk design have greatly increased the reliability of disk drives, leading to the robust designs in common use today.

6.3.14.2  Data managers whose responsibility it is to maintain data have considered the hard disk too unreliable to use as the sole copy of an item, and too expensive to use in multiple, and consequently more reliable, disk arrays. The data on HDDs has consequently been duplicated on multiple tape copies to ensure its survival. As stated above (6.1.4 Practical Aspects of Data Protection Strategies and 7.6 Archival Storage) all data systems must have multiple and separate copies of all data. While experts tend to agree that the most reliable data system consists of a HDD array supported by multiple duplicates on tape, the continued reduction in costs and improvement in reliability make the concept of identical duplicates of data on separate hard disks a possibility. The principle of multiple media remains, however, and disk only storage constitutes a risk.

6.3.15 Reliability

6.3.15.1  Loss of data as a consequence of disk failure and head crashes has made most data professionals suspicious of HDDs, however manufacturers now claim annualised failure rates of less than one percent and an operational life of 40,000 hours (Plend 2003). High reliability drives may have an even longer operational life, termed by manufacturers as “mean time between failure”. Though HDDs are self-contained and sealed and so protected from damage, most failures in disk drives occur in two opposing ways: as a result of wear through extended use, or as power to the drive is turned on or off. The dilemma is whether to leave the disk on, and increase wear, or turn it on and off and increase risk of failure.

6.3.16 System Description, Complexity and Cost

6.3.16.1 As noted in Section 2, Key Digital Principles, the more recent generations of computers have sufficient power to manipulate large audio files. All recent generation computers incorporate hard disks of adequate speed and size, and an external HDD adapter can be plugged into a USB, Firewire or SCSI port. The system complexity and the degree of expertise required to run such systems is not much greater than is necessary for desktop computer operation.

6.3.16.2 When large quantities of audio and audiovisual material required for access are stored on HDDs, the disks are usually incorporated into a RAID (Redundant Array of Inexpensive (or Independent) Disks). RAID increases the reliability of the hard disk system, and the overall access speed by treating the array of disks as one large hard disk. If a disk fails, it can be replaced and all the data on that disk can be reconstructed with data from the rest of the disks in the array. The level of failure the system will tolerate, and the speed of recovery from such failures is a product of the RAID levels. RAID is not designed as a data preservation tool, but as a means of maintaining access through inevitable disk failures. The appropriate RAID level for any particular installation, and the requirement for duplication of controllers, is dependant on the particular circumstance and the frequency of data duplication. A RAID requires that all disks in the array be turned on when any part of the disk is in use. All RAIDs containing archival material, as with all digital data, must be duplicated more than once on other carriers.

Capacity Native tape capacity (GB) # of tapes Recommended
# of
tape drives
Maximum #
of drives
System price (€) Tape price (€) Drive price (€) Cost per GB (€)
10 TB 800 13 2 4 20.480 97 7.625 2,05
50 TB 800 63 4 16 56.800 97 10.175 1,14
100 TB 800 125 8 16 134.050 97 12.725 1,34
200 TB 800 250 12 16 205.350 97 12.725 1,03
500 TB 800 625 18 56 446.938 97 15.975 0,89
1000 TB 800 1250 36 88 864.517 97 15.975 0,86
2000 TB 800 2500 72 176 1.687.690 97 15.975 0,84

Table 2 Section 6.3: Investment Costs of LTO-4 technology based Storage Systems

 

Capacity HW maintenance, year 1 (€) SW maintenance, year 1 (€) HW maintenance, year 2 (€) SW maintenance, year 2 (€) HW maintenance, year 3 (€) SW maintenance, year 3 (€) HW maintenance, year 4 (€) SW maintenance, year 4 (€) HW maintenance, year 5 (€) SW maintenance, year 5 (€)
10 TB 2.420 n/a 2.420 n/a 2.420 n/a 2.514 n/a 2.514 n/a
50 TB 3.454 n/a 4.958 n/a 4.958 n/a 4.958 n/a 4.958 n/a
100 TB 11.808 490 13.817 490 13.817 490 13.817 490 13.817 490
200 TB 15.787 582 19.323 582 19.323 582 19.323 582 19.323 582
500 TB 27.380 1.068 34.111 1.068 34.111 1.068 34.111 1.068 34.111 1.068
1000 TB 47.542 2.115 66.734 2.115 66.734 2.115 66.734 2.115 66.734 2.115
2000 TB 99.272 4.221 99.272 4.221 99.272 4.221 99.272 4.221 99.272 4.221

Table 3 Section 6.3:Yearly Maintenance Costs of LTO-4 technology based Storage Systems

Notes to the tables:

  • Prices are averages of list prices from multiple vendors. A price that a customer has to pay is usually somewhat lower.
  • Prices indicate price of raw capacity. At least double amount of tape media will be needed for backup purposes.
  • Price in the system price column includes cost of tapes and drives for the capacity in question, but does not include any HSM software or hardware
  • The tables indicate only investment costs and maintenance fees that have to be paid to a vendor. In addition to this, also costs from electricity, cooling, machine room, management, etc. must be included in individual calculations. Electricity and cooling of tape library system might cost 10% of purchase price over five year period.

 

Capacity Drive technology Size of drive (GB) # of drives System price (€) Drive price (€) Cost per GB (€)
5 TB SATA 500–1000 5–10 11.884 1.000 2,38
10 TB SATA 750–1000 10–14 19.997 1.000 2,00
50 TB SATA/FATA 1000 50 124.334 1.800 2,49
100 TB SATA/FATA 1000 100 230.914 1.800 2,31
200 TB SATA/FATA 1000 200 456.942 1.800 2,28
500 TB SATA/FATA 1000 500 1.202.726 1.900 2,41
1000 TB SATA/FATA 1000 1000 2.566.513 1.900 2,57
2000 TB SATA/FATA 1000 2000 4.782.584 1.900 2,39

Table 4 Section 6.3: Investment Costs of HDD Based Storage Systems

 

Capacity HW maintenance, year 1 (€) SW maintenance, year 1 (€) HW maintenance, year 2 (€) SW maintenance, year 2 (€) HW maintenance, year 3 (€) SW maintenance, year 3 (€) HW maintenance, year 4 (€) SW maintenance, year 4 (€) HW maintenance, year 5 (€) SW maintenance, year 5 (€)
5 TB 826 750 826 750 826 750 1.845 750 1.845 750
10 TB 1.206 1.125 1.206 1.125 1.206 1.125 2.600 1.125 2.600 1.125
50 TB 5.822 6.125 5.822 6.125 5.822 6.125 12.365 6.125 12.365 6.125
100 TB 10.514 8.500 10.514 8.500 10.514 8.500 22.391 8.500 22.391 8.500
200 TB 21.724 12.750 21.724 12.750 21.724 12.750 44.956 12.750 44.956 12.750
500 TB 57.061 37.250 57.061 37.250 130.394 37.250 130.394 37.250 130.394 37.250
1000 TB 130.203 66.250 130.203 66.250 263.537 66.250 263.537 66.250 263.537 66.250
2000 TB 223.778 124.250 223.778 124.250 477.121 124.250 477.121 124.250 477.121 124.250

Table 5 Section 6.3:Yearly Maintenance Costs of HDD Based Storage Systems

Notes to the tables:

  • Prices are averages of list prices from multiple vendors. A price that a customer has to pay is usually somewhat lower.
  • Price in the system price column includes cost of hard disk drives for the capacity in question.
  • The tables indicate only investment costs and maintenance fees that have to be paid to a vendor. In addition to this also costs from electricity, cooling, machine room, management, etc.must be included in individual calculations. Electricity and cooling of hard disk drive system might cost 30% to 40% of purchase price over five years period.

6.3.17 Disk Only Storage

6.3.17.1  RAID arrays are scalable within the limits of the system, however individual HDDs are infinitely scalable by simply adding more drives. Since the introduction of the IBM 3340 HDD, storage capacity has increased rapidly, almost exponentially, while costs have fallen. These changes, linked with an improvement in reliability, have led some to suggest that HDDs could be used as both the primary storage system and the back up copy. There are three difficulties associated with this approach: Firstly, hard disk life is estimated in terms of usage- time, that is the number of hours of operation. There has been no testing of the life of an infrequently used HDD. Secondly, having data on different types of media is advantageous as it spreads the risk of failure. Therefore the approach should be considered very cautiously. Finally, there is no way of monitoring the condition of the hard disk on the shelf without turning it on at regular intervals and thereby compromising the advantage gained by having the disk turned off (see section 6.3.18 below, Monitoring of Hard Disk Media). Multiple carriers (eg Tape and Hard disk) remain the preferred option. Hard disks should be implemented within an integrated system.

6.3.18 Hard Disk Storage Systems

6.3.18.1  Hard Disk Storage Systems are centralised systems that are used to maximise disk storage utilisation and to provide large capacities and/or high performance. These systems are used in conjunction with server computers so that server have only small amount of internal hard disk storage or do not have it at all. These kind of systems are often used in mid and large size environments as storage for an archiving system. Alternatively an archiving system can share a centralised storage system with a number of other computer systems. The size of a system can vary from 1 terabyte to several petabytes. It should be taken into consideration that performance characteristics of a storage system can vary notably according to its chosen configuration and it is essential that the actual needs for a system are carefully planned beforehand and a qualified professional is used to configure the storage structure and interfaces of a system to produce the best value for ones investment.

6.3.18.2  Centralised disk storage systems are designed to provide better error resilience than independent hard disk drives. These systems provide several alternative levels of RAID protection, their components can be redundant in order to avoid single point of failures, and systems can be locally or geographically distributed to protect valuable assets from different kind of failures and disasters.

6.3.18.3  The connection between the storage system and the computers it serves play important role regarding performance of a system. Generally speaking, two methods used are NAS (Network Attached Storage) and SAN (Storage Area Network).While NAS utilises regular IT network like Ethernet to move data between computer and storage system SAN uses switched Fibre Channel connections. NAS systems can operate at 100 Mbit/s, 1 Gbit/s and 10 Gbit/s speeds while SANs operate at 2 Gbit/s or 4 Gbit/s. Both technologies have clear road map to the future and their performance can be expected to grow in the future. SAN technology is usually chosen for more demanding environments since it gives better performance due to specific design. For example, the in/out (I/O) block size can be controlled more effectively in SAN environments while networking protocols tend to force NAS systems to use quite small I/O blocks. From economical point of view NAS technology is cheaper than SAN technology.

6.3.19 HDD Life

6.3.19.1  As stated above, a life of 40,000 hours is estimated for many commercially available HDDs. Typical commercial use of HDDs would give these disks a replacement life of five years. With improvements such as fluid/ceramic spindle bearings, surface lubrication of disks, and special head parking techniques made on the most recent desktop HDDs, the life of HDDs may be somewhat longer. However, there is no reliable testing of the life span of unused HDD and it would be astute to plan to replace the disks in such a working system within 5 years.

6.3.20 Monitoring of Hard Disk Media

6.3.20.1  An indication of imminent disk failure may be an increase in bad data blocks. It is typical for the latest disks to show bad block errors even from new and most data systems manage the bad blocks by reassigning the address of that block. However, if the quantity of bad blocks increases it may indicate that the disk is beginning to fail. Software exists which will provide a warning of increased bad data blocks, as well as measuring other physical characteristics that may indicate disk problems.

6.3.21 HDD technologies

6.3.21.1  There are four main methods of connecting HDDs and other peripheral devices to computers, USB (Universal Serial Bus), IEEE 1394 (Firewire), SCSI (Small Computer System Interface) and SATA/ATA (Serial Advanced Technology Attachment/AT Attachment). They each have particular advantages in certain situations. USB and Firewire are planned to be all-purpose buses that can be used to connect to personal computer a HDD as well as digital video camera or MP3 player. SCSI and SATA/ATA are mainly used to connect hard disk drives to a computer or disk storage system.

6.3.21.2  SCSI and its successor SAS (Serial Attached SCSI) interface allows faster writing and reading speeds, and facilitates access to larger numbers of drives than the SATA/ATA drives. SCSI disks can accept multiple commands at once on a SCSI bus and does not suffer from request queues like SATA/ATA. The SATA/ATA drives are comparatively cheaper. The read access speed is largely the same and in an audio context neither interface will limit the operation of the digital audio workstation (DAW) more than the other. The performance difference of SCSI/SAS and SATA drives can have meaning in heavily utilised centralised hard disk storage system.

6.3.21.3  Fibre Channel (FC) SCSI/SAS drives are mainly used in demanding use in enterprise or business systems while the cheaper SATA drives are more used in the personal market, but they are also increasingly used in enterprise and business systems to offer more cost-effective storage capacity e.g. in archival storage. In archival storage, the actual decision between (FC) SCSI/SAS and SATA technology is dependent on the actual load of the system. If a system is used to archive small or medium amounts of content that is not accessed intensively a SATA based solution might well be enough. The actual decision must be based on clearly identified demands and negotiations with one’s storage provider.

6.3.21.4  USB and Firewire connected disk can be used to transfer content from one environment to another, but since they are rather unreliable, difficult to monitor and easy to loose they should not be used for archiving even though their pricing may seem very attractive.

6.3.21.5  The interface is not a completely consistent indication of the reliability and performance of a given drive or storage system and the purchaser should be more aware of other operating and configuration parameters of a storage system. It seems to be the case that more reliable drives are associated with the FC SCSI/SAS interface. Nonetheless, HDDs are not in themselves permanently reliable, and all audio data should be backed up on suitable tape (see 6.3.5 Data Tape Performance). (For further discussion see Anderson, Dykes and Riedel 2003).

6.3.21.6  There is one emerging storage technology which may have a prominent position in the near future. Solid- state storage in form of flash memory is developing as a alternative to moving disks and has already become an alternative to a HDD in laptop PCs. Some storage manufacturers have also introduced flash drives in their low cost or midrange storage systems and are planning to introduce flash drives in their high end systems too. Even though flash storage still has some challenges in storage reliability to overcome it might become a viable solution to storage needs of archival community; its price per gigabyte is becoming competitive, it is more environmentally friendly due to lower demand for power, and it does not have moving parts, which could mean longer life time of storage units. A life time of ten years instead of five years for a storage unit could mean lower investment and management costs for an archivist since every other migration to the next storage technology could be skipped. In terms of read and write performance flash storage is already comparable with HDD technology.

6.3.22 Hierarchical Storage Management (HSM)

6.3.22.1  The OAIS Functions of Archival Storage embeds the notion of Hierarchical Storage Management (HSM) in the conceptual model. At the time OAIS was written the situation where large amounts of data could be affordably managed in other ways was not envisaged. The practical issue that underpins the need for HSM is the differing cost of storage media, e.g. where disc storage is expensive, but tape storage is much cheaper. In this situation HSM provides a virtual single store of information, while in reality the copies can be spread across a number of different carrier types according to use and access speeds.

6.3.22.2  However, the cost of hard disc has fallen at a greater rate than the cost of tape, to the point where there is an equivalency in price. Consequently the use of HSM becomes an implementation choice. Under these circumstances a storage system which contains all of the data on a hard disc array, all of which is also stored on a number of tapes, is a very affordable proposition, especially for digital storage systems up to 50 terabytes (and rising every year). For a smaller digital storage facility a fully functional HSM is consequently unnecessary and instead what is required is a much simpler system which manages and maintains copy location information, media age and versions and completely replicates the stored data on hard disc and on tape.

6.3.22.3  For medium to large digital storage systems the need for HSM storage systems remains and continues to be amongst the very expensive components of the digital storage systems.

6.3.23 File Management Software in smaller systems

6.3.23.1  The purpose of file management software in systems where the entire archive is replicated both on hard disc and tape is to keep track of the location, condition, accuracy and age of the tape copies. This basic backup functionality is a lower cost alternative to a classic HSM and may, at least in theory, be more reliable for small systems. However, as the large scale HSM represents a significant market,research and development has been supported by the industry in this area.Small scale file management software is being developed amongst the open source software development community. These include such systems as three most popular open source NAS applications, FreeNAS, Openfiler and NASLite,and the Advanced Maryland Automatic Network Disk Archiver (AMANDA). As with all such open source solutions, the onus is on the user to test the suitability and reliability of such systems, and without further development this publication makes no specific recommendation.

6.3.24 Verification and retrieval

6.3.24.1  In some commercial software, tape read/write error can be reported automatically during the data backup and verification process. This function is normally implemented with cyclic redundancy check, a technology using checksum against data to detect errors for transmission or storage. It is recommended that an error checking function should be implemented in any archival storage system. Error checking is difficult to implement in open source because that capability is linked to specific hardware. A commercially available stand-alone LTO Cartridge Memory Reader is the “Veritape” from MPTapes, Inc. and recently, Fuji Magnetics announced a Chip Reader Diagnostics System for LTO-Cassettes, bundled with software.

6.3.25 Integrity and Checksums

6.3.25.1  A checksum is a calculated value which is used to check that all stored, transmitted or replicated data is without error. The value is calculated according to an appropriate algorithm and transmitted or stored with the data.When the data is subsequently accessed, a new checksum is calculated and compared with the original, and if they match, then no error is indicated. Checksum algorithms come in many types and versions and are recommended, and standard, practice for the detection of accidental or intentional errors in archival files.

6.3.25.2  The cryptographic versions are the only type that have a proven record of trust when protecting against intentional damage to data, and even the simplest of these are now compromised. It has been recently shown that there are ways of creating meaningless bits that will calculate as a given MD5 checksum. This means that an external or internal intruder may replace digital content with meaningless data and that this attack will go unnoticed by the error checking management system until the files are required for use and opened. MD5, although still useful for transmission purposes, is 124 bit and should not be used where security is the issue. SHA-1 is another cryptographic algorithm that is under threat of being compromised, and which it has already been shown can, in theory, be circumvented. The length of SHA-1 is 160 bit: SHA-2 comes in versions with 224, 256, 384, and 512 bit lengths, and are algorithmically similar to SHA-1. The steady growth of computational power means that these checksums may, in the long run, be compromised as well.

6.3.25.3  Even with these compromises, a checksum is a valid approach to detecting accidental errors, and if incorporated into a trusted digital repository, may well be sufficient to uncover intentional damage to data files in low risk scenarios. However, where risks exists, and perhaps even where they do not, monitoring checksums and their viability must be part of preservation planning.

6.4 Digital Preservation Planning

6.4.1 Introduction

6.4.1.1  Once the action has been taken to convert the audio content to a suitable digital storage format for storage on a digital storage system, as defined earlier in this document, there is still a requirement to manage the ongoing preservation of the content. Section 6.3 Archival Storage includes a description of the issues surrounding management of the byte stream, i.e. ensuring that the digitally encoded data retains its logical structure through management of the storage technology.

6.4.1.2  There is, however, another aspect to the preservation of digital information, and that is ensuring that it is still possible to access the content encoded in those files. OAIS calls this function “preservation planning”, and describes it as “the services and functions for monitoring the environmentà and providing recommendations to ensure that the information storedà remains accessible to the Designated User Community over the long term, even if the original computing environment becomes obsolete” (OAIS 2002:4.2).

6.4.1.3  Preservation planning is the process of knowing the technical issues in the repository, identifying the future preservation direction (pathways), and determining when a preservation action, such as format migration, will need to be made.

6.4.2 Future Digital Pathways

6.4.2.1  When a file format becomes obsolete and is at risk of becoming inaccessible due to the unavailability of appropriate software to access the content, there are basically two approaches that can be made; migration, or emulation. In migration the file is modified, or migrated to a new format, so that the content can be recognised and accessed using the available software of the time. In emulation, the access or operating software is modified or designed so that it will open and play the obsolete audio file format on a new system which would not otherwise be able to open the content.

6.4.2.2  Our current understanding leads us to believe that for simple discrete files, such as uncompressed audio files, the most likely approach will be migration but this is not certain and all digital storage approaches and systems should be flexible enough to be responsive to the changing environment. Adequate preservation metadata as described in the PREMIS recommendations or the explicit file typing (including versioning) in BWF/AES31-2-2006 fields will support either approach, as will the standards being developed in AES-X098B which will be released by the Audio Engineering Society as AES57 “AES standard for audio metadata – audio object structures for preservation and restoration”. Harvard University is developing a toolkit which supports the population of the necessary fields which will be released in open source.

6.4.2.3  This aspect of digital preservation is the strongest argument for an absolute adherence to the standard format described. The large investment the audio and IT industries have made in the standard audio format (.wav) means that the requirement for professional software tools which will enable the continued access to content will help to ensure that the sound archive can manage access to their collections. Likewise, the large investment in a single format will also help support the continuance of that format for the longest period, as the industry will not change an entrenched format without significant benefits.

6.4.3 Motivating Factors and Timing

6.4.3.1   Though the wise choice of standard formats, and an observance of industry practices will delay the eventuality, the day will come where it will be necessary to undertake a preservation action of some type which will be needed to maintain access to the audio content stored. The issue for sound archivists concerned with their digital content will be determining when to undertake that step and what precisely to do.

6.4.3.2   A number of initiatives are being developed to help support this need. These include the Global Digital Format Registry (GDFR http://hul.harvard.edu/gdfr/), which exists to support “the effective use, interchange, and preservation of all digitally-encoded content.” Other services provide recommendations about suitable format, such as those provided by the Library of Congress (US) or The National Archives (UK).

6.4.3.3   The factors which will motivate a sound archivist to undertake some sort of preservation action will be the recognition that new software no longer supports the old format, and the industry as a whole moving to select a new format. Knowledge of the events that herald change comes from expert understanding of the technology, the industry and the market and sound archivists are well advised to take heed of the recommendations services such as those noted above.

6.4.3.4  Software and services under development, such as the Automatic Obsolescence Notification System (AONS), will provide advice to collection managers on when changes have occurred in the market requiring action (https://wiki.nla.gov.au/display/APSR/AONS+II+Documentation). The implementation of such services will occur in parallel with the development of the GDFR.

6.5 Data Management and Administration

6.5.1.1  Data Management, in the OAIS, is the services and functions for populating, maintaining, and accessing both descriptive information which identifies and documents archive holdings and administrative data used to manage the archive, in other words the catalogue of content and the statistical record of data content.

6.5.1.2  Administration, in the OAIS, is the services and functions for managing system configuration, monitoring operation, providing customer service and updating archival information. It is also responsible for management processes such as negotiating submission agreement with producer, auditing submission, control physical access, establishing and maintaining archive standards.

6.5.1.3  The management and administration of the digital repository and archival system provides services that allow the sustainability of the system and the preservation of the content stored therein. A requirement of an archival digital storage system would include the ability to interrogate the system to produce result sets of holdings, access usage statistics, contents summaries including sizes and other necessary technical and management information. The data management and administration is critical to a sustainable archival system because this functionality ensures that files preserved and accessed are properly found and identified.

6.5.1.4  It is within this section of the digital storage and preservation system that control over access to content, or security control, is implemented. Many repository software systems incorporate approaches to implementing policies which are stored and managed by the system. It is important to recognise that the rights management information, like the audio content itself, must outlast the system used to store it, and so be transferable to any future replacement preservation and storage system. Information which is encoded in XACML (eXtensible Access Control Markup Language) for example, is both more universally enforceable, and transferable to other systems. XACML is a declarative access control policy language implemented in XML and a processing model, describing how to interpret the policies. XACML is managed by the OASIS standards group (https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml).

6.5.1.5  When selecting, establishing and installing a digital preservation system one of the critical tests should be to determine if the administration of the proposed system is within the capabilities of the host institution. The capability and breadth of functions of a system is often linked with the complexity of use and installation. A system which cannot be adequately managed and maintained is a major risk to the content it manages. It is therefore important that the long term management of a system take account of the available technical expertise required to sustain its use.

6.6 Access

6.6.1 Introduction

6.6.1.1  The OAIS Reference Model defines “access” as the entity that “provides the services and functions that support consumers in determining the existence, description, location and availability of information stored in the OAIS, and allowing consumers to request and receive information products.” In other words, access is the mechanisms and process where content is found and retrieved. IASA-TC 03 “The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy” makes the point that “the primary aim of an archive is to ensure sustained access to stored information”. The preservation of the content is a prerequisite to sustained access to the content, and in a well planned archive access is a direct outcome of it.

6.6.1.2  In its simplest form, access is the ability to locate content and, in response to an authorised request, allow retrieval of the content for listening, or possibly, as long as the rights associated with a work allow it, creating a copy that can be taken away. In the connected digital environment access can be provided remotely. Access, however, is more than just the ability to deliver an item. Most technically constructed archival systems can deliver an audio file on request, but a true access system provides finding and searching capability, delivery mechanisms and allows interaction and negotiation regarding content. It adds a new dimension to access beyond that of conquering distance. In this new services based model of retrieval, access could be considered a dialogue between the provider’s system and the user’s browser.

6.6.2 Integrity in On Line and Off Line Access Environments

6.6.2.1  Prior to the existence of remote access in the online environment, such things as authenticity and integrity were established by individuals in the reading rooms and listening posts of the collecting institutions. The content was delivered by representatives of institutions whose reputation spoke for the integrity of the content. Original materials could be retrieved for examination if the copies were questioned.

6.6.2.2  The online environment still relies to some extent on the trusted nature of the collecting institution, however, an unambiguously original item can never be provided online, and the possibility of undetected tampering or accidental corruption exists within the archive and distribution network. To counter this, various systems exist which mathematically attest to the authenticity or integrity of an item or work.

6.6.2.3  Authenticity is a concern with knowing that something has originated from a particular source. The trusted nature of the institution creating the content attests to the processes, and a certificate authority is issued which a third party can use as a guarantee of authenticity.Various systems exist and are valuable where this could be an issue.

6.6.2.4  Integrity refers to a wish to know whether an item has been damaged or tampered with. Checksums represent the common way of dealing with integrity, and are valuable tools in both the archive and the distribution network (see 6.3.23 Integrity and Check sums). However, as is discussed in 6.3.23, checksums are fallible, and their use requires monitoring on behalf of the archive of latest developments.

6.6.3 Standards and Descriptive Metadata

6.6.3.1 Detailed, appropriate, organised metadata is the key to broad exposure and effective access. In Chapter 3 Metadata, a detailed discussion of metadata in many of its forms and requirements is undertaken, and this should be referred to in developing a delivery system. Ambitious access facilities, using, for example map interfaces or timelines, will only function if there is metadata to support it in a structured and organised form.

6.6.3.2  The most cost effective way to manage and create the appropriate metadata is to ensure the requirements for all the components in the delivery system are established prior to the ingest of the content. In this way the metadata creation steps can be built into the pre-ingest and ingest workflows. The cost of creating a minimal set, as discussed in Section 7.4, is the extra task of adding and structuring the metadata in a system which has already been created.

6.6.4 Formats and Dissemination Information Packages (DIP)

6.6.4.1  The Dissemination Information Package (DIP) is the Information Package received by the Consumer in response to a request for content, or an order. The delivery system should also be able to deliver a result set or a report from a query.

6.6.4.2  Web developers and the access “industry” have developed delivery systems based, naturally, around delivery formats. Delivery formats are not suitable for preservation, and generally, preservation formats are not suitable for delivery. In order to facilitate delivery, separate access copies are created, either routinely, or “on demand” in response to a request. Content may be streamed, or downloaded in compressed delivery formats. The quality of the delivery format is generally proportional to its bandwidth requirements, and collection managers must make decisions about the type of delivery formats based on the user requirements and the infrastructure to support delivery. QuickTime and Real Media formats have proven to be popular streaming formats and MP3 (MPEG 1 Layer 3) a popular downloadable format which may also be streamed. There is no requirement to select only these formats for delivery, and many collection delivery systems provide a choice of formats to the user.

6.6.4.3  For some types of material it may be necessary to create two master WAV files: one, a preservation or archival master that replicates exactly the format and condition of the original the second, a dissemination master that may have been processed in order to improve the audio quality of the content. A second master will allow the creation of dissemination copy as required. It is expected that distribution formats will continue to change and evolve at a faster rate than master formats.

6.6.5 Search Systems and Data Exchange

6.6.5.1  The extent to which content can be discovered sets the limit on the amount of use of the material. In order to ensure broad usage it is necessary to expose content through various means.

6.6.5.2  Remote databases can be searched using Z39.50, a client-server protocol for searching and retrieving information. Z39.50 is widely used in the Library and Higher education sector, and its existence predates the web. Given the extent of its use, it is advisable to establish a Z39.50 compliant client server on databases. However, this protocol is being rapidly replaced in the web environment by SRU/SRW (Search/Retrieval via a URL and Search/Retrieve Web service respectively). SRU is a standard XML-focused search protocol for Internet search queries, utilizing CQL (Contextual Query Language), a standard syntax for representing queries (http://www.loc.gov/standards/sru/). SRW is a web service that provides a SOAP interface for queries in partnership with SRU.Various open source projects support SRU/SRW in relation to the major open source repository software such as DSPACE and FEDORA.

6.6.5.3  OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) is a mechanism for repository interoperability. Repositories expose structured metadata via OAI-PMH which is aggregated and used to support queries on the content. OAI-PMH nodes can be incorporated into the common repositories. OAI-ORE (Object Reuse and Exchange) will be important for the sound and audiovisual archiving community as it addresses the very important requirement to be able to deal efficiently with compound information objects in synchronisation with Web architecture. It allows the description and exchange of aggregations of Web resources.”These aggregations, sometimes called compound digital objects, may combine distributed resources with multiple media types including text, images, data, and video”. http://www.openarchives.org/

6.6.5.4  In order for the sophisticated online environment to work it is necessary to have interoperable metadata and content. This means that there must be some shared understanding of the attributes included, a general schema which is able to operate in a variety of frameworks, and a set of protocols about exchanging content. This is best achieved, as is always in the digital environment, by adhering to the standards, schemas, frameworks and protocols recommended and avoiding proprietary solutions.

6.6.6 Rights and Permissions

6.6.6.1  It is important to note that all access is subject to the rights established in the items and the permission of the owner to use the content.Various rights management approaches exist, from “fingerprinting” the content, to managing the permissions of various individual to access, the physical separation of the storage environment. The particular implementation rights system will depend on the type of content, the technical infrastructure and the owner and user community and it is beyond the scope of this document to define or describe a particular approach.

7: Small Scale Approaches to Digital Storage Systems

7.1 Introduction

7.1.1.1  It is possible to build small scale digital storage systems to meet the requirement of archives with smaller collections and a small recurrent budget. Until recently, only large and comparatively wealthy organisations with sound archives were able to digitise their holdings on a large scale and store them by means of Digital Mass Storage Systems comprising of managed hard disk and data tape. These systems tended to be large and expensive dedicated audio and audio-visual storage systems. In more recent years many national sound archives and large libraries have, with the university and higher education sector, initiated and supported the development of open standards and open source software which supports digital archiving widely. These enterprise systems are now the backbone and the model for all forms of digital archiving. Audio archiving benefits by using these systems and importing our own discipline specific knowledge to them.

7.1.1.2  At the same time as open source and other low cost software solutions are appearing on the market, the cost of data tapes are decreasing, and hard disk drives (HDD) are dropping at an even greater rate. It is now possible to undertake digital archiving of a far more professional character than the inherently risky single carrier target formats such as recordable CD or DVD.

7.1.1.3  This chapter of the guidelines describes how a small scale digital repository meeting the requirements of an OAIS might be established and managed. Chapter 6, Preservation Target Formats and Systems, contains much that is pertinent to this chapter, as does Chapter 3 Metadata, and Chapter 4 Unique and Persistent Identifiers.

7.2 Approaches to Small Scale Digital Archiving

7.2.1 Funding and Technical Knowledge

7.2.1.1  It is quite possible to build a low cost digital preservation system, but this cannot be achieved without at least a small level of technical knowledge and some recurrent resources, albeit at a low level, to make it sustainable. Regardless of how simple or robust a system is, it must be managed and maintained, and it will need to be replaced at some time or risk losing the content it manages.

7.2.1.2  “Digital preservation is as much an economic issue as a technical one. The requirements of ongoing sustainability demand at their base a source of reliable funding, necessary to ensure that the constant, albeit potentially low level, support for the sustainability of the digital content and its supporting repositories, technologies and systems can be maintained for as long as it is required. Such constant funding is not at all typical of the many communities that build these digital collections, many of which tend to be grant funded on an episodic basis. There is therefore a need to develop costing models for sustainability of digital materials according to the specific requirements of the various classes of content, access and sustainability.” (Bradley 2004).

7.2.1.3  It is inevitable and unavoidable that the system and its hardware and software components will require maintenance and management which will demand both technical knowledge and dedicated funds. Any proposal to build and manage an archive of digital audio objects should have a strategy which includes plans for the funding of ongoing maintenance and replacement, and a listing of the risks associated with the loss of technical expertise and how that will be addressed.

7.2.2 Alternative Strategies

7.2.2.1  In the event that there is no adequate way to manage the risks described in the section above an archive may decide to continue with the preservation and digitisation of their collection to look to partnerships to manage the storage risks. An archive may choose to distribute the risk in a number of ways, including; by forming local partnerships so that content is distributed between a number of related collections; by establishing a relationship with a stable well funded archive; by engaging a commercial supplier of storage services (see Section 6.1.6 Long Term Planning).

7.2.2.2  To effectively take advantage of any of the approaches described it would be necessary to establish an agreement about what data and content would be exchanged between the partners, and the form it would take. This agreement should be established well before the need to take advantage of it might occur. An agreement about exchange packages would consider all the relevant information necessary to continue the archival role undertaken by an archive. This would include the data that makes up the audio object itself in its archival form, the technical metadata, descriptive metadata, the structural metadata, rights metadata, and the metadata created to record provenance and change history. It would need to be packaged in a standard form so that it could be used to recreate the archive if data was lost, or so that another archive could take up the role of managing content if that was deemed necessary.

7.2.2.3  The tools to produce such profiles exist using, for example, Metadata Encoding and Transmission Standard (METS), a Library based approach that is widely used, are available.Whether this or other strategies are used, agreement about their form is critical to the success of the strategy.Whether this is used to support remote content replication or to support federation of cooperating archives, the agreement about standard form and exchange is a most effective preservation strategy, spreading the risk of failure, due to natural or man made disaster or just lack of resources at a critical time in the life-cycle of the digital audio object.

7.3 Description of System

7.3.1.1  In Section 6.1.4 Practical Aspects of Data Protection Strategies, the need to address the functional categories defined in the Reference Model for an Open Archival Information System (OAIS, ISO 14721:2003) is argued. The same issues apply to both large and small scale collections as this framework is critical to the development of modular storage systems with interoperable exchange of content. The following section which deals with small scale systems adopts the major functional components of the OAIS reference model to assist in the analysis of the available software and to develop recommendations for necessary development. They are Ingest, Access, Administration, Data Management, Preservation Planning and Archival Storage.

7.3.1.2  The system described consists of some form of repository software which manages the content, at least a minimum set of metadata, as well as hardware, with some recommendations on manual approaches to manage the data’s integrity. The hardware section outlines broadly two situations under which small scale storage systems may be implemented; a single operator digitising onto a single storage device, and a situation where more than one operator requires access to the storage device. Either system assumes compliance with all other components mentioned in the Guidelines, including appropriate analogue to digital converters,adequate sound cards,digital audio workstations (DAW) and appropriate replay devices.

7.3.1.3  The following information describes systems and software that might support a small scale collection as though an institution or collection were undertaking all the tasks.It is important to recognise that the approaches described below do not have to be undertaken by one collection.It is possible to find partners and commercial providers who might support some or all of the tasks described below. It is equally important to recognise that all of these tasks form the complete preservation and archival package and must be undertaken by someone whether locally managed or distributed.

7.3.2 Repository Software

7.3.2.1  A well designed piece of repository software will support a number of the functions identified in the OAIS. There are both commercial providers of the software and open source. The advantage of commercial software is the provider is expected to make the system work, however, these commercial systems have ongoing expenses and may lock the user into proprietary systems from which it is hard to escape. Open source software’s main advantages are that it is free, and the developers adhere to open standards and frameworks which will allow the extraction of content in future upgrades. Its disadvantage is that, though open source communities are helpful, support is the responsibility of the user. It is however, possible to find commercial providers who provide a support service for the open source solutions.

7.3.2.2  Most of these repository software systems will support the tasks identified in access, administration, data management and some aspects of ingest. At the time of writing preservation planning and archival storage is generally not supported by repository software, the former being very often technology or format specific, and the latter dependant on hardware. They are discussed separately in the following sections.

7.3.2.3  Two types of open source software are briefly described, however, this software is under constant development, and the claims and comments made below should be checked against the latest developments made by the software providers. The software described are DSpace and FEDORA.

7.3.2.4  The DSpace repository platform is a very popular and widely adopted repository within the higher education and research sectors, although knowledge of its use within the museums and cultural heritage sectors is limited but growing. One of the reasons for the popularity of DSpace is that it is relatively easy to install and maintain, and has a ready made user-interface that integrates data management and access functions within the system’s architecture. DSpace has a strong international developer community that has evolved to support DSpace and new features are being added constantly.

7.3.2.5  One of the strengths of DSpace is its integrated feature set enabling institutional users to quickly establish a repository and then start adding new items to the collection. This strength,however,is also one of its major weaknesses, in that DSpace has evolved into a monolithic software application, and complex code base, that introduces potential scaling and capacity constraints for some large institutional users. This presents no problems for most small to medium scale collections, and is probably not an issue for any digital audio collection. DSpace currently uses a qualified version of the Dublin Core schema based on the Dublin Core Libraries Working Group Application Profile (LAP)

7.3.2.6  FEDORA (Flexible Extensible Digital Object and Repository Architecture) is an increasingly popular repository system that is designed as a base software architecture upon which a wide range of repository services can be built, including preservation services. Compared to the speedy adoption of DSpace, FEDORA has been slower to gain adopters because it lacks a dedicated user-interface and access service out-of-the-box. There are a number of commercial and opens source providers of web-based front-ends for FEDORA.

7.3.2.7  The main strengths of FEDORA are its flexible and scalable architecture. The experiences of institutional adopters indicate that FEDORA can scale to cope with large collections, yet is sufficiently flexible to store multiple types of digital items and their complex relationships. There are few limitations to the features that can be added to FEDORA, whilst still remaining interoperable with other software applications and systems. It can be configured to support virtually any of the metadata profiles through METS ingest capabilities. The main disadvantage of FEDORA is the high level of software engineering expertise required to contribute to its core development, and it is not readily installed and implemented “out-of-the-box” (Bradley, Lei and Blackall 2007).

7.3.2.8  Tools have been developed to migrate content from DSpace to FEDORA and visa-versa, which theoretically negates any future compatibility issues and supports sharing and other workflows (see http://www.apsr.edu.au/currentprojects/index.htm )

7.4 Basic Metadata

7.4.1.1 Chapter 3 Metadata, outlines the requirements of documentation and management of a collection. As has been stated, metadata is pivotal to all aspects of the life cycle of a digital audio object, and paying strict attention to describing all aspects of the collection is one of the more important steps in its preservation. A detailed metadata record of all technical, process, provenance and descriptive aspects is a vital part of the preservation process. However, it is recognised that there is often a technical imperative to preserve audio collection material, and that this may well be before a metadata management system or policy has been developed. The following very basic recommendations are intended as a first step, a collection of data which is necessary to manage the file, or which must be captured or it would otherwise be lost:

7.4.1.1.1 Unique Identifier: Should be structured, meaningful and human readable as well as unique. A meaningful identifier can also be used to relate objects like: master or preservation files and distribution copies, metadata records, series, etc where a sophisticated system will manage that in the metadata.

7.4.1.1.2 Description: Description of the sound sequence. A small amount of text to simply identify the content of the audio file.

7.4.1.1.3 Technical Data: Format, sampling rate, bit rate, file size. Though this information can be acquired later, making it an explicit part of the record allows management and preservation planning of the collection.

7.4.1.1.4 Coding History: In BWF a number of discrete lines of information describing the original item and the process and technology of creating the digital file that is being archived. (See also 3.1.4 Metadata).

7.4.1.1.5 Process errors: Any error data which the transfer system can collect which describes failings in the transfer process (e.g. uncorrectable errors in CD or DAT transfers).

7.4.1.2 The information described in Unique Identifier, Description, and Technical Data can be recorded in Dublin Core records or the BWF headers. Coding History and Process errors can be recorded in the BeXT chunk of the BWF headers or in related XML encoded documents. The date, and if necessary, time of transfer should be recorded into the BWF header, and the date, and if necessary, time of ingest into the repository should be recorded in the metadata management in the repository. In some circumstances the timestamp information that relates components of a multi¡part recording will be mandatory. It is generally advisable to include time and date information with every event or digital object.

7.5 Preservation Planning

7.5.1.1  Preservation planning, as has been discussed, is the planning and preparation which goes to ensure that the digital audio object remains accessible over the long term, even if the computing storage and access environment becomes obsolete. Preservation planning for a small scale collection which is interested only in the preservation of its own digital audio objects is a relatively straightforward task. The metadata captured above informs the decisions about preservation by making clear the relationship between the original and the preservation copy in the digital repository. The technical information helps with planning. The choice of BWF as the preservation format is made to ensure the longest time possible before any format migration is necessary. It remains only for the collection managers and curators to maintain knowledge of the changes occurring in the digital archiving domain through contact with such associations as IASA.

7.6 Archival Storage

7.6.1.1 The archival storage system sits underneath the repository, technically speaking, and incorporates a suite of sub-processes such as storage media selection, transfer of the Archival Information Package (AIP) to the storage system, data security and validity, backup and data restoration, and reproduction of AIP to new media.

7.6.1.2 The basic principles of archival storage can be summarised as follows

7.6.1.2.1 There should be multiple copies. The system should support a number of duplicate copies of the same item.

7.6.1.2.2 Copies should be remote from the main or original system and from each other. The greater the physical distance between copies the safer in the event of disaster.

7.6.1.2.3 There should be copies on different types of media. If all the copies are on a single type of carrier, such as hard disc, the risk of a single failure mechanism destroying all the copies is great. The risk is spread by having different types of carriers. IT professionals commonly use data tape as the second (and subsequent) copy.

7.6.1.3 The major cost in the data storage systems is not the hardware, but the Hierarchical Storage Management (HSM) System. The OAIS Functions of Archival Storage embeds the notion of HSM in the conceptual model. At the time OAIS was written the situation where large amounts of data could be affordably managed in other ways was not envisaged. The practical issue that underpins the need for HSM is the differing cost of storage media, e.g. where disc storage is expensive, but tape storage is much cheaper.In this situation HSM provides a virtual single store of information,while in reality the copies can be spread across a number of different carrier types according to use and access speeds.

7.6.1.4 However, the cost of disc has fallen at a greater rate than the cost of tape, to the point where there is an equivalency in price. Consequently the use of HSM becomes an implementation choice. Under these circumstances a storage system which contains all of the data on a hard disc array, all of which is also stored on a number of tapes, is a very affordable proposition, especially for a small to medium sized digital audio collection. For this type of system a fully functional HSM is unnecessary and instead what is required is a much simpler system which manages and maintains copy location information, media age and versions (Bradley, Lei and Blackall 2007).

7.7 Practical Hardware Arrangements

7.7.1.1  The following information describes how a practical system might be implemented. As has already been discussed above, the assumption is that all of the audio archival data will be stored on hard drive and all of the audio archival data will also be mirrored on data tape such as LTO.

7.7.2 Hard Disk drives

7.7.2.1  A common and affordable approach to data storage on disk is to connect to a cluster of HDDs (hard disk drive) arranged in a RAID array (see section 6.3.14 Hard Disk Drives). RAID level 1 is little more than two drives mirrored; keeping two copies of the data on different physical hardware; if one disk fails it is available on the other drive. Higher level RAID arrays (2 to 5) implement increasingly complex systems of data redundancy and parity checking that ensures the data integrity is maintained. The higher level RAID arrays achieve the same level of security as level 1, or mirroring, but with significantly less storage space. RAID 5, for example, may have a 25% storage loss (or less depending on implementation), when compared to 50% for RAID 1. Sophisticated arrays are widely available.

7.7.3 Tape Backup

7.7.3.1  No single component of a digital system can be considered reliable, instead the reliability of the system is achieved through multiple redundant copies at every stage. The final and most important component in the storage chain is the data tape. In the recent past LTO has gained popularity for this purpose (see section 6.3.12 Selection and Monitoring of Data Tape Media), however other data tape formats may be appropriate depending on the particular circumstance.

7.7.3.2  All data on disk storage should be duplicated on a suitable storage tape. A minimum of two sets of data tapes must be produced, to be stored physically in different places. As it is not unusual for the second set of tapes to be required in the restoration of the data many established archives make three sets of copies, two to be kept near the system for ease of access and a third set stored remotely to protect against physical disasters. It has become customary that the separate sets of data tapes should be made using different products of which a considerable amount of the same batches are bought at one time. This renders quality control and rescue measures easier, once a batch of a given product should fail. Appropriate volume management software will aid in the back up and retrieval process especially if the system incorporates a number of storage devices.

7.7.3.3  Error checking is difficult to implement in open source and low tech solutions because that capability is linked to specific hardware. Nonetheless, a low-tech possible alternative to proper error testing is described in the following paragraph. The data management software has a catalogue (with a printer attached). The hard disc (in RAID) contains a complete set of data. All data is copied onto identical tape copies. There are at least two copies. As data is copied onto a tape, a unique identifier is printed onto a label (human readable) which is attached to the tape. The same identifier can be recorded onto the header of the tape. The data management system can be scripted to prompt the user to find and insert the tape identified by the system. Rather than checking the tape for errors, the system will verify the content of the tape against the hard disc. The hard disc can check the veracity of its own data content and is aware of any failings itself. If the verification of the tape fails, the system can produce a new tape from the hard disc. Assuming 20 terabytes of storage, the system would verify two tapes a day, every tape and its duplicate can be verified three times per year. In the event of a disc failure requiring the data tapes to replace it, there will be two tapes which have been checked within the previous four months. The risk that both tapes and the hard disc would fail is very low.

7.7.4 Single (or Double) Operator Storage System

7.7.4.1  The simplest archival storage system would be to attach a separate RAID array containing only the audio data to the primary DAW (digital audio workstation). This configuration is only possible for institutions with one operator in the digitising process. A requirement for the success of this approach is a well structured plan for digitisation and a dedicated disk array so as the work can be carried out continuously without major interruptions. This will ensure that the HDD attached to the DAW continuously copies to tape whenever the amount of data to fill the target medium is reached.

7.7.4.2  If two operators and workstations are undertaking the digitisation tasks it will be necessary to provide access to a shared drive or drives. The sharing of such resources can be achieved by defining one of the computers as the server, and configuring it so that it manages the drives, and implementing a single wire sharing capability. Such an approach is relatively easy to implement and allows sharing between two operators, though it requires some procedural agreements to avoid conflicts. Logical organisation of data and strict naming procedures are a necessity of small scale manual storage systems.

7.7.4.3  If a system were established of the size described here. It might be the case that it would be more effective to establish a partnership with a larger archivally established institution, or to contract a storage service provider. Nonetheless, the approach above is possible.

7.7.5 Multiple operator storage system

7.7.5.1  For any number of connections greater than two, a networked system of data storage and backup should be implemented. Such a networked system allows access to multiple users in accordance with the rules set down by the data management system. Small scale networks are relatively common and, with the right level of knowledge, easy and affordable to implement. Reasonable quantities of storage can be achieved with an enterprise level attached storage device. Storage technologies and products can be split into three main types: direct- attached storage (DAS), network-attached storage (NAS) and the storage area network (SAN). NAS has better performance and scalability than DAS and it is cheaper and simpler to configure than SAN. NAS technology is, from a cost benefit view, the most appropriate scalable technology for system of the size under discussion.

7.7.5.2  Most low cost NAS devices exhibit reduced bandwidth when compared to the more expensive devices resulting in slower access times, or a lower number of allowable simultaneous access availability. This should present no major problem to smaller collection as the requirement for simultaneous access remains low, especially if MP3 derivatives of the preservation master copies are used for access.

7.7.5.3  A typical small scale networked storage system may comprise of a server class desktop computer connected to a NAS device. The NAS would have the capability of mounting multiple hard disks in a RAID array. An average low cost NAS would hold between 0.5 and 20 terabytes of disk storage (noting the penalty for RAID is less storage than that indicated by the raw disk size). The digital audio workstations (DAW) access the NAS via an Ethernet switch or similar device which, if configured properly, has the effect of separating the storage facility from the office LAN (local area network) and improves the security of the storage facility. The HDDs would be backed up onto data tape.

7.8 Risks

7.8.1.1  Automated storage systems can be configured to constantly copy and refresh data, discarding data tapes which have become unreliable. Large-scale Digital Mass Storage Systems are professionally designed and run by well resourced organisations which can afford and guarantee all necessary measures for data security.With manual data back up and recovery systems the dangers of data loss associated with self-designed and self-managed manual and semi-automated digitisation systems cannot be overestimated. The responsibility for ensuring that the archived audio data remains valid and accessible falls upon the individual, and requires that they physically check the data tapes on a regular basis. This situation is specifically aggravated by the fact that most research and cultural institutions are notoriously under-financed.

7.8.1.2  Though the design of such systems seems to incorporate a very high level of redundancy, one has to bear in mind that the digital components and carriers may fail at any moment without any warning. Therefore it is imperative to have at any stage of the digitisation process and the further storage at the very minimum two copies of the linear archive file. Any flaw will inevitably lead to the loss of a smaller or greater amount of data, however, if suitable strategies have been put in place, this will not be fatal because the redundant copies are available. In view of the time consuming process of transfer not to mention the inevitable losses of older materials, all efforts have to be made to avoid the necessity of re-digitising materials as an outcome of an inconsistent security architecture or careless conduct in the concrete approach.

7.8.2 Complexity of the System

7.8.2.1  Once implemented and installed data storage systems are relatively easy to operate and maintain. However, at the initial stages of implementation and at any subsequent problem or upgrade, specialised IT support is strongly recommended to ameliorate the risk of poor set up.

7.8.3 Partnerships and Backup

7.8.3.1  As has already been discussed, a partnership which provides data backup capability with an institution with established and trusted digital archival practices is a major manager of risk. A network of repositories which can create and accept such organised packages of information will be a most effective preservation strategy, spreading the risk of failure due to natural or man made disaster, or just lack of resources at a critical time in the life-cycle of the digital object.

7.8.4 Cost and Scalability

7.8.4.1 A small scale system described above can be added to in order to allow the creation of larger storage and management capabilities. Relatively small tape drives which can handle a number of data tapes are available and larger scale robotic systems may make the system expandable. If HDD costs continue to fall the cost of replacing and expanding the disk arrays remains affordable.

7.8.4.2 Partnerships between commercial suppliers and open source providers mean that the sophistication of the repository software can be integrated with the safety of a commercial service provider. DSpace and FEDORA, for example, have both released an open source system that operates with a commercial storage solution company.

7.8.4.3 The cost of establishing a small scale data storage system may seem relatively high in comparison to purchasing an individual CD burner, however, on a bit for bit comparison for the storage of more than a few hundred hours of audio, the relative difference is greatly reduced when costing all the requirements of an archive. In addition, a properly managed data storage facility is an altogether more reliable system and will allow the future transfer of audio data to the next storage solution when that inevitability occurs.

8: Optical discs

8.1 CD/DVD Recordables

8.1.1 Introduction

8.1.1.1  Recordable CD (CD-R) and recordable DVD (DVD-R/+R), have become integral in the recording and distribution of many types of audio and audio-visual materials. Though the CD and DVD are now only one of many types of more affordable and reliable storage technologies, the format remains popular for many reasons, amongst them their ease of use and common familiarity. The CD was initially marketed as the perfect permanent carrier, but this was soon shown not to be the case when many of the early discs failed. Even though subsequent technological development has improved on many of the early manufacturing faults, no credible claim can be made to permanence. In fact, digital archiving experts commonly acknowledge that no carrier is permanent.Instead,the processes of acquiring data, transferring to storage systems and managing and maintaining the data, and providing access and ensuring the integrity of the stored information, presents a new range of risks that must be managed to ensure that the benefits of digital preservation and archiving are realised. Failure to manage these risks appropriately may result in significant loss of data value and content.

8.1.1.2  Recordable CDs and DVDs are often chosen as archival carriers, however, the risk of failure of a storage system based on this type technology is high when compared to other approaches. An integrated digital mass storage system with suitable digital repository management software is recognised as the most appropriate for the long term sustainability of data. There may, however, be circumstances where a collection curator may make a decision to use optical disc for storage.

8.1.1.3  Bearing in mind these constraints, it is possible to use recordable optical disks as reliable carriers for a short period of time providing the following recommendations are carefully adhered to.

8.1.2 CD-R and DVD-R Recording Formats

8.1.2.1  There are two different approaches to the encoding of audio and video on recordable CDs and DVDs, either as an audio “stream”, or as a data file. In the first of these approaches sound is recorded as CD-DA formatted audio, which makes them playable in ordinary CD-players, or to encode it in MPEG formatted DVDs, which may not all play in standard DVD-players. Stand alone recorders will only record these formats, though computer based equipment may optionally produce disks in these standard domestic forms. The use of these formats severely restricts the possibility for on-line access and choosing this option may possibly create a migration problem the next time you need to change carrier. It is not recommended that audio streams be recorded for long term storage.

8.1.2.2  The alternative, recording a file using a computer based audio editing system and writing that file to CD-R or DVD-R is a more reliable approach. Recording files on a 650 MB CD-R allows 59 minutes audio storage for 48 kHz 16 bit linear PCM files, and 39 minutes for 48 kHz 24 bit linear PCM files. Recording the same format files on a 4.7 GB DVD-R allows up to 6 hours of audio storage. For this reason the writing of data files is recommended. Because of the simplicity and ubiquity of linear PCM (interleaved for stereo) IASA recommends the use of a .wav or preferably the BWF .wav files (EBU Tech 3285) if recordable CDs and DVDs are selected as the target format.

8.1.3 Recordability, rewritability, erasability and accessibility

8.1.3.1  CD-R, DVD-R and DVD+R discs are dye-based recordable (write once) discs, but not erasable. CD-RW, DVD-RW and DVD+RW discs are phase-change based repeatedly rewritable discs permitting erasure of earlier data and recording of new data in the same location on the disc. DVD-RAM discs are phase-change rewritable discs formatted for random access, much like a computer hard disc.

8.1.4 Recordable CD and DVD Description

8.1.4.1  CD-Rs and DVD-/+Rs store data in line with microscopic grooves running in a spiral from the centre of the disc to its periphery. All CD/DVD drive types use laser beams to scan these grooves. They differ in the wavelength of the laser beam: DVDs use a narrower track pitch of 0.74µm, compared to 1.6µm on CDs. DVD also takes advantage of new modulation and error correction methods not available when the CD was specified.

8.1.4.2  The mechanical dimension of CDs and DVDs are equal: 120mm in diameter, and 1.2 mm thick. The DVD, however, is made up of two discs of 0.6mm thickness, which are bonded together.

8.1.4.3  CD-R and DVD+R consist of three layers: the clear polycarbonate substrate, the dye layer and the reflective layer. In CD-R the reflective layer is close to the label side of the disc and an additional protective lacquer surface layer covers the fragile surface. DVD-Rs reflective layer is situated in the middle of two polycarbonate layers. In the recording process, a laser of much higher intensity than the reading laser “burns” the organic dye according to the coded signal, leaving a row of minuscule transparent and non-transparent areas aligned along grooves in the disc. All recordable CDs and DVDs contain a reflective layer that allows a reading laser to bounce off the CD/DVD and to be “read” by the pickup sensor in the CD or DVD replay device. Many metals are suitable for use as a reflective layer, although only two have been in widespread use on recordable CD and DVD, gold or silver. The combination of the recorded dye groove with the reflective layer modulates the reading laser in the same way as the injection moulded pits and lands and the reflective aluminium layer of a CD -ROM.

8.1.4.4  The three common organic dyes used in recordable discs are cyanine, phthalocyanine and azo. In a recordable CD each dye gives the media its distinctive look depending on which metal is used for the reflective layer; cyanine (blue) dye appears green on gold media and blue on silver media; phthalocyanine (clear light green) dye appears transparent on gold media, but light green on silver media; azo (deep blue) has developed into different shades of blue, the original being a deep blue, and the more recent Super Azo a brighter shade of blue. Because the dye layer is applied so thinly in recordable DVD the type of dye used on recordable DVDs is not easily distinguishable. However, manufacturers of recordable CD and DVD encode information about the type of dye in the polycarbonate layer. The CD and DVD burners use this information to calibrate laser power, and with suitable software the information can be read by users to more accurately describe aspects of the disc itself. This data may be read by ISRC and ATIP code viewers such as CD Media Code Identifier (http://www.softpedia.com/get/CD-DVD-Tools/CD-DVD-Rip-Other-Tools/CDR-Med... ). This tool allows users to view information such as dye type, disc manufacturer, capacity, write speeds and media type. Clover also provides freeware device, IRSCView (http://www.cloversystems.com/ISRCView.htm) will display the Table of Contents, Control Codes, and ISRC codes on Audio, Mixed Mode, and Enhanced CDs. It provides much less manufacturer information than the CD Media Code Identifier.

schematic of CD-R

Fig 1 Section 8.1: A schematic view of a CD-R (not to scale).

 

schematic view of a CD-RW

Fig 2 Section 8.1: A schematic view of a CD-RW (not to scale).

8.1.4.5  Rewritable CDs and DVDs operate on an entirely different principle. Rewritable discs are erasable and can be rewritten, albeit a finite number of times. The recordable layer is made of germanium, antimony and tellurium. A laser is used to heat the surface to two set temperatures. The higher temperature is known as the melting point (approximately 600 degrees centigrade), while the lower level temperature (approximately 350 degrees centigrade) is described as the crystallisation temperature. Heating the disc, and controlling the cooling rate, produces a track of amorphous or crystalline areas. Due to the different reflectivity these areas will be interpreted by the reading laser like the pit/land structure of a CD-ROM. Earlier rewritable discs and drives could only be written at relatively low speeds and this was encoded and implemented in the first generation of drives and standards. More recent developments have provided a mechanism for burning data onto rewritable discs at a higher speed. Though the older drives will read a new high speed rewritable disc, only the latest generation of disc burners will write a disc of the latest formulation.

8.1.4.6  No trustworthy analysis of the medium or long term reliability of RW discs has been undertaken. Preliminary investigations suggest that the film layer containing the encoded information may degrade at a quicker rate than dye based CD-Rs (Byers 2003:9), other commentators disagree. From a purely practical point of view, CD and DVD rewritable may present a greater risk if used for preservation purposes as they may be overwritten by accident with a resulting loss of the original files.

8.1.5 Optical Disc Standards

8.1.5.1 Adherence to standards is the mechanism by which discs are writable or playable on different manufacturers’ machines. The manufacturers have the responsibility to make the disc in accordance with the particular standards. These standards, however, are not formulated with regard to longevity or reliability of the carrier, but only format interchange. Consequently, a disc recorded and playable on a particular machine may in fact be borderline, or even fail to meet the standard that applies. So, although the manufacturers are responsible for the formulation of a disc, the potential life of any information storage media will only be realised if end users take responsibility for producing a suitable digital copy that falls within the parameters set by those standards. Relying on the technology to meet the standards is not sufficient to ensure optimum disc life.

8.1.5.2  This requirement to ensure that the digital information stored on an optical disc is produced in accordance with the standards is exemplified by the issue of disc and burner compatibility. The standards apply to the recording media rather than the replay and recording technology. Philips warns manufacturers of disc burners that they “must implement a writing strategy giving acceptable results”. However, this can be interpreted in a number of ways, resulting in varying compliance. Philips/Sony attempted to address this issues with the MID (manufacturers identification code). The nature of the production of recordable media means, however, that the only information MID really records is the name of the manufacturer of the stampers that are used in the production of discs. Consequently, it has done little to resolve the issue of disc/burner interaction, which remains something of a problem.

8.1.5.3  The standards that apply to Recordable CD include Orange Book Part II: CD-R Volume 1 CD-WO (CD write once) also known as CD-R standard describing 1x, 2x and 4x nominal CD speed. Orange Book Part II: CD-R Volume 2: Multi-Speed CD-R (CD Recordable) describing the speeds up to 48x nominal CD speed. Orange Book Part III: CD-RW Volume 1 CD-RW (CD Rewritable) describing 1x, 2x and 4x nominal CD speed. Orange Book Part III: CD-RW Volume 2: High Speed CD-RW (CD Rewritable) describing 4x and 10x nominal CD speed. Orange Book Part III: CD-RW Volume 3: Ultra Speed CD-RW (CD Rewritable) describing 8x and 32x nominal CD speed. Green Book. Compact Disc Interactive Full Functional Specification and White Book Video-CD Specification. There are also standards for other proprietary CD formats.

8.1.5.4  The standards that apply to Recordable DVD include ISO/IEC 16824:1999 Information technology -- 120 mm DVD rewritable disk (DVDRAM). ISO/IEC 16825:1999 Information technology–Case for 120 mm DVD-RAM disks. ISO/IEC 17341:2004 Information technology -- 80 mm (1,46 Gbytes per side) and 120 mm (4,70 Gbytes per side) DVD re-recordable disk (DVD+RW ). ISO/IEC 17342:2004 Information technology -- 80 mm (1,46 Gbytes per side) and 120 mm (4,70 Gbytes per side) DVD re-recordable disk (DVD-RW). ISO/IEC 17592:2004 Information technology -- 120 mm (4,7 Gbytes per side) and 80 mm (1,46 Gbytes per side) DVD rewritable disk (DVD-RAM). ISO/ IEC 17594:2004 Information technology–Cases for 120 mm and 80 mm DVDRAM disks. ISO/IEC 20563:2001 Information technology -- 80 mm (1,23 Gbytes per side) and 120 mm (3,95 Gbytes per side) DVD-recordable disk (DVD-R). ISO/IEC 16969:1999 Information technology–Data interchange on 120 mm optical disk cartridges using +RW format–Capacity: 3,0 Gbytes and 6,0 Gbytes . ISO/IEC DTR 18002 – DVD File System Specifications. ISO/IEC 13346, Recordable/ Rewritable Volume and File Structure (ECMA-167) and DVD+R - Recordable Optical Disks, 4.7 GB, recording speed up to 4X (ECMA-349).

8.1.5.5  These standards are in addition to those specified in section 5.6.2 Standards.

8.1.6 System Description, Complexity and Cost

8.1.6.1  As noted in Chapter 2, Key Digital Principles, almost all recent generations of computers have sufficient power to manipulate large audio files. Providing all the system standards regarding the equipment used for conversion and ingest of audio data set out in Chapter 2 are met, the system complexity and the degree of expertise required to run such systems is not much greater than is necessary for desktop computer operation. Many reliable CD and DVD burning programs are available that meet the standards required.

8.1.6.2  The only additional equipment required for the production of recordable CD or DVD is the burner, or drive. The drives may be mounted in the computer cabinet or separate though attached to the computer. The drives communicate with the computer through protocols such as IDE and SCSI for internal drives,and Firewire or USB for stand-alones. Certain drives produce lower error rate CD-Rs and DVD-Rs than others, and it is the responsibility of staff to assess and analyse the results of the disc burning before purchasing (see Section 8.1.9 Errors, Life Expectancy and Testing and Analysis).

8.1.6.3  The low system complexity, easy availability of technology, and inexpensive media makes the CD-R and DVD-R a popular option with sound archives. However, as demonstrated in Chapter 6 Preservation Target Formats and Systems, the cost of a more reliable data storage system is less if averaged across the whole collection, even for quite small collections.

8.1.7 Disc and Drive Compatibility

8.1.7.1  Compatibility between discs and drives may well be an issue when recording data on recordable and rewritable CDs and DVDs. Situations often occur where certain discs produced on a particular drive may produce very poor quality duplicates,or may be unreadable on other drives. Testing of this issue has revealed that this failure rate may be very high. An International Standards Organisation project – ISO N178 Electronic imaging – Classification and verification of information stored on optical media, may address the specific problem of drive compatibility.

8.1.7.2  The reason for poor performance may be related to a number of factors: Early drives do not have the laser power to calibrate on later types of discs; Drives designed for dye based discs cannot write, and often cannot read, rewritable discs; Software issues, aging parts, particularly lasers, and particular implementations may all produce inadequate results;The calibration information encoded into the polycarbonate substrate may not necessarily be precisely accurate. However, even taking these issues into account, a significant number of failures occur which are only explained as technical incompatibilities. The equipment manufacturers’ slightly varied implementation of the disc read standard and the variation in the discs quality mean that a situation can occur where discs and drives are incompatible to the extent that the particular combination may produce failed discs on a particular brand, or batch, of discs.

8.1.7.3  In order to ensure that drives and discs are compatible, it is recommended that a range of brands of reliable and reputable discs are recorded on the selected drive, and these discs are tested to determine error levels. This is discussed in the sections below.

8.1.8 Disc Selection

8.1.8.1  There are three basic types of dye used on write once recordable discs, phthalocyanine, cyanine, and azo. Manufacturers of phthalocyanine discs claim a longer life for their product than the competitors. Some, though not all initial testing supports this view. Some manufacturers use Azo dyes in discs that they claim are archival. Cyanine was the first dye type developed for optical disc recording, and is generally recognised by most manufacturers as having a shorter life expectancy (LE). Dye type, though significant, is only one of the factors determining the life of the media.

8.1.8.2  The variation in the amount of dye used in the dye layer, a result of the manufacturers’ race for even higher recording speeds and higher density recording, is a contributing factor in the long term failure of recordable optical media. Recording speed has increased from X1 to X52 and is still rising, as the recording density has gone from 650MB to 800MB for CD-Rs. It should be noted that discs optimised for high speed recording use less dye, which may indicate a shorter life expectancy. DVD-R uses less dye as a matter of course, as the data rate when writing to a recordable DVD is much higher than for CD-R.

8.1.8.3  It is not, however, just a matter of reducing speed; if discs with a denser dye layer, optimised for writing at lower speeds, are written at higher speeds, they deliver a worse error rate. Though manufacturers indicate the maximum recording speed, writing at that maximum speed may not achieve adequate results. There is an optimum writing speed at which the disc produced obtains the best possible technical measurement for performance. Identifying this speed is best done by trial and error measurement using a reliable disc tester. Typically, the best results will be achieved on a dense dye layer disk written at around 8 times speed.

8.1.8.4  At best, the quality of blank recordable CD and DVD media can be described as variable. The recordable CD and DVD- manufacturing industry has become a market place driven by narrow profit margins and large quantities. Recordable CD and DVD manufacturing equipment has become smaller, cheaper and more self-contained. As a consequence, the production of reliable data carriers for the quality market has largely been replaced by manufacturers of recordable CD and DVD, producing recordable CD and DVD for the low cost market.

8.1.8.5  Many discs that appear to be reputable brands may turn out to have been manufactured by a second party and repackaged for sale. A recordable CD or DVD manufacturer can manipulate the dye, reflective layer and the now expensive polycarbonate components to reduce price or control quality. As a general rule, it has often been recommended that only reliable brand recordable CD and DVD are purchased, however, testing has revealed a range of compliance with agreed standards even amongst them. Instead, it is recommended that the responsible individual or institution insist on dealing with a supplier that is open about the importer or manufacturer they deal with, and who is able to provide contact with the relevant technical personnel in the manufacturing company. Discs that fail the standard specified below should be returned.

8.1.8.6  It is quite difficult to identify the best quality media without high level analysers (Slattery et al., 2004). In most practical circumstances discs must be recorded before they can be tested. Some very high quality CD and DVD testing equipment will analyse an unrecorded disc, but most testing is carried out by recording a test signal and analysing the result. ISO 18925:2002,AES 28-1997, or ANSI/ NAPM IT9.21 is a standard test method to establish the life expectancy of compact discs, and ISO 18927:2002/AES 38-2000 is a standard for estimating method for estimating the life expectancy based on the effects of temperature and relative humidity for recordable compact disc systems. As temperature and humidity aging does not always produce clear results, other approaches have concerned themselves with the susceptibility of recordable dye based discs to light exposure with age, and some manufacturers have undertaken testing in this area. There is, however, no standard for this (Slattery et al., 2004).

8.1.8.7 Summary of Disc Selection

8.1.8.7.1 Purchase a range of best quality discs, based on market research.

8.1.8.7.2 Purchase more than one of each type. (Though price is not necessarily an indicator, always remember that the cost of even the most expensive discs is small compared to the value of the data.)

8.1.8.7.3 Under controlled conditions record some data on each of the discs.

8.1.8.7.4 Test to see which discs perform best with regard to specification in this document. All discs must exceed the recommended quality standards recommended below (see Table 1, Maximum error levels in an archival CDR).

8.1.8.7.5 Test at a number of different writing speeds.

8.1.8.7.6 Keep disc/burner compatibility in mind: different burners may yield different results.

8.1.8.7.7 Choose the three best discs, from at least two dye types (phthalocyanine and azo).

8.1.8.7.8 Record identical copies of the data on the three chosen discs.

8.1.8.7.9 Ensure that delivered supplies of chosen discs are identical with tested sample discs.

8.1.8.7.10 Repeat tests each time a batch of discs are purchased.

8.1.9 Errors, Life Expectancy and Testing and Analysis

8.1.9.1  The only way to know the condition of a digital collection is constant and comprehensive testing. This cannot be stated too strongly; no collection using CD-R or DVD-R/+R as an archival carrier should be without a reliable tester. The error correction capability of most replay equipment will mask the effects of degradation until the errors are well into the uncorrectable region.When this point is reached, all subsequent copies are irreversibly flawed. On the other hand, a comprehensive testing regime allows for best possible planning of preservation strategies by acting on the known, objective and measurable parameters that digital archiving make possible. In the well-documented digital archive, metadata will record the history of all objects, including a record of error measurements and any significant corrections.

8.1.9.2  Life expectancy of CD-R or recordable DVD is a many varied topic. To most end users, a CD-R or DVD-R/+R reaches the end of its life when the drive no longer reproduces the data written on the disc, but because drives are not governed by standards, a CD/DVD that will not play on one drive may well play on another. There are innumerable examples of this. The ANSI/NAPM IT9.21-1996 – Life Expectancy of Compact Discs (CD-ROM)- Method for Estimating Based on Effects of Temperature and Relative Humidity, discusses many of these issues. Alternately, some standards and suppliers specify an acceptable Block Error Rate (BLER). BLER is the number of erroneous blocks per second measured at the input of the C1 decoder (see ISO/IEC 60908) during playback at the standard (x 1) data rate averaged over a 10 second measuring period. Standards ISO/IEC 10149 and ANSI/NAPM IT9.21-1996, or Red Book standard, specify a maximum BLER rate of 220. The standard for recording general data on CD, otherwise known as Yellow Book standard, specify a BLER of 50. For data purposes this lower level is vital.

8.1.9.3  Studies have shown that BLER alone is not a very useful measure when determining LE, because defective discs may exhibit BLER well under 220, or indeed under 50. It is necessary to measure other test parameters, among them E22, E32 (uncorrectable errors), and frame burst errors (FBE, sometimes called Burst Error Length or BERL), which are valid end-of-life indicators.When these parameters exceed the limits specified below, it indicates a need for immediate duplication, assuming the disc containing archival information is still readable.

8.1.9.4  Errors in archival CD-Rs should not exceed that specified in the table below. These are maximum levels after which CD-Rs must be copied. In practice error levels much lower than this are achievable and preferable, and must be met in order for the disc to have any archival life before recopying becomes necessary. A BLER average of 1 and a peak level of less than 20 are easily achievable. Jitter is also a useful diagnostic indicator of the quality of the data recorded on a CD and should be measured after writing. The 3T jitter values should not exceed 35 nS (Fontaine and Poitevineau, 2005).

 

Frame burst errors FBE <6
Block error rate BLER average < 10
Block error rate BLER peak < 50
E 22 (correctable errors) 0
E 32 (uncorrectable errors) 0
3T Jitter <35nS

 Table 1 Section 8.1, Maximum error levels in an archival CD-R

8.1.9.5  The construction of a DVD is significantly different to that of the CD, and though there are many aspects in common the criteria that applies to CDs does not necessarily apply to the DVD. Jitter in DVDs is customarily measured in percentages. Though measured differently, the actual jitter measurement is largely equivalent in the two disc types, the main error measurements, however, are quite different. The two main DVD error measurements are Parity Inner Errors (PIE) and Parity Outer Errors (POE). Industry standards state that the POE should be zero. Other types of error measurement are defined, but at the time of writing no agreed threshold for archival purposes has been developed. The DVD specification also states that any eight consecutive ECC blocks (PI Sum8) may have a maximum of 280 PI errors and jitter should not exceed 9%. However, with regard to the use of recordable CD, archival experience and testing has led to a recommendation in maximum error levels that is approximately 25% of the red book recommendations. An extrapolation on the DVD figures would lead to a recommendation of a maximum of 70 PI errors in any eight consecutive ECC blocks. It is important to recognise that a distributed range of tests on DVD recordable in archival situations has not been undertaken to assess the validity of these figures.

8.1.9.6  Initial investigations indicate that recordable CDs do not necessarily proceed to failure in a linear way and that as a consequence small change in initial error rates could have a greater effect on useful life of the disc. There are several tests that have indicated this to be the case (Trock, 2000), (Bradley, 2001), however, there has not been an extended examination of this proposition. A “longitudinal” examination of recordings over time coupled with artificial aging experiments might bring better information on the factors of disc stability. A factor which continues to add to the lack of consistent research is the lack of an agreed standard for the production of CD/DVD-drives.

8.1.9.7  The comparison of the solid black line to the dashed line (see Fig 1 Section 8.1 overleaf) illustrates that the better the initial recording is the longer the expected lifetime will be. There are several tests that have shown this to be the case (Trock JTS 2000, Bradley IASA/SEAAPAVA 2001), however, there is no empirical proof that this is the case. The dashed line, starting at a higher error level, decays at the same rapid rate, but starting earlier reaches failure level in a much shorter period of time. A “longitudinal” examination of recordings with time, aging experiments, might bring better information on the factors of disk stability. A factor adding to the lack of consistent research is the fact that there is no standard for the production of CD/DVD-drives.

8.1.9.8  Being a composite item containing, amongst other components, organic dyes or other chemical compounds, these optical carriers are bound to deteriorate due to slow chemical reactions. Choosing optical discs as the target medium entails the requirement to set up a monitoring program for the discs and a procedure for recopying discs that approach the limit of LE. The use of recordable and rewritable CD/DVDs as archival carriers cannot be advocated unless a strict testing and monitoring program is set up. It should be noted that testing and analysing, though absolutely necessary, will be time consuming, adding long-term costs to the archival solution.When planning an archival strategy, these costs should be included. Logs of test results should be stored, and occasional testing, perhaps annually, can be carried out on statistically appropriate number of stored discs carrying archival information.When the error rate is shown to be increasing, a transfer to a new carrier can be undertaken of all the discs of that age or type.

accumulated CD-R errors over time

Fig 1. Section 8.1: Accumulated errors in a CD-R over time

8.1.9.9 Summary of Testing

8.1.9.9.1 Test all discs when writing.
8.1.9.9.2  Reject any discs which fail to meet specification.
8.1.9.9.3  Store the relevant test records of all discs.
8.1.9.9.4  Undertake a regular testing of a statistically significant number of stored discs of each different batch of products. 8.1.9.9.5  Undertake a recopying of discs when error rates increase.

8.1.10 Testing of Existing Recorded Discs

8.1.10.1  If data on recordable CD or DVD was not tested at the time of creation, it is critical that tests are made of their current state. Discs must be subjected to rigorous error testing as their current error rates play a major part in determining their further life expectancy. If error rates are measured above the levels expressed in table 1, contents should be immediately transferred to new media.

8.1.11 Testing Equipment

8.1.11.1  Professional testing equipment with dedicated, or at least specified, drives is recommended for accurate testing DVDs and CDs. Such systems are more expensive but are necessary if accurate, reliable and repeatable error measurement are to be achieved. The testing should at least comply with ISO 12142 Electronic imaging – Media error monitoring and reporting techniques for verification of stored data on optical digital data discs. Such testing will not, however, address the problem of the lack of standardisation of optical disc drives. There is at the moment of writing, a standards project with the International Standards Organisation, ISO N178 Electronic imaging – Classification and verification of information stored on optical media, which may address the specific problem of drive compatibility. Although there is test software available on the web as shareware, such software should be carefully evaluated before being relied on in an archival environment. Such software based systems depend on the accuracy of the non-standard computer drives. If a testing system based on computer drives is required, then a proprietary system supplied by the disc manufacturer stands a better chance of being useful. At least one CD/DVD burner company does provide software that allows their drive to be used for the purposes of testing. The results of any testing system that depends on the CD burning drive should be checked against a known, calibrated testing system to ensure adequate compliance.

8.1.11.2  Disc test equipment which accurately measures only the parameters specified in this guidance document are commercially available and of good standard. However, the figures provided by testing these parameters are suitable only for identifying problems. Analysis of problems probably requires access to a high analytical CD and DVD testing facility. It is useful to gain access to this type of equipment, by renting or borrowing, when solving problems, selecting blank media or calibrating in house testing facilities.

8.1.11.3  Kodak, in their web-document “Permanence and Handling of CDs” (Kodak 2002) claim that 95 % of their CD- Rs will maintain a data lifetime of a hundred years in an office environment. The results of these tests are often held to be suspect by archivists, and many have found it difficult to reproduce the tests and achieve the same results. This may be due to different interpretation of the figures and some argument about the validity of the method of estimating lifespan. Even if these tests proved to be true, and in the unlikely event that CD drives are still available 100 years hence, a 5% failure rate is unacceptable in an archive. This conclusion also supports the requirement of an error monitoring program.

8.1.11.4  Accurate, High Quality Production Testers

8.1.11.4.1  At the time of writing the cost of accurate, high quality production testers starts at around US$ 30,000 for the basic models and increases to over US$ 50,000 for many devices. The cost is incurred in the high quality reference drives which are a necessity for accurate and repeatable testing. All testers are aimed at the market of optical disc manufacturers for production control purposes. Actual prices depend on the scale of measurable parameters, many of which are not relevant for testing recordable optical discs as to their archival reliability. Currently, there are three producers of high quality testers:Audio Development (http://www.audiodev.com/), DaTARIUS (http://www.datarius.com/) and Expert Magnetic Corporation (http://www.expertmg.co.jp/). Manufacturers and suppliers should be contacted for quotes.

8.1.11.5  Mid Range Quality Production Testers

8.1.11.5.1 At the time of writing the cost of these devices range from a US$ 3,000 to US$ 11,000 or more. These systems test all the required parameters using standard PC drives which have been specially selected and calibrated. It is recommended that before considering such mid priced testers, the prospective purchaser investigate thoroughly the types of drives and the accuracy of the device. It is also strongly recommended that all mid priced systems be regularly calibrated against a known standard. Currently, a major manufacturer of such mid range testers is Clover Systems (http://www.cloversystems.com/)

8.1.11.6  Downloadable Testers

8.1.11.6.1  There are a number of downloadable testers available online which use a computer’s inbuilt CD/DVD drive to measure error in written CD and DVDs. However, due to the limitations of the software and inaccuracy of the drives, most, if not all, are unsuitable for archival purposes.

8.1.12 Access and Data Migration

8.1.12.1  Discrete carriers like CDs and DVDs are not well suited for on-line access. Making a collection available necessitates staff handling the disks. Handling is one of the worst enemies of this kind of media. Always handle the disks by the edges, and always keep them in their enclosures when not played. The effect of light on the dyes are documented as a deteriorating factor, and excessive temperature and humidity must be avoided, as this may hasten degradation of the disk, and in extreme cases cause delaminating of the polycarbonate layers (Kunej 2001). The disks should be stored in acrylic jewel cases, and cheap plastic sleeves should be avoided as they may create an environment that is detrimental to the disk.

8.1.12.2  Copying for access purposes is however an easily undertaken task, and may be done at many times real time. There are jukeboxes on the market, which, with the appropriate software, will enable online access to the collection, though copying to hard disc may be preferable.

8.2 Magneto-Optical Discs

8.2.1.1  The first (2004) edition of TC-04 described, as a possible target format, Magneto-Optical Discs. By the time of publishing it had reached a capacity of 9.1 GB. This development marked the end for this technology, and the format now has to be considered as endangered. The consequence of this is that the media and carriers will in time become difficult, if not impossible, to obtain. All content on M-O disc should be marked for migration to an appropriate storage system.

8.2.1.2  There has however been developed a new format that uses the same standardised 5.25 inch caddies as the MO disks, called UDO, (Ultra Density Disc). These discs use a phase change technology similar to CD-RW, and differ only from these in that they come in MO style caddies that protect the discs. Some hardware systems allow for use of both MO and UDO technologies in the same robot. A blue laser (405 nm) is used with a double-sided disc. The first UDO was presented in fall 2003 with 30 GB storage capacity. UDO disks are currently available with a 60 GB capacity, with a roadmap promising 120 GB in the next year, and a speculation on 500 GB as the ultimate target

8.2.1.3  Testing and Arrhenius extrapolation have estimated a life expectancy of up to 50 years. As discussed above in relation to other media, such testing should be considered cautiously. It is also much more likely that format obsolescence will be the ultimate limit of long term viability. Though UDO has some adherents, the technology has not penetrated the market to any extent and is consequently a risk for long term archival storage.

8.2.1.4  Though technological developments provide the pathway to long term preservation of our audio content, it behoves the curator, archivist and technician responsible for an archival collection to take a conservative and careful approach in the adoption of any new technology.

9: Partnerships, Project Planning and Resources

9.1.1 Introduction

9.1.1.1  The production and long term preservation of digital audio objects incorporates a number of interrelated parts, many quite complex. These guidelines define the tasks as: Extraction of audio content to create archival digital audio objects; Ingest of the content into a digital storage system including the creation of necessary metadata: Administration and management of the data and system: Archival storage: Preservation planning: and Access.

9.1.1.2  Some institutions have the facility to undertake all the tasks, as well as a collection whose size justifies the expenditure. The alternative is to negotiate partnerships to manage some or all of the tasks on behalf of the collection owners. These partnerships may be with other, larger institutions, could include partnerships with like minded institutions, or could represent a commercial relationship with a supplier.

9.1.1.3  This section of the Guidelines examines the resources required to create and preserve digital audio objects according to technical requirements described in this document. It considers the issues related to size of collections and scale of work, recognising that the professional fulfilment of the requirements as described herein can only be met when the size of the collection held by the respective institutions reaches a critical mass that make autonomous preservation viable. Many institutions, collections or archives have particular expertise and resources in core areas which they can deploy to facilitate the necessary processes. It is recommended that they maximise the benefit from their core business area while carefully examining the areas where services may be better sought elsewhere.

9.1.2 Archival Responsibilities and Collections

9.1.2.1  The first decision to be made is whether an institution should engage in digital audio preservation at all. Often audio or audiovisual collections came about in institutions with a variety of other aims which may not include professional preservation of audio materials. The ever increasing problems related to the physical preservation of an audio collection, the obsolescence of dedicated replay equipment, and digital long term preservation may suggest a rethinking of the collection and preservation policy. Where appropriate alternatives exist, audio collections could be handed over to more specialised institutions. This would not necessarily mean fully relinquishing ownership a collection; the receiving archive could be asked to produce, in return, listening copies that could be held – without significant costs – for further in-house use. Various possibilities of retaining, partly or fully passing on the right of ownership as well as user rights could be applied.

9.1.3 Sharing archival responsibilities

9.1.3.1  Should an institution like to maintain archival responsibility for their collection a number of different scenarios may apply which do not require relinquishing the collection.

9.1.3.2  Producing digital audio objects in-house but entrusting digital preservation to another is one possibility. There are a number of ways this scenario could be enacted. One way, which seems most appropriate in academic institutions and universities, is where several units are engaged in the production and use of digital audio (and audiovisual) documents. Generally, such institutions have a central computer facility, very often with an existing responsibility for managing various digital objects. The data storage facility could take responsibility for the long term preservation of the created audio content. It is important, however, that the central unit would be fully acquainted with the specific of long term preservation of digital audio objects, and develops well defined rules for the production of archival files. The central unit would prescribe recording formats, resolution, annotation procedures, and other archival issues to be followed. Also, long term preservation tasks of that kind could also be fulfilled by private entrepreneurs. This concept would work for newly produced materials, specifically field recordings in various disciplines like anthropology, linguistics, ethnomusicology, and oral history.

9.1.3.3  Another way in which this scenario might be enacted is where a large collection exists with appropriate storage, transfer facilities and technical expertise, but where the infrastructure to support a digital storage facility is not developed enough to build a trustworthy digital repository. Under these circumstances the local facility may undertake the signal extraction and dispatch the resultant digital audio objects to the selected archive.

9.1.3.4  Should institutions, however, already have accumulated, though dispersed, analogue and historical digital originals, signal extraction from these originals to produce digital preservation files could be concentrated in one professionally equipped unit which could also be appended to the central computer unit. If the entire institution does not reach a critical amount of carriers, signal extraction should better be outsourced. The same is true if the institution does not have in-house expertise or equipment for professional digitisation.

9.1.3.5  In any of these scenarios, where a third party archive is to take responsibility for the ingest, management and preservation of the digital audio objects, it is imperative that there is a clear understanding of the roles and responsibilities of the various partners in the work. The ISO 20652:2006 “Data and information transfer systems -- Producer-archive interface -- Methodology abstract standard” identifies, defines and provides structure to the relationships and interactions between an information producer and an archive. It defines the methodology for the structure of actions that are required from the initial time of contact between the producer and the archive until the objects of information are received and validated by the archive. These actions cover the first stage of the ingest process as defined in the open archival information system (OAIS) reference model (see ISO 14721). http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?c...

9.1.4 Critical Mass

9.1.4.1  Critical mass in the field of sound preservation is where the size of collection is sufficient to justify the expenditure to undertake all the tasks in house. It is difficult to quote concrete numbers when defining critical mass; the more professional institutions there are available in a country or region, the higher the critical mass will be. If, however, there are only few institutions engaged in professional audio archiving, or if there is none available at all, then the critical mass would be lower. Critical mass should always relate to specific media formats; coarse grooved discs, microgroove discs, open reel magnetic tape, etc. In fairly developed countries or regions critical mass would be at least several thousands of items, but often institutions with tens of thousands of carriers of one type make a rational decision to outsource signal extraction. Under less developed circumstances the autonomous transfer of few thousand items/hours only can be carried out successfully.

9.1.4.2  The critical mass will also depend on the homogeneity of the material within the respective format. Homogeneous collections can be transferred with some degree of automation. The cost associated with fully automated systems generally suggests outsourcing to institutions or service providers that offer computer controlled parallel transfers. Collections consisting of many different carriers or standard of recording – as often found in research collections – demands reliable manual transfer, which may be available at lower cost in house, provided the specialised expertise is available.

9.1.4.3  Even large, professional sound archives may consider sending parts of their collection to specialised institutions or service providers for the purposes of transfer. This may especially be so for some historical analogue and digital carriers.

9.1.5 Outsourcing

9.1.5.1  Whenever material is outsourced for the purpose of signal extraction, especially to private entrepreneurs, it is important to accurately define the tasks to be fulfilled. This is best achieved by specifying the standards provided by IASA in these Guidelines as part of the contract.

9.1.5.2  When outsourcing any audio processes it is essential to establish a quality control system that provides a high level of assurance that all contracted work has been carried out appropriately. Such measures should be based on stringent delivery of preservation metadata, accompanied by tests of randomly chosen samples, including unannounced visits at service providers and testing of the transfer equipment. Specific attention should be given to test the automated and manual quality control systems established by the supplier, their capacity to manage long term contracts through the use of project management methodology, experience with similar contracts and with specific carriers, equipment maintenance, and finally the balance between cost and quality. Before the start of the production level digitisation phase specific small tests should be performed to ensure all aspects of the process meet criteria before commencing to process on a larger scale.

9.1.5.3  It is a responsibility of a sound archive to manage and control access to its collections in accordance with any legal, moral or ethical constraints which are associated with the content: Outsourcing the processes does not allow the archive to abrogate its responsibilities in this matter. When archival material is given to a third party to undertake any audio processes it is necessary to define in contract the restrictions under which the service provider must operate. For commercial copyright material the legal limitations are probably described in law and can be referred to. Where privacy or other ethical rights are of concern, these should be defined and the service provider should acknowledge their agreement to comply. It is also important to specify how and when copies will be eliminated from the contractor’s storage system when their responsibility comes to an end and the material and content is returned to the owners or archive.

9.1.6 Quantitative Assessment of Project Dimensions

9.1.6.1  Whether preservation is carried out autonomously in-house, or partly or fully outsourced, an indispensable pre-condition for seriously planning preservation is the quantitative assessment of the project. Serious and costly mistakes are often made by underestimating the amount of work needed for the optimal signal extraction from original carriers. Therefore, the first step is to count the numbers of carriers and their playing time.With mechanical carriers, compact cassettes, and optical carriers there is a fairly clear relation between the number of carriers and their respective playing times. This may be more complex in the case of magnetic open reel collections as the playing time is dependant on the length of the tape, the speed of recording and the numbers of tracks. However, with good knowledge of the specific collection, some well founded assumptions can often be made which lead to reasonably accurate estimates. In poorly documented or undocumented collections, a situation often encountered in the estates of prominent persons, this assessment can be extremely time consuming.

9.1.6.2  Once the duration of the carriers to be transferred is assessed, a second important factor is their physical condition. The time factors given in the respective parts of Chapter 5, Signal Extraction from Original Carriers, relate to well preserved items. Any cleaning and restoration measures required may add substantially to transfer times, and must be included in the calculations accordingly.

9.1.7 Hierarchy of transfer to digital

9.1.7.1  Paragraph 16 of IASA-TC 03 “The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy”, describes that, except for lacquer discs, which may fail at any moment without pre- warning, the sequence of transfer within a specific collection is a multi-faceted decision based on the requirement for access to documents, their physical condition and, with ever increasing importance, the availability of equipment, spare parts and professional service support. The project “Sound Directions” has developed “FACET”1 , a tool to asses the respective parameters of a collection to assist in making a decision on a fairly objective and traceable basis. It must be noted, however, that obsolescence of formats and related problems like withdrawal of professional service support, e.g. for R-DAT machines, change rapidly, which calls for constant monitoring of the situation and re-assessment in regular intervals.


1. FACET was developed within project Sound Directions by the Archive of Traditional Music, Indiana University Bloomington, USA. http://www.dlib.indiana.edu/projects/sounddirections/facet/

9.1.8 Long term preservation of digital audio objects

9.1.8.1  It is quite common that, when commencing digital preservation, the costs of long term storage of digital audio objects are permanently and persistently underestimated. At the time of writing professional storage costs are considered to be in the order of a minimum of $US 5/GB/year2 for medium to large scale storage (over 5 TB) Although the hardware cost price has been permanently declining, the cost of the management of the storage, the continuous migration to new generation storage, the hosting in adequate premises (clean room, etc) are always underestimated. As a political target UNESCO has challenged the IT industry to arrive at $US 1/GB/year in the short term, a target which seems far from being met. Some figures detailed in a PrestoSpace study show a trend to stabilize long term storage costs at $US 9/GB/year. As digital audio objects on an average require 2 GB/hour, even future lower preservation costs will still be too high for many cultural institutions.

9.1.8.2  Lower digital storage costs can only be achieved for smaller quantities if the labour costs involved in small scale manual approaches are not incorporated. The systematic use of open source software may make autonomous, not fully automated processes viable in near future also for medium amounts (10-20TB) of storage requirements. The involvement of specialised staff to guarantee the permanent availability of archival files in manual or semi-automatic operation must not be underestimated.

9.1.8.3  Some service providers have recently developed adequate outsourced preservation strategies based on the mutualisation of the use of professional large scale mass storage systems with specific access schemes to users. Their fee is usually based on the size of the digital archives to be stored, the duration of the contract, and the associated services. For small and medium scale archives it can be an attractive solution, as well as for large scale archives before deciding to invest in their own storage solution.


2. Despite the current difference in monetary value USD and EURO are approximately same in the IT world

9.1.9 Calculating overall costs

9.1.9.1  Perhaps the most crucial point when making these decisions is calculation of costs. Unfortunately, no generally applicable concrete figures can be given in this context. In-house costs are difficult to assess as many institutions holding audiovisual collections have infrastructures available (rooms, air conditioning, intranet) the cost of which is incorporated into the general budget, which makes it difficult to calculate overall costs for transfer and/or permanent digital preservation. Labour costs differ significantly even in developed neighbouring countries, which weakens any general conclusions as regard price. Finally, services offered by professional vendors vary considerably, depending on the amount of items per carriers, their state of preservations and hence the possibility to automate the process. Cost of staff, equipment and other resources generally rise over time, while it is possible prices for some automated processes may reduce.

9.1.9.2  Because of the many factors related to a specific preservation project these guidelines refrain from any quotation of price ranges for transfer. These guidelines suggest that holders of collections acquaint themselves thoroughly with the specific situation in their countries or regions and observe the market situation on a constant basis.

9.1.9.3  When seeking prices for audio preservation services, tenders must be well prepared and defined in detail, and any subsequent offers carefully examined. Bids that offer the same service for a fraction of other vendors should be examined with extreme scepticism. Finally, outsourcing can only be successfully managed if a stringent quality assurance system, as described, is established, and any substandard work is rigorously rejected.

9.1.10 Summary

9.1.10.1  In summarising preservation planning, it is strongly recommended that holders of audiovisual collections take the present need for preserving their holdings as an occasion to rethink their overall strategy: All scenarios, from total withdrawal from preservation responsibility,through cooperating in or outsourcing of signal extraction and digital long-term preservation, to taking full autonomous responsibility, should be examined. Each collection is different and institutions are embedded in a variety of environments. These multiple scenarios, which also change over time according to technical development, will make it difficult to decide on a purely economic basis. Generally, all holders of audiovisual collections, specifically of small collections, are strongly encouraged to seek cooperative relationships to manage their preservation requirements. The extent to which responsibility for signal extraction and digital long-term preservation is accepted in-house, should be linked to the general mission of that institution or collection. Memory institutions may decide differently from research collections, which have a strong interest in the availability of audio documents but whose core business does not necessarily include the processes that guarantee their further survival.

References

  • Al Rashid, Shahin 2001 Super Audio CD Production Using Direct Stream Digital Technology, Internet on-line http://www.merging.com/uploads/assets/Merging_pdfs/dsd1.pdf Merging Technologies/Canada Promedia Inc.
  • Anderson, Dave, Jim Dykes and Erik Riedel 2003 More than an interface – SCSI vs.ATA, in Proceedings of the 2nd Annual Conference on File and Storage Technology (FAST), March 2003. Available online at http://www.cs.cmu.edu/~riedel/papers/apr03_ExternalSCSIvsATA.pdf
  • Benson K. Blair (ed) 1988 Audio Engineering Handbook, McGraw Hill, New York.
  • Boston, George IASA Survey of Endangered carriers, (draft). Circulated to IASA TC Members September 2003.
  • Bradley, Kevin (2001) CD-R, Case Study of an Interim Media. IASA/SEAPAAVA Conference. Singapore.
  • Bradley, Kevin (2006) Risks Associated with the Use of Recordable CDs and DVDs as Reliable Storage Media in Archival Collections – Strategies and Alternatives, Memory of the World Programme, Sub-Committee on Technology. Available from http://unesdoc.unesco.org/images/0014/001477/147782E.pdf
  • Bradley, Kevin 1999 Anomolies in the Treatment of Hydrolysed Tapes: Including Non-Chemical Methods of Determining the Decay of Signals in George Boston (ed) Technology and our Audio Visual Heritage; Technology’s role in Preserving the memory of the world JTS 95 A Joint Technical Symposium, pp. 70-83.
  • Bradley, Kevin 2004 Sustainability Issues, APSR Report http://www.apsr.edu.au/documents/APSR_Sustainability_Issues_Paper.pdf
  • Bradley, Kevin, Lei Junran, Blackall, Chris 2007 Towards an Open Source Repository and Preservation System: Recommendations on the Implementation of an Open Source Digital Archival and Preservation System and on Related Software Development UNESCO Memory of the World, http://portal.unesco.org/ci/fr/files/24700/11824297751towards_open_sourc...
  • Brock-Nannestad, G. 2000 The calibration of audio replay equipment for mechanical records, in Michelle Aubert and Richard Billeaud, (eds) Image and Sound Archiving and Access: the challenges of the 3rd Millenium Proceeding of the Joint Technical Symposium Paris 2000, CNC, Paris, pp. 164-175.
  • Byers, F. R. (2003) Care and Handling of CDs and DVDs – A Guide for Librarians and Archivists, NIST Special Publication 500-252.
  • Byers, Fred R. 2003 Care and Handling of CDs and DVDs – A guide for Librarians and Archivists (Draft) National Institute of Standards and Technology, Maryland, USA.
  • Casey, Mike and Gordon, Bruce 2007 Sound Directions: Best Practices for Audio Preservation Indiana University and Harvard University http://www.dlib.indiana.edu/projects/sounddirections/papersPresent/sd_bp...
  • Copeland, Peter 2008 Manual of Analogue Sound Restoration Techniques, British Library Sound Archive http://www.bl.uk/reshelp/findhelprestype/sound/anaudio/manual.html
  • Dack, Dianna 1999 Persistent Identification Schemes National Library of Australia, Canberra available on line http://pandora.nla.gov.au/pan/33822/20030228-0000/www.nla.gov.au/initiat... [accessed 4 Nov 2012]
  • Dempsey, Lorcan 2005 All that is solid melts into flows, Lorcan Dempsey’s weblog On libraries, services and networks. May 31, 2005 Internet on line, http://orweblog.oclc.org/archives/000663.html [accessed January 2009]
  • Dublin Core, http://dublincore.org/
  • DVD Forum 2003 Guideline for Transmission and Control for DVD-Video/Audio through IEEE1394 Bus Version 0.9 [Internet] DVD Forum. Available from http://dvdforum.org/tech-guideline.htm [accessed January 2008]
  • Eargle, John M. 1995 Electroacoustical Reference Data, Kluwer Academic Publishers.
  • Engel, Friedrich 1975 Schallspeicherung auf Magnetband, Agfa-Gevaert, Munich
  • Enke, Sille Bræmer: 2007 Udretning af Deforme Vinylplader (Flattening of deformed vinyl discs), research project, The School of Conservation, The Royal Danish Academy of Fine Arts.
  • Feynman, Richard P. 2000 The pleasure of finding things out, London: Penguin, 2000
  • Fontaine, J.-M. & Poitevineau, J. (2005) Are there criteria to evaluate optical disc quality that are relevant for end-users? AES Convention Paper. Preprint Number 6535.
  • Gunter Waibel. Like Russian dolls: nesting standards for digital preservation, RLG DigiNews, June 15, 2003, Volume 7, Number 3, http://worldcat.org/arcviewer/1/OCC/2007/08/08/0000070511/viewer/file674...
  • Hess, Richard L. 2001 Vignettes; Media Tape Restoration Tips – Cassette track layout Internet on-line http://richardhess.com/notes/
  • King, Gretchen, (N.D.) Magnetic Wire Recordings: A Manual Including Historical Background, Approaches to Transfer and Storage, and Solutions to Common Problems, http://www.music.washington.edu/ethno/wire1.html (accessed 10 October 2008).
  • Klinger, Bill 2002 Stylus Shapes and Sizes: Preliminary Comments on Historical Edison Cylinder Styli. Unpublished paper
  • Kodak 2002 Permanence and Handling of CDs, Internet on-line web.archive.org/web/20041230085156/http://www.kodak.com/global/en/professional/products/storage/pcd/techinf... [accessed 23 June 2011]
  • Kunej, D. (2001) Instability and Vulnerability of CD-R Carriers to Sunlight. Proceedings of the AES 20th International Conference Archiving, Restoration, and New Methods of Recording, Budapest, pp. 18-25.
  • Langford-Smith, F (ed) 1963 Radiotron Designer’s Handbook,Wireless Press, for Amalgamated Wireless Valve Co. Pty. Ltd., Sydney, Fourth Edition, Sixth Impression.
  • LOCKSS (Lots of Copies Keep Stuff Safe) http://www.lockss.org/lockss/Home Accessed November 2008
  • Machine Readable Cataloging (MARC). Library of Congress, http://www.loc.gov/marc/
  • Maxfield, Joseph P. and Henry C. Harrison. 1926 Methods of High Quality Recording and Reproducing of Music and Speech Based on Telephone Research, in Bell System Technical Journal 5, p. 522.
  • McKnight, John G 2001 Choosing and Using MRL Calibration Tapes for Audio Tape Recorder Standardization. MRL (Magnetic Research Laboratories, Inc) Publication Choo&U Ver 5.7 2001-10-25, internet online https://home.comcast.net/~Emrltapes/choo&u.pdf [accessed 23 June 2011]
  • McKnight, John G. 1969 Flux and Flux-frequency Measurements and Standardization in Magnetic Recording, Journal of the SMPTE,Vol. 78, pp. 457-472.
  • Metadata Encoding and Transmission Standard (METS), http://www.loc.gov/standards/mets/
  • Metadata Object Description Schema (MODS), http://www.loc.gov/standards/mods/
  • Morton, David 1998, Armour Research Foundation and the Wire Recorder: How Academic Entrepreneurs Fail, Technology and Culture 39, no. 2 (April 1998, pp. 213-244
  • Open Archives Initiative – Protocol for Metadata Harvesting (OAI-PMH) http://www.openarchives.org/OAI/openarchivesprotocol.html [accessed November 2008]
  • Osaki, H. 1993 Role of Surface Asperities on Durability of Metal-Evaporated Magnetic Tapes in IEEE Trans on Magnetics 29/1 Jan 1993.
  • Paton, Christopher Ann,Young, Stephanie E., Hopkins, Harry P. & Simmons, Robert B. 1997 A Review and Discussion of Selected Acetate Disc Cleaning Methods:Anecdotal, Experiential and Investigative Findings in ARSC Journal XXVIII/i
  • Plendl, Lisa 2003 Crash Course: Keeping The Trusted Hard Drive More Reliable Than Ever,The Ingram Micro Advisor, Internet online http://web.archive.org/web/20081120175220/http://www.macadept.com/images... [accessed 23/06/2011]
  • Powell, James R., Jr. & Stehle, Randall G. 1993 Playback Equalizer Settings for 78 rpm Recordings, Gramophone Adventures, Portage MI, USA
  • PREMIS (PREservation Metadata: Implementation Strategies) Working Group 2005 http://www.oclc.org/ research/projects/pmwg/ [accessed November 2008]
  • Ross, Seamus and Ann Gow 1999 A JISC/NPO Study within the Electronic Libraries (eLib) Programme on the Preservation of Electronic Materials, Library Information Technology Centre, London. Available on line http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf [accessed 7 January 2004]
  • Rumsey ,Francis and John Watkinson 1993 The Digital Interface Handbook Focal Press, Oxford, UK.
  • Schuller, Dietrich (ed) IASA-TC 03 2005 The Safeguarding of the Audio Heritage: Ethics, Principles and Preservation Strategy, International Association of Sound and Audiovisual Archives, IASA Technical Committee,Version 3, September 2005, internet online http://www.iasa-web.org/safeguarding-audio-heritage-ethics-principles-pr... [accessed January 2009]
  • Schuller, Dietrich (1980) Archival Tape Test. Phonographic Bulletin 27/1980, pp. 21-25
  • Slattery, O., Lu, R., Zheng, J., Byers, F. & Tang, X. (2004) Stability Comparison of Recordable Optical Discs – A Study of Error Rates in Harsh Conditions, Journal of Research of the National Institute of Standards and Technology. NIST.
  • Tennant, Roy 2004 A Bibliographic Metadata Infrastructure for the 21st Century, in Library Hi Tech, vol 22 (2) 2004, pp.175-181.
  • Tennant, Roy. The Importance of Being Granular, Library Journal 127(9) (May 15, 2002) p.: 32-34, http:// libraryjournal.reviewsnews.com/index.asp?layout=article&articleId=CA216337
  • Trock, J. (2000) Permanence of CD-R Media, in Aubert, M. & Billeaud, R. (Eds.) The Challenge of the 3rd Millennium JTS 2000. Paris.
  • Trustworthy Repositories Audit and Certification (TRAC): Criteria and Checklist http://www.crl.edu/sites/default/files/attachments/pages/trac_0.pdf
  • Watkinson, John: 1991 R-DAT Focal Press
  • Webb, Colin 2003 Guidelines for the Preservation of Digital Heritage UNESCO Memory of the World, Internet on-line http://unesdoc.unesco.org/images/0013/001300/130071e.pdf [accessed November 2003].
  • Wheatley, P. (2004) Institutional Repositories in the context of Digital Preservation. DPC Reports. Digital Preservation Coalition.
  • Wright, Richard and Adrian Williams 2001 Archive Preservation and Exploitation Requirements, PRESTO-W2¡BBC- 001218, PRESTO - Preservation Technologies for European Broadcast Archives 2001 Internet on-line, http://presto.joanneum.ac.at/Public/D2.pdf