Achieving a New Vision for Electronic Records
at the National Archives of Canada

Prepared by:

Sheila Powell
Records Disposition Division

Ann Martin
Government Archives Division

13 June 1996

	
Achieving a New Vision for Electronic Records
at the National Archives of Canada

1.	Introduction

The National Archives of Canada  (hereafter NA) is facing a crisis in its ability to preserve the archival electronic records of the Government of Canada and electronic records of national significance created by the private sector.  This report will outline the current electronic records situation and propose a strategy that will enable the NA to resolve the outstanding issues and move forward into the twenty-first century as a world leader in electronic records archives.

This report is the fourth, and last, designed to address the NA’s 1995-96 Strategic Priority B-3:   To develop a program for the acquisition and preservation of electronic records.  The first report, by Sheila Powell, set out non-institution specific elements of an integrated  electronic records program.  Powell examined the management and archival issues that face archives that preserve electronic records, as discussed in archival and information management literature.  The report concluded that an archives must establish a clear mission for its archival electronic records program, be adaptable to changing organizational, technological, social, and economic conditions, assign a person or group of persons the responsibility and accountability for the operation of the program, and frequently evaluate the effectiveness of the program.  It also identified common archival elements of electronic records programs:   macro-appraisal according to function; active archival involvement in standards development and systems design; the need to make electronic records available to researchers; and the importance of continual development of staff knowledge and skills relating to electronic records.  Finally, the report set out three possible strategies for the operation of an archival electronic records program:  a strictly custodial approach, where the archives takes complete responsibility for the preservation of electronic records of archival value; a “distributed custody” approach, where preservation of electronic records is the responsibility of the creators of the records; and  a combination of the custodial and distributed custody approaches, in which  decisions as to the custodial future of electronic records are made in cooperation and partnership with creating agencies according to the degree of appropriateness of the custodial strategy to a particular series of electronic records, based on a range of factors, such as the agency’s electronic  record keeping capabilities and its organizational culture. 

A review of  current NA operations relating to archival electronic records, including existing policies, procedures, and short-term plans, was undertaken by Ann Martin in her report, Report on the Status of the Electronic Records Program at the National Archives of Canada.  This report identified current practices at the NA, as well as the absence of official policies and procedures, and other program anomalies.  These were highlighted as issues which required resolution by the relevant operational areas.  The list of outstanding issues identified in the report is attached as Appendix I.  The combined review of current practices and unresolved issues provide a comprehensive picture of the electronic records program at the NA.

Margaret Hedstrom, in her report entitled Electronic Records at the National Archives of Canada:  Program Review and Strategic Directions, indicated five key strategies that could guide the program during the next five years.  These strategies include:  using a broader definition of  “electronic records program” which includes partnerships with federal government institutions; adopting a user-centred, access-driven view of electronic records services; obtaining and building a technology infrastructure of hardware, software, telecommunications, and skills adequate to the demands of the program; developing expertise and providing national and international leadership on clearly defined archival electronic records issues; and using electronic records issues to build synergistic relationships with other elements of NA programs.

This final report suggests a direction for the future of electronic records at the NA, based on the information and observations made by the three authors in their reports.

1.1	Definitions

It is important to have a clear understanding of the terms electronic records and archival electronic records program. Electronic records are defined as:

recorded information created or received and maintained by an organisation or person in the transaction of business or the conduct of affairs and kept as evidence of such activity, that is processed or stored in a computer system and that requires an electronic system for retrieval.

For the purposes of this report, an archival electronic records program is defined as:  

an integrated system of policies, procedures, and activities, organized and guided  by an archival institution, the goal of which is the preservation of records in electronic format designated as being of archival value.

2.	The Crisis in the Archival Electronic Records Program at the National Archives of Canada

The challenge to an archival electronic records program is maintaining the functionality and contextuality of electronic records  in the face of software and hardware obsolescence.  This is further complicated by the pace of technological change and the knowledge that government and private organizations are already using complex electronic recordkeeping systems, electronic communication, shared relational databases, and a myriad of multi-media software dependent documents.  The situation will only become more complicated since the Treasury Board Secretariat has indicated that electronic commerce will become the preferred means for the government to conduct its business by 1998.  Non-government organizations and individuals are also moving rapidly to electronic recordkeeping:  the very existence of a nationally significant private documentary heritage at the NA is in jeopardy.  Private records of  national significance are now kept, or soon will be kept, in electronic form only.

Meanwhile, the NA has continued to handle electronic records with skills, tools, and attitudes more appropriate to the data files acquired in the 1970s.  The NA has failed to put in place the structures, standards, practices, and skills required to ensure the preservation of the electronic records in its custody, or where they have been established, the NA has not had the resources or commitment to carry them out.  The symptoms of this crisis are telling:

•	The NA has a backlog of at least 1,000 data files awaiting processing.

•	Government Archives Division alone already expects to acquire over 50 electronic systems which are slated for transfer to the NA by the year 2000.    Further government-wide appraisal work, which includes appraisal of office automation systems, as well as private sector acquisition will only add to this total.

•	The NA is technologically unable to acquire some important electronic records already designated as archival.  For example, two 1992 disposition authorities provided for the transfer of individual and corporate tax returns from Revenue Canada to the NA.  To date, the NA has been unable to effect the transfer of the records.

•	The NA cannot respond adequately to reference requests for electronic records and has not developed service standards.   The response to a request for Indian registration records took over 14 months  to complete.

•	The NA has no coordinated means of anticipating technological change as it affects electronic recordkeeping, nor can it anticipate and plan for solutions to  archival issues resulting from technological change.

•	Work relating to electronic records is performed in all but one branch (MSB) of the NA, without any mechanism to plan or perform the work in a coordinated manner.

•	There are no standards for the skills that are required of staff to perform archival tasks on electronic records, and no strategy for the acquisition of these skills and this knowledge by staff.

Due to its inability to respond in a meaningful way to these electronic records issues, the NA is passively accepting the loss of an increasingly important component of the national memory.  As a result, the NA faces losing credibility as an institution dedicated to the preservation of the Canadian archival heritage.  This is unfortunate, since the NA has a well-deserved reputation as one of the oldest and best archival electronic records programs in the world.   NA employees and alumni have written widely-read articles and studies on electronic records, many of which have set the standard for theory and practice.  Staff members have travelled around the world, speaking about our archival electronic records program at the invitation of other governments and national archives.  The NA is an obligatory stop on any archival electronic records “grand tour.”  Despite our reputation, however, the litany of problems set out above is proof that the current situation is unacceptable.

3.	Towards a Solution to the Crisis:  Achieving a New Vision for Electronic Records

The authors recommend a two-phase approach to resolving the crisis in the archival electronic records program.  These phases will allow the NA to reach its ultimate goal, which is referred to in this report as the Vision.  The Vision is the long-term conception of what the ideal NA archival electronic records program should be.  The Vision must be decided upon by Senior Management Committee first, since this ultimate goal will determine what preliminary work must be done in order to achieve it.  Phase One consists of the concrete tasks which must be carried out immediately to prepare the NA to develop a more mature archival electronic records program.  In Phase Two, the NA would move beyond the immediate problems tackled in Phase One to implement a sophisticated and integrated approach to electronic records.   
 
Implementing the proposed Vision incorporates a number of common threads relating to electronic records:  the need to integrate electronic records across the functional structure of the NA;  the importance of understanding and maintaining over time the nature of electronic records and electronic recordkeeping systems; the technological hurdles that must be overcome in order to preserve electronic records over time; the need to explore new ways of approaching traditional archival functions in order to help electronic records over the hurdles; and the importance of a staff with expertise in archival theory and practice as it relates to all records, and specifically to electronic records.  This process of renewal will prepare the NA to allocate its resources effectively in the face of rapid technological change and shrinking budgets. 

3.1	The Vision for the National Archives of Canada’s Archival Electronic Records Program 

The  Vision for the NA’s archival electronic records program consists of its broad objectives and articulates the identity of the NA in relation to other institutions that preserve electronic information and/or data.  It is the ideal that the NA wants to achieve in the long term, after important preparatory work has been accomplished.

When the Vision for its archival electronic records program is in place, the NA will:

•	Re-affirm its identity and reflect its mandate as an archival institution as it preserves and makes available electronic records of archival and historic value created or received and maintained by an organisation or person in the transaction of business or the conduct of affairs and kept as evidence of such activity, rendering the NA distinct from data archives and data libraries.

•	Ensure and facilitate the identification, preservation, and accessibility to Canadians of public and private electronic records of archival and historical value.

•	Support creators of electronic records by suggesting ways in which the content, structure, and context of creation and use of the records can be present from the time of creation and preserved over time.

•	Manage electronic records under its control according to archival standards and practices and integrate their management into the existing management structure for all other archival holdings in all media. 

•	Build upon the 1993 NA policy statement Leaving Archival Electronic Records in Institutions to acquire,  preserve, and make available electronic records in a flexible, post-custodial environment, and take advantage of non-traditional practices borrowed from other models of archival programs when appropriate in order to ensure the best possible preservation of electronic records. 

•	Recruit, train, and maintain a body of employees with a sound knowledge base in archival theory and practice in general and in electronic records specifically, and ensure that the staff’s knowledge and skills are developed on a continuing basis.

In order to reach these ultimate goals, a series of short- (Phase One) and medium- term (Phase Two) tasks must be undertaken.

3.2	Phase One:   First Steps to Achieving the Archival Electronic Records Program 

During Phase One, the NA: 

•	Reviews, documents, and compiles existing NA policies, procedures, and standards relating to all tasks associated with electronic records, including advice, disposition, technical appraisal, acquisition, processing, arrangement, description, storage, and reference.

•	Reviews the mandates of all areas of the NA that are involved in electronic records activities to ensure that work is carried out in the most appropriate areas, and adjusts mandates, responsibilities, and human, financial, and technical resources accordingly.

•	Assesses current NA practices and holdings in order to determine the current level of spending on electronic records as a means of facilitating planning.

•	Determines  requirements for electronic recordkeeping systems and the electronic records maintained therein in order to ensure that the integrity (reliability) of electronic records is present upon creation and can be preserved over time (authenticity).

•	Determines requirements for technological evaluations of electronic records which facilitate the arrangement, description, and accessibility of the records.

•	Establishes standards for the accessibility of electronic records to users, including standards for documentation in existing descriptive systems.

•	Addresses the need for the NA to proceed with RAD-compliant descriptions of electronic records in the face of the current state of Chapter 9, Rules for Archival Description.

•	Identifies the knowledge and skills required of staff performing functions relating to archival electronic records.

3.3	Phase Two: Developing a Framework for a Mature Archival Electronic Records Program 
 
During Phase Two, the NA:

•	Enters into cooperative partnerships with record creators in order to: 

1) provide advice on setting up electronic recordkeeping systems that ensure that the integrity of electronic records is present upon creation and is preservable over time;
2)  identify significant functions and related electronic recordkeeping systems through macro-appraisal, which isolates archival from non-archival records; and
3) evaluate technological specifications of the archival records in order to determine the most appropriate strategy for preservation.

•	Coordinates the planning and execution of archival activities relating to electronic records through a “champion” of electronic records who would act for a fixed length of time in order to provide momentum to the electronic records program.

•	Establishes and carries out a planned and coordinated research agenda for electronic records,  and explores opportunities to carry out this research in partnership with federal government departments, other archives (such as the US National Archives and Records Administration), and university archival science programs.

•	Ensures staff are trained and given development opportunities to ensure they reach required levels of skill and knowledge, and ensures new staff meet required skill and knowledge levels.

4.	Implementation

Should this proposal to adopt a new vision for electronic records at the NA be accepted, implementation should commence as soon as possible.  It is recommended that the implementation of Phases One and Two be carried out under the leadership of one individual, who would be the “champion” of electronic records at the NA.  Once the integrated archival electronic records program is firmly established, there will be no further need for this position.

The individual assigned to this position, whether recruited from within the NA or from another federal agency, or from the private sector, must possess excellent administrative skills, a strong knowledge of electronic systems and information management, and an interest in and a sensitivity to the unique and multi-faceted role of archives in relation to electronic records.  This person must be able to interact confidently and effectively with NA senior managers and other staff, and have the necessary leadership qualities to establish the integrated electronic records program that is proposed in this report. 

5.	Conclusion

The situation outlined in Ann Martin’s Report on the Status of the Electronic Records Program at the National Archives Of Canada cannot be allowed to continue.  For change to occur, NA senior management must recognize that the status quo is untenable and that the NA must renew itself in order to adapt to the challenges posed by electronic records. This report recommends the adoption of a new Vision for electronic records at the NA.  This Vision will guide the implementation phases and result in a successful integrated archival electronic records program that will ensure that the NA continues to be recognized as a world leader in the field of archival electronic records.

	Appendix II

	List of Issuesof Issues

•	Issue 1 - Interdependency of Work Within the NA

Many steps in the advice-appraisal-transfer-processing-storage-conservation-reference chain require the participation of several divisions.  This interdependency requires a commitment to, and an understanding of, shared corporate goals.

•	Issue 2 - Managerial Accountability

There is currently no accountability for the electronic records program as a whole.  Program accountability is a necessary component for evaluating the success, and the failures, of any process.

•	Issue 3 - Co-ordination of Roles and Responsibilities Regarding the Provision of Advice

Consistency in the messages provided by IMSP, RDD, GAD, VSA, and the FRCs is required in order to ensure that regardless of the level at which the advice is given, it has the same end -- the acquisition of the best quality archival record.

•	Issue 4 - Provision of NA-Approved Advice

In assuming a leadership role, the NA must provide advice based upon approved policies and best practices.  These must address the current state of record keeping in the federal government.

•	Issue 5 - Sharing of Appraisal Information and Decisions

With the MYDP process and the concept of macro-appraisal firmly in place, archivists and their supervisors are confident that they can appraise electronic records for their archival value.  However, with electronic records, the possibilities of different approaches to appraisal are much greater than for textual records.   Consequently, creative solutions and precedents must be shared within RDD, GAD, MSS, and VSA to ensure that archivists do not "reinvent the wheel" each time they appraise a new system.

•	Issue 6 - Requirement for Detailed Technical Appraisals

The amount and type of information collected in the technical appraisal has an impact on the amount of time spent processing, selecting, arranging, and describing electronic systems.  The body of knowledge collected and analysed at the technical appraisal stage must be sufficient to expedite the processing of records.

•	Issue 7 - Defining Specific Terms and Methods of Transfer

Though the terms and conditions of transfer allow for the method of transfer to change due to technological advances in both the originating agency and the NA, the terms of transfer should specify the manner in which the transfer to the NA will be accomplished.  This will depend on the NA's decisions about the desired level of functionality of electronic records transferred to its custody and control.

•	Issue 8 - Development of Monitoring Guidelines

The implementation of the policy to leave electronic records in creating institutions is dependent upon guidelines that indicate how monitoring agreements would be implemented and enforced.

•	Issue 9 - Pro-active Acquisition of Textual Private Sector Electronic Records

Acquiring electronic records in the private sector presents a challenge since these records are not subject to consistent disposition procedures.  Regardless of the donor, MSS archivists must be active in looking for electronic records in what were previously wholly textual environments.  In addition, MSS must be prepared to extract the information from these systems to ensure that the NA receives the records of highest archival value.

•	Issue 10 - Deposit System for Electronic Records

The NA must ensure that electronic records on deposit are stored and maintained in a way that will ensure their long-term readability and accessibility.  If the NA cannot read the records, they cannot be accessioned as archival records when the donor is ready to transfer them to the NA.

•	Issue 11 - Delay in Processing Records

Electronic records, particularly in GAD and MSS where there is a great diversity of systems, are taking a long time to process.  Delays in processing only delay the identification of other problems (eg. the records are not those which were identified for transfer, the data is unreadable or the NA does not possess the appropriate hardware and software to run the system)  which may have occurred at time of transfer.

•	Issue 12 - Development of Proficiency in Processing Records

NA archivists and technical staff must develop expertise in accessioning, verifying, selecting, and arranging electronic archival records using a variety of tools.  Technical capabilities (eg. hardware, software, links to appropriate servers) must be available to archivists at all times.

•	Issue 13 - Providing Access to Technical Expertise

Though most of the technical resources associated with processing electronic records are located in a branch other than AGRB, they must be available to GAD and MSS archivists when the electronic holdings are ready to be processed.

•	Issue 14 - Preservation on Different Physical Storage Formats

Currently, the NA is storing both copy 1 and copy 2 on 8mm tape.  This is contrary to the 1992 Working Group on Conservation Standards and Technologies' recommendation to use two different physical storage formats.

•	Issue 15 - Ongoing and Retrospective Tape Destruction

Although the identification of obsolete tapes has begun, it is an extensive task to look at each GOO and its attached tapes.  In addition, approximately 4,000 tapes that are not attached to any GOO description will need to be evaluated and either provided with descriptions or disposed of.

•	Issue 16 - Processing the MRA Legacy

For those electronic records originally accessioned by MRA which, upon reflection, do have archival value, there must be a commitment of time and resources in order to bring these records under control and make them accessible to researchers.

•	Issue 17 - Choosing a Long-term Physical Preservation Format

The NA will be choosing “long-term” physical preservation formats.  Once a choice has been made, the NA will be in a better position to forecast costs associated with the preservation copying program.  Planning for new physical formats will be an on-going concern if the institution is to keep pace with technology.

•	Issue 18 - Systematic Tracking of "Native Formats"

The NA does not currently track the type of software used to generate our electronic holdings.  Consequently, it is ill-equipped to decide what types of hardware and software capabilities are required to support the holdings.

•	Issue 19 - Choosing a Logical Storage Format and Migration Strategy

The NA must continue to investigate logical storage formats and identify a long-term strategy for viewing, operating, and migrating electronic records which have been acquired to date, as well as those that will be acquired in the near future.

•	Issue 20 - Description of Electronic Records

Description of electronic records must be in a RAD-compliant manner in order for them to be included in the multi-level, multiple-media descriptions of our holdings.

•	Issue 21 -Establishing Standards for Access to Electronic Records

The establishment of reference standards for electronic records will enable the NA to provide access to electronic records in a more systematic fashion.

•	Issue 22 - Delivery of Reference Services  

Electronic records offer new possibilities for the delivery of reference services.  Concepts such as public use databases and on-line transfer of data are two examples of resource efficient delivery mechanisms.