Assessing the Need:
What Information and Activities Should We Preserve?

Michael L. Miller

Abstract
Archivists, records managers, librarians, and technology advocates have developed visions of what electronic records could and perhaps even should be preserved for various purposes.  This paper looks at the issue from a very pragmatic point of view of someone who has worked in both a Federal agency that produces and disseminates huge quantities of information and an archival program that has attempted to deal with the flood of Federal records for roughly a quarter of a century. The author examines some questions underlying the title of this paper with the goal of highlighting areas of potential agreement and disagreement on how the issue of preservation should be addressed.   The author argues that we need to look to limit the amount of documentation we seek to retain over time and match our preservation plans to our management capabilities at this time.

Introduction
The request to write this paper brings to mind two stories.  The first is about myself.  When I was in the military back in the late 1960s, I bought a (for the time) very elaborate 35mm camera, with multiple lenses, filters, external flash and lots of other gadgets.  As was typical of cameras of that time, this “fancy” camera did not have automatic light metering and focusing.  After a few years of taking “artistic” photos, I found myself using the camera primarily for snapshots of friends, a task for which it was unsuited (in my hands anyway).  I eventually bought a much more modest 35mm camera that was entirely automatic and fit my needs very well.

The second story is much older, one of Aesop’s Fables, the story of “A Boy and the Filberts.”  

	“A boy put his hand into a jar of Filberts and grasped as many as his fist could possibly hold.  	But when he tried to pull it out again, he found he couldn’t do so, for the neck of the jar was too 	small to allow of the passage of so large a handful.  Unwilling to lose his nuts, but unable to 	withdraw his hand, he burst into tears.  A bystander, who saw where the trouble lay, said to him, 	“Come my boy, don’t be so greedy:  be content with half the amount, and you’ll be able to get 	your hand out without difficulty.”

		Do not attempt too much at once.”

What ties these two together in my mind is the need to think through very clearly exactly what we need to preserve and why, when it comes to electronic records.  The question in the title of this paper is not new - it is at the core of the archival discipline.  In its most basic form, the question of what we should preserve is one of determining value - value to whom, for what purpose(s) and for how long.  Some would say it is the most critical question facing archivists, and most would agree that at least it is the most fun to debate.  The volumes of literature on the subject bear witness to that fact.  This paper is not meant to resolve the ongoing debate, but rather to frame the debate and identify some of the critical issues from the perspective of someone with both records management and archival backgrounds.

The question “What Information and Activities Should We Preserve?” seems simple enough, but the underlying issues are far more complex.  I see three basic sets of questions:

	What is the “what” that we are considering preserving?
	What do we mean by “preserve” in this context? 
	Who makes up the “we” that will be doing the preservation?

The format of this paper will be to look at each of these themes in detail, raising a series of questions about the theme and then offering a position from the point of view of this archivist/records manager.  Hopefully this will serve as the springboard for broader discussions.

However, before even looking at what we might want to preserve, it is worth examining an underlying  presupposition - namely that some electronic information is not worth preserving.  Some have indicated that the storage capacity of modern computers and the distributed nature of the Web would allow us to keep practically everything, making the question this paper addresses irrelevant.  Traditionally archivists, records managers, and librarians have assumed that some documents would not be preserved.  The question has always been how are the documents selected, and by whom, and for what purposes.  Do electronic records resolve the need to be selective?  

In my view, selection will be necessary in the future. The only question is how that selection will be accomplished.  Selection of documentation for preservation can be based on some definition of value -  for organizational needs,  specified purposes, potential or actual use, or other considerations.  Preservation can also be based around management and cost issues so that older information is preserved only as long as it does not require extensive, or in some cases, any additional resources.  In many cases, preservation is in fact random, especially the preservation of paper records, which can be “preserved” because people neglect them by throwing them into an attic or basement where they are discovered decades, even centuries later.  Fortunately or unfortunately, the last scenario “preservation through neglect” is not an option for electronic records, making some agreement on what to preserve even more important than it was in the past. 

Examining the “What” 
Margaret Hedstrom subtitled one of her many articles on electronic records issues “Deciding What is Essential and Imagining What is Possible.”  The challenge of imagining what is possible and deciding what is essential is crucial to all areas of electronic information, but especially when it comes to laying out the potential types of activities for documentation.  It is here that we need to be sure to get beyond Terry Cook’s warning about the problem of using paper minds to evaluate electronic records.

The “what” question can be subdivided into two related questions.  The first question is the “imagining what is possible part” and the second is the “deciding what is essential” part.

	What are the basic types of documentation that might be preserved and why?
	What qualities and functionality of that documentation warrant preservation and for what 
	purposes? 

The question that is not included is what specific activities deserve to be documented.  That is a deliberate omission.  Whether, in the grand scheme of things, AIDS research is more worthy of documentation than drag racing is not a question to be addressed in a general paper. All activities have their champions who can explain better than I why the record of their activity deserves to be preserved.  My goal here is to lay out a framework for deciding what types of documentation to keep to meet your needs assuming that you have decided to document the activity.  

The following is a brief list of the types of documentation that should be considered for preservation, at least in the “imagining” stage.  In some cases the categories overlap, but the goal here is ensuring breadth of coverage not taxonomic exactitude.  The terminology may smack of archives-centricness, but hopefully the descriptions will allow all information professionals to recognize the types of documentation under discussion.  These categories are not presented because I think that they all need to be preserved, but because each has been suggested for preservation at one time or another for certain reasons.

Final Product Created for Use by External Audiences.  This category would include any type of electronic product meant for use as an “information resource” by an audience external to the creating organization.  Such final products may be only in electronic form or may be available in a variety of media.  Examples include newspapers, books, reports, catalogs, data products such as Census files, final processed observational data, digital photographs, data products released as part of scientific collaboration, and Web pages.  The basic characteristics of all of these items is that they are final, finished products meant primarily for use by others and not as part of documentation of the creating organization’s regular day-to-day activities.  The primary goal of these products is to disseminate information. 

Final Documentation Needed as Evidence of a Transaction, Activity, or Program.  This category would include final, finished documentation of the organization’s activities and programs.  They are distinguished from the first category because their primary purpose is to conduct a “business transaction” not to simply provide information to an external audience (although they may do that as well).  The documentation may consist of a single document, a series of documents (file), or an entire recordkeeping system.  Examples would include letters to constituents, a will or deed, an international treaty, bills for services, an electronic commerce interchange, a court decision, the records of a court case, or the administrative record documenting the development of a government regulation.  The key characteristics of such documents are that they are used primarily to accomplish a business purpose, are relied upon by the organization as the documentation of their activity.  Generally speaking, the organization or parties involved determine what constitutes the “official record” of the transaction based on law, regulation, or best practices.  Once created, the record needs to be retained primarily as evidence that the transaction took place and was properly carried out.

Although these two products differ by purpose and intended audience they share a number of characteristics.  First, they are the “official positions” or statements of the creating organization.  This makes them useful for some types of inquiries (e.g., what was the Supreme Court decision in a specific case), but less useful for others (e.g., how did the court really arrive at the decision).  Second, as the official positions of the organizations, they may well deserve to be preserved, even after they have been superseded.  Third,  both types of products sit atop a pyramid of other documentation that supports them.

Administrative Data.  Consists of data created and collected by an organization to carry out its functions and activities.  Such data will serve as a major information resource for the organization and will also be used for analytical purposes and to develop products discussed above.  However, despite the variety of uses to which they are put, all of these databases have an ongoing organizational functionality that requires that they be maintained over time to meet the business needs of the organization.  Examples include budget, personnel, payroll, and other administrative systems; systems to track accounts, systems to track and manage programmatic functions such as the issuance of permits, systems to track the issuance of passports, and other types of systems that document how a specific organizational activity is managed.  In their active life these systems support the work of the creating organizations, but they may be of interest to those researching specific areas documented by the system.

Software.  This category includes off-the-shelf commercial software, commercial software as modified by the organization to meet organizational needs, custom designed software, expert systems, computer models, and other software used by the organization.  A record of software has two purposes.  First, it allows information that is in a software dependent format to be used.  Second, a knowledge of software used by an organization and the capabilities and limitations of the software is critical to understanding the abilities and inabilities of users to accomplish certain  tasks.  Some examples of how information about software and how it works may help.  First, it has been the policy of the Environmental Protect Agency (EPA) and other Federal agencies to print to paper any e-mail message qualifying as a Federal record.  To properly evaluate the resulting paper copies, it is necessary to know that the system printed the date the message was originally created, not the date it was sent, even when the two dates were different.  At a higher level, a copy of the expert system used by EPA permit writers to develop permit documentation will reveal a lot about the process used to issue permits and why some applications would be denied.  Finally a record of the changes to the massive modeling system used to produce the national economic models or to the system used to interpolate data in Census studies will be crucial to understanding the validity of the data.  

Explanatory Material Needed to Understand the Final Products.  Electronic documents, information, or objects frequently are not self-explanatory.   To use or understand them generally requires additional information that varies by the type of electronic record under discussion.  For databases, one needs information about data elements and codes for example, while metadata about how and why a record was created is fundamental to understanding its role in the activity it documents.  To be truly useful, scientific data needs contextual information about its accuracy, the instrumentation used to collect it, and data limitations.  Bottom line is that to fully understand any document (electronic or otherwise) there must be sufficient explanatory material retained with it. 

Source Materials.  This category includes those materials that were mined to produce a final product.  It may include raw data that is later corrected, edited, and enhanced or finalized; databases from which subsets are extracted; geographic information system (GIS) data layers used to create specific products; and a data warehouse used to conduct and analysis, historical data used for projections, and data used in computer modeling.

Product Development Materials.  This category includes the documentation created during the development of a final product, but not required or judged to be part of the final product or official documentation.  Examples include notes, working files, drafts, records of receipt of electronic mail messages, and other documents created during the business process.  In the case of scientific data, it might include the results of experiments, measurements, or testing, not required to support publication of the results.  In socio-economic analysis it might include data analyses, the results of computer modeling, and the data used to test hypotheses.  In land use studies, it might be the results of GIS runs used to identify potential sites for a waste facility.  Although these materials were judged not to be necessary for the final documentation of the activity in question, they were consciously created as part of the development process, and may provide valuable information about how a specific activity was carried out.

Data Management Information.  This category covers the documentation created by the computer system or application independent of the will of the individual user.  In document creation it would include the basic information created on document size, initial creation date, the number of editing sessions, a record of changes made to the document during the session, and other information captured automatically.  In developing a GIS product it is the record of how the actual processing took place.  In the case of EDI transmissions it is the logs of the various translations, transfers, receipts, and other stages in the transmission and processing of the EDI transaction.  Relating to the Web, it includes such diverse records as the record of the session and the sites visited, the records of what information was available on the Web at a particular time, and information captured about those visiting the Web site.  In database applications it includes audit trails, records of who accessed the system, and changes to the data.  Systems management information would include information about systems users, passwords, log-ons, and other information used to manage the system itself.

Tying all of these different types of information together is the fact that they were created to aid in the management of computer systems and the reconstruction of data in case of system failure.  However, all of them can be used to document various aspects of an individual user’s activities, and may, in some cases, actually shed light on how a transaction occurred or an activity was completed.  As an example, proving the integrity of a recordkeeping system might involve keeping a record of access to the system, the passwords and PINs of persons having access, and any changes made to the data.

Informal Electronic Communications.  Much e-mail traffic consists of sharing of information and views, exploring areas of common interest, and personal communications of all sorts.  For many, e-mail has replaced both the telephone and the letter as the preferred way to communicate with peers, family, and friends.  One need only think of the variety of listservs, chat rooms, personal group lists, and informal collegue-to-collegue communications to understand that there is a wealth of communication going on that may be business related that is not linked to a specific “business transaction” or “product.”  Even so, these communications may provide tremendous insight into the workings and cultures of many segments of our society.  The informal and “private” nature of the medium means that many people are far freer in expressing themselves over e-mail than they are in other media.  Although there are a series of issues relating to the privacy of persons involved in such exchanges, those who want to capture the flavor of the late 20th century will find them a wealth of information about how we live and work.

What Does It Mean to Preserve?
Preservation of electronic information has had a checkered history to this point.  Those trying to preserve information electronically have had to deal with evolving hardware and software, fragile media, and the need to maintain not only the information itself, but the documentation and software necessary to support it.  Whether the preservation is done by data libraries, data centers, or archives, it has presented problems. It is this history of limited success in maintaining electronic records that serves as the context for my remarks.  While the future is brighter in this area, the present offers significant challenges that make me wary of attempting too much.

It is my firm belief that the best documentation is the result of a clearly bounded understanding of the entity (e.g., organization, event, city, phenomenon, person, etc.) to be documented and a firm vision of what it is that is to be documented, and the level of detail in documentation that is sufficient.  In government archives, the current formulation of this theory is known as functional appraisal, where the archivist looks at the major functional activities of an agency (e.g., issuing permits for pesticide uses) and determines what documentation should be preserved to document that activity.  The short hand version of the theory is that the archivist has a better chance of preserving a useful record for the future if s/he simply selects the documentation of what the organization does that is important in its view, rather than trying to determine what records might be useful in the future based on the types of information they contain.  There are exceptions, but this is the general rule.

For example, the Environmental Protection Agency (EPA) creates extensive files concerning each pesticide, its formulation, product testing, potential risks, and other aspects that are taken into consideration in reviewing the application.  They were appraised based on the documentation they provide of how the Agency carried out an extremely sensitive function, as the strawberry growers in California know full well.  However, historians of agriculture, science, labor, and issues that we haven’t even thought of yet, may find extensive information about their subjects in these files.  

This is not always an easy task. The National Archives and Records Administration (NARA) has developed its bounded area of concern - records of Federal agencies that constitute essential evidence.  What that means in concrete terms is being investigated now.  This does however raise another bias of mine, which is I don’t believe in saving everything. It can be argued that every type of documentation might be useful in resolving some issue.  In my mind, that does not justify keeping everything and letting technology allow us to store it and search it.  Storage is not the major issue - but information management, ongoing access in the midst of changing technologies, and ability to comprehend what has been stored are very real issues. Therefore I would argue that we need a process to select what we will retain.

The first step in understanding the preservation issue is to look at why we might seek to preserve information.  I offer a few of the many potential uses of the information we preserve, arranged into what I perceive as levels of system functionality to serve those needs.  Level I is the simplest and Level III the most difficult.

Level I:   Provide information 
	Provide answers to particular questions.
	Provide raw material for research
	Provide a historical “record” of what took place (corporate memory).
Level II: Serve as the official, legal record of activities
	Support the ongoing operation of the business
	Serve as the legal record of the transaction or a portion of it.
	Permit oversight and support accountability
Level III:  Allow for the recreation of the transaction to check validity of results

I see each of these three levels as being progressively more difficult and expensive to maintain.  I would also argue that, as time goes on, the primary uses of the information gradually move to Level I uses, and the number of users that have an interest in Level II and III uses will decline.  Therefore we should be thinking in terms of a strategy that gradually reduces the functionality of the systems as well as the amount of information we preserve to keep them in line with research needs.

This viewpoint is not universally accepted.  A major reason is that it limits accountability and oversight.  To the extent that we think in terms of official accountability for activities, a “statute of limitations” can be applied that limits the length of time that records need to be maintained for accountability purposes.  This “predictability” about the uses of the records is essential to their management.  For example, the EPA limits the length of time it can require the regulated community to produce records required under regulations.  This allows the regulated companies the ability to manage their records.  Yet for those investigating wrongdoing - corporate knowledge of health risks from smoking for example - always want as much material maintained as possible forever, or until all of the issues have been resolved.  This perspective finds all information potentially useful and would require the retention of all information for extended periods of time to ensure that it will be available “just in case.”   This desire runs counter to the need to rationally manage one’s records and information resources.

If the final product sits upon a pyramid of supporting documentation, and the final product is an obvious candidate for preservation, the question becomes what supporting documentation (if any) should be maintained, for what period of time, in what format, and for what purpose.  Each of these four aspects deserve detailed examination.  Let me say at the outset that I would argue that there are instances when each of these documentation sources is worth preserving at least from a theoretical point of view.  I would also argue that, in the vast majority of cases, the final products of the activities will serve the majority of research uses very well, especially as time goes on.  The basic issue is to develop guidelines as to what the minimum level of necessary documentation might be.  Those whose job it is to preserve information and/or records should not be deluded into thinking that what constitutes appropriate and indeed necessary documentation in some special cases should become the standard for documenting all activities.  

The first question then is the level of richness in documentation that we wish to maintain.  The most basic question concerning electronic records and the level of richness to be preserved, is whether we see electronic records as a way of preserving more information about more activities or whether we see it as a way to preserve the same level of documentation in a format that is easier to access and use.  The more we wish to preserve electronically the greater the management problems.

Technically, the maintenance of final products is relatively easy, compared with the maintenance of supporting materials. The document creator specifically wanted to see the final products preserved, at least for a time, so these items are better documented and managed than the materials used to create them.  Supporting documents may provide crucial information about the evolution of a document for example, or they may be a jumble of information that is difficult if not impossible to understand as context.  The maintenance of supporting documentation works best when the period of preservation is relatively short and the scope of the attempt is relatively narrow.  At this point in the evolution of electronic records our ability to identify and link all the supporting materials for a specific activity is limited in practice by two practical problems. Programs wishing to preserve supporting documents on a regular basis will have to deal with software that is geared toward allowing maximum capability to change and update documents and information, and records creators who are not used to thinking of the preliminary documents as records that need to be managed.

The second  question concerns the length of time the documentation should be preserved.  Archivists used to speak of “permanent” records, but now speak of records of continuing value.  In effect preservation begins at the moment of creation and it is simply a question of how long specific information or documents need to be retained - how long do they have continuing value?  Preservation becomes an ongoing process where the level of documentation will gradually diminish over time based on need.  Viewed this way, it is very necessary to clearly lay out for the specific activities and classes of information what exactly is to be preserved and for how long.  It is quite possible to agree that for some activities, relatively full documentation should be preserved for a specific period of time - several years perhaps, with the idea that unless there is some specific issue, a major investigation or problem with an audit, or a newspaper story, the documentation will be reduced in complexity after a specific period of time.

The third question concerns the level of functionality that should be retained over time.  In the case of word processing documents, must we preserve the capability to modify those documents, a capability (functionality) that the creator would have had?  If we need to retain records of system audits until the transactions of a system have been audited, do we need to maintain them after that, or is a simple record of the findings of the audit sufficient?  For example, it is possible to argue that for a period of time, researchers should be able to easily replicate the analysis done on a GIS in support of the siting of a waste-water treatment plant for example.  However, preserving the complete functionality of the system rather than simply the results of the analysis can be difficult, especially if it must be maintained across system migrations.  This is why the idea of gradually diminishing levels of documentation has always made sense.  Yes, there would be full functionality to recreate the analysis for a while, because the actual decision may actually be in doubt for a while or subject to litigation.  But given the costs of maintaining that functionality, it would be allowed to diminish over time as the need to document what actually took place and why, superseded the need to be able to second guess decision makers.

Another commonly debated question of functionality is how we go about preserving documents.  At one end of the scale we have ASCII text, which is easy to preserve, but does not capture the look and structure of the documents.  A second strategy might be to use SGML as a standard and convert document to that (or HTML) for preservation.  A third option would be to preserve the documents in a read only format using a viewer, or imaging the documents to preserve their format.  A fourth is to ask whether electronic media are the best way to preserve records or outdated information over time.  For documents that are not active and are difficult to maintain (older expert systems for example) does it ever make sense to preserve what is needed in paper form?  Perhaps only the manuals explaining how the system was used would be sufficient. Even if there was a need to maintain electronic access, the manuals would be easier to maintain electronically that the systems themselves. Are all of these approaches viable options, depending on the needs of the users?  Is it better to leave everyone to develop their own strategy or to try to set guidelines on when each is most applicable?

Finally, we return to the question of purpose - why are we retaining these records.  To the extent that we are attempting to preserve information that people can use to answer questions, there are a number of alternatives that would simplify preservation.  A creating organization may have to retain extensive “overhead” to prove that the electronic records on their system are indeed the authentic records of the organization and are trustworthy in their content.  But once the records have served their business function, is it still necessary to preserve the full “recordkeeping functionality” of the system?  As an example, members of the regulated community producing reformulated gasoline can submit their mandatory reports via electronic data interchange (EDI).  These submissions can be used for civil and criminal prosecution so the ability to manage them as official records and ensure their complete accuracy is very important.  However, the EPA can only initiate criminal and civil penalties for a specific period of time - generally seven years.  After that, the submissions still have value to those studying the reformulated gas industry and developing new regulations, but they cannot be used for prosecution.  After the seven years, it makes sense to remove them from the recordkeeping system, and maintain them in a data library with proper security to ensure that the data were not changed.  

To sum up, preservation should be seen as an ongoing process, not only in terms of the need to perform it on an ongoing basis, but also because it requires constant reevaluation of what level of richness of information and functionality we need to preserve.  It becomes a question of resources.  The simpler the materials we choose to preserve and the more standard the formats, the easier our jobs will be.  Given limited resources, we need to develop a clear set of priorities as to what actually should be preserved.  These priorities should be based on predictable use of the records and a clear understanding of their value to particular constituencies for particular purposes.  

Who is the “We” That Will Do the Preservation?
This question is the subject of a later paper, but to explain my approach to the preservation question, I need to at least pose my position on this matter.  The “we” should be considered extremely broadly at least in the first instance.  It should include records creators, data and traditional libraries, and archives of all descriptions.  This broad-brush approach has two major advantages and one major disadvantage.  The first advantage is a philosophical one.  A diverse group of preservationists with differing interests and values for documentation will result in diverse things being saved, at least initially.  A preservation program with a well-defined focus and clear boundaries will have a better chance of preserving the richness of events than an institution with an extremely broad preservation mandate.

The second is a practical one.  If the program preserving documentation in electronic form can focus on a specific defined area, it has a better chance to resolve the technical issues inherent in any electronic records program.  An organization that has a single e-mail system can devise a system-specific means of preserving the messages from that system more easily that NARA can devise a system for preserving the products of multiple systems.

The rather obvious disadvantage is that there is no guarantee that many of the “preservationist” programs will last themselves.  As a result documentation may well be lost.  This is certainly true, but the final result may not be any different than if preservation were left to a small select group of programs.  If anything, the approach buys time, which in preservation is a major advantage. Theoretically, additional time may help sort out what documentation is actually necessary.  My own view is that preservation, even archival preservation, is for the length of time the documentation continues to have value.  Therefore I assume that the value of certain types of documentation currently preserved will diminish over time, resulting in certain records being discarded in any case.  From a technical perspective, the additional time allows for new approaches for preserving electronic records to evolve.

However, we must be aware that records creators are usually not in the preservation business.  They have their own  work to accomplish and the records and information they generate are a by-product of that activity.  They will preserve the records and information as long as it serves their purposes and the preservation does not entail significant costs.  However,  creating groups may balk at the expense of migrating older records no longer used to a new hardware or software platform.  Records creators have a difficult time understanding why it is necessary to preserve a record of the limited functionality of an old information system when they migrate to a new one.  In sum, records creators may help for the early stages of the preservation process; but long-term preservation has been and will remain the work of specialists in preserevation.

Conclusion
This paper has attempted to lay out some of the major choices we face in determining what information and activities should be preserved to document the digital age.  I am drawn back to the boy in Aesop’s fable who had to give up some of the  Filberts that were in his grasp in order to get any at all.  I find that many of us are enamored of what the technology can do - if applied as the vendors describe.  However, many parts of our culture are far behind the cutting edge of technology.  Partly it is because records creators do not choose to implement many of the options available to them, because they don’t see the need.  Even when they do, the effectiveness of the technology solutions still stands or falls on the capability and dedication of the people implementing it.  As only one example, it is clear that records creators will try to “outwit” records management software by entering insufficient indexing and metadata if they do not see the need for it or do not clearly understand how the system works. 

The parameters of the debate are bounded on one side by the ever-expanding potential of the technology and on the other by the much more slowly expanding capabilities of the people employing the technologies.  The question comes down to not what should we preserve, but what can we feasibly manage at this point.  Manage means two things.  First is to physically preserve the information and records in a technical sense, ensuring that it is accessible despite the passage of time and changing technology.  This means an ongoing program of preservation.  But management also means that the information and documentation has the context to make it understandable as well as accessible.  Today much of the context for electronic records is provided by the individuals who created them and worked with them.  We will need to get beyond that limitation if we are going to be successful in preserving electronic records over the long haul.  

-----------------------
Endnotes