New approaches to television archiving

Worldwide, more than 30 million hours of unique television programming are broadcast every year, yet only a tiny fraction of it is preserved for future reference, and only a fraction of that preserved footage is publicly accessible. Most television broadcasts are simply lost forever, though television archivists have been working to preserve selected programs for fifty years. Recent reductions in the cost of storage of digital video could allow preservation of this portion of our culture for a small fraction of the worldwide library budget, and improvements in the distribution of online video could enable much greater collaboration between archival institutions.

Contents
Non–commercial broadcasters, educational institutions, and libraries
For–profit organizations with television archives
Governmental institutions
Fans and amateurs
Collaborative possibilities
Cataloging
Technical standards and low–cost approaches to preservation and access
Legal strategies
Building a social consensus about television archiving

"At present, chance determines what television programs survive. Future scholars will have to reply on incomplete evidence when they assess the achievements and failures of our culture." — James H. Billington, Librarian of Congress [ 1]

Television remains the most powerful medium in our culture, but we have almost no memory for it: broadcasts are ephemeral, and therefore difficult to analyze, fact–check, and evaluate. Broadly accessible television archives could transform television into a medium with a permanently preserved history that is searchable, accessible, and therefore accountable.

The value of archiving television has been recognized for decades. The Library of Congress began accepting television programs for copyright deposit in 1949, and early archiving efforts were started in the 1950s [ 2]. Eight years ago, the Library of Congress (LC) produced "Television and Video Preservation 1997: A Study of the Current State of American Television and Video Preservation" [3]. That report, which is still available online, accurately described both the problems facing television archivists, and the importance of television archiving to students, historians, and scholars.
Unfortunately, the economic, institutional, and legal problems described in the LC’s report still exist, and most of the work outlined in the Report’s plan to solve them has yet to be done. Archives are still under–funded, cataloging is still expensive, technical standards are still in flux, and most holdings have yet to be digitized. Archiving is not a priority for most program owners and local stations, and rights issues make access to archived material problematic.

Changes in the media used for archival storage create new possibilities for preservation of and access to cultural materials. The shift from clay to papyrus, which allowed for the creation of the Library of Alexandria, and from paper to microfilm (which some in the archive community count as a major disaster), are both examples of this. A similar change is now underway in the world of television and video archiving.

In the years since the report by the Library of Congress, the price of raw disk storage has dropped by more than 99 percent, from $180 to under 50 cents per gigabyte [ 4], and more than 25 million broadband connections have been added in the U.S. alone [ 5]. It is now possible to cost effectively store every broadcast, and to offer Internet access to television archives to students and adults all over the world.
The price of raw disk storage has dropped by more than 99 percent, from $180 to under 50 cents per gigabyte ...
The UC Berkeley study "How Much Information 2003" estimates that the world’s 21,264 television stations broadcast a total of 31 million hours per year of unique programming, and that digitizing this would require 39.8 to 68.9 petabytes of storage per year [ 6]. At roughly $4 million per petabyte for mirrored online storage, this puts the cost of saving all broadcasts at $160 to $280 million per year.
Of course, there are additional costs involved in archiving, but to put those dollar figures in perspective, $160 to $280 million per year is less than one percent of the current worldwide library budget of approximately $31 billion, or about two to three percent of the more than $9 billion libraries spend every year on materials stock, e–content, and subscriptions [ 7].

As of late 2004, many parallel initiatives exist to preserve and provide access to some portion of television broadcasts, or discrete collections of tapes and programs. The organizations and people behind these efforts can be roughly grouped into four categories: non–profits, for–profits, governmental organizations, and amateurs (primarily fans).

These four groups interact with and influence each other in complex ways, but they have been limited in their ability to collaborate. The shift from tape to disk as an archival medium and the emergence of broadband networks mean that it is possible to distribute the work of television archiving across many different organizations, and to provide universal access to holdings at a reasonable cost.

For example, non–profits might begin working with fans and amateurs to help collect broadcasts and add metadata; commercial entities might begin to help non–profits by simplifying rights clearance procedures; and, governmental organizations involved in archiving might work with lawmakers to develop new models of access to their collections. This is not meant to underplay the challenge and cost of cataloging, management, and access. Rather, the point is that technological changes since the time of the Library of Congress’ 1997 report can enable large–scale television archiving in new ways that are worthy of increased institutional support.
What follows is a short survey of the types of institutions involved in television archiving; a brief exploration of some of the technical, cataloging and indexing, and legal issues common to most archiving projects; and some possible next steps towards ensuring effective preservation and access of television broadcasts.

Non–commercial broadcasters, educational institutions, and libraries

Existing non–profit institutions, museums, libraries, and universities house some of the largest television archives in existence. The staff in these institutions has a deep knowledge of the practicalities of preserving video and providing access to it.
Several of these organizations, including the Vanderbilt Television News Archive, the Museum of Television and Radio, the Peabody Awards Collection, and the UCLA Film & Television Archive, provide Internet access to their catalog data. For researchers, such organizations often have better access to material than broadcasters or program owners.
Most of these organizations are in the process of digitizing their collections, or are actively considering their digitization options. Once digitized, these collections could become Internet accessible if legal and contractual questions can be resolved with program owners. For archivists, this opens the possibility of funding preservation activities by charging fees for online access.

Unfortunately, many of these non–profit archives only allow onsite access to their collections, and they are not usually able to help researchers with rights clearances. One side effect of this limitation is that these archives are often difficult to use for educational purposes. Restricting access to onsite visitors creates enormous barriers to use; students and even faculty lack budgets for travel to these archives. Developers of educational software or distance learning programs, and teachers who want to show footage in their classrooms, face onerous and expensive clearance procedures.

Non–profits, universities, museums, and libraries face a combination of legal and fiscal constraints that greatly limits their ability to offer comprehensive, widely accessible archival services.

For–profit organizations with television archives

Commercial organizations with archives of television broadcasts include program owners, broadcast networks, local television stations, video monitoring services, and stock footage houses.

For program owners, broadcast networks, and local stations, providing access to their archives is incidental to their business. While some news networks and program producers allow online access and ordering of broadcast segments, most of what is in these archives is not easily accessible by the public.

For stock footage houses and video monitoring services, archiving (at least for some period of time), is their business, and they tend to offer easier access, often via the Internet, to their collections. Many of these organizations are able to provide off–site access to their collections, can clear footage for use in many different contexts, and can respond quickly to customer requests. Most of them now offer some Internet–based access to their catalogs, and a few even allow viewing of footage via the Internet. Unfortunately, most monitoring services retain footage for only a limited period of time, typically six months to one year.
A third category of commercial companies is just beginning to emerge — Internet–based video search engines. Startups such as BlinkX and SingingFish, and established companies such as Yahoo, Google, and Microsoft are all exploring opportunities in providing online access to video [8].

Several other developments in the commercial sector are likely to improve access to archival holdings. The adoption of media asset management systems by television networks and program producers will make it easier for them to track their holdings, retrieve materials on demand, and sell footage. Development of better rights clearance procedures will simplify transactions. The continued growth of broadband subscriptions and the growth of desktop video will expand the number of potential users.

Relative to non–profit archives, commercial services also have weaknesses. They usually focus on "hits" rather than on comprehensive archiving. They also tend to be expensive relative to the budget of typical academic researchers, and many of them hold only material that their organization owns or has broadcast.

As in the non–profit sector, commercial approaches to archiving are limited by economics: providing public access to archival footage has never proven sufficiently profitable to make it a priority for commercial broadcasters.

Governmental institutions

The Library of Congress, the National Archives, and many state– and local–level organizations collect and preserve broadcasts and other video footage, and make it accessible to the public.

The largest of these is the Library of Congress (LC), which has amassed an enormous television archive through a combination of legacy acquisitions, and the collection of depository copies made for purposes of copyright registration. The LC currently has over 1.1 million moving image items in its collection, and over 314,000 television titles.
Though governmental archiving efforts don’t face the same constraints that non–profits and for–profits do, they do face severe economic challenges and must frequently rely on outside funding. The Library of Congress facility in Culpeper, Va., for example, would not exist if not for funding from the Packard Humanities Institute [ 9].
The Library of Congress currently has over 1.1 million moving image items in its collection, and over 314,000 television titles.
Government archives also affect the practices of other institutions, such as publicly funded broadcasters. In the U.S., the American Television and Radio Archive legislation lets the Library of Congress and its contractors collect footage, and new regulations require the Library to post notices of off–air recordings it makes [ 10].

In Europe, certain government institutions and individual legislators are discussing the idea that broadcasters have a public responsibility to offer access to their holdings, particularly those holdings produced with public funds [ 11].

Governmental organizations, as well as their contractors, can play a vital role in building television archives. But as of late 2004, it seems unlikely that any governmental body in the U.S. will create a comprehensive, Internet–accessible archive in the next ten years.

Fans and amateurs

Small–scale, ad hoc projects to make selected broadcasts available on the Internet are becoming increasingly comprehensive and effective. These loosely affiliated fan and amateur efforts often use peer–to–peer (P2P) and other file–sharing technologies.

These efforts often provide access to materials that are not easily available elsewhere. The area of sound recordings has shown [ 12] that this can be a powerful approach to collecting, digitizing, and making accessible large amounts of material.
Already, some of the most accessible collections of popular culture shows are essentially pirated copies of DVDs and off–air recordings. For home broadband users, an entire season’s worth of a particular television show (typically less than five hours of video) can be downloaded in less than an hour.

Copyright owners find the fan and amateur approach to archiving dangerous and unacceptable. If history is any guide, it seems likely that broadcasters will move to restrict P2P video services, much as the music industry has attempted to limit the sharing of music files.

But not all fan and amateur activity need violate copyright. National broadcasting authorities, such as the BBC [ 13], plan to make a portion of their collections available on the Internet under a Creative Commons license. This will increase amateur and fan involvement in television archiving, because these users will find, highlight, and use some of the best portions of that content. Additionally, there is the possibility that fans will create new intellectual property by making television shows and programs partly from sampling previous programs, the same way musicians have in recent years.
The BBC plans to make a portion of their collections available on the Internet under a Creative Commons license.
These fan and amateur approaches may not offer systematic cataloging or preservation, and the recording standards tend to be very inconsistent. Yet these efforts could be leveraged by non–profit and government institutions. If organizations want to support television archiving activities, a promising low–cost way of generating metadata could be to give the fan base easy access to collections. Giving fans simple ways to contribute footage may be one of the best ways to grow collections economically.

It is also worth recalling the history of film preservation: amateur collectors gradually gained the respect of professional archivists as it became clear that many films would have been lost to history without amateur preservation efforts. Then as now, professional archivists were forced to distance themselves from amateur collectors in order to maintain good relations with studios and copyright holders. In the case of film, the collaboration between professional archivists and amateur collectors took decades to establish; perhaps in the case of television, it need not take so long.

Collaborative possibilities

Melding the various current approaches to television archiving will be crucial to expanding the scope of preservation and the ease of access to collections. Four areas for potential cooperation stand out:

cataloging;

technical standards;

legal strategies for access and right clearance; and,

building a social consensus about television archiving.

Although each of the following subjects could be discussed in great detail, what follows is brief overview of each of these issues, with an eye towards aspects of each one that might be addressed more successfully on a collaborative basis.

Cataloging

Cataloging is fundamental to libraries and archives, but it is also tremendously difficult and expensive. Currently, operators of television archives must devote a substantial fraction of their budgets to cataloging activities [ 14].

Catalogs have a primary function as a tool to help users access collections. But they can also give archivists management information on what portions of a collection most need migration and/or digitization. Catalog data is therefore related to archiving budgets, and may be used to support grant applications.

The lack of a comprehensive unified catalog, known as a "union catalog," of archived television broadcasts is one major barrier to assessing the overall state of television archiving [ 15].

Sharing catalog data via an Internet–accessible union catalog could help reduce duplication of effort by television archivists. Rather than having several institutions produce a description of an item, shared catalogs make it possible to use catalog records generated at other institutions. Perhaps more important, shared catalog data can help archivists avoid digitizing materials that have already been digitized by other institutions.

As of late 2004, the most promising project to create a union catalog is the Moving Image Collections (MIC) [ 16], a joint project supported by the Library of Congress, the National Science Foundation, and the Association of Moving Image Archivists. MIC supports a wide variety of cataloging and metadata standards and is in use by a broad community of archivists. Television archivists can support MIC by contributing their records to the project, and by insisting that vendors that perform digitization services offer MIC–compliant solutions.
Over the long term, automated ways of cataloging are likely to become more effective. Human efforts are increasingly being supplemented by technologies such as voice recognition for transcription and indexing, and the use of closed caption data. For example, BlinkX is offering access to more than 40,000 hours of video that has been indexed with automated tools. As noted above, search engine (i.e., quasi–catalog) companies like Yahoo, Google, and Microsoft are also working to extend their online search tools to cover video broadcasts.

Technical standards and low–cost approaches to preservation and access

Online access to text depends on the use of common standards. Video access will too.

Organizations such as the International Federation of Television Archives [ 17] and the Joint Technical Symposium [ 18] have taken the lead in developing standards for archival preservation. An excellent summary of technical standards has been developed by Media Matters, LLC, a New York City–based firm specializing in digitization, with support from the Mellon Foundation [19].

Translating the conclusions of these organizations into low–cost, standards–based, commodity hardware and software solutions still remains to be done. The BBC’s September 2004 white paper on archiving interactive television [ 20] explores one cost–effective method of recording off the air. Another promising approach to large–scale digitization is provided by Media Matters [ 21], which has contracts with the Peabody Archive, the Library of Congress, and other large tape libraries. Other systems will be described in forthcoming publications of the Association of Moving Image Archivists’ Television Interest Group.

Technical approaches to improving access to digitized collections will depend heavily on legal issues being resolved. These range from the status of video streams, to the use of digital rights management technologies, to the implications of the broadcast flag, which could make off–air recording of digital TV problematic [ 22].

Legal strategies

Not all television archivists share the same legal concerns. Applicable laws vary according to the type of institution, the type of content being archived, how material is collected, how access is provided, by country, and by many other considerations that make it difficult to generalize about legal issues.

Moreover, there is a deep divide between commercial organizations concerned with protecting their intellectual property, and non–commercial archives that wish to provide broad public access.

Yet all sides in the copyright debate could benefit from the creation of a system that would simplify access to and reuse of archival footage, and streamline rights clearance procedures. An organization that handles rights for television, something like ASCAP or BMI does for the music industry, could reduce the barriers to access and use, and perhaps even help to fund preservation efforts.
Whether such rights managements and clearance services could be created by existing commercial organizations, or would need to be housed in a new entity, is unclear. Either way, improved economic incentives for program owners could help solve many problems related to preservation and access.

Other legal questions are open to debate, but will need input from commercial, non–commercial, and governmental archives if the answers are to have legitimacy. Among the legal issues most in need of attention:

What are the implications of the broadcast flag for preservationists?

Under what circumstances might video streams be classified as loans of material rather than as performances?

Might DRM technologies be used to provide loans of material over the Internet?

While industry associations representing copyright holders and groups that are working to expand fair use and the public domain are likely to remain in bitter conflict, there are also organizations such as the Association of Moving Image Archivists’ Copyright Interest Group that may prove an neutral forum for exploring the legal issues of television archiving.

Building a social consensus about television archiving

Perhaps the most important missing ingredient needed for the creation of a comprehensive, broadly accessible system of television archives is a social consensus that 1) television broadcasts are an important part of our culture deserving of systematic preservation and widespread access, just like books, periodicals, sound recordings, and film, and 2) that such a system is technically, legally, and economically feasible.

Several institutions are working on creating that consensus, such as the National Television Preservation Foundation, the National Academy of Television Arts and Sciences, the International Federation of Television Archives, and the Association of Moving Image Archivists’ Television Interest Group. Although they do not yet have the visibility or financial backing they need, they do have a model they can follow.

Since the early 1980s, the film community has made tremendous strides in building a social consensus that film is worthy of preservation. That consensus led to large–scale funding, and the creation of institutional structures whose mission is to ensure that films are preserved.

By imitating the successful work in the film community, and innovating in the areas of cataloging, digitization, and legal issues, television archivists can make sure that what is broadcast today remains available for future generations of students, historians, and scholars. After nearly five decades of effort, comprehensive approaches to television archiving are finally within reach.

About the author

Jeff Ubois is a staff research associate at UC Berkeley’s School of Information Management and Systems, and a co–chair of the Association of Moving Image Archivists’ Television Interest Group. He writes about issues in television archiving and digital video at http://www.archival.tv.

Acknowledgments

This paper was written under the auspices of the Information Quality Project of the School of Information Management and Systems at UC Berkeley, which has been generously supported by Brewster Kahle, and the Kahle/Austin Foundation. Brewster Kahle, Peter Lyman, Rick Prelinger, Gary Lynch, Howard Besser, Peter Kaufman, Nina Davis, Laura Quilter, Deirdre Mulligan, Sherwin Siy, Lila Bailey, Gregory Lukow, Abby Smith, Isabel Walcott, and Ben Gross provided ideas and comments.

Notes

1. William Murphy, 1997. "Television/Video Preservation 1997: A Report on the Current State of American Television and Video Preservation," p. xi, at http://www.loc.gov/film/tvstudy.html, accessed 27 December 2004.

2. Personal conversation with David Francis, former Chief of the Motion Picture, Broadcasting and Recorded Sound Division of the Library of Congress.
3. Murphy, 1997.

4. See, for example, http://www.littletechshoppe.com/ns1625/winchest.html and http://www.pricewatch.com for data on commodity hard drive prices.

5. Federal Communications Commission, Wireline Competition Bureau, Industry Analysis and Technology Division, 2004. "Local Telephone Competition and Broadband Deployment," at http://www.fcc.gov/wcb/iatd/comp.html.

6. Peter Lyman and Hal R. Varian, 2003. "How Much Information," at http://www.sims.berkeley.edu/research/projects/how-much-info-2003/execsum.htmon accessed 27 November 2004.
7. Online Computer Library Center (OCLC), 2003. "2003 Environmental Scan, section on worldwide library spending," at http://www.oclc.org/membership/escan/economic/educationlibraryspending.htm, accessed 27 December 2004.
8. Kevin Delaney and Martin Peers, 2004. "TV’s Future May Be Web Search Engines That Hunt for Video," Wall Street Journal (16 December), p. B1.
9. Library of Congress, 1998. "Packard Foundation Grants $10 Million To Establish National Audio–Visual Conservation Center," at http://www.loc.gov/today/pr/1998/98-087.html, accessed 27 December 2004.
10. Those new regulations also expressly allow the collection of sound recordings, material from cable channels, and the Internet. See Copyright Office, Library of Congress, 2004. "Acquisition and Deposit of Unpublished Audio and Audiovisual Transmission Programs," Federal Register, volume 69, number 206 (26 October), pp. 62411–62412, and at http://www.copyright.gov/fedreg/2004/69fr62411.html, accessed 27 December 2004. This states
"the final rule announced herein includes a new provision requiring the Library to maintain on its Web site, at http://www.loc.gov/rr/record, for audio recordings, or http://www.loc.gov/rr/mopic, for audiovisual recordings, a list of the transmission programs that it has recorded under this authority. A radio, cable, satellite, or Internet transmission program that has been recorded by the Library shall be included on the list within fourteen days of the recording by the Library."
11. Joe Figueiredo, 2004. "Dutch parliamentarians favour releasing public broadcast images into public domain," DM Europe (20 October), at http://www.dmeurope.com/default.asp?ArticleID=3911, accessed 27 December 2004.
12. These amateur efforts are not necessarily illegal or lacking in quality control. For example, Etree ( http://www.etree.org/) takes pains to ensure that the archival copies it maintains have been cleared with performers, and has created systems for maintaining high–quality metadata about each performance.
13. British Broadcasting Corporation (BBC), 2004. "BBC Creative Archive pioneers new approach to public access rights in digital age," press release (May), at http://www.bbc.co.uk/pressoffice/pressreleases/stories/2004/05_may/26/creative_archive.shtml, accessed 27 December 2004.
14. For example, the Vanderbilt Television News Archive says that it takes roughly five hours of human time to catalog one hour of news footage, and that 3.5 out of six staff members devote some portion of their time to cataloging activities.
15. Murphy, 1997. The first major finding of the study is that "Educational access remains largely unattainable for a variety of reasons, including underfunding in public archives, lack of descriptive cataloging and reference copies, copyright interests and very restrictive usage policies."
16. See the MIC Web site at http://mic.imtc.gatech.edu/.

17. See the Web site of the International Federation of Television Archives at http://www.fiatifta.org/, accessed 27 December 2004.

18. Joint Technical Symposium, 2004. "What is the Joint Technical Symposium," at http://www.jts2004.org/english/what_is.htm, accessed 27 December 2004.
19. Media Matters, LLC, 2004. "Digital Video Preservation Report of the Dance Heritage Coalition," at http://www.danceheritage.org/preservation/Digital_Video_Preservation_Report.doc, accessed 27 December 2004.
20. Duncan Chisholm, Chris Gibson, Jeff Hunter, Andrew Pearce, and Cathy Smith, 2004. "Archiving Interactive Digital Television," at http://www.bbc.co.uk/rd/pubs/whp/whp096.shtml, accessed 27 December 2004.
21. See the Media Matters Web site at http://www.media-matters.net.

22. See especially the comments by the Association of Research Libraries (at http://www.arl.org/info/frn/copy/DRMBFR.html, accessed 27 December 2004) and the American Library Association (at http://www.ala.org/ala/washoff/WOissues/copyrightb/broadcastflag/broadcastflag.htm, accessed 27 December 2004) for discussions about the possible negative effects of the broadcast flag on libraries, distance learning programs, and video archives.

References

British Broadcasting Corporation (BBC), 2004. "BBC Creative Archive pioneers new approach to public access rights in digital age," press release (May), at http://www.bbc.co.uk/pressoffice/pressreleases/stories/2004/05_may/26/creative_archive.shtml, accessed 27 December 2004.
Duncan Chisholm, Chris Gibson, Jeff Hunter, Andrew Pearce, and Cathy Smith, 2004. "Archiving Interactive Digital Television," at http://www.bbc.co.uk/rd/pubs/whp/whp096.shtml, accessed 27 December 2004.
Copyright Office, Library of Congress, 2004. "Acquisition and Deposit of Unpublished Audio and Audiovisual Transmission Programs," Federal Register, volume 69, number 206 (26 October), pp. 62411–62412, and at http://www.copyright.gov/fedreg/2004/69fr62411.html, accessed 27 December 2004.
Federal Communications Commission, Wireline Competition Bureau, Industry Analysis and Technology Division, 2004. "Local Telephone Competition and Broadband Deployment," at http://www.fcc.gov/wcb/iatd/comp.html.

Joe Figueiredo, 2004. "Dutch parliamentarians favour releasing public broadcast images into public domain," DM Europe (20 October), at http://www.dmeurope.com/default.asp?ArticleID=3911, accessed 27 December 2004.
Joint Technical Symposium, 2004. "What is the Joint Technical Symposium," at http://www.jts2004.org/english/what_is.htm, accessed 27 December 2004.
Library of Congress, 1998. "Packard Foundation Grants $10 Million To Establish National Audio–Visual Conservation Center," at http://www.loc.gov/today/pr/1998/98-087.html, accessed 27 December 2004.
Peter Lyman and Hal R. Varian, 2003. "How Much Information," at http://www.sims.berkeley.edu/research/projects/how-much-info-2003/execsum.htmon accessed 27 November 2004.
Media Matters, LLC, 2004. "Digital Video Preservation Report of the Dance Heritage Coalition," at http://www.danceheritage.org/preservation/Digital_Video_Preservation_Report.doc, accessed 27 December 2004.
William Murphy, 1997. "Television/Video Preservation 1997: A Report on the Current State of American Television and Video Preservation," p. xi, at http://www.loc.gov/film/tvstudy.html, accessed 27 December 2004.

Online Computer Library Center (OCLC), 2003. "2003 Environmental Scan, section on worldwide library spending," at http://www.oclc.org/membership/escan/economic/educationlibraryspending.htm, accessed 27 December 2004.

Editorial history
Paper received 26 January 2005; accepted 7 February 2005.

This work is licensed under a Creative Commons License.
New approaches to television archiving by Jeff Ubois
First Monday, volume 10, number 3 (March 2005),
URL: http://firstmonday.org/issues/issue10_3/ubois/index.html