SAN JOSE – Compliance, security and legal risks are leaving organizations of all types at risk of losing their digital information, according to a warning from 276 long-term archive practitioners who participated in the Storage Networking Industry Association’s 100 Year Archive Requirements Survey Report.

“Large organizations faced with retaining and preserving huge amounts of digital information for very long periods of time are at the front edge of a troubling crisis,” said Vincent Franceschini, chairman of the board of SNIA. “It is one thing to manage a domain of digital records that an archivist can personally guard and shepherd…it is quite another to meet the archival challenges of today’s enterprise data centers.”

He added that these data centers can be characterized as environments with petabytes of distributed information, high data growth rates, many facilities and many departments with uncoordinated responsibilities and requirements, as well as a lack of business-level budget, interest and focus on its archives.

“All these operating challenges are now compounded by high risk. Yes, risk of failure and fines from legal discovery, compliance requirements or security threats. Add to this the risk of losing information that may be of great value to the organization, and the picture looks pretty daunting,” Franceschini said.

The drivers behind protecting an organization’s digital assets include the protection and preservation of an organization’s history; meeting regulatory requirements; concern with litigation protection; protection of business or intellectual property assets; and protection of customer privacy. Eighty-one per cent of respondents said that database information was considered to be most at risk for loss.

The recently released report captures the operating practices, requirements and issues facing organizations managing large amounts of information. This information can be managed for extended periods of time, from 10 years to “forever.”

The survey found that long-term digital information retention needs were real, with 80 per cent of respondents indicating they have information that they must keep for over 50 years. Sixty-eight per cent said they must keep their digital data for more than 100 years. Over 40 per cent of respondents noted they were keeping e-mail records for over 10 years. SNIA defined long term as greater than 15 years, a period beyond which multiple physical media and logical format migrations must take place. Only 30 per cent declared they were migrating information at regular intervals.

“The digital crisis is exacerbated by time. In 10 years, 50 years, 200 years, which applications will still be around? What computer and storage system will be able to read and utilize the old information, providing that it is not corrupted by then?” asked Franceschini.

“Even finding a single piece of content and all the linked objects that contain associated content amid trillions of distributed information objects is, at best, a costly adventure. Many standards and best practices exist today documenting the practices of preserving digital information. Yet, none of them address the core problems caused by inadequacies and inefficiencies in the supporting storage infrastructure, and surrounding management systems.”

Seventy per cent of respondents said they were ‘highly dissatisfied’ with the ability to read their retained information in 50 years.

The survey noted the top four ways that organizations can lose their digital information: they cannot read it, cannot interpret it correctly, cannot validate its authenticity and cannot find it.

As well, those surveyed felt that current practices were too manual, prone to error, costly and lacked adequate coordination across the organization. In addition, information classification and collaboration between those who own information and administrating groups, were both recognized as very important practices that can be implemented now.

To solve these technical challenges around preserving digital information, SNIA is working on producing a reference model for long-term digital information retention; similar in format to the Open Archival Information System (OAIS) document that covers the storage domain portion of long-term retention. Building off OAIS, it will define storage architectures and services to provide a robust, scalable and long-term digital information repository.

“The plan is to completely change the concept of physical migration through technology,” said Franceschini.

He added that the 100 Year Archive Task Force plans to leverage the OAIS architectural model’s concept of an “archival information package”, in creating a storage container called Self-Describing, Self-Contained Data Format, (SD-SCDF). It is planned that implementation of SD-SCDF will be enabled by integration with the eXtensible Access Method (XAM) application-to-storage interface standard currently in development by SNIA.

Additionally, market education will also be important in retaining digital data. The 100 Year Archive Task Force has a Web site and a speaker’s bureau, and will also be presenting at events worldwide.

“Based on the findings of this research, information professionals can embrace a collaborative relationship between all departments in the organization, with the goal of setting requirements for information,” Franceschini said. “Putting collaborative practices in place to identify and classify your information is essential to setting requirements so that IT knows the business requirements.”

a>>