Managing Data Growth & Data Archival

Recently I have faced one scenario around data archival concerns which made me think and look little deep into data archival solutions & need for the same.

Problem Statement
Business now tracks almost every piece of data that are created as part of any business interactions. Now large terabyte online transaction processing database systems are becoming common. The rate at which data is growing has increased rapidly. Now it takes months rather than years for the data in a typical transactional database to grow from terabytes to petabytes to exabytes. It has resulted in greater challenges in terms of data management.  It has a negative impact on the performance of the application as data operations takes more time to complete. (We could alleviate the problem to a certain extent using caching solutions,  splitting data based on geographies/data range/other conditions or other means.)
Huge data growth also increased the operational cost in terms of provisioning of additional costly storage space; the associated cost involved in physical space/cooling cost/other things. It also brings in additional burden to database management  operations like data movements, database upgrades plus other regular administrative tasks. At the same time, keeping old sensitive information like customer credit card information/healthcare reports in a large production database systems beyond certain period will also pose serious security risk if it is not managed properly.

Solution Considerations
One of the straightforward approaches to the above problem is to remove the unwanted data on a periodic basis. But it will not address the concern completely. As per one study, majority of the data operation involves only data that is created within two-three years. The access to the data that is three-six years older is very minimal, and one more than six years is rare. So  scalable approach is the adoption of tiered storage approach where active data is placed in the costly SAN storage  tier  but inactive one in low-cost storage tier. Let's look at the solution consideration  little deeper for building full fledged data archival solution.

• One of the primary consideration for data archival solution is to understand the data entities & its relationship in the source system.  Without well-defined meta-data management approach, data archiving exercise is bound to fail.
• Next is the data retention consideration. Data retention requirements should be looked from the business requirements angle  for instance in terms of compliance requirements ( like PCI-DSS, SOX plus others) plus other considerations. It will determine data retention considerations like how much data needs to reside in primary data storage space, when it needs to be moved to secondary low-cost inactive storage space, then to tape and finally purged.
• Data retrieval- Even if data moves to secondary storage space, data needs to be retrieved for processing even though it may be for limited scenarios. If applications that may need this data is primarily java based ones, then support of jdbc based retrieval may need to be supported. So concerns around the retrieval mechanism need to be  considered while looking at the overall solution.
• Data Store & Data Storage Format - There are multiple options here. One of the default option is to have similar data store as the production one. For example, if the production data store is oracle DB, it may be better to use the oracle based one for inactive part also. But a better option would be to leverage Hadoop Stack so that commodity hardware could be leveraged for storage and also could scale easily. And at the same time, real-time access to data would not be required for the majority of the cases, so obvious time delay associated with batch mode data retrieval is acceptable. 

In addition to above considerations, data governance of inactive data should be given same kind of priority as the active one. Otherwise, there is always a tendency to ignore the same for inactive data which may lead to unnecessary cost escalation later on due to things like security breaches, higher cost for data retrieval, non adherence to compliance requirements.