Managing Data Growth & Data Archival

Recently I have faced one scenario around data archival concerns which made me think and look little deep into data archival solutions & need for the same.

Problem Statement
Business now tracks almost every piece of data that are created as part of any business interactions. Now large terabyte online transaction processing database systems are becoming common. The rate at which data is growing has increased rapidly. Now it takes months rather than years for the data in a typical transactional database to grow from terabytes to petabytes to exabytes. It has resulted in greater challenges in terms of data management.  It has a negative impact on the performance of the application as data operations takes more time to complete. (We could alleviate the problem to a certain extent using caching solutions,  splitting data based on geographies/data range/other conditions or other means.)
Huge data growth also increased the operational cost in terms of provisioning of additional costly storage space; the associated cost involved in physical space/cooling cost/other things. It also brings in additional burden to database management  operations like data movements, database upgrades plus other regular administrative tasks. At the same time, keeping old sensitive information like customer credit card information/healthcare reports in a large production database systems beyond certain period will also pose serious security risk if it is not managed properly.

Solution Considerations
One of the straightforward approaches to the above problem is to remove the unwanted data on a periodic basis. But it will not address the concern completely. As per one study, majority of the data operation involves only data that is created within two-three years. The access to the data that is three-six years older is very minimal, and one more than six years is rare. So  scalable approach is the adoption of tiered storage approach where active data is placed in the costly SAN storage  tier  but inactive one in low-cost storage tier. Let's look at the solution consideration  little deeper for building full fledged data archival solution.

• One of the primary consideration for data archival solution is to understand the data entities & its relationship in the source system.  Without well-defined meta-data management approach, data archiving exercise is bound to fail.
• Next is the data retention consideration. Data retention requirements should be looked from the business requirements angle  for instance in terms of compliance requirements ( like PCI-DSS, SOX plus others) plus other considerations. It will determine data retention considerations like how much data needs to reside in primary data storage space, when it needs to be moved to secondary low-cost inactive storage space, then to tape and finally purged.
• Data retrieval- Even if data moves to secondary storage space, data needs to be retrieved for processing even though it may be for limited scenarios. If applications that may need this data is primarily java based ones, then support of jdbc based retrieval may need to be supported. So concerns around the retrieval mechanism need to be  considered while looking at the overall solution.
• Data Store & Data Storage Format - There are multiple options here. One of the default option is to have similar data store as the production one. For example, if the production data store is oracle DB, it may be better to use the oracle based one for inactive part also. But a better option would be to leverage Hadoop Stack so that commodity hardware could be leveraged for storage and also could scale easily. And at the same time, real-time access to data would not be required for the majority of the cases, so obvious time delay associated with batch mode data retrieval is acceptable. 

In addition to above considerations, data governance of inactive data should be given same kind of priority as the active one. Otherwise, there is always a tendency to ignore the same for inactive data which may lead to unnecessary cost escalation later on due to things like security breaches, higher cost for data retrieval, non adherence to compliance requirements.

Bulk Data Movement-Data Integration Principles

Service Oriented Integration is the preferred option for integration as it aids in natural interoperability of business capabilities. However, there are scenarios like bulk data movement where batch mode integration techniques like ETL needs to be considered.  For the same, I recommend considering following fundamental industry standard data integration principles while you architect, design & implement the solution for  batch mode integration scenarios.

• Data Quality First
• As a first step in the data integration initiative, it is recommended to perform the data quality analysis of the source system to understand its data quality level. It includes aspects like completeness, consistency, accuracy, precision, accessibility and timeliness of the data. It helps in two ways- 
• Devising approaches for improving data quality of the source systems
• Aids in the definition of data integration architecture which takes into account of inherent data quality issues.
• Write Once-Read Many
• There may be scenarios where data needs to be extracted from the source system, and then it is fed to multiple target systems after undergoing validations and transformations. If there are multiple target systems which have a dependency on the source system, it may lead to requests for creation of multiple extraction components at source systems. It will increase the complexity at source systems and also escalate the development & maintenance costs. To avoid getting into this trap, it is recommended to adopt architecture principle of Write Once-Read Many. It means to build only one extraction component per source type.
• Extract Everything
• It is observed that there is a  tendency to grab data only required for current needs. However, as we know, consumers of the source system may evolve so is their data needs. It is strongly recommended to analyze the data entities managed by source system and then consider both current & future needs of the current & potential consuming systems. Based on the analysis, extract entire data set from the source system which may be required to meet the same.
• Target Based Load Process
• This principle promotes the design of the load process  by giving consideration to target system first then looking at the subject areas within that target based system.

 One of the recommended reading in this space -  Data Integration Blueprint and Modeling: Techniques for a Scalable and Sustainable Architecture. 

Major Dimensions in large scale SOA based engagement which needs consideration before you embark on the journey

This paper cover major dimensions which should be considered for large scale SOA based engagements. It is based on the experience in executing large engagements in the same space and also based on performing architectural reviews of such initiative as part of consulting engagements.

Enterprise Architectural RoadMap-Transition to Target State
It was observed in multiple business transformation engagements, it starts well by starting enterprise architecture definition starting by capturing current state of the architecture and then defining the target states in terms of Business architecture, Information Architecture and Technology Architecture. But in the most of the cases, they have defined the target state which they want to achieve in 3-5 years which is a very long period. In almost majority of the cases, those initiatives failed to meet even 50% of intended objectives. Is that because of the issues with Target State Architecture? In most of the cases where I have involved in the review of such initiatives, it has been observed that it is not target state but failure is due to lack of transition architecture.
It is very important to define transition architecture in terms of Business/Information/Technology Architecture which represents intermediary state between current state and target state. Ideally it should be better to define intermediary state one want to achieve in a year’s time. After the first transition state milestone, team should do review that if they have achieved all objectives of intermediary architecture state and ensure if they are in right track to achieve the target state architecture. Based on the feedback of such iterations, target state should be tweaked. At the same time, minor changes could be made to exploit the opportunities and also handle threats associated with changes in PESTEL world so that they stay relevant in terms of achieving target business objective. At the same time, intermediary state should be defined in a way that business should be able to see visible benefits so that it will keep their interest and continue investing in the long term initiative.

Architecture Principles- Form the basis of the governance
In any major architecture initiative, the definition of the architecture principles should be one of the first activities to be performed. It is key to the success of architecture governance strategy. The architecture principle reflect the consensus across the enterprise and embody the spirit and thinking of the enterprise architecture. As per TOGAF framework, it act as
 Principles that govern the architecture process, affecting the development, maintenance, and use of the enterprise architecture
 Principles that govern the implementation of the architecture, establishing the first tenets and related guidance for designing and developing information systems
This is extremely important step as it provides guidance and also put constraints on any architectural decisions. It has been observed that most the cases this is one of the neglected activity and if done it is cut and copy paste from enterprise architecture literature or just done for the sake of process compliance. In the worst case, it is also observed that there is total neglect for the same. If there are not commonly accepted architecture principles in place, this will lead to inconsistent decision making and also complete failure of the architecture governance mechanism. So it is imperative to define architecture principle upfront at organizational level in terms of IT Principles which should dictate Architecture Principles. At the same time, care should be taken to have buy- in from all relevant stakeholders before formally adopting the same. If the organization is mature one, there will be already one defined at enterprise level which could be readily leveraged for the same. It has observed that in such organizations, architecture governance mechanism is not just for sake of process compliance but it truly bring business value to the organization.

Architectural Patterns and Practices- Enables in faster adoption with discipline
After the adoption of architecture principles and definition of target and transition state architectures, team will be very clear about where they are headed to. This provides a great opportunity for another level of optimization in large scale IT initiative which may involve execution of multiple projects in parallel by different globally distributed teams. Optimization steps could include following
 Form Solution Architecture group which act as an extension of Enterprise Architecture group. It should contain members from globally distributed team. Ideally it would be better to have at-least one from each location so that they could also play the role of local evangelist who could share relevant information and also in forming the consensus with local team.
 Identify the common set of architecturally significant requirements for each transition architectural state and also potential ones for target architecture state.
 Start defining solution patterns to address those requirements by complying with architecture principles.
This activity will help in bringing the consistency in terms of design choices for addressing similar concerns and also help in reusing IT resources which includes both software & hardware components. This could potentially bring in additional cost savings.

Setting the stage for adoption
In large scale adoption of service orientation, one of the key aspects to consider is the ability to work as a team not within a project team but across the multiple project teams. This is critical for successful delivery of service oriented solutions. So this calls for additional roles that are required for ensuring team work on a larger scale and maintain trust among members of this larger group so that they could rely on each other for successful delivery.
Another important aspect is to ensure that whole team is fully aware of the overall objective of the business program and high level understanding of transition and target state architecture. The appropriate team must have deeper understanding of various elements of these architectural states and also should be well versed with solution patterns, associated best practices and methodologies. At the same time, it is important to adopt common communication framework based on common vocabulary, concepts and methods. The team should be provided appropriate training and followed with internal assessment to measure the understanding. This is not about SOA architecture concepts or service technologies but more about contextually significant ones. This would ensure the common understanding which is very important for team work at such a large scale. At the same time, it is important to have appropriate governance mechanism which ensures that development team follows the solution patterns and adhere to architectural standards. So enforcing discipline in terms of how consistently team applies its knowledge is also key.

Conclusion
This white paper tries to cover set of recommendations which could be considered when you embark on larger IT initiative. As always there is no one right way to address any concern but many ones. Above one is one such way to address common concerns we normally would encounter with such initiatives.

Intrinsic Interoperability of models

One of  the interesting problem within an typical enterprise is the issue of lost in translation.....basically objective of building business capability and corresponding IT capability normally tends to get lost during flow of activities across teams (say...Business --- Enterprise Architects ---- Project Architects ---Technical Leads --- Designers --- Developers). Now the question is why is it happening? Obvious answer is decision taken at each level is not getting properly communicated and adhered to. At the same time, there is no obvious mechanism to detect this failure early enough to avoid any accident. 

Let’s look at an analogy. Say "God" has given you the responsibility to create a new solar system with planets like Earth and also with built-in cities and towns with all amenities including housing. Now "God" wants to validate if everything is created as per his vision …..basically by looking at different views from planets to continents within the planet to countries within the continent to states within the country  then to cities within the state.  If you had designed with tools with features like Google Maps which will provide ability to zoom into details starting from high level view of planets to continent to country, then could zoom into state level, then zoom into city level ...zoom into detailed view of city.....it will clearly show how things are tied together.  Beauty of such a model is that all the views are intrinsically tied together which ensures design decision at various levels are implemented as planned.

Now coming back to the problem....in IT, as we know each team creates different models(like business motivational model,  business process model, service model, security model, information model, component model plus others) which is best suited to convey their decisions. One of the inherent issue is there is no standards that tie together those models in terms of relationships between elements across the models, trace-ability between models. This is because most of standards are defined by different groups for a specific problem and there is lack of standards that tie them together unless you use single standard like UML alone. But UML alone is not enough to capture all aspects of business and IT capabilities.  Another aspect is about diagram interchange as each team may be using different set of tools with support for subset of adopted standards. (For instance, component designer may be using uml modelling tools which may not have support for BPMN standards.)  This is also adds to the problem as all relevant models may not be available within a single tool so context get lost. To address the problem, I would suggest following as steps

• Adopt modelling standards at enterprise level which aids intrinsic interoperability
• Bring in internal modelling standards for bridging the gaps in areas like relationships between elements across the models, trace-ability between models. 
• Adopt tools which support almost all modelling standards adopted at enterprise level. At the same time, places where there are gaps...build utilities which enables diagram interchange or creates some kind of mapping information

As a long term solution, there should be standard which ties various models which is required in a typical enterprise. Hope to see some activity in standards body on the same.

Social Media, Social Data, Social Commerce, Social Intelligence, Social Next Thing !!!

We have seen impact of social media platforms like Twitter in popular uprisings in Northern Africa and Middle east. Recently I have seen how effectively  a dominant local player in food & beverages business countering an extremely negative report through Facebook rather than  going full offensive through media or legal route. These are clear indication of increasing maturity level of adoption of social media platform  for commercial and non commercial purpose. So ones who do not have proper social media strategy may miss the boat. Now the question is how effectively new entrant could adopt social media strategy.  For me, there are two aspects to it.

• First is adoption of social media platform. It should start with well defined end objective rather than "me too" strategy. It could be things like improving customer connect , improving brand awareness , etc. Next step is identifying the target customer segment for example based on  geography, age group, etc. Then devise strategy for the same. It could be sharing thought leadership by IT Services companies using blogging platform, Pushing news items in terms of short message (predominantly )one way communication by media players using twitter, Building a community to listen & talk to end customer base using facebook or similar kinda  platform by retail majors . 

• Second is making sense of social data. This has  many flavors to it. Basic one or rather level one of maturity  is identifying the impact of things like marketing campaigns, new product launches, brand perception , etc. Next level could be things like identifying people who are really influencing the community around them and using that data to devise the strategies to make them satisfied.  For me, nirvana state would be integration of traditional customer data ,which is generated within enterprise  apps, and social data ,which is captured from  social community initiatives and also listening to external social platforms. This kinda integration will provide a 360 degree view into customer. This will  aid in creating individual specific offerings which has greater chances of success.

I believe there are tons of opportunity in this space as things are still evolving and have great scope of innovation.

MapReduce & Database

Quiet an interesting article on how map reduce algorithm could be implemented using Pipelined Table functions in Oracle database. Streaming, pipe lining and parallel execution of table function could improve the performance.  This will be interesting alternative on large scale data processing by leveraging  database infrastructure especially in those cases where oracle database is in use by choice or not :)

Malware Attack!!!!

Do you ever subjected to malware attack which then locked your taskmanager to prevent you from remove the malware exes? The tips for handling the same.You could access the taskmanager indirectly using tasklist command. Then identify the details of problematic process and kill with taskkill command. After that use MSCONFIG command to get access to sys configuration, modify the processes under startup & services tab. Then remove the problematic exe and also cleanup temp folders. After this, you may get back control over your computer Then install anti malware to do the final clean up act :)