Making your business data driven requires monitoring a lot more data than you are used to, so you will have a big-data hump to jump. The bigger is the data at play, the more is the need to handle bigger interfaces, multiple data sources etc. and it requires a strong data management strategy to manage various other considerations. Depending on the business and relevant data, technology consideration may vary. Following are certain areas that you could consider for technology strategy to pursue data driven project. Use this as a basic roadmap, every business is specific and it can have more or less things to worry about.
So key technology considerations for today’s CIOâs Â include:
One of the primary thing that will make entire data driven project workable is the database considerations. This is dependent on the risks associated with the database.
Consider the following:
Coexisting with existing architecture:
One thing that you have to ask yourself is how will the required technology square with the existing infrastructure. Technology integration if not planned well could toast a well-run business. So, careful consideration is needed as it has direct dependency on performance and cost of the organization. ETL (Extract, Transform and Load) tools must act as a bridge between relational environment such as Oracle and analytics data warehouse such as Teradata.
– Storage and Hardware:
To make the engine work requires lot of processing around compressions, deduplication and cache management. These functions are critical for making data analytics work efficiently. Data analytics is the backbone for most of data driven projects. There are various vendors out there with tools that are sophisticated to handle up to 45 fold compressions, and reinflation, making processing and storage tedious. So, consideration around tools and their hardware and storage need is critical. So, each tool must be carefully studied for its footprint on the organization and resource allocations should be made according to the complexity of the tools and tasks.
– Query and Analytics:
Query complication varies dependent on used case. Some queries do not require a lot of pre/post processing and some queries require deep analytics, pre and post processing. Each used case comes with its own requirement and therefore must be dealt with accordingly. Some cases may even require help of visualization tools to make data consumable. So, careful considerations must be made on low and high bar requirements of the used cases. Query and Analytics requirement will have indirect impact on the cost as well as infrastructure requirement for the business.
– Scale and Manageability:
Business often have to accumulate data from disparate data sources and analyze it in different environment making entire model difficult to scale and manageable. It is another big task to understand the complications around data modeling. It encompasses infrastructure requirements, tool requirements, talent requirements etc. to provision for future growth. A deep consideration should be given to infrastructure scalability and manageability for post data driven model business. It is a delicate task and must be done carefully for accurate measurements.
There are many other decisions to be made when considering the information architecture design as it relates to big data storage/analysis. These include choosing between relational or non-relational data stores; virtualized on-premise servers or external clouds; in-memory or disk-based processing; uncompressed data formats (quicker access) or compressed (cheaper storage). Companies also need to decide whether or not to shard â split tables by row and distribute them across multiple servers â to improve performance. Other choices to be made include choosing either column-oriented or row-oriented as the dominant processing method and hybrid platform or greenfield approach. Solution could be the best mix of above stated combinations. So, a careful run of thought must be given to data requirements.
– Column-oriented Database:
As opposed to relational (row-based databases), column-oriented database group stores data that share similar attributes, e.g. one record contains the age for every customer. This type of data organization is conducive to performing many selective queries rapidly, a benchmark of big data analytics.
– In-memory Database:
In-memory is another way to speed up processing by turning to database platforms using CPU memory for data storage instead of physical disks. This cuts down the number of cycles required for data retrieval, aggregation and processing, enabling complex queries to be executed much faster. They are expensive system and has a use when high processing rate in real-time is a priority. Many trading desks use this model to process real-time trades.
âNot Only SQLâ provides a foundation Semi-structured model of data handling for inconsistent or sparse data. It’s not structured data, and therefore does not require fixed-table schemas, join operations and can scale horizontally across nodes (locally or in the cloud). NoSQL offerings come in different shapes and sizes, with open-source and licensed options and keep the needs of various social and Web platforms in mind.
– Database Appliances:
They are readily usable data nodes that are self-contained combinations of hardware and software to extend storage capabilities of relational systems or to provide an engineered system for new big data capabilities such as columnar, in-memory databases.
– Map Reduce:
Is a technique used for distributing computation of large data sets across a cluster of commodity processing nodes. Processing can be performed in parallel, as the workload is reduced into discrete independent operations, allowing some workloads to be most effectively delivered via a cloud-based infrastructure. It comes really handy when dealing with big-data problem at low cost.
Other constraints that are somewhat linked to technology but must be considered are resource requirement, market risks associated with tools etc. This could be considered as an ad-hoc task but it holds the similar pain point and must be taken seriously. Some other such decisions are if resource/support is cheap or expensive, risks with technologies being adopted and affordability of the tools.
– Resource Availability:
As you nail down on technologies needed to fuel the data engine, one question to ask is: whether the knowledgeable resources are available in abundance or will it be a nightmare to find someone to help out. It is always helpful to adopt technologies that are popular and has more resources available at lesser cost. It is a simple math of demand and supply but it ultimately helps a lot later down the road.
– Associated Risks with tools:
As we all know world is changing rapidly at the rate difficult to keep pace. With this change comes changing technological landscape. It is crucial to consider the maturity of the tools and technologies being considered. Installing something new and fresh holds the risk of lower adoption and hence lesser support. Similarly, old school technologies are always vulnerable to disruption and run-down by competition. So, technology that stays somewhere in the middle should be adopted.
– Indirect Costs & Affordability:
Another interesting point that is linked to technology is the cost and affordability associated with a particular technology. License agreements, scaling costs, and cost to manage organizational change are some of the important consideration that needs to be taken care of. What starts cheap need not be cheap later on and vice-verse, so carefully planning customer growth, market change etc would help in understanding the complete long term terms and costs with adopting any technology.
So, getting your organization on rails of data driven engine is fun, super cool, and sustainable but holds some serious technology considerations that must be considered before jumping on a particular technology.
Here is a video from IDC’s Crawford Del Prete discussing CIO Strategies around Big Data and Analytics