20 Best Practices in Customer Feedback Programs: Building a Customer-Centric Company

Customer feedback programs (sometimes referred to as Voice of the Customer Programs, Customer Loyalty Programs) are widely used by many companies. These customer feedback programs are designed to help them understand their customers’ attitudes and experiences to ensure they are delivering a great customer experience. The ultimate goal of a customer feedback program is to maximize customer loyalty, consequently improving business performance (Hayes, 2010).

Chief Customer Officers and the like look to industry professionals for help and guidance to implement or improve their customer feedback programs. These industry professionals, in turn, offer a list of best practices for implementing/running customer feedback programs (I’m guessing there are as many of these best practice lists as there are industry professionals). I wanted to create a list of best practices that was driven by empirical evidence. Does adoption of best practices actually lead to more effective programs? How do we define “effective”? Are some best practices more critical than others? I addressed these questions through a systematic study of customer feedback programs and what makes them work. I surveyed customer feedback professionals across a wide range of companies (including Microsoft, Oracle, Akamai) about their customer feedback program. Using these data, I was able to understand why some programs are good (high loyalty) and others are not (low loyalty).  If you are a customer feedback professional, you can take the best practices survey here: http://businessoverbroadway.com/resources/self-assessment-survey to understand how your company stacks up against best practices standards in customer feedback programs.

I will present the major findings of the study here, but for the interested reader, the full study can be found in my book, Beyond the Ultimate Question. While no best practices list can promise results (my list is no different), research shows that the following 20 best practices will greatly improve your chances of achieving improved customer relationship management and increasing customer loyalty.

Components of Customer Feedback Programs

Before I talk about how to best structure a customer feedback program, let us take a 30,000-ft view of an enterprise-wide customer feedback program. A customer feedback program involves more than simply surveying customers. To be useful, a customer feedback program must successfully manage many moving parts of the program, each impacting the effectiveness of the overall program. The elements of customer feedback programs can be grouped into six major areas or components. These components are: Strategy, Governance, Business Process Integration, Method, Reporting, and Applied Research. Figure 1 below represents the components of customer feedback programs.

Components of a Customer Feedback Program
Figure 1. Elements of a Customer Feedback Program

Strategy involves the executive-level actions that set the overarching guidelines around the company’s mission and vision regarding the company objectives. Governance deals with the organization’s policies surrounding the customer feedback program. Business Process Integration deals with the extent to which the customer feedback program is integrated into the daily business processes. Method deals with the way in which customer feedback data are collected. Reporting is involved in the way in which customer feedback data are summarized and disseminated throughout the company. Finally, Applied Research focuses on the extent to which companies gain additional operational and business insight through systematic research using their customer feedback data.

Best Practices Study and General Findings

While many companies have a formal customer feedback program, only some of them experience improvements in customer loyalty while the other companies find that their customer loyalty remains flat. To understand why this difference occurs, I conducted a study to understand how loyalty leading companies, compared to loyalty lagging companies, structure their customer feedback programs (see Hayes (2009) for details of the study methodology).

A total of 277 customer feedback professionals from midsize to large companies completed a survey about their company’s customer feedback program. The respondents indicated whether their company adopts 28 specific business practices related to their customer feedback program (e.g., senior executive is champion of customer feedback program; Web-based surveys are used to collect customer feedback). Additionally, respondents were asked to provide an estimate of their company’s customer loyalty ranking within their industry; this question was used to segment customers into loyalty leaders (companies with a loyalty ranking of 70% or higher) and loyalty laggards (companies with a loyalty ranking below 70%).

Table 1. Adoption Rates of Customer Feedback Program Practices of Loyalty Leaders and Loyalty Laggards
Table 1. Adoption Rates of Customer Feedback Program Practices of Loyalty Leaders and Loyalty Laggards

The survey results revealed real differences between loyalty leaders and loyalty laggards in their customer feedback programs (See Table 1). There were statistically significant differences in adoption rates between loyalty leaders and loyalty laggards across many of the business practices. Loyalty leading companies were more likely to adopt specific practices compared to their loyalty lagging counterparts, especially in areas related to strategy/governance, integration and applied research. In upcoming posts, I will explore each component of the customer feedback program and present best practices for each.

Take the Customer Feedback Programs Best Practices Survey

If you are a customer feedback professional, you can take the best practices survey to receive free feedback on your company’s customer feedback program. This self-assessment survey assesses the extent to which your company adopts best practices throughout their program. Go here to take the free survey: http://businessoverbroadway.com/resources/self-assessment-survey.

Source: 20 Best Practices in Customer Feedback Programs: Building a Customer-Centric Company by bobehayes

May 25, 17: #AnalyticsClub #Newsletter (Events, Tips, News & more..)

[  COVER OF THE WEEK ]

image
Ethics  Source

[ LOCAL EVENTS & SESSIONS]

More WEB events? Click Here

[ AnalyticsWeek BYTES]

>> Enterprise Data Modeling Made Easy by jelaniharper

>> Which Machine Learning to use? A #cheatsheet by v1shal

>> Are U.S. Hospitals Delivering a Better Patient Experience? by bobehayes

Wanna write? Click Here

[ NEWS BYTES]

>>
 [Bootstrap Heroes] G-Square brings in a bot and plug-and-play element into analytics – YourStory.com Under  Financial Analytics

>>
 Hybrid Cloud Security: It’s Much More than Cloud Connectors | CSO … – CSO Online Under  Cloud Security

>>
 White House: Want data science with impact? Spend ‘a ridiculous … – FedScoop Under  Data Science

More NEWS ? Click Here

[ FEATURED COURSE]

Introduction to Apache Spark

image

Learn the fundamentals and architecture of Apache Spark, the leading cluster-computing framework among professionals…. more

[ FEATURED READ]

The Future of the Professions: How Technology Will Transform the Work of Human Experts

image

This book predicts the decline of today’s professions and describes the people and systems that will replace them. In an Internet society, according to Richard Susskind and Daniel Susskind, we will neither need nor want … more

[ TIPS & TRICKS OF THE WEEK]

Save yourself from zombie apocalypse from unscalable models
One living and breathing zombie in today’s analytical models is the pulsating absence of error bars. Not every model is scalable or holds ground with increasing data. Error bars that is tagged to almost every models should be duly calibrated. As business models rake in more data the error bars keep it sensible and in check. If error bars are not accounted for, we will make our models susceptible to failure leading us to halloween that we never wants to see.

[ DATA SCIENCE Q&A]

Q:Explain what a false positive and a false negative are. Why is it important these from each other? Provide examples when false positives are more important than false negatives, false negatives are more important than false positives and when these two types of errors are equally important
A: * False positive
Improperly reporting the presence of a condition when it’s not in reality. Example: HIV positive test when the patient is actually HIV negative

* False negative
Improperly reporting the absence of a condition when in reality it’s the case. Example: not detecting a disease when the patient has this disease.

When false positives are more important than false negatives:
– In a non-contagious disease, where treatment delay doesn’t have any long-term consequences but the treatment itself is grueling
– HIV test: psychological impact

When false negatives are more important than false positives:
– If early treatment is important for good outcomes
– In quality control: a defective item passes through the cracks!
– Software testing: a test to catch a virus has failed

Source

[ VIDEO OF THE WEEK]

#BigData @AnalyticsWeek #FutureOfData #Podcast with Eloy Sasot, News Corp

 #BigData @AnalyticsWeek #FutureOfData #Podcast with Eloy Sasot, News Corp

Subscribe to  Youtube

[ QUOTE OF THE WEEK]

Data are becoming the new raw material of business. – Craig Mundie

[ PODCAST OF THE WEEK]

#BigData @AnalyticsWeek #FutureOfData #Podcast with Michael OConnell, @Tibco

 #BigData @AnalyticsWeek #FutureOfData #Podcast with Michael OConnell, @Tibco

Subscribe 

iTunes  GooglePlay

[ FACT OF THE WEEK]

The Hadoop (open source software for distributed computing) market is forecast to grow at a compound annual growth rate 58% surpassing $1 billion by 2020.

Sourced from: Analytics.CLUB #WEB Newsletter

Improving Employee Empowerment Begins with Measurement

empowermentI read an article last week on employee empowerment by Annette Franz. She reflected on the merits of employee empowerment and also provided excellent examples of how employers can improve the customer experience by empowering their employees; she sites examples from the likes of Ritz-Carlton, Hyatt and Diamond Resorts, to name a few. Before employers institute ways to improve employee empowerment, however, they need to understand the level of empowerment their employees currently experience. How do employers know if their employees feel empowered? An effective way is to simply ask them.

Employee Empowerment Questionnaire (EEQ)

Twenty years ago (yes, 20 years ago), I developed an Employee Empowerment Questionnaire (EEQ) that includes 8 questions you can use in your employee survey to measure employee empowerment. The EEQ was designed to measure the degree to which employees believe that they have the authority to act on their own to increase quality (my definition of employee empowerment). Employees are asked to indicate “the extent to which you agree or disagree with each of the following statements on a 1 (strongly disagree to) to 5 (strongly agree) scale”:

  1. I am allowed to do almost anything to do a high-quality job.
  2. I have the authority to correct problems when they occur.
  3. I am allowed to be creative when I deal with problems at work.
  4. I do not have to go through a lot of red tape to change things.
  5. I have a lot of control over how I do my job.
  6. I do not need to get management’s approval before I handle problems.
  7. I am encouraged to handle job-related problems by myself.
  8. I can make changes on my job whenever I want.

The EEQ is calculated by averaging the rating across all eight questions. EEQ scores can range from 1 (no empowerment) to 5 (high empowerment). Studies using the EEQ show that it has high reliability (Cronbach’s alpha = .85 and .94 in two independent samples) and is related to important organizational variables; using the EEQ, I found that employees who feel empowered at work, compared to their counterparts, report higher job satisfaction and lower intentions to quit.

Using the EEQ for Diagnostic and Prescriptive Purposes

Employers can use the EEQ for diagnostic as well as prescriptive purposes. Comparing different employee groups, employers can identify if there is a general “empowerment problem” in their organization or if it is isolated to specific areas/roles. This simple segmentation exercise can help employers know where they need to pinpoint improvement efforts. For example, in a study of employees working for a federal government agency, I found that employees in supervisory roles reported higher empowerment (Mean EEQ = 3.71) compared to non-supervisors (Mean EEQ = 3.04). For this agency, improvement efforts around empowerment might experience the greatest ROI when focused on employees in non-supervisory roles.

In addition to acting as a diagnostic tool, results of the EEQ can prescribe ways to improve employee empowerment. While these eight questions, taken as a whole, measure one underlying construct, each question’s content shows employers how they can empower employees:

  1. Minimize red tape around change management.
  2. Allow employees to make mistakes in the name of satisfying customers.
  3. Reward employees who solve problems without the permission of management.
  4. Give employees rules of engagement but let them be creative when dealing with unique customer problems.

Summary

Employee empowerment remains an important topic of discussion in the world of customer experience management; employee empowerment is predictive of important organizational outcomes like employee job satisfaction and employee loyalty, outcomes that are associated with a better customer experience and increased customer loyalty. The Employee Empowerment Questionnaire (EEQ) allows companies to diagnose their empowerment problem and can help prescribe remedies to improve employee empowerment (e.g., minimizing bureaucratic red tape, allowing for mistakes, rewarding creative problem-solving).

As part of an annual employee survey, the EEQ can provide executives the insights they need to improve employee satisfaction and loyalty and, consequently, customer satisfaction and loyalty. To read more about the development of the Employee Empowerment Questionnaire, click here to download the free article.

Originally Posted at: Improving Employee Empowerment Begins with Measurement by bobehayes

Three Big Data Trends Analysts Can Use in 2016 and Beyond

One of the byproducts of technology’s continued expansion is a high volume of data generated by the web, mobile devices, cloud computing and the Internet of Things (IoT). Converting this “big data” into usable information has created its own side industry, one that businesses can use to drive strategy and better understand customer behavior.

The big data industry requires analysts to stay up to date with the machinery, tools and concepts associated with big data, and how each can be used to grow the field. Let’s explore three trends currently shaping the future of the big data industry:

Big Data Analytics Degrees

Mostly due to lack of know-how, businesses aren’t tapping into the full potential of big data. In fact, most companies only analyze about 12 percent of the emails, text messages, social media, documents or other data-collecting channels available to them (Forrester). Many universities now offer programs for big data analytics degrees to directly acknowledge this skills gap. The programs are designed to administer analytical talent, train and teach the skillsets – such as programming language proficiency, quantitative analysis tool expertise and statistical knowledge – needed to interpret big data. Analysts predict the demand for industry education will only grow, making it essential for universities to adopt analytics-based degree programs.

Predicting Consumer Behaviors

Big data allows businesses to access and extract key insights about their consumer’s behavior. Predictive analytics challenges businesses to take data interpretation a step further by not only looking for patterns and trends, but using them to predict future purchasing habits or actions. In essence, predictive analytics, which is a branch of big data and data mining, allows businesses to make more data-based predictions, optimize processes for better business outcomes and anticipate potential risk.

Another benefit of predictive analytics is the impact it will have on industries such as health informatics. Health informatics uses electronic health record (EHR) systems to solve problems in healthcare such as effectively tracking a patient’s medical history. By documenting records in electronic format, doctors can easily track and assess a patient’s medical history from any certified access port. This allows doctors to make assumptions about a patient’s health using predictive analytics based on documented results.

Cognitive Machine Improvements

A key trend evolving in 2016 is cognitive improvement in machinery. As humans, we crave relationship and identify with brands, ideas and concepts that are relatable and easy to use. We expect technology will adapt to this need by “humanizing” the way machines retain memories and interpret and process information.

Cognitive improvement aims to solve computing errors, yet still predict and improve outcomes as humans would. It also looks to solve human mistakes, such as medical errors or miscalculated analytics reports. A great example of cognitive improvement is IBM’s Watson supercomputer. It’s classified as the leading cognitive machine to answer complex questions using natural language.

The rise of big data mirrors the rise of tech. In 2016, we will start to see trends in big data education, as wells as a shift in data prediction patterns and error solutions. The future is bright for business and analytic intelligence, and it all starts with big data.

Dr. Athanasios Gentimis

Dr. Athanasios (Thanos) Gentimis is an Assistant Professor of Math and Analytics at Florida Polytechnic University. Dr. Gentimis received a Ph.D. in Theoretical Mathematics from the University of Florida, and is knowledgeable in several computer programming/technical languages that include C++, FORTRAN, Python and MATLAB.

Source: Three Big Data Trends Analysts Can Use in 2016 and Beyond by agentimis

May 18, 17: #AnalyticsClub #Newsletter (Events, Tips, News & more..)

[  COVER OF THE WEEK ]

image
Data Storage  Source

[ LOCAL EVENTS & SESSIONS]

More WEB events? Click Here

[ AnalyticsWeek BYTES]

>> Godzilla Vs. Megalon: Is There Really a Battle Between R and SAS for Corporate and Data Scientist Attention? by tony

>> Apr 20, 17: #AnalyticsClub #Newsletter (Events, Tips, News & more..) by admin

>> Why Entrepreneurship Should Be Compulsory In Schools by v1shal

Wanna write? Click Here

[ NEWS BYTES]

>>
 Fit enough: Internet of things’ expansion continues – NewsOK.com Under  Internet Of Things

>>
 Verisk Analytics, Inc., Acquires The GeoInformation Group – Yahoo Sports Under  Risk Analytics

>>
 Will Amazon Go Get Retail Tech Going? – Read IT Quik Under  Sentiment Analysis

More NEWS ? Click Here

[ FEATURED COURSE]

Learning from data: Machine learning course

image

This is an introductory course in machine learning (ML) that covers the basic theory, algorithms, and applications. ML is a key technology in Big Data, and in many financial, medical, commercial, and scientific applicati… more

[ FEATURED READ]

Storytelling with Data: A Data Visualization Guide for Business Professionals

image

Storytelling with Data teaches you the fundamentals of data visualization and how to communicate effectively with data. You’ll discover the power of storytelling and the way to make data a pivotal point in your story. Th… more

[ TIPS & TRICKS OF THE WEEK]

Analytics Strategy that is Startup Compliant
With right tools, capturing data is easy but not being able to handle data could lead to chaos. One of the most reliable startup strategy for adopting data analytics is TUM or The Ultimate Metric. This is the metric that matters the most to your startup. Some advantages of TUM: It answers the most important business question, it cleans up your goals, it inspires innovation and helps you understand the entire quantified business.

[ DATA SCIENCE Q&A]

Q:Explain what a local optimum is and why it is important in a specific context,
such as K-means clustering. What are specific ways of determining if you have a local optimum problem? What can be done to avoid local optima?

A: * A solution that is optimal in within a neighboring set of candidate solutions
* In contrast with global optimum: the optimal solution among all others

* K-means clustering context:
It’s proven that the objective cost function will always decrease until a local optimum is reached.
Results will depend on the initial random cluster assignment

* Determining if you have a local optimum problem:
Tendency of premature convergence
Different initialization induces different optima

* Avoid local optima in a K-means context: repeat K-means and take the solution that has the lowest cost

Source

[ VIDEO OF THE WEEK]

#GlobalBusiness at the speed of The #BigAnalytics

 #GlobalBusiness at the speed of The #BigAnalytics

Subscribe to  Youtube

[ QUOTE OF THE WEEK]

Processed data is information. Processed information is knowledge Processed knowledge is Wisdom. – Ankala V. Subbarao

[ PODCAST OF THE WEEK]

#BigData @AnalyticsWeek #FutureOfData #Podcast with Juan Gorricho, @disney

 #BigData @AnalyticsWeek #FutureOfData #Podcast with Juan Gorricho, @disney

Subscribe 

iTunes  GooglePlay

[ FACT OF THE WEEK]

According to estimates, the volume of business data worldwide, across all companies, doubles every 1.2 years.

Sourced from: Analytics.CLUB #WEB Newsletter

Is Service Quality More Important than Product Quality?

This past weekend, Apple released the iPad 3. My daughter and I visited the Apple store in downtown San Francisco to take a peek at their new device. Of course, the store was packed full of Apple fans, each trying out the new iPad. This particular in-store experience got me thinking about the role of product vs. customer service/tech support in driving customer loyalty to a brand or company.

Product vs. Tech Support

Figure 1. Descriptive Statistics and Correlations among Variables

There has been much talk about how companies need to focus on customer service/tech support to help differentiate themselves from their competitors. While I believe that customer service/tech support is important in improving the customer relationship to increase customer loyalty, this focus on customer service has distracted attention from the importance of the product.

I will illustrate my point using some data on PC manufacturers I collected a few years ago. I have three variables for this analysis:

  1. Advocacy Loyalty
  2. PC Quality
  3. Tech Support Quality

Advocacy Loyalty was measured using 4 items (e.g., overall sat, recommend, buy again, and choose again for first time) using a 0 to 10 scale. PC Quality and Tech Support Quality were each measured on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale. PC Quality was the average of three questions (PC meets expectations, PC is reliable, PC has features I want). Tech Support Quality was the average of six questions (tech support timely, knowledgeable, courteous, understands needs, always there when needed).

Figure 2. Path Diagram of Study Variables

Product is More Important than Technical Support in Driving Advocacy Loyalty

The descriptive statistics and correlations among these variables are located in Figure 1. A path diagram of these variables is presented in Figure 2. As you can see, when comparing the impact of each of the customer touch points on advocacy loyalty, PC quality has the largest impact (.68) while Tech Support as the smallest impact (.21) on advocacy loyalty.

As the results show, advocacy loyalty is more highly correlated with PC quality (.79) than with technical support quality (.59). Even when examining the partial correlations among these variables (controlling for the effect of the third variable), PC quality is much more closely linked to advocacy loyalty (.68) than is technical support quality (.22).

Summary

People don’t frequent a store primarily because of the service. They flock to a company because of the primary product the company provides. Can we please temper all this talk about how important service quality is relative to the product?  Ask anybody at that Apple store why they were there and they would tell you it was because of the products, not the service.

Originally Posted at: Is Service Quality More Important than Product Quality?

SAS enlarges its palette for big data analysis

SAS offers new tools for training, as well as for banking and network security.

SAS Institute did big data decades before big data was the buzz, and now the company is expanding on the ways large-scale computerized analysis can help organizations.

As part of its annual SAS Global Forum, being held in Dallas this week, the company has released new software customized for banking and cybersecurity, for training more people to understand SAS analytics, and for helping non-data scientists do predictive analysis with visual tools.

Founded in 1976, SAS was one of the first companies to offer analytics software for businesses. A private company that generated US$3 billion in revenue in 2014, SAS has devoted considerable research and development funds to enhance its core Statistical Analysis System (SAS) platform over the years. The new releases are the latest fruits of these labors.

With the aim of getting more people trained in the SAS ways, the company has posted its training software, SAS University Edition, on the Amazon Web Services Marketplace. Using AWS eliminates the work of setting up the software on a personal computer, and first-time users of AWS can use the 12-month free tier program, to train on the software at no cost.

SAS launched the University Edition a year ago, and it has since been downloaded over 245,000 times, according to the company.

With the release, SAS is taking aim at one of the chief problems organizations face today when it comes to data analysis, that of finding qualified talent. By 2018, the U.S. alone will face a shortage of anywhere from 140,000 to 190,000 people with analytical expertise, The McKinsey Global Institute consultancy has estimated.

Predictive analytics is becoming necessary even in fields where it hasn’t been heavily used heretofore. One example is information technology security. Security managers for large organizations are growing increasingly frustrated at learning of breaches only after they happen. SAS is betting that applying predictive and behavioral analytics to operational IT data, such as server logs, can help identify and deter break-ins and other malicious activity, as they unfold.

Last week, SAS announced that it’s building a new software package, called SAS Cybersecurity, which will process large of amounts of real-time data from network operations. The software, which will be generally available by the end of the year, will build a model of routine activity, which it then can use to identify and flag suspicious behavior.

SAS is also customizing its software for the banking industry. A new package, called SAS Model Risk Management, provides a detailed model of a how a bank operates so that the bank can better understand its financial risks, as well as convey these risks to regulators.

SAS also plans to broaden its user base by making its software more appealing beyond computer statisticians and data scientists. To this end, the company has paired its data exploration software, called SAS Visual Analytics, with its software for developing predictive models, called SAS Visual Statistics. The pairing can allow non-data scientists, such as line of business analysts and risk managers, to predict future trends based on current data.

The combined products can also be tied in with SAS In-Memory Analytics, software designed to allow large amounts of data to be held entirely in the server’s memory, speeding analysis. It can also work with data on Hadoop clusters, relational database systems or SAS servers.

QVC, the TV and online retailer, has already paired the two products. At its Italian operations, QVC streamlined its supply chain operations by allowing its sales staff to spot buying trends more easily, and spend less time building reports, according to SAS.

The combined package of SAS Visual Analytics and SAS Visual Statistics will be available in May.

Originally posted via “SAS enlarges its palette for big data analysis”

Source: SAS enlarges its palette for big data analysis

May 11, 17: #AnalyticsClub #Newsletter (Events, Tips, News & more..)

[  COVER OF THE WEEK ]

image
Data Mining  Source

[ NEWS BYTES]

>>
 Incentives need to change for firms to take cyber-security more … – The Economist Under  cyber security

>>
 Xilinx Expands into Wide Range of Vision-Guided Machine Learning Applications with reVISION – Design and Reuse (press release) Under  Machine Learning

>>
 Neustar forms marketing analytics partnership with Facebook 26 September 2016 – Research Magazine Under  Marketing Analytics

More NEWS ? Click Here

[ FEATURED COURSE]

CS229 – Machine Learning

image

This course provides a broad introduction to machine learning and statistical pattern recognition. … more

[ FEATURED READ]

Data Science from Scratch: First Principles with Python

image

Data science libraries, frameworks, modules, and toolkits are great for doing data science, but they’re also a good way to dive into the discipline without actually understanding data science. In this book, you’ll learn … more

[ TIPS & TRICKS OF THE WEEK]

Keeping Biases Checked during the last mile of decision making
Today a data driven leader, a data scientist or a data driven expert is always put to test by helping his team solve a problem using his skills and expertise. Believe it or not but a part of that decision tree is derived from the intuition that adds a bias in our judgement that makes the suggestions tainted. Most skilled professionals do understand and handle the biases well, but in few cases, we give into tiny traps and could find ourselves trapped in those biases which impairs the judgement. So, it is important that we keep the intuition bias in check when working on a data problem.

[ DATA SCIENCE Q&A]

Q:What is the Law of Large Numbers?
A: * A theorem that describes the result of performing the same experiment a large number of times
* Forms the basis of frequency-style thinking
* It says that the sample mean, the sample variance and the sample standard deviation converge to what they are trying to estimate
* Example: roll a dice, expected value is 3.5. For a large number of experiments, the average converges to 3.5

Source

[ VIDEO OF THE WEEK]

#BigData @AnalyticsWeek #FutureOfData #Podcast with Dr. Nipa Basu, @DnBUS

 #BigData @AnalyticsWeek #FutureOfData #Podcast with Dr. Nipa Basu, @DnBUS

Subscribe to  Youtube

[ QUOTE OF THE WEEK]

Without big data, you are blind and deaf and in the middle of a freeway. – Geoffrey Moore

[ PODCAST OF THE WEEK]

#BigData @AnalyticsWeek #FutureOfData #Podcast with Scott Zoldi, @fico

 #BigData @AnalyticsWeek #FutureOfData #Podcast with Scott Zoldi, @fico

Subscribe 

iTunes  GooglePlay

[ FACT OF THE WEEK]

Every person in the world having more than 215m high-resolution MRI scans a day.

Sourced from: Analytics.CLUB #WEB Newsletter

Can Hadoop be Apple easy?

Hadoop is now on the minds of executives who care deeply about the power of their rapidly accumulating data. It has already inspired a broad range of big data experiments, established a beachhead as a production system in the enterprise and garnered tremendous optimism for expanded use.

However, it is also starting to create tremendous frustration. A recent analyst report showed less enthusiasm for Hadoop pilots this year than last. Many companies are getting lost on their way to big data glory. Instead, they find themselves in a confusing place of complexity and befuddlement. What’s going on?

While there are heady predictions that by 2020, 75 percent of the Fortune 2000 will be running a 1,000-node Hadoop cluster, there is also evidence that Hadoop is not being adopted as easily as one would think. In 2013, six years after the birth of Hadoop, Gartner said that only 10 percent of the organizations it surveyed were using Hadoop. According to the most recent Gartner survey, less than 50 percent of 284 respondents have invested in Hadoop technology or even plan to do so.

data-center-Tim-Dorr-Flickr

The current attempts to transform Hadoop into a full-blown enterprise product only accomplish the basics and leave the most challenging activities, the operations part, to the users, who, for good reason, wonder what to do next. Now we get to the problem. Hadoop is still complex to run at scale and in production.

Once you get Hadoop running, the real work is just beginning. In order to provide value to the business you need to maintain a cluster that is always up and high performance while being transparent to the end-user. You must make sure the jobs don’t get in each other’s way. You need to support different types of jobs that compete for resources. You have to monitor and troubleshoot the work as it flows through the system. This means doing all sorts of work that is managed, controlled, and monitored by experts. These tasks include diagnosing problems with users’ jobs, handling resource contention between users, resolving problems with jobs that block each other, etc.

How can companies get past the painful stage and start achieving the cost and big data benefits that Hadoop promises? When we look at the advanced practitioners, those companies that have ample data and ample resources to pursue the benefits of Hadoop, we find evidence that the current ways of using Hadoop still require significant end-customer involvement and hands-on support in order to be successful.

For example, Netflix created the Genie project to streamline the use of Amazon Elastic MapReduce by its data scientists, whom Netflix wanted to insulate from the complexity of creating and managing clusters. The Genie project fills the gaps between what Amazon offers and what Netflix actually needs to run diverse workloads in an efficient manner. After a user describes the nature of a desired workload by using metadata, Genie matches the workload with clusters that are best suited to run it, thereby granting the user’s wish.

Once Hadoop finds its “genie,” it can solve the problem of turning Hadoop into a useful tool that can be run at scale and in production. The reason Hadoop adoption and the move into production is going slowly is that these hard problems are being figured out over and over again, stalling progress. By filling this gap for Hadoop, users can do just what they want to do, and learn things about data, without having to waste time learning about Hadoop.

To read the original article on Venture Beat, click here.

Source: Can Hadoop be Apple easy?

Deriving “Inherently Intelligent” Information from Artificial Intelligence

The emergence of big data and scalable data lakes has made it easy for organizations to focus on amassing enormous quantities of data–almost to the exclusion of the analytic insight which renders big data an asset.

According to Paxata co-founder and Chief Product Officer Nenshad Bardoliwalla,“People are collecting this data but they have no idea what’s actually in the data lake, so they can’t take advantage of it.”

Instead of focusing on data and its collection, enterprises should focus on information and its insight, which is the natural outcome of intelligent analytics. Data preparation exists at the nexus between ingesting data and obtaining valuable information from them, and is the critical requisite which has traditionally kept data in the backrooms of IT and away from the business users that need them.

Self-service data preparation tools, however, enable business users to actuate most aspects of preparation—including integration, data quality measures, data governance adherence, and transformation—themselves. The incorporation of myriad facets of artificial intelligence including machine learning, natural language processing, and semantic ontologies both automates and expedites these processes, delivering their vaunted capabilities to the business users who have the most to gain from them.

“How do I get information that allows me to very rapidly do analysis or get insight without having to make this a PhD thesis for every person in the company?” asked Paxata co-founder and CEO Prakash Nanduri. “That’s actually the challenge that’s facing our industry these days.”

Preparing Analytics with Artificial Intelligence
Contemporary artificial intelligence and its accessibility to the enterprise today is the answer to Nanduri’s question, and the key to intelligent information. Transitioning from initial data ingestion to analytic insight in business-viable time frames requires the leveraging of the aforementioned artificial intelligence capabilities in smart data preparation platforms. These tools effectively obsolete the manual data preparation that otherwise threatens to consume the time of data scientists and IT departments. “We cannot do any analysis until we have complete, clean, contextual, and consumable data,” Nanduri maintained, enumerating (at a high level) the responsibility of data preparation platforms. Artificial intelligence facilitates those necessities with smart systems that learn from both data-derived precedents and user input, natural language, and evolving semantic models “to do all the heavy lifting for the human beings,” Nanduri said.

Utilizing Natural Language
Artificial intelligence algorithms are at the core of modern data preparation platforms such as Paxata that have largely replaced manual preparation. “There are a series of algorithmic techniques that can automate the process of turning data into information,” Bardoliwalla explained. Those algorithms exploit natural language processing in three key ways that offer enhanced user experiences for self-service:

  • User experience is directly improved with search capabilities via natural language processing that hasten aspects of data discovery.
  • The aforementioned algorithms are invaluable for joining relevant data sets to one another for integration purposes, while suggesting to end users the best way to do so.
  • NLP is also used to standardized terms that may have been entered in different ways, yet which have the same meaning across different systems in a manner that reinforces data quality.

“I always like to say that I just want the system to do it for me,” Bardoliwalla remarked. “Just look at my data, tell me where all the variations are, then recommend the right answer.” The human involvement in this process is vital, particularly with these type of machine learning algorithms that provide recommendations that users choose from-—and which then become the basis for future actions.

At Scale
Perhaps one of the most discernible advantages of smart data management platforms is their ability to utilize artificial intelligence technologies at scale. Scale is one of the critical prerequisites for making such options enterprise grade, and encompasses affirmative responses to critical questions Nanduri asked of these tools, such as can they “handle security, can you handle lineage, can you handle mixed work loads, can you deal with full automation, do you allow for both interactive workloads and batch jobs, and have a full audit trail?” The key to accounting for these different facets of enterprise-grade data preparation at scale is a distributed computing environment that relies on in-memory techniques to account for the most exacting demands of big data. The scalable nature of the algorithms that power such platforms is optimized in that setting. “It’s not enough to run these algorithms on my desk top,” Bardoliwalla commented. “You have to be able to run this on a billion rows, and standardize a billion rows, or join a billion row data set with a 500 billion row data set.”

Shifting the ETL Paradigm with “Point and Click” Transformation
Such scalability is virtually useless without the swiftness to account for the real-time and near real-time needs of modern business. “With a series of technologies we built on top of Apache Spark including a compiler and optimizer, including columnar caching, including our own transformations that are coded as RDDs which is the core Spark abstraction, we have built a very intelligent distributed computing layer… that allows us to interact with sub-second response time on very large volumes of data,” Bardoliwalla mentioned. The most cogent example of this intersection of scale and expeditiousness is in transforming data, which is typically an arduous, time consuming process utilizing traditional ETL methods. Whereas ETL in relational environments requires exhaustive modeling for all possible questions of data in advance—and significant re-calibration times for additional requirements or questions—the incorporation of a semantic model across all data sources voids such concerns. “Instead of presupposing what semantics are, and premodeling the transformations necessary to get uniform semantics, in a Paxata model we build from the ground up and infer our way into a standardized model,” Bardoliwalla revealed. “We are able to allow the data to emerge into information based on the precedents and the algorithmic recommendations people are doing.”

Overcoming the Dark
The significance of self-service data preparation is not easy to summarize. It involves, yet transcends, its alignment with the overall tendency within the data management landscape to empower the business and facilitate timely control over its data at scale. It is predicated on, yet supersedes, the placement of the foremost technologies in the data space—artificial intelligence and all of its particulars—in the hands of those same people. Similarly, it is about more than the comprehensive nature of these solutions and their ability to reinforce parts of data quality, data governance, transformation, and data integration.

Quintessentially, it symbolizes a much needed victory over dark data, and helps to bring to light information assets that might otherwise remain untapped through a process vital to analytics itself.

“The entire ETL paradigm is broken,” Bardoliwalla said. “The reason we have dark data in the enterprise is because the vast majority of people cannot use the tools that are already available to them to turn data into information. They have to rely on the elite few who have these capabilities, but don’t have the business context.”

In light of this situation, smart data preparation is not only ameliorating ETL, but also fulfilling a longstanding industry need to truly democratize data and their management.

Source: Deriving “Inherently Intelligent” Information from Artificial Intelligence by jelaniharper