February 23, 2009 by Steve.

This is the second correspondence on last week’s Predictive Analytics World (PAW) in San Francisco. About a year and a half ago, I wrote a book review on Super Crunchers by Yale economist Ian Ayres, in which I noted that super crunching is the amalgam of predictive modeling and randomized experiments. Randomization to treatment and control groups allows investigators to minimize the risk of study bias so that the only important differences between groups out of the gate are that one is named treatment while the other is called control. Predictive modeling by itself allows analysts to infer relationships and correlation; the addition of experiments sharpens the focus to cause and effect. The combination of predictive modeling and experiments is thus a very potent tool in the business learning arsenal of hypothesize/experiment/learn.

The power of analytics plus experiments was understood well by PAW participants. Conference chair Eric Siegel noted the importance of experiments in demonstrating the value of predictive modeling, citing the oft-told story of Harrah’s Entertainment that “not using a control group” is rationale for termination. Siegel also detailed the champion/challenger experimental analogy used by enterprise decision management practitioners.

SAS’s Anne Milley improved her standing with me quite a bit with a short but incisive presentation. Anne’s just now starting to get over an unfortunate remark on the risk of using the open source analytics platform R in a January NY Times article.

In this talk, she quotes Derek Bok, president of Harvard University from 1970-1991: “If you think education is expensive, try ignorance”. Anne proceeds to frame predictive analytics in a broader context of applying scientific principles to business. This framework for business analytics is one of:

1)      Observe, Define, Measure

2)      Experiment

3)      Act

She also proposes an Analytics Center of Excellence to promote dialog between producers and consumers of analytics, sagely noting that the social is every bit as important as the analytical, and that data quality is king. Sounds like someone who’s been around the modeling block more than a few times.

John McConnell of Analytical People discusses the popular CRISP-DM (CRoss-Industry Standard

Process for Data Mining) methodology in his study of customer retention. The steps of the CRISP-DM feedback loop include Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation and Deployment. Randomized experiments or other rigorous designs are part and parcel of the evaluation step.

Jun Zhong, VP Targeting and Analytics, Card Services Customer Marketing, Wells Fargo, uses randomized experiments as well as propensity adjustments for his response modeling so he can distinguish re-active purchasers from pro-active and non purchasers to best allocate scarce targeting dollars.

Finally, Andreas Weigend, former Chief Scientist of Amazon.com is a big proponent of the scientific method for learning in business. His talk, The Unrealized Power of Data, articulated a methodology, PHAME, for measuring the power of data. Weigend’s approach,

ProblemàHypothesisàActionàMetricsàExperiments, supplements top-down problem definition, hypotheses formulation and evaluation metrics with the bottom-up performance measurement of experiments in a learning feedback loop. Tom Davenport would be proud.

Posted in Stats and BI | No Comments »

February 23, 2009 by Steve.

I just returned from two days at Predictive Analytics World in San Francisco. I must admit, my expectations weren’t very high. I’m not enthusiastic about version one of just about anything, a reticence that has generally served me well. This time was an exception. Kudos to conference chair Eric Siegel and producers Prediction Impact, Inc. and Rising Media Ltd.

A hobbled Siegel kicked-off the conference on Wednesday with his keynote: Five Ways to Lower Costs with Predictive Analytics. His focus was on variations of response and churn modeling along with risk management. Uplift modeling has to do with understanding whether a positive response was caused by modeling solicitation or would have occurred anyway. It is, of course, cheaper not to intervene with those inclined to respond. Siegel is a proponent of proving the value of analytics by contrasting results of experimental and control groups.

Though not a keynote, John Elder’s multiple case study talk: The High ROI of Data Mining Solutions for Innovative Organizations, offered wisdom for practical predictive analysis. Elder sees the primary functions of data mining as 1) eliminate bad; 2) discover good; and 3) streamline/automate for efficiency. His examples for Anheuser Busch, Walt Disney World, the IRS, and hedge fund optimization were quite interesting, but his broader message was on the alignment of business and technology to ensure success.  Critical factors include committed business champions, a strong interdisciplinary team, data vigilance, and a methodology that analyzes across multiple time periods.

Usama Fayyad, CEO of Open Insights, LLC and late Chief Data Officer of Yahoo, followed the data path in his keynote: New Challenges in Predictive Analytics. At both Yahoo and Open Insights, data strategy involves turning data into insights and strategic business assets. For Yahoo, the magnitude of data handled – over 25 terabytes and hundreds of millions of events per day – is staggering. Fayyad sees Yahoo’s “model” as the funnel from awareness–àpurchase, involving brand advertising and marketing.

The confluence of search, behavioral targeting and social media has both expanded and complicated traditional modeling. Fayyad sites Flickr photo sharing, with 90 million users organizing, generating, and distributing their own content using the “wisdom of crowds”. This new generation of on-line marketing involves individual targeting + social networks that lead to social targeting. The opportunities – and challenges – for predictive modeling in this evolving context are enormous.

Finally, Andreas Weigend, former Chief Scientist of Amazon.com, delivered a passionate talk entitled: The Unrealized Power of Data. Weigend’s point of departure is the evolution of Customer Relationship Management (CRM) to Customer Managed Relationships (CMR). Observing that 50% of Amazon customers do not come onto the site with the intention of buying, Weigend opines that companies must fully utilize both existing and new data to impact their bottom lines and strengthen relationships with customers. He argues that location data, individual self-reports and individual relationship information are special keys to marketing 2.0 – viral or social marketing.

Weigend is also a big proponent of scientific methods for learning in business. His PHAME approach will be covered in a subsequent blog.

Posted in Stats and BI | No Comments »

February 16, 2009 by Steve.

Two recent WSJ articles offer insight into Stats Man’s Corner theme of hypothesize/test/learn for business. I’ll cover one today and the other in a future blog.

More Smokers Quit if Paid, Study Shows, WSJ, Feb 12, 2009, details results of a study published in the New England Journal of Medicine that calibrates the success of getting smokers to quit by offering financial incentives. Over 20% of adults in the U.S. smoke, costing their employers $3,400 per smoker annually. 480,000 Americans die each year from smoking-related diseases, so smoking remains a significant health and business problem.

The study tracked a group of 878 smoking General Electric employees for 18 months from 2005 and 2006. The employees were first given information on smoking cessation programs and then randomly divided into two groups. Employees in the intervention group were offered cash incentive payments of up to $750 over the course of the investigation to abstain from smoking, while those in the control group were provided no such cash subsidy. The maximum $750 payment for included $100 for completing the program, $250 for not smoking six months after enrolling in the study, and an additional $400 for another six months of abstinence. Smoking habits were self-reported, with validation from saliva and urine testing.

The results of the experiment were somewhat heartening, with 14.7% of the intervention group in contrast to 5% of the controls reporting smoking cessation for the first year of the study – a significant difference. At the conclusion of the 18 months, the figures were 9.4% and 3.6% respectively. The study raises important policy questions, but the fact that individuals were assigned to intervention and control groups at random supports the internal validity of the results: other unmeasured or potentially conflicting explanations for the differences in cessation between groups should be minimized. Steven Schroeder, director of the Smoking Cessation Leadership Center at UCSF, remarked that the study “shows that incentives work”. At the same time, the study offers little in terms of the external validity or generalizability of findings. Are the positive results specific to the population tested? To study time frame payouts? What will the results look like in five years?

Lead researcher Kevin Volpp, a physician and faculty member of the prestigious Wharton School of the University of Pennsylvania, is also Director of the Leonard Davis Institute of Health Economics Center for Health Incentives (LDI CHI). The charter of LDI CHI is to facilitate research that makes significant contributions to reducing the disease burden from major public health problems such as tobacco cessation, obesity, and medication non-adherence for cardiovascular and other diseases through better understanding of how to design and apply incentives and other behavioral economic approaches to improving health. The center has three primary missions:

1.      To advance knowledge about incentive design

2.      To develop and test scalable and cost-effective applications

3.      To work with private and public sector entities such as large employers, insurers and health systems to improve health care delivery and the health of the population

The LDI CHC is research engage — combining evidence-based health care with the behavioral economics of incentives and nudge, and our now-familiar business tool chest of hypothesize/test/learn. A powerful learning and change platform indeed.

Posted in Stats and BI | No Comments »

February 11, 2009 by Steve.

Nassim Nicholas Taleb was right. The world financial system was recently devastated by unpredicted catastrophes of grand proportion – a financial black swan – and struggles today to recover and explain the carnage. Taleb predicted such an event in his 2007 bestseller The Black Swan, The Impact of the Highly Improbable. In 2005, Taleb took on the investment community with his highly entertaining Fooled by Randomness: The Hidden Role of Chance in the Markets and in Life., in which he assails the financial services community for its dumb luck, its hubris and its reckless conduct.  With no shortage of ego, Taleb iterates his financial doom and gloom theme with a vengeance in a Feb 2, 2009 Forbes interview, castigating the industry for its overconfidence and its lax risk-control models built on faulty assumptions. For Taleb, the extraordinary booms and busts experienced over the last 20 years are far outside the tidy predictions of our cherished bell curve models. Fat tails are, unfortunately, facts of financial life.

I wrote an article on Fooled By Randomness for Information Management last year in which I focused less on the financial implications of Taleb’s meanderings than his insights on human behavior. It seems people are often deluded in attributing causes of their own behavior. Success is generally interpreted in a strict causal trail of events – I did such-and-such which resulted in a favorable outcome — while failure is simply bad luck, or random. Survivorship bias occurs when the weak die young, thus skewing results toward the successes still alive. It’s all too easy to forget the deceased. Hindsight bias, a particularly pernicious human foible, allows failings to be adroitly “predicted” after the fact – I knew it all along. And the narrative fallacy allows us to fool ourselves with anecdotes and stories, which are much “easier” than rigorous evidence. Finally, humans expect progress to be linear, when it often plays out like an S Curve: “Tomato ketchup in a bottle – None will come and then the lot’ll”.

The antidote for these flawed explanations of cause and effect? None other than the hypothesize/test/learn cycle outlined in an earlier blog.

L. Gordon Crovitz, Information Age columnist for the WSJ, offers hope in a February 9, 2009 article with observations from the latest Technology, Entertainment and Design (TED) conference. He cites linked data, an exciting concept proposed by Tim Berners-Lee, inventor of the World Wide Web, which will facilitate correlating digital data in disparate formats. Linked data will enable analysts to better formulate and test business performance hypotheses using the Web, thus providing a more scientific basis for decision-making. Indeed, Berners-Lee thinks linked data may be the salvation of financial services, promoting an intelligent antidote to flying blind with new — and risky — financial instruments.

Posted in Stats and BI | No Comments »

February 4, 2009 by Steve.

I got an email the last week in January from the R help list announcing the release of the newest version of glmnet, a statistical learning algorithm that fits lasso and elastic net regularization paths for squared error, binomial and multinomial models via coordinate descent. Don’t be ashamed if you find that description a bit abstruse: just know you’re not alone! Suffice it to say that glmnet is a state-of-the-art modeling package that handles the prediction of interval and categorical dependent variables efficiently.

The package’s creator is Trevor Hastie, co-author with Jerome Friedman and Rob Tibshirani of the accompanying arcane-sounding paper: Regularized Paths for Generalized Linear Models via Coordinate Descent, published last summer. Hastie, Friedman and Tibshirani are also eminent professors of Statistics at Stanford University, the top-rated such department in the country. Last Fall, I attended a statistical learning seminar with Hastie and Tibshirani where similar models were presented at a dizzying pace.

So the R user community had just been provided access to a latest learning algorithm hot off the development presses from three world-renowned practitioners – for free. And glmnet is readily accessible from the internet, installing on existing R platforms painlessly. No commercial stats package that I know of – certainly not the market leader – is even close to releasing a competitive offering. I’d say that’s a pretty good deal for stats types like me, and a benefit to working with a fertile, world-wide open source initiative like R.

After installing glmnet on my PC, I tested it against a 1988 Current Population Survey (CPS) data set that consists of 25,631 cases. My objective was to predict the log of weekly wages from experience and education, both measured in years. I first divided the base data set into two subsets, a training set with two thirds of the cases randomly selected, and a test one with the remainder of the records. I then developed two separate models with the training data – one a straight linear model with an interaction term, the other using cubic spline mappings of experience and education. Once model parameters were developed with the training data set, I evaluated and graphed the results using the separate test data.

The plot on the left shows the linear plane generated by glmnet; the one on the right depicts the curvilinear plane from the cubic spline mapping. The linear model seems naïve in contrast to the cubic spline alternative which provides a much closer fit between actual and predicted wages. Indeed, preliminary exploration of the training data set confirmed the curvilinear nature of the relationships between education, experience and wages, with wages actually declining for high- end experience. The linear model incorrectly details uniformly increasing wages across the ranges of both education and experience.

The relationship on the left is thus mis-specified and produces predictions out of synch with actual outcomes. A naive linear specification like this is, unfortunately, more the rule than exception for BI analysts using Excel or other standard BI tools for their models. Prudent analysts will turn to the sophisticated packages of platforms like R for predictions that closely reflect the subtlety of their data.

For those interested, Hastie and Tibshirani are offering a new two-day seminar,Statistical Learning and Data Mining III (http://www-stat.stanford.edu/~hastie/sldm.html), March 16-17, 2009 in Palo Alto, CA.

Posted in Stats and BI | No Comments »

February 2, 2009 by Steve.

I’ve subscribed to the Harvard Business Review for about five years. When the monthly magazine arrives in the mail, it often seems there are either several articles pertinent for business intelligence or none at all. The February 2009 edition was one of the former.

The article: Why Good Leaders Make Bad Decisions, cites neuroscience research to observe that leaders often make decisions through the unconscious processes of pattern recognition and emotional tagging. Pattern recognition uses assumptions from prior experiences to categorize a current decision situation, often suggesting solutions similar to those that worked in the past. Emotional tagging is about the emotionally-committed preferences of the decision-maker, which of course can have substantial impact on the action taken. These processes, which in ways are similar to rules of thumb or heuristics, may produce effective decisions. They may also, however, be sources of systematic bias that can lead to faulty decisions.

The article cites examples from a book by one of the authors, Sydney Finkelstein, of Dartmouth’s Tuck School of Business, that conducted post-mortems of flawed business decisions. The authors suggest that businesses should build safeguards into their management decision processes driven by red flag conditions to guard against such sources of bias.

Tom Davenport’s article: How to Design Smart Business Experiments, offers a scientific antidote to the “on a wing and a prayer” approach to business decision-making. The culture of hypothesize/experiment/learn for operational decisions is closely aligned with the Evidence-Based Management philosophy espoused by Stanford professors Jeff Pfeffer and Bob Sutton. Indeed, the hypothesize/test foundation for business decisions is promoted in the Balanced Scorecard, Super Crunchers, and Enterprise Decision Management.

Davenport espouses a cycle for putting ideas to the test that includes:

1)      Create/Refine Hypotheses

2)      Design Experiment

3)      Execute Experiment

4)      Analyze Results

5)      Plan Rollout

6)      Rollout

Findings from all steps in the process are submitted to a Learning Library for posterity and, hopefully, for reuse.

Testing generally makes the most sense for smaller, operational decisions that are repeated often – the core of business transactions. At eBay, Amazon, and Google, randomized testing is the norm for website development. Sears has tested several formats for including its merchandise in Kmart stores, and vice-versa. Capital One has been in the testing vanguard since 1988, using experiments to design new offerings, moving to the top ranks of credit card companies by its “ability to turn a business into a scientific laboratory…..subject to testing using thousands of experiments.” And Harrah’s Entertainment has given teeth to its hypothesize/test/learn culture by a mandate that “not using a control group” is rationale for termination.

Business Intelligence, which distinguishes between exploratory and confirmatory analytics, loves the focus given its efforts by an evidenced-based culture built on hypotheses and experimentation.  Evidence-based companies generally embrace BI early and significantly; return on investment (ROI) exercises for BI are strategic and substantive.

Posted in Stats and BI | No Comments »