Where Einstein Meets Edison

Perpetual experimentalists: How successful entrepreneurs understand customers

Perpetual experimentalists: How successful entrepreneurs understand customers

Apr 24, 2014

Image courtesy of HelloWallet

Common wisdom in entrepreneurship, most markedly promoted by books such as Lean Startup, emphasizes the importance of rapid experimentation and iteration in many aspects of startup development, from selecting a business model to understanding customer behavior. The latter has become increasingly prevalent over the past decade. Even large, established companies like Amazon and Google engage continuously in these behavioral experiments, or A/B tests, and base fundamental decisions around product and service design and pricing on their outcomes. Some companies even employ Chief Scientists to construct and analyze these tests.

But what do these experiments look like? And what is the value they provide?

Increasingly, such experimentation employs behavioral economics to better understand consumers. Dan Ariely, author of Predictably Irrational, explains that the field of behavioral economics is, at its core, a science identifying patterns in behavior and ultimately determining tactics to alter behavior. Ariely says, “traditionally, it was thought that if you just tell people information about something it will change the way they behave.” Essentially, the more people know about something, the more likely they will be to act in accordance with that knowledge and in their best interest. However, this has been proven over and over not to be the case in areas as wide-ranging as health and wellness, employment, safe driving, and personal finance. As such, a marketing, sales, or product/service design approach based solely on educating consumers is insufficient. On the other hand, an empirical approach to understanding consumer behavior can help pinpoint the specific tactics and approaches that resonate with customers. The types of experiments conducted across and within organizations vary greatly, but organizations that utilize this approach are highly committed to it. Several examples of companies that have successfully deployed such practices include Mint.com (acquired by Intuit), HelloWallet, and RedBeacon (acquired by Home Depot).

Opt-in versus opt-out

Mint.com, a personal finance management platform with over $13 million users, has utilized experimentation to determine many of its product and service features. Group Product Manager Vince Maniago described the importance of framing and user choice in determining financial health outcomes. In a recent experiment, Mint.com sought to determine whether automatically opting users into daily finance tips emails would help people make better spending/saving decisions day-to-day versus allowing individuals to opt-in of their own accord. While automatic opt-in has been effective in other contexts (e.g., annual 401k participation), it was much less impactful in this case, and those who opted in of their own accord were more than two times more likely to read the emails regularly.

Framing the daily tips as a voluntary promotion of positive financial behavior proved much more beneficial than the automatic opt-in approach that is most commonly employed. This was particularly true if they opted in while in a “hot state,” or a state of heightened awareness about their finances (e.g., tax season).

The power of association

HelloWallet also seeks to improve individuals’ personal finance behaviors, but does so in partnership with their employers. The company is heavily science- and data-driven, and has identified some interesting behavior related to people’s existing psychological associations. Principal Scientist Stephen Wendel shared two cases where associations made all the difference:

  • In an effort to help users curb spending on the weekends, when they are most prone to overspend, HelloWallet ran an experiment to see whether an email reminder to spend less sent on Fridays would be effective. While seemingly logical, this intervention proved not only to be ineffective, but actually increased spending as it reminded consumers of their existing association between the weekends and shopping and caused them to hit the ATM!
  • In developing ways to provide feedback to customers, HelloWallet saw great success in using a scoring system that users associated with the experience of academic grading. Introduction of financial health scores caused the average user to save an additional $200 in the first month alone.

Industry transformation through testing

RedBeacon is a two-sided online marketplace for handymen (e.g., plumbers, carpenters, etc.), called “pros,” and homeowners or others requiring their services. The ability to change behavior was imperative for RedBeacon from the outset, as individuals were not accustomed to booking such services online and pros had never utilized an online platform to get jobs. Rather, word of mouth and community-based advertising were the norm. In order to transform the industry, RedBeacon developed a thorough understanding of both of their customer bases through experimentation.

Chief Technology Officer Aaron Lee discussed the quote process as an area that was particularly challenging and benefited from testing.

  • Users: Originally, users were asked to submit requests for jobs via free-form text, presumably to allow for increased detail/specificity. However, this led to a great deal of difficulty for the pro in pricing jobs, as individuals tended to miss crucial information in their request. As a result, RedBeacon moved to a mad lib/decision tree job input design where users could select pre-populated characteristics about their job in 4-5 clicks. While less customizable, this actually enhanced the speed and accuracy of quotes for both users and pros. By monitoring user behavior closely, the team was able to determine the optimal design.
  • Pros: Given their past experience with providing job quotes, pros would often rush to submit pricing in order to get in front of the client prior to competing pros. This led to hasty and poorly composed responses. To remedy this, RedBeacon tested a 30-minute safe period where if a pro started responding to a request, s/he would be guaranteed one of a limited number of responses that would reach the client. This 30-minute window allowed pros time to develop a higher quality response. As with all of the above examples, the right answer in this case was unclear and it took a rigorous empirical approach to test an intervention in order to arrive at a business decision.

This scientific approach to consumer behavior continues to become increasingly sophisticated and the pace with which experiments are conducted has ramped significantly. For example, Mint.com has developed a “hypothesis testing system” to allow them to design and deploy statistically significant, large scale tests in a matter of days based on virtually any customer attribute available in its data store (e.g., individuals who shop at Walmart more than at Target). As technology becomes more sophisticated and these approaches continue to show strong results, behavioral economic principles and experimentation will become standard practice across a broader spectrum of businesses, both new and established. For startups, this means a faster and more reliable way to refine products and services to meet customer needs.