The Lead Singer Swap – How to A/B Test like a Rock Star

stpContributed by John Zilch

If you follow baseball, you’re familiar with the herculean slugger who also happens to be a clubhouse cancer. If you follow business, you’ve heard of at least a few cases of visionary CEOs who are distrusted and often feared by his/her subordinates. And if you follow music, the case of the gifted yet narcissistic musician is a familiar tale.

In 1994, a four-piece rock band out of California named Stone Temple Pilots (commonly known as STP) was suffering from this ailment. On one end, they had released two of the most celebrated rock records of the decade, selling over ten million albums and captivating the music industry with their genre-crossing sound. Yet, behind the scenes the band was a wreck. Guitarists Dean and Robert DeLeo along with drummer Eric Kretz, despite writing the vast majority of the music for the band, were often overshadowed by the flamboyant lead singer Scott Weiland. While a mesmerizing presence on stage, Weiland was both unreliable and antagonistic to his bandmates behind the curtains.

So the DeLeo brothers did something novel to the music world: they ran a test. Rather than ousting Scott Weiland from STP, they began a second band, employing Kretz on drums and hiring an experienced lead singer, Dave Coutts. They named their side project “Talk Show” and prepared to simultaneously record, release and promote two albums, each with a different lead singer.

To marketing and product professionals, this style of test is commonly known as an A/B test. In this type of test, two scenarios are tested that are identical with the exception of one variant. Tests are then run for the variant against samples of the same audience. For instance, a marketer may send the same email offer to sample groups using two different subject lines to test the open rates. A user experience designer may place a button in two different locations on a website to test which gets more clicks. As long as the samples for each test are large enough to provide statistically significant results, the results may be used to make important business decisions. (Act-On – one of the leading marketing automation providers in the industry – detail other A/B best practices in a helpful eBook found on their website.)

To control the test the best they could, the DeLeo brothers wrote thirty songs together and randomly assigned half of the songs to one band and the remaining songs to the other. They then recorded both albums under the same record label and toured with both bands. In their A/B test, Weiland was test A and the Coutts was test B. (Unfortunately, the audience couldn’t be partitioned for the test, providing one exception to the A/B test rule.)

The DeLeo brothers were banking on Talk Show success to allow them to disband STP and rid themselves of the infighting that was plaguing their career. Unfortunately, the test results didn’t agree. The album recorded with Coutts (self-titled as “Talk Show”) was a commercial and critical failure while STP’s record (“Tiny Music from the Vatican Gift Shop”) received higher marks from both critics and fans. Listening to the records, it was immediately obvious why. Weiland’s lyrics were eccentric and his vocals seemed to convey a different emotion in each track. Coutts was mostly dry and vapid.

One of the cardinal rules around A/B testing is to listen to the data and not let personal bias influence conclusions. The DeLeo brothers obliged. They put an abrupt end to Talk Show and begrudgingly continued on with Stone Temple Pilots. The band would slog through two more records before breaking up in 2002. Stone Temple Pilots may not have been built for the long haul, but the data spoke for itself: Their band was most effective with Weiland at the helm.


jz-headshotJohn Zilch is a Product Manager at Dun & Bradstreet, a cloud-based data management platform for marketing and sales leaders. In his current role, John specializes in data management services, inbound enrichment and platform integration. John loves building world-class software products for world-class companies such as Intuit, Pegasystems and now Dun & Bradstreet. Follow John on Twitter at @JohnZilch and LinkedIn at www.linkedin.com/in/johnzilch.

 

Image from Wikipedia.

One thought on “The Lead Singer Swap – How to A/B Test like a Rock Star

  1. I have to disagree with your assessment of Dave Coutts being “dry and vapid”.

    Coutts biggest detriment was comparison to a successful entity. Second was a polarized, staunch fan base. Both of which have little to do with the quality of Dave’s work.

    I don’t own anything with Weiland’s work on it, but I’ve been trying for years to track Coutts down to pay for more of his work.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s