The money was gone.
When I approached the Craps table at the casino, the excitement was intoxicating. The guy shooting the die, people excitedly collecting their winnings, and the fast-paced action of the table made me want to get a bet in. I misunderstood the odds and that was my fault.
The game had seemed so straight-forward – 11 potential numbers (two to twelve) each with an equal chance of hitting. I looked at the table and saw a payout of 30 to 1 if the shooter rolled a two. I figured I had one in eleven odds of my number coming up and it seemed like a great bet. I quickly dropped $20 in chips and watched my money disappear as the dealer collected my chips.
Later, I realized that the chance of rolling a two is only 2.78% versus 16.67% odds on rolling a seven. I had neglected to realize that there is only one possible combination that yields a two (when each die displays one). The statistics were real. They were fair. I had misinterpreted them. My $20 was gone, but it was a cheap lesson. I have come to learn that statistics are rarely what they seem.
Similar to the excitement of a casino, vacation rentals are an exciting industry and have plenty of supporting statistics. However it is absolutely vital that you understand the statistics (odds) when you make decisions impacting your company’s future or your family’s livelihood.
To be transparent, I am a huge fan of statistics. When used correctly, statistics can help guide our business decisions and give us our best chance at success. These statistics can help us understand where to invest marketing dollars, how to forecast sales properly, and make educated business decisions.
The vacation rental vertical needs statistics, but it's equally important to understand the ways statistics can be misused. Below you'll find some examples to help ensure correct analysis.
"The VR industry needs statistics. Also to understand how stats can be misused." - @DavidAngotti
1. Information Omission
"More Than 80% Of Dentists Recommend Colgate.”
People who read the above statement on billboards concluded that 80% of dentists recommend Colgate over and above other brands and the remaining 20% of dentists would recommend different brands rather than Colgate. Those are two very different statements.
Not surprisingly, the incorrect assumption that Colgate was recommended over other brands of toothpaste resulted in higher sales and undeserved credibility.
The Advertising Standards Authority analyzed Colgate’s claims and found them to be misleading[1. Colgate gets the brush off for 'misleading' ads, The Telegraph, January 2017]. When you look at it closely, Colgate had conveniently omitted a vital piece of information – the dentists in question were asked to select all the toothpaste brands they would recommend. The dentists were not recommending Colgate instead of another brand, but rather in addition to the other brands.
[Matt's Note: Interestingly, finding this picture of the Colgate statement was almost impossible. I guess this support's David's observation that information omission can cause serious harm.]
Vacation Rental Application
As an industry, both property managers and vacation rental home owners must be vigilant to understand the statistics and information we are presented with. For example, HomeAway recently stated that renting out your vacation home for as little as 18 weeks per year can cover more than 75% of the mortgage[2. Vacation Rentals Provide Valuable Income Source for Owners this Summer, HomeAway, June 2016].
While accurate, this statistic is far from a full picture. As a vacation home owner and former property manager, I understand that HomeAway is likely referring to a highly seasonal destination that earns the lion's share of its revenue in that 18 week period. The revenue is oftentimes sufficient to cover 75% of the annual mortgage expense, but not the utilities, maintenance, upgrades, HOA dues, management percentage (if not self-managing), and the myriad of other expenses that homeowners know all too well.
Obviously, HomeAway has a vested interest in seeing the vacation rental supply expand to meet the ever-growing market demand. When they published this statistic, the PR team at HomeAway probably understood that the statistic was accurate even if potentially misunderstood (not unlike Colgate).
2. Inaccurate/Biased Sampling
"58% of American adults have visited the Colosseum in Rome”
Does that seem impossible? Well, what if I told you that the survey was taken on Delta flight leaving Rome for Atlanta? So to be more clear, 58% of American adults leaving Rome on a flight for Atlanta have visited the Colosseum in Rome. As ridiculous as the above example seems, inaccurate sampling is a real issue and not always easily identifiable.
Vacation Rental Application
An industry example of a sampling error is Airbnb’s recent statement regarding Smart Pricing. In an interview, Airbnb’s Vice President of Product recently declared, "Hosts who implement Airbnb's Smart Pricing have seen a 13% increase in their revenue."[3. Airbnb gives pricing tips to users, expects revenue boost, CNBC, May 2016]
If we accept this statistic at face value, we would be a fool not to implement Smart Pricing. However, there are some major issues with this statistic.
The unfortunate reality is this statistic is not based on careful, random selection with appropriate control groups. Instead, Airbnb allowed hosts to self-select and use the tool. The selection bias has a profound impact on the usefulness of the statistic.
By design, the group that was most likely to opt-in for Smart Pricing had the most to gain. These hosts had not properly researched and set prices based on best practices or demand. On the contrary, expert-level hosts with advanced pricing strategies were the most resistant to the new tool.
To make matters worse, if a host tried out the tool and realized the Smart Pricing rate output didn't work and deactivated it, they were not included in the study. The statistic conveniently includes a self-selected group that was most likely to experience gains and excludes those that would experience loss.
Also, a control group would have helped us understand how much of the 13% increase was due to the tool versus overall industry growth. What impact did Airbnb brand growth, higher review counts over time, and other outside factors have on the statistics?
In addition to selection bias and the listed external factors, did Airbnb directly or indirectly increase the ranking of properties that participated in this program? If so, that would be an unsustainable practice once a large segment opted in.
In the same way the planeload of American tourists is not a representative sample of the whole country, a small segment of self-selected hosts is not a valid statistic to make decisions based on.
3. Confirmation Bias
Clearly, the marketing and PR departments of large companies have a vested interest in convincing people to use their service. But they also have a great deal of responsibility, seeing that our industry is so young and naive.
I highly recommend that every vacation rental read this article by James Clear on common mental errors and biases to better understand how every human being has biases and weaknesses associated with decision-making. The post will be enlightening and definitely color the way you interpret statistics and make business decisions.
While I won’t dig into each of the biases in the above article, I do want to briefly discuss confirmation bias. This is basically the idea that we are all likely to favor information that reinforces our current belief set.
In the article, the author says:
"It is not natural for us to formulate a hypothesis and then test various ways to prove it false. Instead, it is far more likely that we will form one hypothesis, assume it is true, and only seek out and believe information that supports it."
For example, a couple has been dreaming of owning a beach condo for the past decade. Their confirmation bias will inevitably cause them to cling to the facts that justify the purchase and reinforce the decision to buy. Since our brains are wired to oppose changes in belief, the couple will discount valid arguments that oppose buying a second home and cling to the facts that support it.
4. The Four Statistic Filters
Statistics are all around us. Should we ignore them? No. Should we be scared of them? No! Instead, let’s use a framework to help us apply appropriate skepticism and determine an action plan. By using common sense, it will allow us to filter out the bad statistics and take action on the good ones!
For example, at the VRMA National Conference in Orlando, Simon Lehman said the following:
"60%+ of all travel bookings will happen on mobile devices by 2020."
Instead of blindly trusting the above statistic, I recommend applying the four statistic filters (questions) below:
- Does this person or company stand to profit from me believing this statistic? No. Simon Lehman has an excellent reputation of sharing valuable insight and his company Phocuswright exists to provides and sell valuable data.
- Does this statistic pass the common-sense filter? Absolutely. We know from our Google Analytics that mobile traffic and conversions is steadily increasing and 60% seems possible by 2020.
- Are there similar statistics that support or undermine this data? Yes. There are plenty of statistics that point to increasing mobile usage.
- If I take action on this statistic, is there a significant risk? Does inaction produce risk? If my website has a poor mobile experience, choosing to ignore this statistic could dramatically decrease the number of conversions coming through my site. In turn, this could increase my dependency on mobile-friendly listing sites.
The fast-changing and exciting business of vacation rentals dictates that we carefully analyze each decision and be ready for change! By applying this simple list of questions, we can quickly, and more confidently determine if an industry statistic should be trusted and ultimately drive us to action.
I have to admit, I'm a sucker for marketing and for hard-hitting statistics. If I saw a billboard for toothpaste saying that 80% of dentists recommend it, I'd buy it immediately. But I've also now learned that not all statistics are as they might appear. I've copy/pasted David's four filters to my Evernote for quick reference (in case anyone wants it too).