This post is written by Wes Melton, Co-Founder and CTO of SmokyMountains.com. Wes' post is designed to help owners and managers understand the nuances of optimization and to encourage the larger corporations (with greater resources and data) to continue exploring the worlds of Machine Learning and Artificial Intelligence.
Last year we hit the vacation rental conference circuit hard.
We had the pleasure of networking with a myriad of companies clawing to secure their spot in this explosive industry. My friends and colleagues at these companies seemed to all be working on a lot of the same problems, each with their own flavor and/or approach to solving it.
One of the more common themes I heard from both AirBnB and HomeAway/Expedia presenters when giving advice to both single home self-managers and to multi-unit Property Managers on how to maximize leads in the search results on their sites was this:
"Make sure to pick the best photo for your listing."
Seems logical enough at first doesn't it? The worse your photo is, the more likely a traveler will bypass it in favor of a better photo down the list. Oppositely, the best photos get the best results.
However, I began to realize that this common advice has two layers of challenges to it.
1. Choosing the "best" photo in a set is a subjective task
Whether you are a homeowner or Property Manager, it is extremely hard to pick a photo that represents the statistical "best" in terms of click through performance. Human instinct is good, but how can we really be sure we are picking the best?
In most cases, the odds of someone accurately picking the "best" (or the statistically highest click-through rate image) is approximately 1-in-n due to Random Chance (n being the number of photos you have in your listing). Random Chance suggests that if you were highly successful in choosing the generally 5 ‘best’ photos in a gallery of 30 photos, then once narrowed down, a home owner or manager really has a 1-in-5 chance of picking precisely the image that will yield the highest performance on a listing site.
If our chance of picking the 'best' photo out of our own gallery is only 20%, that means we have an 80% chance of picking the wrong listing photo.
2. The best photo may change based on the other photos around you
The second challenge behind this "pick the most click-worthy photo" statement is much more complex and it has to do with the layout of listing sites. The definition of the "best" photo in a private gallery of 30 is almost always changing based on the photos sitting above and beneath it on the listing site.
Said in another way, an owner could successfully isolate the top-performing photo in their own set...only to be stifled by the fact that this selected photo is not a top-performer in the context of competitor listings. It's pretty much impossible for any independent owner or manager to determine this layer without access to the full data flow and variables of the listings feed.
This revelation led me to the following belief...
Listing Sites should be picking our featured listing photos
Across my visits to these conferences, every time I heard a presenter from any of the listing sites instructing folks to just, "pick the best photo", something inside of me got curious. "Surely they're working on a mathematical approach to this,” I told myself, “surely, they are just giving people the best advice for current-state, until they can do it for us with solid data?”
So, after attending a HomeAway breakfast in Phoenix last fall, I walked up to one of their top product managers and asked them, "So when are you rolling out algorithmic photo selection for the listings?" A blank stare and short conversation later I learned that, as far as what was communicated to me at that time, it wasn't on their radar.
As surprising as the conversation was, the reality did not change: Listing Sites have the best opportunity and data to train aMachine Learning model that is capable of picking the quantitative best photo (in terms of click through performance) for every single listing on their site with 95% accuracy depending on performance tuning.
Being extra generous, if we assume that the average vacation rental professional is currently operating on 50% odds of choosing the 'best' photo that will earn the most clicks on the search results, then a 95% accuracy is a theoretical uplift of 90% over base performance (!!!!). That leaves a ton of potential improvement on the table for both the listing sites as well as HomeOwners and Property Managers.
Progressing from traditional A/B testing to Machine Learning based Optimizations
Thinking about the problem at hand, it got me curious: “How can we solve this problem algorithmically?”
While there are a number of Machine Learning (ML) models and similar AI (Artificial Intelligence) models that could provide valuable results, the solution I decided to experiment with requires little-to-no additional infrastructure, can be implemented cheaply, and the results are quantitative and easy to analyze.
The Multi-Armed Bandit Problem
And the ramifications of this experiment go far beyond just photos. Choosing the right featured photo is a perfect example of the what the computer science world calls the "Multi-Armed Bandit Problem."
In its simplest form, using slot machines as the analogy, the challenge is as follows: "How do you pick the slot machine (listing photo in this case) that is currently delivering the highest 'reward' (clicks in this case) while minimizing loss (lost clicks in this case)?"
For our use case, this assumes different photos of a property (if used as the featured listing photo) will have different CTRs (rewards). In the current world, you essentially pick the “best” photo at random (i.e. without knowing what the reward or loss is from their decision). However, in an algorithmic world, we can test multiple options and choose the ‘best’ solution with the highest reward, using intelligent data-science (not gut feeling) to grow our businesses.
While there are a number of Machine Learning models out there that solve this problem effectively, I decided to try out a model (largely based on the work found here and here) which, once implemented, simply never stops testing, and is essentially, always learning.
To go even further, “always learning” means that in seasonal or changing markets, the algorithm will automatically choose a different photo if it detects a different photo is currently out-performing the previous 'best' photo for each listing. In addition, assuming other listings are updating photos, this algorithm can react and find the more optimal listing photo in the current listing conditions: this is something that an owner or manager would never be able to do.
The bad news is that as individual vacation rental owners or Property Managers, typically our websites will never have enough traffic or click volume to run a meaningful test that is able to learn and adapt over-time. But listing sites (and certainly the big ones!) have enough traffic to quantitatively let a Machine Learning algorithm choose the right photo quickly, effectively, and most importantly, with higher accuracy than will ever be possible for a single person sitting behind a computer trying to "pick the best photo" for their listing.
In our own testing at SmokyMountains.com, we've seen the algorithm do exactly what we would hope: Explore multiple options by delivering a unique experience to each traveler in the test, and then -- once it has determined quantitatively
Here's what Machine Learning could soon do for your vacation rental business
In the graph below, we ran a test to simulate a normal search results page on a listing site. We then let the algorithm start choosing images to serve to travelers for a single listing, measuring the photo that received the highest percentage of clicks-to-views for that listing.
Since this was a simulation, in order to validate the algorithm, we had to manually assign the winner before the test started. That is to say, we pretended as if Photo_1 was historically our best performing photo and we assigned it the highest CTR in order to see if the algorithm was able to learn correctly which photo was the top performer.
Spoiler alert: it worked. For the purposes of the simulation above, we set up our model in an excessively random environment to best simulate “real world conditions.” The algorithm still finds that Photo_1 is the top performer over time, and starts showing it the majority of the time as the other photos level off only getting shown a small portion of the time for testing on-going performance changes.
Occasionally we’ve observed a sub-optimal photo in terms of performance gets picked initially (which is visible in the graph above), but over time the algorithm generally finds the global maxima and picks the image with the highest conversion rate – again this is typically only in scenarios where you have two images with very similar CTRs (32% vs 35%).
After rounds of testing, our model was super effective at finding the “best” photo from a CTR perspective with at least 95% accuracy – that’s exciting stuff!
So what do we do for now?
While I’m guessing that the major listing sites are already working on this type of algorithmic based conversion optimization (and I’m hopeful some may be already utilizing it!), the best odds an everyday owner or manager has for picking “the best” performing photo is to follow what others have done and attempt to hack your way to figuring out which photos are the best performers. This is certainly better than nothing. But it's not nearly as amazing as having a computer do it for you.
I’ve watched the technical team speak at AirBnB speak at tech-industry conferences, and know these companies have the talent to make ML/AI driven optimizations a reality on their sites.
While I’m not sure where they’re at in their evolutions as companies, I firmly believe there are measurable financial rewards for them as well as every owner or manager who lists with them once they decide to move away from choosing listing photos via ‘best chance’ to a statistics driven model that can choose the right one nearly every time.