Soon, Algorithms Will Select Your Best Featured Image (And Here’s How It Works)

Author: Wes Melton
Category: Harness Tools
May 18, 2017

This post is written by Wes Melton, Co-Founder and CTO of Wes' post is designed to help owners and managers understand the nuances of optimization and to encourage the larger corporations (with greater resources and data) to continue exploring the worlds of Machine Learning and Artificial Intelligence.

Last year we hit the vacation rental conference circuit hard.

We had the pleasure of networking with a myriad of companies clawing to secure their spot in this explosive industry. My friends and colleagues at these companies seemed to all be working on a lot of the same problems, each with their own flavor and/or approach to solving it.

One of the more common themes I heard from both AirBnB and HomeAway/Expedia presenters when giving advice to both single home self-managers and to multi-unit Property Managers on how to maximize leads in the search results on their sites was this:

"Make sure to pick the best photo for your listing."

Seems logical enough at first doesn't it? The worse your photo is, the more likely a traveler will bypass it in favor of a better photo down the list. Oppositely, the best photos get the best results.

However, I began to realize that this common advice has two layers of challenges to it.

1. Choosing the "best" photo in a set is a subjective task

Whether you are a homeowner or Property Manager, it is extremely hard to pick a photo that represents the statistical "best" in terms of click through performance. Human instinct is good, but how can we really be sure we are picking the best?

In most cases, the odds of someone accurately picking the "best" (or the statistically highest click-through rate image) is approximately 1-in-n due to Random Chance (n being the number of photos you have in your listing). Random Chance suggests that if you were highly successful in choosing the generally 5 ‘best’ photos in a gallery of 30 photos, then once narrowed down, a home owner or manager really has a 1-in-5 chance of picking precisely the image that will yield the highest performance on a listing site.

If our chance of picking the 'best' photo out of our own gallery is only 20%, that means we have an 80% chance of picking the wrong listing photo.

2. The best photo may change based on the other photos around you

The second challenge behind this "pick the most click-worthy photo" statement is much more complex and it has to do with the layout of listing sites. The definition of the "best" photo in a private gallery of 30 is almost always changing based on the photos sitting above and beneath it on the listing site.

Said in another way, an owner could successfully isolate the top-performing photo in their own set...only to be stifled by the fact that this selected photo is not a top-performer in the context of competitor listings. It's pretty much impossible for any independent owner or manager to determine this layer without access to the full data flow and variables of the listings feed.

This revelation led me to the following belief...

Listing Sites should be picking our featured listing photos

Across my visits to these conferences, every time I heard a presenter from any of the listing sites instructing folks to just, "pick the best photo", something inside of me got curious. "Surely they're working on a mathematical approach to this,” I told myself, “surely, they are just giving people the best advice for current-state, until they can do it for us with solid data?”

So, after attending a HomeAway breakfast in Phoenix last fall, I walked up to one of their top product managers and asked them, "So when are you rolling out algorithmic photo selection for the listings?" A blank stare and short conversation later I learned that, as far as what was communicated to me at that time, it wasn't on their radar.

As surprising as the conversation was, the reality did not change: Listing Sites have the best opportunity and data to train aMachine Learning model that is capable of picking the quantitative best photo (in terms of click through performance) for every single listing on their site with 95% accuracy depending on performance tuning.

Being extra generous, if we assume that the average vacation rental professional is currently operating on 50% odds of choosing the 'best' photo that will earn the most clicks on the search results, then a 95% accuracy is a theoretical uplift of 90% over base performance (!!!!). That leaves a ton of potential improvement on the table for both the listing sites as well as HomeOwners and Property Managers.

Progressing from traditional A/B testing to Machine Learning based Optimizations

Thinking about the problem at hand, it got me curious: “How can we solve this problem algorithmically?”

While there are a number of Machine Learning (ML) models and similar AI (Artificial Intelligence) models that could provide valuable results, the solution I decided to experiment with requires little-to-no additional infrastructure, can be implemented cheaply, and the results are quantitative and easy to analyze.

The Multi-Armed Bandit Problem

The Multi-Armed Bandit

And the ramifications of this experiment go far beyond just photos. Choosing the right featured photo is a perfect example of the what the computer science world calls the "Multi-Armed Bandit Problem."

In its simplest form, using slot machines as the analogy, the challenge is as follows: "How do you pick the slot machine (listing photo in this case) that is currently delivering the highest 'reward' (clicks in this case) while minimizing loss (lost clicks in this case)?"

For our use case, this assumes different photos of a property (if used as the featured listing photo) will have different CTRs (rewards). In the current world, you essentially pick the “best” photo at random (i.e. without knowing what the reward or loss is from their decision). However, in an algorithmic world, we can test multiple options and choose the ‘best’ solution with the highest reward, using intelligent data-science (not gut feeling) to grow our businesses.

While there are a number of Machine Learning models out there that solve this problem effectively, I decided to try out a model (largely based on the work found here and here) which, once implemented, simply never stops testing, and is essentially, always learning.

To go even further, “always learning” means that in seasonal or changing markets, the algorithm will automatically choose a different photo if it detects a different photo is currently out-performing the previous 'best' photo for each listing. In addition, assuming other listings are updating photos, this algorithm can react and find the more optimal listing photo in the current listing conditions: this is something that an owner or manager would never be able to do.

The bad news is that as individual vacation rental owners or Property Managers, typically our websites will never have enough traffic or click volume to run a meaningful test that is able to learn and adapt over-time. But listing sites (and certainly the big ones!) have enough traffic to quantitatively let a Machine Learning algorithm choose the right photo quickly, effectively, and most importantly, with higher accuracy than will ever be possible for a single person sitting behind a computer trying to "pick the best photo" for their listing.

In our own testing at, we've seen the algorithm do exactly what we would hope: Explore multiple options by delivering a unique experience to each traveler in the test, and then -- once it has determined quantitativelywhich photo has the highest CTR on the site -- promote that photo as the 'winner.' We are fortunate to have a big enough data sample size AND to control the variables around the test. Here's what we learned...

Here's what Machine Learning could soon do for your vacation rental business

In the graph below, we ran a test to simulate a normal search results page on a listing site. We then let the algorithm start choosing images to serve to travelers for a single listing, measuring the photo that received the highest percentage of clicks-to-views for that listing.

Since this was a simulation, in order to validate the algorithm, we had to manually assign the winner before the test started. That is to say, we pretended as if Photo_1 was historically our best performing photo and we assigned it the highest CTR in order to see if the algorithm was able to learn correctly which photo was the top performer.

Spoiler alert: it worked. For the purposes of the simulation above, we set up our model in an excessively random environment to best simulate “real world conditions.” The algorithm still finds that Photo_1 is the top performer over time, and starts showing it the majority of the time as the other photos level off only getting shown a small portion of the time for testing on-going performance changes.

Occasionally we’ve observed a sub-optimal photo in terms of performance gets picked initially (which is visible in the graph above), but over time the algorithm generally finds the global maxima and picks the image with the highest conversion rate – again this is typically only in scenarios where you have two images with very similar CTRs (32% vs 35%).

After rounds of testing, our model was super effective at finding the “best” photo from a CTR perspective with at least 95% accuracy – that’s exciting stuff!

So what do we do for now?

While I’m guessing that the major listing sites are already working on this type of algorithmic based conversion optimization (and I’m hopeful some may be already utilizing it!), the best odds an everyday owner or manager has for picking “the best” performing photo is to follow what others have done and attempt to hack your way to figuring out which photos are the best performers. This is certainly better than nothing. But it's not nearly as amazing as having a computer do it for you.

I’ve watched the technical team speak at AirBnB speak at tech-industry conferences, and know these companies have the talent to make ML/AI driven optimizations a reality on their sites.

While I’m not sure where they’re at in their evolutions as companies, I firmly believe there are measurable financial rewards for them as well as every owner or manager who lists with them once they decide to move away from choosing listing photos via ‘best chance’ to a statistics driven model that can choose the right one nearly every time.

About the author 

Wes Melton

Wes Melton is a builder of things, technology expert, and Lego fanatic. Previously a technology consultant to national brands, he now spends his days building his brand, writing code, and contributing to the broader business and technology community.

  1. Forgive me for complaining:

    Listing sites already pick enough–and don’t know my traveler or destination, which changes how guests behave

    Facebook uses algorithms to decide what goes on my wall, and what order comments go

    Yelp uses algorithms to decide orders of reviews

    TripAdvisor same as yelp

    I want to pick my pictures, because I know what client I want (and not someone attracted to a picture who’s never heard of my destination, worse guests!)

    I want comments and feeds to be in order

    I want to decide what I want to see

    Algorithms help the listing companies and not individuals who work hard to get what they truly want. (And decide how to rank you when really its for them to make rental fees not in your best interest)

    Sorry for the negatives!

    1. Hi!

      Could you expound upon, “Algorithms help the listing companies and not individuals who work hard to get what they truly want”?

      Wouldn’t an algorithm that increases clicks to your listing in theory generate more bookings?

      If guest qualification is your fear, pricing should ideally handle a lot of that.

      1. I’ve had far less clicks on vrbo this year since algorithms. (They strictly want what gets them commission and not what’s best for owners rental). Ex feb 2016 had 940 clicks. This feb was 240.

        TripAdvisor really dramatic, sense algorithms it’s DEA. ( 3 views a week? Used to be 100)

        And algorithms don’t take into account that it can be the same person clicking, they aren’t “new” clicks.

        For example Lilly Pulitzer pops up on my Facebook every time. If I click 3 times, those aren’t new clicks so it’s a false number

        1. PS I know it’s the way things are done right now, sorry to be so negative. But “numbers” don’t necessarily mean quality, which algorithms don’t take into account. It’s just numbers. And doesn’t really mean conversion where it count either

          1. Hi Blamona!

            “And doesn’t really mean conversion where it count either” <– This largely depends on the team designing and deploying the conversion tests. If success metrics are defined correctly, a good test will absolutely deliver both quality and quantity.

          2. I get what you’re saying, but listing companies don’t have owner’s best interest in mind when using algorithms to decide where to rank you.

            This would work for own personal websites where you are in control, and not interests of someone else (like listing sights)

            And in my case on Facebook, 1 of my pages has gotten 120,000 looks, has 400 comments. I have to read each comment every time someone comments because algorithms decide what order to put comments in. I just want recent!

            I know it’s the way of the world, the problem with listing sites is they have different interests than the actual owner does.

          3. “listing companies don’t have owner’s best interest in mind” — is a different but very valid argument. Listing sites have their own interest in mind. It’s merely a matter of how one utilizes the algorithms, which never lie.

          4. My concern with the statement is that I’m afraid it doesn’t represent the issue well. We run a listing site generating a large volume of revenue in our market for our partners.

            As the owner of a listing site, I can tell you I am VERY concerned with the impact to owners.

            The second problem is that, while I do understand that some VRs feel (rightfully or not) shafted by recent updates on the big sites, they are moving more and more inline with what the consumer wants, which is necessary for survival.

            So, if us “listing sites” truly don’t care about our owners (which speaking at least for ourselves, we do), it’s because we have to first appeal to the consumer or the home owner will really suffer when bookings fall off due to lack of product-market fit as consumer demands evolve.

        2. Hi Blamona!

          So this algorithm wouldn’t affect your rank – it would work to determine which listing photo will get you the most views of your property.

          In regards to, “And algorithms don’t take into account that it can be the same person clicking, they aren’t “new” clicks.”, that’s not accurate. You can choose which clicks to recognize – if they’re counting all clicks it was a decision that was made by the conversion team.

    2. Thanks, Blamona!

      I couldn’t agree more! If that ever happens I can see my bookings dropping dramatically. My apartment (in my house) is in a neighbourhood in Vancouver. 20 minute drive to downtown. My guests want nothing to do with staying downtown.

      The problem for me is that there is a misconception that Vancouver IS the downtown core. TripAdvisor suggestions for travellers always mention Gastown, Yaletown, Capilano Canyon for instance. Places most Vancouverites never visit – for us they’r tourist traps. Very close to me are VanDusen Gardens, Queen Elizabeth Park, the UBC botanical gardens and tree-top walk and the four best beaches with free parking.

      I’m sure the people who would be organizing the photos don’t know this either. Downtown Vancouver has turned into a maze of office and apartment towers and very little else.

      When I get an inquiry asking how far is it to get downtown I send them a page of links to all the wonderful things to do on “our” side of town and I usually get the booking – and very grateful guests.

  2. I agree with Blamona on this. Sure, algorithms have their uses but once they come into play, people will be trying to game the algorithm (just as using the algorithm in the first place is trying to game the listing site visitor’s choice).
    A photo may work well for a while because of the context i.e. it stands out from those around it. That may be because the subject matter is different or the image is supersaturated. It may not be the photo that gets the click but the short descriptive text next to the photo. Or it may simply be because a friend recommended the property and so it was clicked.

    The statement “over time the algorithm generally finds the global maxima and picks the image with the highest conversion rate” is assuming that it is the image that is creating the highest conversion rate. There are so many variables as to why a listing site does something. Visitors do not a booking based on one image. It will be based on the copy, the other images, the mood they are in, what their friends said, what they read in the newspaper or saw on that travel program last night.

    Do we really want to replace the pleasure of subjective choice with an algorithm that may not actually be doing what we hope it to be doing? At any given time an algorithm may tell you that more clicks are coming when a particular photo is the headline photo but that is not necessarily telling you anything useful at all.
    That is the problem with statistics and time. You can make many different conclusions from statistics and the time aspect can vary results too. Time of day, time of month, time of year. Many false assumptions can be made about how time affects us.

    Matt believes that our choice will come down to deciding who will be our gatekeeper. That is hardly a choice. That is relinquishing your freedom to a machine. I am sure he is right that many people will and do relinquish their freedom of choice. Those people probably stay in hotels! In the idiosyncratic world of the holiday home, I hope we don’t go too far down the algorithmic highway to a bland world where choices are made for us.

    1. Hi Nick!

      From your response I’m a little concerned maybe I didn’t explain well enough the specific implication of this algorithm that I was describing.

      The specific use case I was describing was only on the search results page (in this scenario, though it certainly has broad use cases) – so while I would agree that a certain image wouldn’t directly cause someone to book most likely, I do think that basic logic would agree that if you increase more views to your listing, then you should see an increase in bookings if the conversion rate of the listing stays equal.

      Your statement, “That is the problem with statistics and time. You can make many different conclusions from statistics and the time aspect can vary results too. Time of day, time of month, time of year.” is exactly why an algorithm, and not people making changes based on gut feel once in a while, need to be doing this. The algorithm we’ve deployed that is described above is able to react to all of those variables that you’ve described, in real time, better than any human can currently with the incredibly limited visibility to conversion data on the listing sites.

      Also – while I understand the “pleasure of subjective choice”, it’s simply impossible to know with any significance if the photo you are subjectively picking to show up in the search results is driving the maximum number of clicks to your listing that are possible, especially in a search results page that is constantly changing. Only an algorithm that can react in real time will be powerful enough to maximize views of your listing and therefore maximize conversions as well.

      While I definitely can understand the fear component, we’ve already deployed very similar technologies to entire portions of our site and have seen very measurable, positive results.

      My fear with the responses on this topic from the community so far is that they seem to be an emotional reaction to a general disdain for the listing sites and a lack of complete understanding of how optimizations like this could drive real benefits for everyone.

      I think the only way to stay super relevant and competitive in an evolving market as a PM/VR is to stick to quantifiable – objective – results, and be suspicious of our subjective ones until they’re proven.

      1. Hi Wes,

        I can accept that this may work on your own listing site if the context is constant i.e. the other photos (and everything else for that matter) on a page of listings are constant for a period of time long enough to run a test. My experience of the giant listing sites is that a page of listings returned for any enquiry (which is usually location and date based) is never constant. In a constantly changing context, the photo that the algorithm “thinks” is best may only be so because the others (which have changed) have become worse by comparison.

        On a smaller site this might not be so because the presentation on a listing page may be more consistent. I guess what I am really saying is that I do not think that a listing site is perhaps the best place to test photos because it is almost impossible to have a true control. There are testing sites like Peek for evaluating the navigation and ease of use of websites. Something similar might be a better way of testing the drawing power of photos and a lot simpler than trying to create what seems to me a futile algorithm.

        On a one or two property site, it is not that hard to simply change photos and see if one works better but even that has the problem of a different photo at a different time of year may elicit a different response. Matt’s website has tested photos in the past by just asking members to give their opinion. Whether that tells you that a photo is “better” is unknowable except in the longish run that a test needs to have any valid meaning. The trouble is that the long run is not really that long because tastes and perceptions change so rapidly.

        I suggest that it may simply be better to change photos regularly. It would not be hard to have a core bunch of informational photos i.e. the bedroom/ bathroom/ living room/ kitchen/ pool / building photos with a few alternatives and a revolving bunch of “mood” photos. In other words you have a pool of 50-80 images but the algorithm delivers about 25-30 at any one time in such a way that the core informational photos form around two thirds of those delivered. That would make much more sense to me than trying to pick a single winner in an endlessly changing listing page landscape.

        Your fear that my reaction is an emotional reaction to the major listing sites is not quite right. These listing sites are a fact of life that have brought many benefits and problems as well. My emotional reaction is more about the idea that machine learning and artificial intelligence are appropriate or even effective in every area of human life.

        I suspect that we are being subjected to a lot of propaganda about the effectiveness of machines. This usually takes the form of how helpful it is for consumers (more choice, cheaper etc) and beneficial for producers (efficiency, better ROI etc) but really it is about enabling scale for very large organisations who want to remove the human element. The human element is being marginalised under this onslaught without much thought as to the quality of outcome as long as it is “cheaper” and more “efficient”. Those two words are cover for an awful lot of economic dishonesty.

        Thank you, though, Wes for a really interesting post. I am not convinced in this case but despite my scepticism will keep an open mind.

        1. Hi Nick!

          I would encourage you to study “The Law Of Really Big Numbers”. I think it would help with addressing some of your concerns/criticisms to the approach.

          Your thoughts on the big sites vs little sites are actually backwards to proven studies where this algo has been run in the wild.

          The big sites have so much volume they’re able to determine quickly and accurately and then learn (aka react/change) accurately as well because of how much traffic.

          I appreciate your thoughts, but largely believe this is the way of the future as it’s already implemented in various forms on a number of huge sites outside of our vertical with proven results. It’s not a question of if this works, it’s a question of when leading VR listing sites will implement.


    2. Nick, I don’t think the question is “do we want” to use algorithms? because they’re going to become (in fact, in many ways, they already are) an integrated component of life (whether we like it or not). The headline you clicked for the morning news, the stoplight that optimizes traffic patterns, the image that convinced you to buy your (now) favorite pants….etc. My question is, “which of these inevitable gatekeepers do we choose to trust?” and “with that taken into consideration, how can we engineer our lives to be as independent as possible?” It’s a big responsibility being in charge of creating those algorithms for sure.

      1. Hi Matt – I grant you that algorithms are an essential part of life but I question whether they are the answer to everything. Your choice of “The headline you clicked for the morning news” is a case in point. Algorithms serve up the news they think we want to read or see. That not only leads to confirmation bias but must have contributed to the whole “fake news” thing.
        I am also not so pessimistic as to believe that gatekeepers are either inevitable or that we have to trust them! I’m also not sure that I want to engineer my life as opposed to simply grab it by the horns and live it without having to measure it down to the last nanometer.

        Independence is an attractive thought (the romance of freedom) but a slippery concept because, in practice, we are all highly dependent upon one another. Your website is a grand example of that. Life is about relationships which implies dependencies. When the shit hits the fan you want a Capt Sulley to be at the controls, not a machine.

        1. YEP. My personal take is that these algorithms are BEST when they create a better experience for both the buyer AND the seller. If it’s only good for one side (and bad for the other) it’s sub-optimal. This begs the question: who are HomeAway’s customers: travelers or property owners/managers? In @blamona:disqus’s case below, I believe that’s where the confusion lies.

          1. That is a very good question Matt. In my view the customers who count for the listing sites are the consumers who rent the listed properties. The property owners/managers are the producers. They are the exact equivalent of farmers who have been squeezed with little though for the long term consequences by the giant supermarket chains under the endless mantra of “Cheaper Prices” for consumers.
            It’s a hard one to argue. Farmers have responded in an extraordinary fashion over the last 50 years with the aid of technology, plant breeding, larger farms and so on. The externalities of environmental damage have been ignored until relatively recently and the food wastage is immense – about one third, mostly in the developed world. It could be argued that other side effects are widespread obesity and diabetes.
            A direct possible comparison in our industry is the huge amount of new “investment” in property specifically for renting out as vacation rentals. The AirBnB effect. In Australia the number of mortgages that are interest only has reached 40%. Many of those mortgages are sitting on a mere 5% deposit. This is not investment but rampant speculation. I doubt it is a sustainable proposition and will most likely end in tears. The externality in this case is that first time buyers of homes to live in are struggling to compete with “investors” who receive tax subsidies which adds to the upward pressure on real estate prices.

            For the VR industry there is a sustained push back taking place across the world as politicians respond to the pleas for “affordable” housing and also by the hotel industry wanting to crimp the runaway success of AirBnB.

            The quaint idea that listing sites have interests in common was demolished by the promotion encouraging consumers to book without thought because they could always cancel whenever they felt like it. That is more or less fine for hotels that have inventory but inappropriate for holiday homes that are not metropolitan or are in seasonal destinations.

            I apologise for getting off track regarding the algorithms that never lie but your question was a good one regarding our owner/manager interests and those of the listing sites. I do not want to demonise AirBnB or any listing site but I think it is as well to not have any illusions about the level of priority that they direct towards their suppliers.

  3. At the risk of revealing too much about myself… Tinder does this. 🙂

    But as of now, it’s the Tinderer’s choice whether to turn on that feature or not. I chose to keep it turned off simply because I wanted to have a photo that captured who I am in my heart of hearts, not the “best performing” one. My thought was I wanted to weed out those I’m not compatible with.

    But at the same time, I DO realize that more right swipes = larger pool of potential matches = greater chance of meeting someone compatible. So, admittedly, maybe my logic is a bit faulty there.

    All of this is to say I see where you’re coming from, Nick and Blamona. There’s something a bit icky about giving over your agency when it comes to how you (or your property) are represented. But what if it actually worked?

    Maybe the major OTAs will start off by making it an OPTION, as Tinder did, for homeowners to use algorithms to determine their best photo. Let the results speak for themselves.

Comments are closed.

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}