Search & Personalization @ Vudu

Customer Problem

As of 2016, Vudu was the #3 Transaction Video on Demand (TVOD) service in the US and had recently launched an Advertising Video on Demand (AVOD) service. CX & NPS surveys showed that customers wanted a seamless content discovery experience. We embarked on an initiative to improve the Search & Recommendations experience. Our vision was that customers should be able to discover content to watch right within their home screens via personalized recommendations. If customers wanted to search for a specific title, then our goal was to make them succeed faster and succeed often in their search missions.

Product Gaps

Search on Vudu was served by a vendor-provided service and was poorly optimized for living room experiences. There was no content personalization on key screens in the Vudu interface. Merchandisers curated content by providing inputs to content operators in spreadsheets. Content operators manually programmed 8 platforms to create content pages with rows of movies and TV products. Each platform had 5–10 curated pages with 5–15 rows on each page. Page designs could not be shared across platforms. Due to these system inefficiencies, Vudu could not expand to more platforms and customers had a sub-par content discovery experience on existing platforms.

Metrics

Succeed Faster: Time to first stream/purchase, Search effort (# of characters, searches)

Succeed Often: Mean reciprocal rank for top 1000 queries, random sets

Operator Metrics: Content programming time per platform, # of platforms supported

Path to Innovation

Search User Experience: In 2017, we launched a new search engine called Hooli. We tested Hooli against the incumbent and demonstrated a 5% increase in search attributed revenue in an A/B test. We continued to test & launch improvements to the search experience. Key innovations over the next 2 years were

• Instant search which rendered search results after 3 characters were inserted
• Actor & genre query processing
• New facets to search within your library and free to watch content
• Pre-search experience: Past 10 searches
• Post-search experience: Improving the null search experience
• Collections feature to organize content. We also improved the search & browse APIs which could then be deployed in our operator experience infrastructure to create content rows. I focus the rest of the post on personalization.

Personalization Infrastructure

To enable personalization at scale, we invested in 3 key areas of platform development: i) Personalized content ordering of rows ii) Machine Learned Ranking of content within rows iii) Building a new operator experience. The program was called Zoltar; we loved movie-themed names at Vudu:-)

Milestones

• Launched Email Recommendations: In Q1/Q2 2017, we setup marketing A/B tests with personalized emails against a holdout group of curated emails and showed a 30% increase in sell-through rate.
• Next, I established a data partnership with a vendor to classify content in the Vudu catalog into micro-genres.
• Launched Personalization in Roku app: My team designed algorithms to identify top 5 microgenres for a user and launched it as a [For You] experience in the Roku app demonstrating 10% incremental revenue in an A/B test. Having secured executive buy-in for an infrastructure upgrade to personalization, I invested in stakeholder relationships across content operations, engineering, marketing & merchandising to ensure wider adoption of the new system.

Operator Experience

Traffic to the [For You] page was low since users had to navigate to it using their TV remotes. A revamp of the operator experience was necessary to introduce personalization into highly trafficked pages that drove bulk of transactions and streams. During customer research, we identified new front-end requirements such as variable image sizes & promotion tags. Merchandisers wanted the ability to create personalized storefronts for seasonal events, deals, promotions and for targeted segments such as kids and families. In the key sections of our apps such as the homepage and AVOD landing page, merchandisers wanted to retain the ability to insert rows and to curate content within specified rows. They needed these capabilities to fulfill content promotion agreements with studios and to highlight deals.

During product discovery, I proposed page design templates which support merchandising rows, personalized rows & hybrid rows. Merchandising & hybrid rows can be pinned at a specified position on a page. Merchandising rows & hybrid rows use content search filters to select content followed by a matrix factorization algorithm to generate a personalized rank for each user. Using hybrid rows, merchandisers can pin top n content items followed by personalized ranking of the remaining items. To create personalized rows, content operators can specify the number of personalized rows on a page and choose from a catalog of personalization algorithms including [Top N for you], [Top genre rows] and [Because you watched]. Content operators can specify image sizes for each row and specify promotion tags at the content level. They can share page design templates across different platforms.

Impact on Metrics

Vudu has expanded from 8 to 15 platforms over the past 2 years serving more customers with more personalized content on their chosen platforms. Content operators refresh pages daily as opposed to weekly. Moreover, content operators can easily setup A/B tests by creating different page layouts for test & control groups. Vudu executed more than 20 personalization A/B tests in 2019. Analysis showed that the top 3 Zoltar-enabled drivers of growth were addition of new platforms & storefronts, click-through rate from content in personalized pages (+12%) and reduced time to first stream start (-8%). Personalization enabled by Zoltar now drives more than 50% of revenue and streams at Vudu.

Business Outcomes

Zoltar was key to helping Vudu transition from a service with no personalization to one where all key experiences are personalized. The system led to the following outcomes:

• 10% increase in net promoter score as users are more conveniently able to discover content
• Improved app ratings (2.8 to 4.7 on iOS mobile) as merchandisers are more easily able to program different page layouts for iOS with a greater weightage for physical discs, AVOD. Note that iOS effectively blocks digital transactions due by charging an app 30% for a digital purchase.
• 18% in-app incremental revenue as measured by efficiencies in existing apps and new app platform rollouts such as Apple and Android TV.
• $2M per year in operational cost reduction as measured by time savings for programming content on 15 platforms.

Zoltar required product, process innovations and change management making it one of the most exciting & valuable innovations of my career.