If there’s one thing Spotify excels at, it’s knowing the customer. Many a Twitter user has eulogized the Swedish company’s use of big data and machine learning to provide deeply personalized features like the individualized Discover Weekly playlist, joking that the music streaming service knows them more intimately than their spouse.
The feature uses proprietary algorithms to understand a user’s music tastes and point them towards new genres, artists and tracks.
Meanwhile, the social networking aspect of the platform is a boon to artists thirsty for visibility. The Release Radar feature launched in 2016 as a companion to Discover Weekly, delivering a customized playlist every Friday with new releases from artists users previously listened to or actively follow.
The wizards behind these flagship products are Spotify’s data scientists and user researchers, who work together to understand what users are doing on the platform and why, in order to serve up more of what they like.
“Having those two things together gives us a really well-rounded, holistic view of our users, what their wants and needs are, and what they’re doing,” Peter Gilks, director of product insights at Spotify, said at the Qualtrics X4 Summit in Salt Lake City, Utah.
Ad delivery as a user experience
Through its desktop site and mobile app, Spotify logs over 100 billion data points per day based on the activities of its 207 million active users around the world. More than half of them are free, ad-supported users who receive ads in the form of audio, video and display banners.
“The ads they receive, how they receive them, the formats they come in and when they receive them is actually a critical part of their overall user experience,” explained Gilks. “Making sure that is a positive experience and not a negative one is something that our team focuses on a lot.”
Despite being a cross-industry poster child for how tech companies should handle big data, Spotify learned a valuable lesson last summer when it decided to introduce a new feature: a ‘Skip’ button that would allow its free users to skip ads they found irrelevant. From the company’s perspective, this would mutually benefit advertisers, listeners and the business.
Photo credit: Spotify
Listeners would be fed better-targeted ads while “advertisers can reach a more engaged group of listeners, and Spotify can keep both groups happy by having a relevant ad delivery system,” explained Colette Kolenda, a user researcher at Spotify. She studies how the platform’s free users experience and interact with ads while listening to music.
Kolenda’s team piloted a prototype of the ‘skip’ feature in a test market of highly engaged users in Australia. In one iteration, the button displayed the word ‘Skip,’ and in the other iteration, the button appeared as a symbol identical to the one in the media player used for skipping tracks.
The product insights team was convinced that uptake of the new feature would be universal, immediate and game-changing. It wasn’t.
“Once we had the two designs, we wanted to understand, are they usable?” explained Kolenda. “[Usability] has two components: users have to physically be able to engage with this feature and they have to understand what they’re doing.”
Photo credit: Spotify
After testing the prototypes on passersby at a local park, Spotify decided to move forward with the symbol button, and started developing the new feature to test it at scale. Before it could launch a test, the team defined some KPIs to measure what success would look like.
“There were a few questions we wanted to be able to answer, such as what percentage of users were using our new feature, what percentage of ads were being skipped and what is the impact on how these listeners would use Spotify,” said Kolenda.
Using Voice of Customer research
Instead of the enthusiastic uptake they’d hoped for, some users would skip ads once or twice, then stop. Others never even tried the feature. Clearly, there was a lack of product education at play, and differing levels of awareness between users.
Kolenda and her team flew to Australia to visit Spotify listeners in their homes to glean more information.
One Spotify listener, Ben, was a “power user” who skipped a lot of ads – he was in the top 10th percentile for early adopters of the new feature, until he stopped. Served a video ad, Ben found that he couldn’t skip it, and so assumed that he could no longer skip ads. As a customer, he didn’t understand the different ad models the company was using.
Read more: 4 Brands that Practically Invented Customer-Centric Advertising
“From an hour-long interview with Ben, we learned that we had to have a consistent experience,” said Kolenda. “Audio and video ads had to be skippable in the same intuitive way.”
One thing companies struggle to do is put themselves in their user’s shoes, as Spotify’s case study shows. When introducing a new product or feature, product managers are so intimately familiar with the product that they aren't cognizant of things like awareness and education around the product that affect how a user experiences that new feature.
“This was our breakthrough insight, a moment when we realized we had to change our methodology in order to get at this fuller, richer story of what was going on behind the what and why data,” said Kolenda.
Up to this point, the user research and data science teams had been working side by side exchanging insights, but without collaborating on research.
“What we decided to do instead was work together to combine data science and user research methodologies at the same time pointed at the same population to understand user’s different behaviors,” explained Kristie Savage, a product data scientist for ad formats at Spotify.
Merging quant and qual to tell the whole story
In the next iteration of the experiment, the user research team ran a diary study while the data scientists tracked the transactional data of the study participants. A diary study is an open-ended questionnaire delivered over a period of time, which Spotify administered to participants daily over a period of three weeks.
“We asked them questions about the kind of music they listened to, what playlists they were listening to on Spotify and their experience with the ads,” said Kolenda.
The data showed that one user, Matt, consistently skipped six ads per listening session. In the diary study, he wrote: “I can only skip six ads because I only get six song skips. I guess the ad skips must follow the same rule.”
“On Spotify’s free version you can only skip six songs per hour streamed. This did not apply to ads, but Matt applied his own mental model about what he knew about Spotify towards a completely different product,” explained Savage.
Findings like these underscored the importance of having the user research and data science teams “point different methodologies at the same question” to understand gaps in the user experience. In Matt’s case, for instance, an education component was lacking.
Spotify started sending personalized PSAs to users based on their usage of the ad skip feature. Those skipping ads in a scattershot fashion would be prompted with a banner from Spotify saying, “Freedom to skip. There’s no limit to the number of ads you can skip on Spotify,” while those who hadn’t used the feature at all received the following nudge: “Listen to the ads you like, skip the ones you don’t. It’s that simple.”
Today, Spotify allows users to skip an unlimited number of audio ads and to skip video ads after 5 seconds of playback.