Spotify

Case Study: Independent Usability Study

View Full Case Study

Overview

Roles

UX Researcher

Problem

This report details the results of two studies that I conducted on Spotify’s mobile application. These include a moderated usability test (5 participants) with the goal of discovering how experienced users perform at tasks that gradually increase in complexity, as well as an online survey (42 participants) intended to gather quantitative user data.

The studies uncovered various ongoing issues with the application, both in design and engineering. Based on my findings, I propose several UX Recommendations as well as steps to take for future research.

Key Insights

Research Process

Test Objectives
Study Methods
Participants
Moderated Usability Test Tasks
  1. Follow an artist of your choice. “Like” and download one of their albums. Wait for the album to download. Finally, delete the downloaded album.
  2. Let’s say you really like this artist and want to follow them outside of Spotify. Please use the Spotify app to find and follow one of their social media accounts – it doesn’t matter which platform.
  3. Find and Like an existing workout playlist.
  4. Create a playlist and add 5 songs to it. Move the bottom song to the top of the playlist. Remove one of the songs you added. Finally, copy the playlist's Share link to your clipboard.
  5. Add 5 songs to your queue from your Starred playlist. Skip ahead to the 2nd song. Remove one of the songs. Finally, clear the queue.
Moderated Usability Test Findings
  1. Experienced users demonstrated firm grasp of basic Spotify interactions, and were able to easily overcome trickier tasks or tasks with app-side problems.
  2. Participants tended to rate the expected easiness for each task fairly high, and tended to rate the experienced easiness lower when they encountered challenges – even if their performance didn’t suffer.
    • Tasks 4 and 5 demonstrated the most notable dips in perceived easiness, and both correlated with higher (worse) success scores and more moderator assistance.
  3. All participants gave a SUS score above the “Acceptable” rating (70), and their average SUS score was much higher than 70; together these results indicate that experienced users tend to view the app as more usable.
    • Future studies should compare average SUS Scores by experience level, to assess how performance and confidence affect perceived usability.
Online Survey Metrics
Online Survey Execution
  1. Wrote 10 question survey in Google Forms to pull quantitative data from target audience.
  2. Posted Forms link to several forums on Reddit (Spotify, Music, and Design-themed pages) to somewhat diversify responders.
  3. After an adequate number of responses (was looking for 30-50, stopped at 42), set the Forms page to no longer receive responses.
Online Survey Findings
  1. Although only 81% of participants use Premium, nearly 90% of participants use Spotify at least 5 days per week; this demonstrates that there is a portion of highly active users who still don’t pay for Premium.
    • This portion of users should be focused on in future studies, as a way to increase Spotify’s overall revenue.
  2. Participants overall gave Spotify a fairly high NPS of 40, which is consistent with the fact that they tended to list Spotify as their most used music/audio app. This score is generally consistent with other consumer statistics websites.
    • While not unexpected, this data will be extremely useful in future tests with Spotify-specific target groups.
    • Users with Premium gave a significantly higher NPS of 50, likely due to the improved features that come with the paid model. This demonstrates the tradeoff of a diminished free experience, and may justify the ROI of improving the free version or attempting to increase subscription to the paid version.

Summary

UX Recommendations
Goal/Intended Result Recommendation
Improve “My Library” Tabs Individual Liked Songs section should be added to bar with Albums/Playlists/Artists. Visibility of Music/Playlists tabs should be improved with green underlines consistent with the lower bar.
Improve Visibility of Social Links Low visibility in current state; access to About page should be available at very top of artist’s page, near 3-dot icon and Follow button.
Improve Playlist Editing (A) Hold and drag to automatically move song in a playlist; could add a delay so that dragging must be intentional, OR (B) Move Edit option higher, to top of playlist UI, so as to reduce overall clicks while editing.
Improve Playlist Editing (A) Allow editing playlist order while it is not in “custom sort”, OR (B) Show hypothetical ability to move playlist songs (grayed out hamburger icon) and notify the user of the correct steps to enable editing
Fix “Clear Queue” Interaction Clear Queue button should only be grayed when not clickable, rather than disappear completely. Back-end logic needs to be fixed to improve overall usability.
Next Steps
  1. Rerun the study with new participants. Now that defined metrics and recommendations have been established, they can be tested against alternative user demographics to compare performance.
  2. Test future groups with new tasks evaluating the legitimacy/efficacy of UX Recommendations. Create alternative designs and A/B test the design changes.
  3. Compare the Net Promoter Score generated in this study to the NPS for other music/audio streaming apps, as well as the NPS scores generated for each future participant group.