Ranking task is available in unmoderated studies that let you ask participants to order a set of options based on their preferences or priorities. Instead of selecting a single favorite, users arrange multiple items from most to least preferred. This allows you to gather more nuanced insights about user preferences across multiple options.
Why use ranking questions?
- Capture relative preferences between multiple items
- Identify trade-offs users make when choosing
- Prioritize features, designs, or offerings based on user preferences
How Hubble calculates ranking scores
In Hubble, ranking scores are automatically calculated once responses are collected. We apply a weighted average calculation, assuming that lower rank numbers reflect stronger preference. This means:
- The weighted average takes into account all participants’ rankings.
- A lower weighted average indicates higher overall preference.
- You don't need to manually transform the data. The scoring is handled automatically when you review your results.
This allows you to quickly identify which options are most preferred, based on how participants ranked them.
How weighted average is calculated
Once responses are collected, ranking data needs to be aggregated to identify overall preferences. The core idea behind scoring ranking questions is as the following:
Convert ranks into scores, assign weights, and calculate averages.
Step 1: Collect ranks from participants
For the sake of simplicity, we'll use an example below:
| Participant | Option A | Option B | Option C |
|---|---|---|---|
| User 1 | 1 | 2 | 3 |
| User 2 | 2 | 1 | 3 |
| User 3 | 1 | 3 | 2 |
Step 2: Sum the ranks for each option
- Option A: 1 (User 1) + 2 (User 2) + 1 (User 3) = 4
- Option B: 2 + 1 + 3 = 6
- Option C: 3 + 3 + 2 = 8
Step 3: Divide by the total number of respondents
There are 3 respondents in this example.
- Option A: 4 ÷ 3 = 1.33
- Option B: 6 ÷ 3 = 2.00
- Option C: 8 ÷ 3 = 2.67
Step 4: Interpret the scores
- The lower the average score, the stronger the overall preference.
- In this example, Option A (1.33) is most preferred, followed by Option B (2.00), then Option C (2.67).
Alternative (Reverse) Weighting
An alternative approach is to assign points in reverse, where higher points reflect stronger preference. For example:
- Rank 1 → maximum points (e.g., 3 points if there are 3 options)
- Rank 2 → next lower points (2 points)
- Rank 3 → lowest points (1 point)
This method produces a "preference score" where higher total points indicate higher preference.
Both methods are valid, and it's just different perspectives and how you want to interpret the question and the results. In Hubble, we assume lower ranks = better, and lower weighted average would be the preferred option.
Tips on better leveraging ranking questions
Avoid asking users to rank too many items (ideally no more than 5–7):
As the number of items increases, it becomes cognitively harder for participants to meaningfully compare and rank options. For example, the difference between items ranked 10th and 11th could be marginal.
Too many items can lead to random or inconsistent rankings, participant fatigue, and lower data quality. Keeping the list manageable ensures more thoughtful, reliable input.
Rankings capture relative preference but not intensity of preference:
A ranking tells you the order of preference but doesn’t reveal how strongly one option is favored over another.
For example, a participant might feel very strongly about their top choice but see little difference between the remaining options, and thus ranking alone won’t capture this nuance.
Follow-up questions can help explain why certain items were ranked higher or lower:
Adding open-ended follow-up questions allows participants to elaborate on their reasoning, provide context, and clarify trade-offs behind their rankings.
This qualitative input helps you better understand user motivations and adds valuable depth to the ranking data.