Does your app let reviewers rate videos / mockups either 1 to 5 stars or 1-10 numerically?

Ratings are super helpful because then we know how good something currently is ... and how far or close it is to being high quality. That saves a lot of resources because if an item is already 4.7 out of 5 stars, then spending much more time on feedback is often a waste because is often becomes nitpicking over minor differences of personal preferences that don't matter.

Likewise, if an item is rated only a 1.9 out of 5, it is usually a waste of resources to implement most suggestions that are annotated because even 2 or 3 rounds of suggested tweaks to a weak premise or a weak core design won't get the item close to good enough. Instead a complete overhaul or do-over is needed. Without a rating, the organization or creator often won't know just how off-track an item is, and will waste time trying to improve something that ultimately isn't going to work. Reviewers tend to not have the heart to put into writing that they hate something a creator has created, or that they greatly dislike it or think it sucks.

But most people are plenty comfortable clicking a rating for a variety of reasons. One reason is that people tend to feel more anonymous when clicking a ratings UI. Another reason is that it's a number, and they're not having to write in their own words that the item is quite bad. Another is that clicking takes a split second, whereas writing a sentence about how bad something is takes far longer. Thus part-way through it, there's a lot more opportunity to balk at crapping on the design in your own words.

This is also true for organizations getting feedback from people inside of their organization. If the head of a department sends the item to people in lower tiers of the department for feedback, a lot of people will balk at or avoid saying in writing how bad they really think the item is for fear of possibly getting on the wrong side of the boss if the boss might possibly be attached to the design or approach. But if they're just 1 of several or more people clicking a ratings UI that will compile into an average, most people will click the number they truly think.

It's also very helpful and even essential in overcoming the problem of vagueness of words and phrases. Even if you ask reviewers to "please let me know how good you think it is," often the replies will be vague like: "I think it's ok." The problem is that some people use ok to mean it's good, but not great, and others use it to mean they think it's very much below average. Other examples of vague replies are: "It could be better" or "I think it has potential" or "I don't really like it." When you ask people in those situations what they'd rate the item in question, you get a very wide range of numbers, showing that people use vague phrases to mean really different things.

Ratings are also kind of key to knowing if new versions are on the whole better than the last version or not. For example, on version 3 a few team members make a dozen suggestions with annotations, and half of them are implemented. With only annotations and no ratings, sending the new version out will result in new annotations, but usually won't tell you if the reviewers on the whole think the item has gotten worse or better. For 3 reviewers, the changes could result in them lowering their ratings from 5 stars to 2 stars. For five others, it increases from 4 stars to 5 stars. That lowers the average by a full star lower.

But you wouldn't know it from annotations. You wouldn't even know it if you asked people to say whether they like the new version better or not, because 5 would say better and only 3 would say worse. You'd think it overall has gotten significantly better, rather than the reality that it's dropped a whole star lower.

Anyway, if you don't currently have a ratings feature, it's actually super low-cost to do. A single developer can usually write the code for it in 2 days including displaying the average and the number of ratings like: Average: 2.4 (7 ratings).

The UI for this is one of the most common UIs out there, so you wouldn't need to spend any time trying to come up with a UI for it. In fact, many frameworks and coding languages such as Python or Ruby on Rails have pre-made modules/plugins that handle this that are either part of the framework or available as an open-source plug-in. But even if not, it's really very low cost to do yourself. One thing I'd recommend is that when reviewers receive an item, don't show the average rating until the reviewer does her own rating. That way the average rating doesn't influence the reviewer. In fact, the reviewers don't even need to see the average rating at all: only the core app users need to see it. So you could just leave that part out of the reviewer UI.

Track order wrong

1 comment