Jump to content

Misleading Star ratings?


schludermann

Recommended Posts

I think that the Star ratings used at survey police are not very helpful and can be misleading.  This is because there are several survey panels in obvious decline from recent reports but they still have 3-5 stars.  The rating system should help make good decisions, not lead to bad discoveries.

I think this is because SurveyPolice doesn't use a rolling score. A rolling score uses the most recent data to represent a metric.  In the customer service world, this is a very common way to show current relevancy and avoid masking poor behavior.

 

I also think a single star rating is not helpful enough when determining if a Panel is suitable.  It may be very simple but it is not descriptive enough to help gauge how good a panel is.

This is because there are multiple behaviors that come into the appropriateness of a panel, as follows, please add to the list:

1. Yield - How much can you make if you comeback everyday, say over a quarter. Are the reward thresholds discouraging?

2. Withdraws - How hard is it to claim a reward, options, latency for reward arrival, does the reward cost have a fee? How long do rewards persist before being forfeited?

3. Customer service - How fast do they respond, do they rely on canned responses, are the responses relevant or nonsensical, Are they personable?

4. Technical performance - Duration until a survey loads, too many redirects, survey hangs, can a survey restart after hang or quit.

5. Survey completions - Does the survey panel track all attempts or only completed attempts, do they use a sensible survey description if you need to identify a survey to support?

6. Is the panel tracking you - Many survey panels use trackers, some as high as 15 and read each others cookie files. Some surveys will not run if tracking is controlled by you.

7. Is the Terms and conditions reasonable - There are many that allow for undisclosed, capricious and arbitrary account closure.

8. Is the panel part of a larger organization - There is a lot of consolidation happening, is it good? We should be able to spot trends.

 

I think as a group, we could come up with a collective alpha/numeric code 6-8 characters long, perhaps pairings. 

SurveyPolice uses 10 levels of star division, so I don't think it would be hard.  The star system can remain for those that want that type of rating.

I would like to suggest using letter for qualitative ratings and numbers for quantitative ratings.  Each position always has the same meaning (support, payout, yield, technical...)

Perhaps and adhoc group could come together to hash out a rating system?

 

Kris

 

Link to comment
Share on other sites

I don't usually pay attention to the star rating. I read the actual reviews and see if the company sounds like one that is worth my time or not. I also only pay attention to recent reviews. We have the ability to edit our reviews so it's partially on us to make sure we are keeping the ratings current. Just my two cents ;)

  • Like 4
Link to comment
Share on other sites

13 minutes ago, spider said:

I don't usually pay attention to the star rating. I read the actual reviews and see if the company sounds like one that is worth my time or not. I also only pay attention to recent reviews. We have the ability to edit our reviews so it's partially on us to make sure we are keeping the ratings current. Just my two cents ;)

I also read the reviews and make my decisions to join based on them

  • Like 3
Link to comment
Share on other sites

I never pay attention to stars on sites anymore. Netflix ruined that for me, hahahaha-3 star movies indeed, lol. All kidding aside, I take the time to read reviews that people bother to write. I figure if their opinion meant enough for them to take the time to do that-it's worth reading. Star ratings only take a sec and not that much thought. Plus, the variance is also based on the type of person rating, their tolerances for/lack of tolerances will vary from another's.

  • Like 2
Link to comment
Share on other sites

7 hours ago, schludermann said:

I think this is because SurveyPolice doesn't use a rolling score. A rolling score uses the most recent data to represent a metric.  In the customer service world, this is a very common way to show current relevancy and avoid masking poor behavior.

Hi Kris,

SurveyPolice uses a very sophisticated algorithm to calculate star ratings and it does incorporate a 'rolling score'. Reviews left today are certainly weighted more heavily than those left 2 years ago, for example, and this is just a basic feature of what our algorithm involves.

Our algorithm is one of the strongest you will find when it comes to rating systems. We use a very impressive method to calculate scores and even did extra consultation with a mathematician who has a PHD in statistics, who fully stood behind the methodology we are using. Our confidence level in our own scoring method could not be higher!

If a star rating seems too high, it may be because there are a large number of reviewers who rated the panel very highly in the past, or there are still a steady stream of users who are assigning it the highest score possible, even if there are others who are giving it a very low score. Ratings should not fluctuate wildly, and a single review should not be able to "make" or "break" a panel's score, which is exactly how we've set things up.

With regards to a qualitative scoring system, we had this many years ago, but opted to simplify things. We agree that a higher level of detail could help paint a clearer picture for scores, though this is not always the case; asking users to provide more details on their experience with something basic like customer service for instance, might not apply to them... nevermind whether they think something like the TOS of a website is fair (when the vast majority of users never read them). It's a delicate balance for sure, and we hope that users who want to get the full story on a panel take time to read some of the actual reviews, rather than fully relying on few yellow stars.

 

Link to comment
Share on other sites

I found the best thing to do is check a site out. I may think it's the best survey site and someone else may think it stinks. All of us are different so we will not like the same places. I take the pros and cons from here and find a middle ground if I can.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...