|
The Cheshire Cat posted:Honestly it could be anything. The thing about any numeric scale is the vast majority of people will only use the very top or bottom rating because people who just felt a thing was "average" don't tend to care enough to leave a review. This is why so many rating systems now are just "like" or "dislike". Also because people have been trained to, when customer service is all set up as 'give the max or this person gets poo poo on', the only time you're not gonna give the max is when they were a real shitheel.
|
#
?
Apr 6, 2019 09:18
|
|
|
|
| # ? Apr 28, 2024 18:54 |
|
|
Range voting becomes approval voting when everyone votes strategically. Film at eleven. (It’s still never worse than starting with approval voting or with any ranked voting method.)
|
|
#
?
Apr 6, 2019 09:25
|
|
|
Tunicate posted:Also because people have been trained to, when customer service is all set up as 'give the max or this person gets poo poo on', the only time you're not gonna give the max is when they were a real shitheel.
|
|
#
?
Apr 6, 2019 10:20
|
|
|
Plus there's this hot new corporate bullshit thing called NPS which basically means that under a straight 1-10 scale, anything under a 7 is considered a negative. e: Shamefully got the abbreviation wrong Somfin has a new favorite as of 12:08 on Apr 6, 2019 |
|
#
?
Apr 6, 2019 10:36
|
|
|
Somfin posted:Plus there's this hot new corporate bullshit thing called NPM which basically means that under a straight 1-10 scale, anything under a 7 is considered a negative. NPS? I’ve found it useful
|
|
#
?
Apr 6, 2019 10:39
|
|
|
The Cheshire Cat posted:Honestly it could be anything. The thing about any numeric scale is the vast majority of people will only use the very top or bottom rating because people who just felt a thing was "average" don't tend to care enough to leave a review. This is why so many rating systems now are just "like" or "dislike". Llamadeus posted:Also the fact that the ratings are aggregated encourages the extremes of the scale to maximize moving the average (eg if something sits at an average of 6 but you think it's an 8 then rating it 10 is twice as effective in bringing the score up towards 8) There are some sites that avoid this, but I think only the ones where the main purpose is for people to review things. e.g. movie review site Letterboxd, where a) everyone is kinda pretentious enough to spend time thinking if the movie is truly four stars or only three and a half and b) they only ever show the aggregated score next to a histogram:  which can take on all kinds of different shapes
|
|
#
?
Apr 6, 2019 11:13
|
|
|
Furia posted:NPS? I’ve found it useful Yeah, that one. But the fact that the scale is 1-6 = "gently caress YOU", 7-8 = *crickets*, 9-10 = "Okay you're fine" is just reinforcing what I'm talking about
|
|
#
?
Apr 6, 2019 12:09
|
|
|
NPS is mathematically terrible. Avoid.
|
|
#
?
Apr 6, 2019 14:49
|
|
|
I'm a fan of NPS myself. I find it much more informative than a pass/fail, and it actually does what is says: nobody who's giving you a 6 or below will be a Promoter of what you're selling. It's effectively a "barely passing" grade and shouldn't be thought of as a good thing -- risk of customer flight is huge at that point. Admittedly, I'd never rely on just the one NPS number on its own and would want to see the actual percentages for all three categories. But -- unlike about 90% of other corporate metrics -- it does its job well.
|
|
#
?
Apr 6, 2019 14:50
|
|
|
I can’t find my preferred explanation of why the NPS math is bad right now, but this one looks decent: http://businessoverbroadway.com/2018/05/07/data-science-reveals-three-problems-with-the-nps-dogma/ E: this is the one I’ve found best explains the terrible numerics: https://blog.usejournal.com/net-promoter-score-considered-harmful-and-what-ux-professionals-can-do-about-it-fe7a132f4430 Subjunctive has a new favorite as of 15:01 on Apr 6, 2019 |
|
#
?
Apr 6, 2019 14:57
|
|
|
Trabant posted:I'm a fan of NPS myself. I find it much more informative than a pass/fail, and it actually does what is says: nobody who's giving you a 6 or below will be a Promoter of what you're selling. It's effectively a "barely passing" grade and shouldn't be thought of as a good thing -- risk of customer flight is huge at that point. Do you like it because it measures what you want? Or because it is a quantitative measurement you can directly act on, and use to support and guide your decisions? Studies show that 99% of all cases is the second option. Sorry, didn’t mean to type studies, but just my bullshit experience, YMMV
|
|
#
?
Apr 6, 2019 15:13
|
|
|
As a customer, another mark against NPS is that it doesn't really define what is meant by "recommend". I usually assume it means without solicitation ("Hey Joe, have you heard of this great company XYZ? I think they're awesome!"), but maybe execs think it means that I'd recommend the company if prompted in some way. Because of this interpretation, I pretty much always mark 0 for these.
|
|
#
?
Apr 6, 2019 15:40
|
|
|
Always give all max scores in the name of worker solidarity
|
|
#
?
Apr 6, 2019 15:49
|
|
|
Subjunctive posted:I can’t find my preferred explanation of why the NPS math is bad right now, but this one looks decent: http://businessoverbroadway.com/2018/05/07/data-science-reveals-three-problems-with-the-nps-dogma/ quote:This is what Kate Rutter calls Analytics Theatre. Creating dramatic swings in numbers for the sake of drama, but not because they help us make our products or services better. This is fantastic and a name that we badly need.
|
|
#
?
Apr 6, 2019 15:49
|
|
|
Goon Danton posted:Always give all max scores in the name of worker solidarity
|
|
#
?
Apr 6, 2019 16:14
|
|
|
Salespeople are workers too, and prime ministers keep their jobs based on elections instead of metrics. 
|
|
#
?
Apr 6, 2019 16:17
|
|
|
I am delighted by the idea of extending worker solidarity to middle management sales execs and project/product/program managers because after all stuff like NPS is a way for the C suite to get middle management to knife fight each other for scraps of cash and maybe the nearest suggestion of becoming C suite themselves.
|
|
#
?
Apr 6, 2019 16:32
|
|
|
Anyone who seriously defends a metric like NPS(r) that can’t distinguish between everyone rating things 0/10 and 6/10 is at best naive about the tool and more likely just captured because they’ve used it to make decisions in the past and don’t want to admit they were mistaken. It’s a sickness. Just ask about satisfaction on a small scale (3 or 5 choices) and spend your effort identifying the experiences and behaviours that most correspond to different ratings. You get signals with equivalent predictive power about growth, except they’re less noisy and you can compare different numbers in a sane way.
|
|
#
?
Apr 6, 2019 16:43
|
|
|
I hate all of you and support guillotining anyone who genuinely cares about Customer Satisfaction by any metric rather than the other billions of us who care about Please Just Pay Me Enough And Can I Go Home? Or: mash 10 gently caress management your csr doesn’t owe you poo poo
|
|
#
?
Apr 6, 2019 18:24
|
|
|
There are lots of reasons to want to know if your users like your product other than justifying ill treatment of support staff.
|
|
#
?
Apr 6, 2019 18:27
|
|
|
This escalated quickly
|
|
#
?
Apr 6, 2019 20:03
|
|
|
I'd say it escalated below average. 10/10
|
|
#
?
Apr 6, 2019 20:26
|
|
|
I like NPS because it makes sense conversationally but I guess I’m now oppressing people or something I feel like that’s quite an escalation
|
|
#
?
Apr 6, 2019 20:29
|
|
|
I've always hated those arbitrary, unit-less numeric scales. Rate my experience from one to ten? Well, it was perfectly ordinary, just like every other time I've made a similar [call, purchase, whatever], so I'll say five, I guess? Oops, turns out the people collecting the numbers think that means you hated it with burning passion. I've never understood that ten-point "pain scale" either. I get that bigger number = more ouch, but what distinguishes, say, a six from a seven? Can people really quantify it with that level of precision (scaled to some maximum "ten" that's going to be different for everyone anyway) while they're in agony?
|
|
#
?
Apr 6, 2019 21:00
|
|
|
Powered Descent posted:I've always hated those arbitrary, unit-less numeric scales. It's really not a thing that's useful as a one-off data point. The thing about those scales is they CAN be used over time to measure trends in an individual - if someone rates their pain as a 5 when they first come in but later rates it as a 7, you know something has gotten worse because their own subjective experience has changed, even if you don't really have any sense for how bad a "5" or a "7" actually is. The thing is that a lot of these systems get adopted outside of the original context they were developed for and are used incorrectly. Like review scales were always pretty arbitrary but at least you kind of knew that someone like Roger Ebert would have carefully considered exactly what star rating he was giving in a review, because he's a professional whose evaluation is supposed to go a bit more in depth than "like/dislike". But when you apply that to the general public? Most people don't have that same sense of professional obligation so the scale is meaningless to them.
|
|
#
?
Apr 6, 2019 21:13
|
|
|
People fundamentally don't understand that numerical data isn't automatically meaningful. It's been like that forever, and probably won't ever get better.
|
|
#
?
Apr 6, 2019 21:19
|
|
|
I remember reading that wine reviewers hate the 100 point scale, but consumers love it. I suppose you can take advantage of the fact that an 89 is statistically indistinguishable from a 90, but the prices may be very different.
|
|
#
?
Apr 6, 2019 21:25
|
|
|
Is this a good time to mention that my employer's Employee Satisfaction Survey asks the following question? I would recommend my closest manager to others 1(Do not agree in the slightest) 2 3 4 5 6 7 8 9 (Agree completely) Yes, my manager got a terrible result, something like 3 on average. And yes, anything below 7 is considered a fail. And yes, he's still on the job a year later.
|
|
#
?
Apr 6, 2019 21:50
|
|
|
The system works.
|
|
#
?
Apr 6, 2019 21:59
|
|
|
Hippie Hedgehog posted:Is this a good time to mention that my employer's Employee Satisfaction Survey asks the following question? This seems like the weirdest question because like, who "recommends" a manager, ever? Although I expect that's probably why the scores came in so low.
|
|
#
?
Apr 6, 2019 22:05
|
|
|
Somfin posted:Plus there's this hot new corporate bullshit thing called NPS which basically means that under a straight 1-10 scale, anything under a 7 is considered a negative. We use NPS scores for our customer service surveys and they're dumb as hell because ours is: 10, 9 = great! Promoter, 1 in the numerator, 1 in the denominator 8, 7 = neutral. 0 in the numerator, 1 in the denominator 6 and below = detractor. -1 in the numerator, 1 in the denominator. So neutral is bad. It's peak corporate bullshit.
|
|
#
?
Apr 6, 2019 22:50
|
|
|
we can talk data and statistical methods or whatever but I think we can all agree the best way to determine if NPS is valid is to do a survey of how likely people are to recommend its use to a friend
|
|
#
?
Apr 6, 2019 23:43
|
|
|
The core problem is that you have businesses using it to punish their employees. If they aren't absolutely perfect all the time and get perfect numbers (which literally never happens because, you know, people aren't perfect?) then they're rated bad and shafted on raises or whatever. It became just another excuse to stagnate wages.
|
|
#
?
Apr 6, 2019 23:54
|
|
|
I think this was from the movie about the Armenian Genocide, which was brigaded by Turkish nationalists whose opinion is generally "the genocide didn't happen but they deserved it but if we admit we did it we'll have to pay reparations". It had something like 40,000 1/10 ratings before it was released.
|
|
#
?
Apr 7, 2019 00:25
|
|
|
Spoeank posted:We use NPS scores for our customer service surveys and they're dumb as hell because ours is: What happens if you get a 0 in the denominator? 
|
|
#
?
Apr 7, 2019 04:44
|
|
|
Spoeank posted:We use NPS scores for our customer service surveys and they're dumb as hell because ours is: That is the actual standard, yeah. Powered Descent posted:What happens if you get a 0 in the denominator? You score infinity. The obvious correct response is to never send out the test in the first place.
|
|
#
?
Apr 7, 2019 04:48
|
|
|
The Cheshire Cat posted:This seems like the weirdest question because like, who "recommends" a manager, ever? No the scores were fine for most managers, just seldom above 7 because employees don't expect to answer a customer satisfaction survey... I hadn't heard about NPS before and realized now that it's really odd to use it for manager feedback. Everyone gets the summary of company-wide scores vs your unit's scores emailed to them. It's a weird kind of survey in several ways. They used the same scoring scale for all the questions but very few were "I would recommend X". Most were more generic like "I feel that <company> top management are doing a good job" or "I am proud to work at <company>", or "I am empowered to do the best work I can do" etc.
|
|
#
?
Apr 7, 2019 08:59
|
|
|
[turning to goon sitting next to me in the audience] the gently caress is NPS
|
|
#
?
Apr 7, 2019 09:30
|
|
|
Phlegmish posted:[turning to goon sitting next to me in the audience] the gently caress is NPS Net Promoter Score
|
|
#
?
Apr 7, 2019 09:33
|
|
|
|
| # ? Apr 28, 2024 18:54 |
|
|
Pretty much any measurement will work somewhat immediately after introduction. After a short while, people will optimize for the measurement instead of what the measurement is intended to reflect, and it becomes useless. See also: any performance-based bonus ever.
|
|
#
?
Apr 7, 2019 09:56
|
|