FutureIQ Knowledgebase

Things Go South When You Trust DATA Too Much - Goodhart's Law - FutureIQ

3,439 views Wait, is this logic right? • Mar 22, 2023
Open in YouTube
Slog Reference: Goodhart's Law

Description

Goodhart's law suggests that when the measure becomes the primary target, it ceases to become a good measure. Here is Navin & Shrikant explaining Goodhart's law with different examples and proving how data can be manipulated and can cause more adverse effects than intended.

Learn about the Cobra effect, Mcnamara's fallacy and other examples of Goodhart's laws in this simple, fun and short video.

Hope you enjoyed FutureIQ by Navin Kabra and Shrikant Joshi. Do hit us up on Twitter:
@ngkabra http://twitter.com/ngkabra
@shrikant https://twitter.com/shrikant

Listen to The FutureIQ Podcast on the podcast provider of your choice: https://tapthe.link/FutureIQRSS

Watch other episode:
Learn about the law that helps you detect fraud: https://youtu.be/JHA2QpJKGN4

Links:
Cobra effect: https://tapthe.link/CobraEffect
Keith Rabois' paired indicators: https://tapthe.link/KRPairedIndicators

Chapters:
Chapters:
00:00 Introduction
00:26 The cobra effect
03:04 Example 2
04:10 Applications of Goodhart's law
04:58 Example 3
05:35 Is data bad?
06:31 Example 4
08:37 Measurements vs Gut instincts
09:53 Example 5
11:00 Example 6
12:20 Conclusion

#thefutureiq #cobraeffect

Related Slog Matches

Goodhart's Law

Fuzzy Text

100.00

Transcript

hi my name is srikant and with me today is Naveen Cabra and we are here to explore some principles of life so Naveen what do you have for us today so shrika today I want to talk about something called good Hearts law which talks about how when you are measuring something uh I mean usually that's a good thing having data to make decisions yeah but sometimes it can go horribly wrong okay so let me start with an example uh during the times of the British Raj in India in Delhi there was a problem of too many Cobras so what the British did was they gave an incentive for uh people if they bring in a dead
Cobra you get a uh incentive you get actual money okay so they were measuring actual dead cobras so the way to reduce cobras they decided was to find dead cobras was to measure a dead cobras and then pay makes sense now the problem is that when you are measuring something like this what happens is that there is a real life thing which is the actual cobras which are causing problems and then there is this measure movement in a spreadsheet somewhere and the measurement becomes the primary target right so what these people started doing uh I mean initially whole bunch of people killed cobras brought it took money then some people realized that they can make money this way so they
started breeding cobras oh and bringing those cobras that they bred to make money so the measurement became the target instead exactly okay right and people started focusing on the measurement rather than the real problem I can see how that could be a problem uh in in real life in in the current world scenarios as well for example uh in in job appraisals maybe am I right all all kinds of problems so we'll take a bunch of examples but let me finish uh this example because it gets even worse oh it does oh because what happened was that quickly the British realized that this is not working the way they intended because people are trying to game the
system yeah to do this so they canceled the program okay and what happens to the cobras that were already bred at that point the people just released them so this program ended up increasing the number of cobras okay okay that is terrifying and that is scary not just because I am terrified of snakes but imagine Cobra is being released in the wild which was not supposed to be there in the first place so the I mean the reason you should really be terrified this is still you know like a really old example maybe even an apocryphal but uh there is a page on Wikipedia called the Cobra effect which gives a very long list of
examples like this where in real life it has caused problems so for example uh Canadian system right the healthcare system is uh nationalized they were paying incentives to orphanages to take care of Orphans right so you got about a half a dollar per month per orphan okay but the second tell me this created more orphans no [Laughter] it's not that bad but it's still bad right yeah um psychiatric hospitals were paid two and a half dollars per mental patient so more than 20 000 orphans got put in psychiatrist hospitals and labeled as mental patients just so that people could earn money on them okay am I am I bad for thinking that it is better to
have movement patients than orphans I'm really sorry you're thinking this way but that is the first but anyway anyway coming coming back to the the concept of good heart Salah that we were discussing and of course we'll put the link in the description for you to check out the Cobra effect it's on Wikipedia so if you do a quick search you'll find it but coming back to goodnight's law um how does good heart's law apply on a day-to-day basis yeah so um the thing is that um everywhere right from education where you are uh you know everything is based on marks from entrances to colleges where it is all based on an entrance exam two
companies where it is either um Etc right everything uh I mean earlier it used to be just if your boss thinks you're doing a good job you will get it but now suddenly we are in the age of data and everything is supposed to be data driven yeah which is a good thing usually yeah okay it reduces subjectivity but often it can be a problem so another example from real life is that to reduce bugs in software you give an incentive to software programmers to per bug fixed yeah I can see how that can go wrong pretty quickly because there is the joke among software developer circles that you solve one bug and create four yeah
so no I mean actually there is a Dilbert where Molly is saying oh I just you know uh created uh wrote uh this for myself right so just add bugs yeah yeah right so this is a problem everywhere and you have to be careful all right so um this this basically reminds me of something I heard as a student in college there are three kinds of Lies there are lies there are damned lies I'm sorry for my language there and then there is statistics so uh does good hearts Lord then say that statistics is the cause of everything that is problematic in terms of evaluating with data well not exactly right I mean uh you might get the impression that I am
saying stop using statistics and data for things like that but what good heart's law says is that when a measure becomes a Target that's when it ceases to be a good measure right so you have to prevent it from becoming a Target otherwise having data is a good thing so what you have to do is be careful of how you are using your measures example yeah so um one example is that a company does not want fraud right a credit card company so customer service is given incentive to weed out the frauds and only have good customers right now according to goodhart's law what is likely to happen what no so what happens actually is that
the customer service people start treating each and every customer as if they are a criminal right now this is you know and if it's not it's not me when I call into the customer service hotline and it's not me who gets treated like a criminal all the time it's everybody right yeah hopefully but uh so uh here is the thing as in if they are going to treat everybody like criminal they will of course weed out the real criminals but they will also build out a lot of real customers and yeah yeah and so uh who is I don't know the pronunciation okay okay he has a very good rule for this he says that you should do measurements but
have an opposite measurement also for every measure to prevent it from being misused okay okay so the example in terms of this credit card thing uh so if you are treating every customer as a potential fraudster what is going so that is going to cause this good measure to improve huh what measure goes down because of this customer satisfaction correct so now measure both measure number of frauds measure customer satisfaction and make sure this goes down without causing this to go down right this is called paired indicator so if you search on Google for paired indicators you will find a bunch of examples and how to do it don't worry we'll put in a lot of these resources
that we're discussing in the descriptions as well so you can check the descriptions for more information on those but why you are saying this I realized there is an argument to be made for not measuring anything at all because if measures are going to cause problems why measure them at all why not why not just go by got Instinct I guess no so uh usually our gut instinct can be very wrong we have all kinds of biases we have all kinds of ambiguities and we tend to be overconfident right this is something we will cover in a lot of our future episodes uh but the important thing is that in general the switch from instinctive decision making to data
driven decision making has been by and large good okay just don't want to overdo it right so what today's episode should be telling you about this is that use data but don't make the data the only source of the decision right have have humans looking at the data have humans who are experienced they should be allowed to override the data now when I think about it there are many real world scenarios where this has the potential to go horribly and possibly comically wrong yeah right yeah like I mean the biggest real world scenario is the Vietnam War where Robert McNamara at the uh I think he was the Secretary of Defense at that time he
said I am going to take all decisions in a data driven manner so everything about the Vietnam War was based on data data comes in and then people who have no experience with actual War are taking decisions purely based on the data and that's why U.S did so badly in the war okay in fact it is such a bad example uh and lots of people are dead because of that it's has a name of for itself it's called McNamara's fallacy which is over Reliance on data without relying on human experts all right we'll of course put that link in that description as well but how about how about how about in business does it does good heart's
law appear in in business scenarios as well of course business scenarios uh all the time right okay I mean you put sales Target and people uh do various things for sales uh I mean I I just thought of a very fun example it's not directly business but you can imagine how it will come in business right there was a famous archaeologist and he was looking for hominid skulls and the local Javanese people he gave them incentive uh for every piece of hominid skull that they found uh he would pay them money right you can't even imagine how this is going to go wrong I can I think I can imagine because when they wanted cobras they killed cobras
no but women it's culture like it has to be you know 70 000 years old and all that what they did he was paying them per piece so they took full skulls and broke them into a little pieces business you you have one of your incentives is you know number of customers number of sign ups or something like that and a sales person has a bad incentive you can imagine how they can start doing funny things with such yeah in fact I think I may have come across some of these people myself as uh previous uh jobs let's let's not go there because we'll probably be watching this but uh yeah uh and and I
think I have seen a couple of chain stores also doing this like fast food chains he was doing something like this although I'm trying to reconcile this uh right any other Salient points significant points that you want to mention about the good Hearts law no I think I just want to repeat the main fundamental thing which is that yes data is a good thing do not give up on data but make sure that it doesn't become a Target all right make sure that that is not the only thing that the decision is based on so either used pair indicators so pair it with a to make sure that it's not being gamed or put humans uh on top
of it so that the final decision is by an expert who understands the full context usually the problem is that the data doesn't understand the full context and that's a good note to end this video on data doesn't understand the full context if you want the full context check the description I am Shri Khan this is Naveen and this was future IQ thank you