tag:blogger.com,1999:blog-9093436161326155359.post3655762173725402318..comments2024-03-28T06:43:02.954+00:00Comments on Variable Variability: Publish or perish is illegal in Germany, for good reasonVictor Venemahttp://www.blogger.com/profile/02842816166712285801noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-9093436161326155359.post-35678999300764166402016-10-04T13:44:49.041+01:002016-10-04T13:44:49.041+01:00Goodhart's Law is important to mention. The ab...Goodhart's Law is important to mention. The above problem is not just a scientific one, but a destructive micro-management technique that is quite common. I like the BBC Reith lecture on this a lot: <a href="http://www.bbc.co.uk/radio4/reith2002/lecture1.shtml" rel="nofollow">A question of trust</a>.<br />Victor Venemahttps://www.blogger.com/profile/02842816166712285801noreply@blogger.comtag:blogger.com,1999:blog-9093436161326155359.post-23672366967280625402016-10-03T21:50:59.940+01:002016-10-03T21:50:59.940+01:00Victor
Thank you for this article. I have two thi...<br />Victor<br /><br />Thank you for this article. I have two things to say: the first refers to the concept of 'Quality' and second to Goodhart's Law.<br /><br />In Zen and the Art of Motorcycle Maintenance, the protagonist seeks a definition of 'Quality': struggling to find if it can be specified exactly or if it is just a matter of opinion. In the end he arrives at the idea that 'Quality' is a measure of the amount of attention that another human being has paid to something.There are some quantitative aspects to quality - but the key thing is that is essentially a human to human communication. In scientific literature it requires a human being to read a paper to determine its quality - no matter how many citations it has.<br /><br />The Second point is Goodhart's Law<br /><br />https://en.wikipedia.org/wiki/Goodhart%27s_law<br /><br />This states that once one uses a simple measurement as a proxy for the thing one means to measure, people will 'game' the system and the measurement will become worthless. This kind of 'quantitative thinking' is like a cultural infestation - thank you for speaking so clearly about its ill effects.<br /><br />My comment on the effect is here:<br /><br />https://protonsforbreakfast.wordpress.com/2015/10/05/target-culture-and-the-road-to-hell/<br /><br />Michael<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9093436161326155359.post-23417351529467358922016-09-14T17:49:34.171+01:002016-09-14T17:49:34.171+01:00Some years ago I was asked (indirectly) to advise ...Some years ago I was asked (indirectly) to advise on an incentive bonus scheme that a small computer company wanted to set up for its workforce.<br />The initial idea from the finance department was to grade everyone on a number of criteria;- Reliability, initiative, punctuality etc on a scale from 1 to 10 and then combine the scores in some weighted system to give the amount of bonus a person would get.<br /><br />I had two critiques of this'<br />1)- The combination of different measures, some numerical like punctuality and some subjective assessments like reliability was mixing apples and oranges, and then forcing it into a ten point scale that was arbitrary. My experience with other quality assessment systems has led me to the view that three levels is the most that is applicable to quality assessment. Basically you have; Good/Normal/Bad. Or in systems were negative assessment is not acceptable, Excellent/Good/Satisfactory. You can argue about the borderlines, but if your criteria are to be effective the number in the central group, the normals, will be the majority. The extremes, Good/Bad will be a smaller number than the average/normal group. Any attempt to make finer distinctions is invariable a futile exercise. It provides a false sense of increased accuracy without any real improvement in the information.<br />2)- The way the company worked was it sent teams out to install hardware. The membership of teams was fluid and was altered by the need for specific skills or numbers and friendship links. So a person might have a high score in one team, but low in another depending on the type of work and how well they got on with the team leader and assessor. Because the bonus was based on the individual score but was dependent on team dynamics it was intrinsically flawed, rewarding a person for something the team as a whole might be responsible for. That could be divisive.<br /><br />The feedback I got was not positive. The idea that only three grades of assessment might be possible for quality was rejected. It seemed obvious to those considering the scoring system that by having a ten point scale they had increased the accuracy and usefulness of the information, not imposed and arbitrary and misleading metric.<br />The point about personal assessments being affected by the team context was taken as an indication that the personal score could be used to decide when to move someone from a team to improve their score.<br /><br />It is extremely difficult to make any accurate measure of the quality of work done by people who are not working on a fixed, defined task. If creativity of scientific or clinical knowledge is involved it get harder. That does not prevent those who wish to measure quality from finding some aspect of the activity that can be reduced to a numerical measure and then asserting that the number is a meaningful and accurate proxy for the quality they claim to be measuring. And that the finer the gradations of the number the more accurate the measurement.<br /><br />Historically the one thing that does have a strong correlation with successful scientific research is throwing money at it. It is noticeable how often modern biochemicals, drugs and organic compounds have as date and location of the first isolation and synthesis, the big German investment in Chemical research in the 1920s.<br /><br />I do not know how well the incentive scheme worked at the computer company. It was of course a means of capping the wage bill. By having a bonus scheme the percentage of the company earnings that went to the workforce could be controlled by using the incentive scores to distribute a smaller slice of the company cake.<br /><br />I concluded some time ago that the accountants had taken over the asylum.izenhttps://izenmeme.wordpress.com/noreply@blogger.com