Forum

Lack of evidence fo...
 
Notifications
Clear all

Lack of evidence for unconcious bias

3 Indlæg
1 Brugere
0 Likes
11.9 K Set ialt
torsten2
(@torsten2)
Medlem
Tilmeldt: 6 år siden
Indlæg: 7
Topic starter  

Copenhagen 12 Dec 2018

 

To the LERU Secretary-General, Professor Kurt Deketelaere,

 

Please find below a review of LERU Advice Paper No. 23 of January 2018, “Implicit bias in academia: A challenge to the meritocratic principle and to women’s careers – And what to do about it”, which I have undertaken out of interest in the problem at hand. To my great surprise, the LERU paper turned out to be seriously lacking in quality.

The paper claims that there is "ample evidence that implicit bias is a (if not the) major cause of less favorable assessment of women’s academic capacities in research, teaching and leadership" (p. 4), and purports to review the available evidence and show "how implicit bias plays a role in processes where important career impacting decisions are made" (p. 3). I have read the literature which is referenced in support of this statement, and must conclude that none of the publications support the notion of unconscious or implicit bias against women. In addition to this general misrepresentation of the results of the referenced studies, the paper contains several other examples of inaccuracies and misrepresentations as you will see in the table below.

This is not to say that unconscious bias does not exist - it may or may not, and everyone is entitled to their opinion about this matter. The LERU paper is, however, not presented as an opinion paper but as an academic review of the available evidence. This sets standards for quality which the paper clearly does not meet.

Before going into details with each study, let me highlight the LERU paper's definition of implicit or unconscious bias. The paper states:

"In this paper the term implicit bias is used to mean that human beings are not neutral in their judgement and behaviour but instead have experience-based associations and preferences (or aversions) without being consciously aware of them". (p. 3, my emphasis).

and further

"A large body of research points to one significant problem: women’s and men’s academic achievements and potential are often unintentionally evaluated differently. This phenomenon is known as bias, and can be defined as a mechanism of skewed information processing, under the influence of context and accumulated experience. It is bias (‘implicit bias’, also referred to as ‘unconscious bias’) ...." (p. 6, my emphasis)

To be absolutely clear, the paper uses the words "bias", "implicit bias", and "unconscious bias" synonymously. In the following, I shall use "implicit bias" and "unconscious bias synonymously", and use the word "bias" to signify any kind of bias whether conscious or unconscious.

 

 

 

 

Table 1. Review of publications pertaining to the LERU paper's main claim

Study

Content of study and LERU paper representation of study

Conclusion

Steinpreis 1999 (1)

The LERU paper uses data from a publication labelled Steinpreis et al. 1999 in text. No such reference is found in the reference list. Assuming that it is (1), the following can be said:

 

Two CVs were sent to 238 academic psychologists. The CVs came from a real-life scientist but names were changed to traditional male and female names. The participants were asked to evaluate the applicants' hireability, teaching, research and service experience.

 

There was an effect of the applicant's gender (as indicated by the name) for the temporary job application, whereas there was no effect of the applicant's gender on the evaluation of the tenure application.

 

The study did not assess the impact of participants' gender attitudes. No attempt was made to study unconscious bias. Whether the bias was conscious or unconscious cannot be determined and therefore the study cannot be said to contribute evidence for either hypothesis.

 

Moss-Racusin 2012 used much the same methodology but added assessment of participants' gender attitudes, see below.

 

The LERU paper states the following about the article:

 

“Research shows that female psychologists (Steinpreis et al., 1999) .... are just as likely to discriminate against female candidates as their male counterparts (when assessing CVs with randomised female versus male names). Steinpreis et al. (1999) show that negative bias especially affected women at the early-career stages”

 

There was, however, no negative bias in late-career stage. The statement that "negative bias especially affected women in early-career stages" is therefore inaccurate.

There is no support in this publication for the existence of unconscious  gender bias.

 

The LERU paper’s presentation of the study results is inaccurate also in other aspects.

Moss-Racusin 2012 (2)

In this study an application for a position as laboratory manager was sent to a sample of 127 professors. All professors received the same application, the only difference being that the application had been randomly assigned a male or a female name. Participants were asked to rate the applicant's competence, hireability and potential salary and mentoring that would be offered. The professors' "subtle bias" against women was measured with questions like

 

·          "It is easy to understand the anger of women's groups in America."

·          "It is easy to understand why women's groups are still concerned about societal limitations of women's opportunities."

·          "Over the past few years, the government and news media have been showing more concern about the treatment of women than is warranted by women's actual experiences."

 

The male-named applications were rated higher on all parameters than the female-named applicants.

 

Further analyses showed that the bias related to gender attitude was present only for female applications, not male. In other words: professors who had a negative attitude towards women as measured with the questions above, had a bias against the female applicants. Looking at the actual content of the questions that assess the bias, it is hard to argue that the bias is subtle. It is in fact rather clear that a person who is of the opinion that "the government and news media have been showing more concern about the treatment of women than is warranted by women's actual experiences" and/or who disagrees that "It is easy to understand the anger of women's groups in America", is not unconsciously biased against women - this person is openly and consciously biased.

Whereas the study demonstrated bias against the applications which were assigned a female name, there is no evidence in this publication to support a hypothesis of unconscious  bias.

 

 

van den Brink 2011a (3)

This study is mainly based on in-depth interviews with 64 professors who had served on committees appointing professors in the Netherlands. The interviewees offered their views on the evaluation process. The paper brings a number of verbatim quotes, which seem to indicate overt, conscious bias against women. There is no attempt to quantify the observations, and no attempt to go behind the explicit statements given by the interviewees. The study does not address unconscious  bias.

 

Thus, there is no support in this publication for the existence of unconscious  gender bias.

There is no support in this publication for the existence of unconscious  gender bias.

van den Brink 2011b (4)

This study is mainly based on interviews with 21 scouts for candidates to fill positions as professors in academic medicine. The analysis “revealed a dominant pattern of recruitment by invitation by male scouts, leading to three gender mechanisms of inclusion and exclusion through formal/informal networking”. The paper brings a number of verbatim quotes, which seem to indicate overt, conscious bias against women. The study does not address unconscious  bias.

 

Thus, there is no support in this publication for the existence of unconscious  gender bias.

There is no support in this publication for the existence of unconscious  gender bias.

MacNell 2015 (5)

Two instructors, one female and one male, taught an online anthropology/sociology course to 4 groups of students. Two groups were informed of the correct gender of the instructors, the other groups were told the opposite gender. Regardless of the actual gender of the instructor, the students rated the perceived female instructor lower than the perceived male instructor. The findings were, however, not statistically significant at the conventional 5% level. There was no attempt to study unconscious  bias.

 

The LERU paper references the study like this:

 

"In an online experiment, teachers with an ascribed male identity were systematically rated higher than female teachers (McNeal [sic] et al., 2014), although their actual

performance was the same"

 

This statement is inaccurate considering that the findings were statistically non-significant.

There is no support in this publication for the existence of unconscious gender bias.

 

The LERU paper’s presentation of the study results is inaccurate also in other aspects.

Knobloch 2013 (6)

243 scholars (MA or PhD level) were asked to rate 15 abstracts from a scientific conference. The abstracts were labelled with constructed author names that indicated male or female gender. The participants' Gender Role Attitudes were assessed with questions like  “It is more important for a wife to help her husband’s career than to have one herself” and “A wife’s most important task is caring for her children.”

 

There was a gender bias in that the scientific quality of abstracts was rated higher for male than for female authors. The effect vanished when Gender Role Attitude was included in the statistical analysis model. In other words: the gender bias could be attributed to the participants' explicit conservative gender norms.

There is no support in this publication for the existence of unconscious  gender bias.

Maliniak 2013 (7)

Citations of 3,000 articles were analyzed. After controlling for a large number of background factors, articles published by women were cited less often that articles published by men. Maliniak et al. offer two explanations: 1. that women cite themselves less than men, and 2. that there are networks of men that cite each other and women that cite each other.

 

"Men tend to cite male-authored articles more than female-authored articles and women tend to cite female-authored articles more than male-authored articles58 This difference alone could account for the gender gap in citations since the number of men in IR is significantly higher than women."

 

There is no attempt to study unconscious  bias.

 

The LERU paper references the study as evidence that "papers, ostensibly authored by males, acquire higher scores on quality than papers ostensibly authored by females". The use of the word ostensibly implies that the design is experimental along the lines of e.g. Knobloch 2013, but this is not the case. Further, the papers did not acquire scores.

 

This is an analysis of the actual number of citations of real-life publication. There is no experiment, it is a purely observational study.

There is no support in this publication for the existence of unconscious gender bias.

 

The LERU paper’s presentation of the study results is inaccurate also in other aspects.

 

 

West 2013 (8)

The proportion of females among PhDs and the proportion of women among all with tenure in different time periods was compared to the proportion of female authorships in the same periods. The women had less authorships than should be expected based on their share of PhDs but more than should be expected based on their share of tenure. That is the main finding in Table 2. In some subgroups men had more first and last authorships than expected based on their overall proportion of  authorships.

 

The study did not take into account the seniority of the women. The proportion of women in  academia increased substantially during the studied time period. Without thoroughly analyzing the effect of this as well as tenure, this data cannot be meaningfully interpreted. Even if the statistical analysis were made more meaningful, it is not clear how one would interpret for example a finding of less publications per year among women than among men, everything else being equal. Do men work harder? Do men produce more quantity but less quality? The dataset simply does not hold information to answer such questions. The authors of the publication clearly acknowledge this saying: "the data do not allow us to uncover mechanisms that produce the gender disparities we find" (p. 6).

 

The LERU paper references the study as evidence that "papers, ostensibly authored by males, acquire higher scores on quality than papers ostensibly authored by females". The use the of the word ostensibly implies that the design is experimental along the lines of e.g. Knobloch 2013, but this is not the case. Further, the papers did not acquire scores.

There is no support in this publication for the existence of unconscious gender bias.

 

The LERU paper’s presentation of the study results is inaccurate also in other aspects.

Zogmaister 2008 (9)

The study investigates the impact of various priming stimuli on for example IAT responses to questions about group loyalty. There is no mention of evaluation of researchers, academic networks or for that matter gender.

 

The LERU paper references the study as evidence that "Particularly in evaluations of early-stage researchers, in-group loyalty and academic networks plays a major role (cf. Zogmaister et al., 2008)".

 

Since there is no mention of researcher, academic networks or gender in the publication, the text in the LERU paper is a misrepresentation of the study.

There is no support in this publication for the existence of unconscious gender bias.

 

The LERU paper misrepresents the study also in other aspects.

Heilman 2008 (10)

This is a literature review of research about job appraisals. There is no original data.

 

The LERU paper states the following about this and the next study

 

“For women to be deemed equivalently hireable, competent, or worthy of promotion in male gender-typed professions, they must demonstrate a higher level of achievement than identically qualified men (Heilman & Haynes, 2008; Kaatz et al., 2014)”

 

The review does not discuss women's hireability, competence, or promoteability in relation to their level of achievement. There is no discussion of unconscious  bias.

 

The text in the LERU paper misrepresents the content of the publication.

There is no support in this publication for the existence of unconscious gender bias.

 

The LERU paper misrepresents the study also in other aspects.

Kaatz 2014 (11)

This is a review paper that does not present any new data. One of the reviewed papers is Heilman et al. 2008, which is summarized like this:

 

“Further experiments find that, before women are deemed equivalently hirable, competent, or worthy of promotion in male gender-typed professions, they must demonstrate a higher level of achievement than identically credentialed men”

 

Thus, the LERU paper’s text about Heilman et al. is taken verbatim from Kaatz et al. whereby LERU, instead of correcting the error, propagates the misrepresentation. One must suspect that the authors of the LERU paper did not read Heilman et al.

There is no support in this publication for the existence of unconscious gender bias.

 

 

The above table contains all references cited in the LERU paper to support its main claim that "implicit bias is a (if not the) major cause of less favourable assessment of women’s academic capacities in research, teaching and leadership".

Whereas some of the above papers demonstrated gender bias, none of them attempted to demonstrate a contribution from unconscious bias. Such studies cannot be said to contribute evidence for either hypothesis, therefore they do not provide evidence for the existence of unconscious gender bias. In the case of Moss-Racusin (2) and Knobloch (6), the bias was explained by the participant’s explicit gender norms which leaves no room for unconscious bias. The two studies by van den Brink (3;4) clearly addressed conscious bias only. Two publications were review papers (10;11), one of which quoted the other for statements that were not to be found in the publication.

It is concluded that none of the studies provide evidence of unconscious bias as defined by the LERU paper. The main claim of the paper is thus unsubstantiated by the scientific evidence. In view of the lack of reservation as to the validity of the paper's conclusion and the misrepresentation of the published data, I suggest that the paper is harmful to the scientific credibility of the research universities that constitute the League and should be withdrawn.

 

Respectfully,

 

Torsten Skov

MD, PhD in epidemiology, BA in philosophy

Mørkhøj Parkalle 20E

DK-2860 Søborg

Denmark

E-mail: torsten.skov@begrund.dk

 

 

Reference List

 

       (1)    Steinpreis RE, Anders KA, Ritzke D. The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: A national empirical study. Sex Roles 1999;41(7/8):509-28.

       (2)    Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty's subtle gender biases favor male students. Proceedings of the National Academy of Sciences 2012 Oct 9;109(41):16474.

       (3)    van den Brink M, Benschop Y. Gender practices in the construction of academic excellence: Sheep with five legs. Organization 2011 Jul 29;19(4):507-24.

       (4)    van den Brink M. Scouting for talent: Appointment practices of women professors in academic medicine. Social Science & Medicine 2011;72(12):2033-40.

       (5)    MacNell L, Driscoll A, Hunt AN. Whats in a Name: Exposing Gender Bias in Student Ratings of Teaching. Innovative Higher Education 2015;40(4):291-303.

       (6)    Knobloch-Westerwick S, Glynn CJ, Huge M. The Matilda Effect in Science Communication: An Experiment on Gender Bias in Publication Quality Perceptions and Collaboration Interest. Science Communication 2013 Feb 6;35(5):603-25.

       (7)    Maliniak D, Powers R, Walter BF. The Gender Citation Gap in International Relations. 2013;2013/08/28(4):889-922.

       (8)    West JD, Jacquet J, King MM, Correll SJ, Bergstrom CT. The role of gender in scholarly authorship. PLoS One 2013;8(7):e66212.

       (9)    Zogmaister C, Arcuri L, Castelli L, Smith ER. The Impact of Loyalty and Equality on Implicit Ingroup Favoritism. Group Processes & Intergroup Relations 2008 Oct 1;11(4):493-512.

     (10)    Heilman ME, Haynes MC. Subjectivity in the appraisal process: A facilitator of gender bias in work settings. In: Borgida E, Fiske ST, editors. Beyond common sense: Psychological science in the courtroom. Blackwell Publishing Ltd.; 2008.

     (11)    Kaatz A, Gutierrez B, Carnes M. Threats to objectivity in peer review: The case of gender. Trends Pharmacol Sci 2014 Aug;35(8):371-3.

 

 


   
Citer
torsten2
(@torsten2)
Medlem
Tilmeldt: 6 år siden
Indlæg: 7
Topic starter  

LERU Office
Minderbroedersstraat 8
3000 Leuven, Belgium

10 January 2019

Dear Dr. Torsten Skov,

Thank you for your comments on the LERU advice paper 23. We have carefully considered your comments and
arrive at the conclusion that you seem to have misunderstood the central issue of bias. This misunderstanding
on your part, moreover, leads you to levy criticism against the LERU paper for claims that the paper does not
make. Thus, we respectfully can only conclude that our paper is systematically misrepresented in your
comments. Please see below for a detailed account. We consider this correspondence now closed.

Yours sincerely,

the authors of the LERU Advice paper 23

Bias
We define bias as information processing skewed by automatic looking for and/or interpreting information that
is consistent with one’s own beliefs. These beliefs are implicitly present and can be made explicit, for example by
endorsing or rejecting explicit statements about lack of equality between men and women. We carefully use the
term "implicit bias", not "unconscious bias" customary as a technical term in UK contexts (with only a brief
reference to the latter). The issue of “conscious” vs. “unconscious” is not discussed in our paper.
In contrast, you understand "unconscious", a word we avoid, in the literal sense, and accuse us of citing evidence
for bias which is not unconscious, although we refer to this evidence as showing (implicit) bias.

References
All the references mentioned give evidence for bias and this is how we mention them in the text:
Steinpreis et al. (1999): thank you for pointing out the omission in the reference section. Our statement that
“negative bias especially affected women at the early career stages” is correct; the study shows main effects of
gender on judgments of hireability, and absence of main effects (p > .05), but still effects on tenurability (cf. Fig.
1 and page 521).
Moss-Racusin et al. (2012): we do not discuss any hypothesis about unconscious bias in the paper.
Van den Brink (2011): we do not refer to this research in connection with unconscious bias.
MacNeil et al. (2015): apart from not discussing unconscious bias, we speak about “systematically rated”, not
“significantly rated”, even though the authors themselves conclude that “students rated the male identity
significantly higher than the female identity”, in contrast to your statement.
Knobloch et al. (2013): we do not discuss this work in connection with unconscious bias.
Maliniak et al. (2013), West et al. (2013): something went wrong during the editing process here. Our text should
read: papers, authored by males, acquire higher citation scores than papers authored by females. Moreover,
men predominate in the prestigious first and last author positions and in single-author papers (Maliniak et al.,
2013, West et al., 2013). Thank you for pointing this out. We do not discuss unconscious bias in this context.
Page 2 / 2
Zogmaister et al. (2008): our paper states that “in-group loyalty and academic networks play a major role”.
Zogmaister et al. discuss this as “favouritism” (and how it can be diminished). Clearly, favouritism results from
in-group loyalty in academic networks. We do not discuss any unconscious bias here.
Heilman & Haynes 2008, Kaatz et al. 2014: the reference is correct and the wording is in agreement with Kaatz et
al. (2014). There is no discussion of unconscious bias here.


   
SvarCiter
torsten2
(@torsten2)
Medlem
Tilmeldt: 6 år siden
Indlæg: 7
Topic starter  

18 Jan 2019

Dear Dr. Maes,

Thanks for your answer to my review of the LERU paper. I have given it some thought and made a few comments below. I am aware that you consider the correspondence closed, thus I expect no reply. But you are of course welcome.

In my review I took great care to describe the proposed hypothesis in LERU's own words. The LERU paper makes no distinction between implicit bias and unconscious bias. I quoted the text in the review, and I will do it again:

"A large body of research points to one significant problem: women’s and men’s academic achievements and potential are often unintentionally evaluated differently. This phenomenon is known as bias, and can be defined as a mechanism of skewed information processing, under the influence of context and accumulated experience. It is bias (‘implicit bias’, also referred to as ‘unconscious bias’) ...." (p. 6)

So, as I said in the review, the LERU report uses the words "bias", "implicit bias", and "unconscious bias" synonymously.

I also made it abundantly clear that I used the words "implicit bias" and "unconscious bias" synonymously. You may substitute "unconscious bias" with "implicit bias" throughout the review, the arguments and conclusion will remain the same. So your reply to me that I misunderstood the whole thing because I used the term "unconscious bias" instead of "implicit bias" is, if I may say so, on the far side.

You are offering a clarification of what you mean by implicit bias which is slightly different from the definition in the paper, but since you do not indicate that you will revise the paper, I do not see that this revision or clarification is of any consequence for the evaluation of the paper as it stands.

Even granted your revision/clarification of what implicit bias is, my comments would be almost the same. It is in effect very simple: none of the referenced studies distinguish between implicit and explicit bias and the studies' setup do not allow any such distinction to be made post hoc. The studies may not contradict your theory that implicit bias plays a major role in impeding women's careers in academia, but they do not support the theory either. So the studies may show bias (some of them do) but whether it is explicit or implicit, one cannot know.

You are offering the interpretation of the studies that the findings are due to implicit bias, but you seem to be blind to the fact that an assumed interpretation does not turn the studies into evidence. They can only be interpreted as evidence for implicit bias if you are already convinced that implicit bias is the explanation - which is circular reasoning.

You may ask what difference it makes whether bias is implicit or not? For me: none, in the sense that I believe you could remove the words "implicit" and "unconscious" from the whole paper. You could argue that there is bias and suggest all the remedies R1 to R9 to overcome the problem. None of them rely on the bias being implicit.

So the question is, why did you introduce the theory at all? The paper says: "something stops women on their way to the top in academia, and we seek to explain this phenomenon with reference to implicit bias" (p. 6). And indeed you conclude that implicit bias does play a major role. But why "implicit bias" and not just "bias"? After all there is some evidence of bias in the studies you reference but no evidence specifically of implicit bias, so why this insistence that the bias is implicit?

In the absence of your explanation, I can only guess (and I emphasize that this is only a guess): footnote 1 on page 6 mentions Kahneman's "Thinking fast and slow" as inspiration for the paper's definition of unconscious/implicit bias. If I may elaborate a bit on this, Kahneman proposed a two-system theory of the human mind: System 1, which employs heuristics for quick decision making that does not need conscious processing, and System 2, which examines the available information more thoroughly, and makes deliberate and reasoned decisions. Importantly, System 2 can overrule the automatic reactions of System 1; it can, as it were, overcome the biases of System 1. But it is, in Kahneman's words, slow and lazy. We are, in other words, steeped in our prejudices unless we make an  effort to overcome them.

Most of our behavior is decided by System 1. We wouldn't get much done if we were to rely on System 2 for everything. And System 1 relies on our prejudices, for example about gender roles, what women can and cannot do. So here you are: the biases that we see are due to System 1.

This model fills a gap that seems to have developed as expression of overt sexist views have become increasingly rare in academia. Few professors would vent the opinion that women are unsuited to be professors. Something else must be at play than explicit bias to explain why women have less favorable careers in academia than men, and here the theory of implicit bias is a godsent. Even though people do not express biased views, they may still make biased decisions for example when hiring staff.

I have no issue with the above reasoning - except that it is still only a theory, a hypothesis. You at LERU must also have felt that something was missing since you ventured on supporting the theory with empirical evidence from the literature. That was a mistake that got you into circular arguments and materially false statements such as "There is ample evidence that implicit gender bias plays a role in academia" (p. 3).

What you could have done instead is the following: you could have presented the same studies and said they showed bias against women in academia. If you wanted to preempt objections that this is an outdated discussion or that the findings must be wrong because one does not hear sexist comments in the public sphere or even in private conversations any more, you could have added a section about the psychological theory of implicit bias and simply said that absence of explicit bias does not rule out that implicit bias plays a role when people make decisions about whom to hire. If gender bias is demonstrated, it is no argument against the findings, that there does not seem to be any explicit bias around.

I have posted my review of the LERU paper on a discussion forum (begrund.dk). Unless I hear otherwise from you within a week I shall assume that you have no objections to me posting your reply letter there.

Regards

Torsten Skov

 


   
SvarCiter
Del: