Can AI Be Biased or Discriminatory?
The answer to whether Artificial Intelligence can be biased or discriminatory is categorical: yes, AI can reproduce and amplify patterns of discrimination and prejudice.
NinaKalkazu.org
Summary
The answer to whether Artificial Intelligence (AI) can be biased or discriminatory is categorical: yes, AI can reproduce and amplify patterns of discrimination and prejudice [19, 122]. Although automated systems are often associated with a promise of neutrality and technical objectivity, they learn and make decisions from historical data produced by human societies that already contain structural inequalities [19, 103, 123]. This article analyzes how algorithmic bias manifests across multiple social dimensions—such as gender, race, age, and social class—examining real case studies in crucial areas like recruitment, criminal justice, healthcare, credit lending, and facial recognition [20, 25, 28, 35, 37, 52]. A taxonomy of failures in machine learning models is presented [52, 77], along with mitigation strategies and emerging regulatory frameworks necessary to ensure equity in the digital age [39, 43, 45, 87, 129].
1. Introduction
The rapid integration of AI into decision-making in public and private sectors has been driven by the promise of greater efficiency and elimination of human subjectivity [122]. However, far from acting as neutral judges, algorithms operate as sociotechnical mirrors: AI does not create bias from scratch, but codifies, scales, and automates the asymmetries of the past [19, 103, 123].
Because machine learning models depend on vast volumes of data to identify patterns and predict behaviors, any historical bias, cultural stereotype, or underrepresentation present in the training sample is incorporated into the system's operational logic [19, 123]. The result is the perpetuation of systemic discrimination under the guise of supposedly impartial mathematical decisions [103, 122].
This problem spans various categories protected by international human rights norms, such as gender, race, ethnicity, age, and socioeconomic status [20, 25, 28, 124]. Understanding the nature of algorithmic bias is an indispensable step to prevent new technological architectures from intensifying oppression and digital segregation [103, 124].
2. The Genesis of Algorithmic Bias: How AI Learns and Amplifies Prejudice
Algorithmic bias occurs when a system generates results that disproportionately favor or harm certain social groups [19, 54, 123]. The genesis of this phenomenon is subdivided into several stages of the development pipeline [51, 77]:
Dependence on Biased Historical Data: If an institution's decision history reflects discriminatory practices from the past, the model will assimilate these practices as optimization rules [19, 36, 62, 123].
Reconstruction Through Proxy Variables: Direct elimination of sensitive attributes (such as race or gender) from the dataset does not prevent discrimination, as the model infers these characteristics through correlated variables, such as zip code or consumption history [70, 71, 123].
Feedback Loops (Feedback Loop Bias): When biased algorithmic decisions shape future reality, the system begins to learn from the impact it itself caused, creating a perpetual cycle of automated discrimination [61, 78].
Distributional Amplification: In generative models, the algorithm does not merely mirror the distribution of training data; it frequently exaggerates stereotypes, associating prestigious positions with white males and subordinate roles with minorities [30, 67, 68, 79].
3. The Multiple Dimensions of Algorithmic Discrimination
3.1. Gender and Age Bias in the Labor Market and Social Services
Gender and age discrimination by AI is widely documented in recruitment and public services [20, 23, 25, 36]:
Automated Resume Screening: Amazon's selection tool, trained on historical data from predominantly male hires, learned to penalize resumes containing the word "women's" (as in "women's chess club") and to devalue graduates from women's colleges [36, 62, 125]. On the Workday platform, a class action alleges systemic discrimination by age, race, and disability [25, 26, 60].
Age and Gender Prejudice: Studies on language models like ChatGPT and Gemma revealed that AI generates descriptions and evaluations that disadvantage older women [20, 64] and uses terms of greater clinical severity for men in social care notes while softening women's conditions [23, 58, 59].
Credit Decisions: The Apple Card algorithm offered significantly lower credit limits to women compared to their male spouses with identical or superior financial profiles [29].
3.2. Racial Bias and Facial Recognition: Algorithmic Racism
Algorithmic racism manifests critically in computer vision technologies and public safety [35, 69, 93, 103]:
The Gender Shades Project: Scientist Joy Buolamwini demonstrated that commercial facial recognition systems had error rates of only 0.8% for light-skinned men, but the rate rose to as high as 34.7% for dark-skinned women due to underrepresentation in training data [35, 69, 93, 105].
Wrongful Arrests in Criminal Justice: Facial recognition failures led to wrongful arrests in the United States, such as the unjust detention of Porcha Woodruff, a pregnant Black woman wrongfully detained in Detroit [69, 95].
Image Display Algorithms: Twitter's automatic image cropping algorithm repeatedly prioritized white faces over Black faces in thumbnail generation [37, 38].
3.3. Social Class, Credit, and Predictive Justice
The intersection between social class and race manifests in access to financial resources and criminal risk assessment [28, 70, 72, 126]:
Criminal Recidivism Assessment (COMPAS): The COMPAS algorithm classified Black defendants who did not reoffend as "high risk" at a rate of 44.9%, compared to 23.5% for white defendants, while labeling white repeat offenders as "low risk" [28, 72, 126].
Mortgage Discrimination: Studies in the U.S. real estate market revealed that algorithm-granted loans charged higher interest rates to Black and Latino borrowers with risk profiles identical to white borrowers [70, 127].
3.4. Disparities in Healthcare and Medical Diagnosis
The use of AI in healthcare has revealed serious failures with direct impacts on quality of life [22, 27, 37, 56]:
Skin Cancer Diagnosis: Models trained predominantly on light skin showed significant accuracy losses when analyzing lesions on darker skin (Fitzpatrick types IV to VI), increasing the risk of late diagnoses [22].
Healthcare Resource Allocation by Spending Proxy: A health algorithm affecting millions of patients used past medical expenses as a proxy for care needs; because Black patients generated lower expenses due to access barriers, the system classified Black patients as less needy than white patients with the same clinical severity [37, 53].
Psychiatric Treatment Plans: Tests with models like Claude, ChatGPT, and Gemini revealed that indicating African American race led models to omit indicated medication or suggest less effective treatment plans [27, 56].
4. Typology of Bias in Machine Learning
Algorithmic failures are organized according to the moment in the pipeline when they occur [52, 77]:
Type of Bias | Entry Phase | Practical Manifestation |
|---|---|---|
Training Data Bias | Data collection and curation | The model inherits unbalanced history (e.g., Amazon's HR tool) [36, 62, 77]. |
Representation Bias | Sampling and fine-tuning | Demographic underrepresentation (e.g., facial recognition and dermatological diagnosis) [22, 35, 69, 77]. |
Proxy Variable Failure | Feature engineering | Correlated attributes reconstruct protected categories (e.g., zip codes in credit) [70, 71, 78]. |
Feedback Loop Bias | Deployment and learning | The model learns from its own biased recommendations (e.g., Workday process) [61, 78]. |
Distributional Amplification | Generative process | The model exaggerates existing imbalance in the dataset (e.g., Stable Diffusion) [30, 67, 68, 79]. |
5. Mitigation Strategies and Legal Governance
Overcoming algorithmic discrimination requires an integrated approach [39, 43, 45, 87, 131]:
5.1. Technical Tools and Audits
Toolkits have been developed for bias auditing and mitigation [39, 40, 43, 45]:
Microsoft Fairlearn: Evaluates demographic parity metrics and adjusts models [43].
IBM AI Fairness 360 (AIF360): Includes over 70 fairness metrics and mitigation algorithms [45].
Aequitas: Auditing of predictive models in the public sector and justice [40].
Google What-If Tool: Interface for counterfactual analysis and parity testing [39].
5.2. Human Diversity and Representative Sampling
Effective mitigation requires demographically stratified sampling and the inclusion of diverse development teams and red-teaming networks to identify failures before deployment [81, 82, 84, 131].
5.3. Legislation and Governance
International regulation imposes clear limits on automated decisions [87, 88, 129]:
EU AI Act: Classifies AI in recruitment, credit, healthcare, and law enforcement as high risk, requiring audits and human oversight [87, 131].
GDPR and LGPD: Article 22 of GDPR and provisions of the LGPD, Brazil's data protection law, guarantee the right to explanation and to contest strictly automated decisions [129].
6. Conclusion
Artificial Intelligence is not inherently neutral [103, 122]. Because it is developed by human beings and fed by historical data, AI reproduces and amplifies prejudices of gender, race, and social class on an industrial scale [19, 103, 123]. To ensure fair technology, it is essential to adopt independent audits, diversity in development teams, and strict compliance with legislation protecting fundamental rights [87, 129, 131, 132].
