Data science and machine learning rely on data. Commercial applications using data must follow legal and ethical guidelines for gathering, processing, and retaining customer or public data. The topic I propose is a review of data governance practices, machine learning ethics, and how informatics professionals, machine learning engineers, systems designers, and policy activists can monitor data governance, privacy policies, and machine model behavior.

Laws such as the GDPR and CCPA have defined the rights of customers to control what companies know about them (European Union, 2016; California Legislature, 2018). These and similar laws ensure that people have the right to know what data exists about them and the right to be forgotten. The new EU Artificial Intelligence Act (AI Act), which came into force on August 1, 2024, is likely the first of many new laws that will attempt to regulate ML and AI (European Union, 2024). Like the GDPR and CCPA, it will impose requirements on data collection and governance, but it will also now impose requirements on machine learning model governance.

Because the EU AI Act is a new law, its consequences may not yet be immediately apparent, but it addresses issues that have concerned machine learning practitioners, ethicists, and privacy activists for years. In 2016, Microsoft released the Tay Twitter bot, which began tweeting antisemitic and racist messages within 24 hours (Prahl & Goh, 2021). Buolamwini and Gebru (2018) examined bias in commercial facial recognition systems from IBM, Microsoft, and Face++, which consistently showed significant accuracy disparities based on gender and skin type..

In recent years, interest in generative AI has exploded, with ChatGPT being one of the first LLMs to reach the status of a killer app (OpenAI, n.d.; Roumeliotis & Tselikas, 2023). Slightly before this surge in popularity, Bender et al. (2021) issued their famous warnings that large language models could display dangerous behaviors if trained on uncurated large datasets scraped from the internet. Since then, many scientists, activists, and concerned citizens have expressed fears and doubts about ethical problems and intellectual property rights issues related to large models trained on datasets containing unacknowledged copyrighted material and toxic content (Birhane et al., 2024; Gebru, 2019; Raji et al., 2020)

There are no easy answers to the ethical problems posed by ML and AI because AI ethics reflects deeper issues that have always existed in human society. There are voices claiming that racism, prejudice, and other forms of discrimination have been resolved and no longer require laws or public policies. However, a simple search engine query on any sensitive topic will likely reveal content filled with hatred. That AI tends to be an accurate mirror of human society is not an excuse. AI is now a part of society, and humans will learn to imitate the behavior of AI just as AI learns to imitate human behavior. Any ML system allowed to replicate bigoted human behavior without intervention is certain to amplify the harm humans already inflict on each other.

There is no excuse for a machine learning engineer or system designer to neglect precaution. Now may be our best, perhaps only, chance to set an ethical and human trajectory for automated systems to follow for the rest of our lives. We have a responsibility in the industry to avoid the easy path of unconsciously profiting from uncontrolled AI. This requires investing in curated datasets collected with the consent of data authors and oversight by a diverse external community. We must monitor deployed models’ behavior and measure not only their profitability but also their societal benefit.

Article Analysis

Gebru and Buolamwini (2018)

Buolamwini and Timnit Gebru (2018) investigated bias in commercial AI gender classification systems by analyzing their performance across different gender and skin tone subgroups. The study found that darker-skinned females experience significantly higher error rates compared to lighter-skinned males, revealing systemic bias in these systems. The findings highlight the need for more diverse and balanced datasets, transparency in AI development, and improved algorithms to ensure fair and equitable performance across all demographic groups in facial recognition technology.

Timnit Gebru and Joy Buolamwini are two of the most influential researchers in AI ethics. Buolamwini is the founder of the Algorithmic Justice League (Algorithmic Justice League, n.d.). Timnit Gebru was famously driven out of Google in 2020, supposedly because of the then-unpublished paper, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” (Heikkilä, 2020; Bender et al., 2021).

Boyd (2021)

Boyd (2021) investigates how “Datasheets for Datasets,” a tool for dataset documentation, can help ML engineers recognize and understand ethical issues in training data. Through a study where ML engineers were tasked with ethical decision-making using problematic datasets, it was found that those provided with Datasheets were more likely to notice ethical concerns early and frequently, demonstrating how such tools can enhance ethical sensitivity and support ethical decision-making in AI development.

Boyd (2021) follows up on suggestions from Gebru et al. (2021), Bender and Friedman (2018), and Yang et al. (2018) for ethical disclosure guidelines and evaluates how machine learning engineers and data scientists become sensitive to ethical issues related to datasets and machine learning methodologies. This study is important for anyone seeking to build a data science team based on ethical principles.

Prahl and Goh (2021)

Prahl and Goh (2021) analyzes corporate communication strategies following AI failures using 23 case studies. It explores various response strategies such as apology, corrective action, denial, and a new “mirror strategy,” which attributes AI errors to societal issues. The study highlights how AI crises differ from traditional ones in terms of accountability and suggests that new crisis communication frameworks may be needed to effectively manage AI-related issues.

While informatics professionals, data scientists, and engineers may not directly handle PR issues caused by AI disasters, they will still be called upon to explain what went wrong when such events occur. Therefore, it is wise to consider potential failures during the design and development of ML algorithms and to document steps for addressing issues before they arise. It is only common sense that those responsible for a system should be prepared to apologize when necessary and show a willingness and ability to identify and fix mistakes.

Reddy et al. (2020)

Reddy et al. (2020) proposes a framework to address ethical, regulatory, and safety concerns related to AI in healthcare. It highlights challenges such as bias, privacy, and trust, and suggests a governance model centered on fairness, transparency, trustworthiness, and accountability. The model aims to guide AI’s integration into clinical care by ensuring responsible use, ethical compliance, and safeguarding patient rights, thereby fostering trust among healthcare providers and patients in AI-driven tools. 

While the proposed framework is aimed at AI in healthcare, the principles of fairness, transparency, trustworthiness, and accountability apply to AI, ML and data science in any context. Specialists in other industries than the healthcare industry should find this framework adaptable.

References

Algorithmic Justice League. (n.d.). Algorithmic Justice League. Retrieved September 1, 2024, from https://www.ajl.org/

Bender, E. M., & Friedman, B. (2018). Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6, 587–604. https://doi.org/10.1162/tacl_a_00041

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

Birhane, A., Dehdashtian, S., Prabhu, V., & Boddeti, V. (2024). The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 1229–1244. https://doi.org/10.1145/3630106.3658968

Boyd, K. L. (2021). Datasheets for Datasets help ML Engineers Notice and Understand Ethical Issues in Training Data. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1–27. https://doi.org/10.1145/3479582

Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency (pp. 77-91). PMLR. https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

California Legislature. (2018). California Consumer Privacy Act of 2018 (Assembly Bill No. 375). https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201720180AB375

European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A02016R0679-20160504&qid=1532348683434

European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 12 July 2024 on artificial intelligence (Artificial Intelligence Act). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

Gebru, T. (2019). Oxford Handbook on AI Ethics Book Chapter on Race and Gender. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1908.06165

Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.

Heikkilä, M. (2020, December 4). Google reportedly forced out a top AI ethics researcher, and thousands of her colleagues are protesting. MIT Technology Review. https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/

Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature machine intelligence, 1(9), 389-399. https://arxiv.org/abs/1906.11668

OpenAI. (n.d.). ChatGPT. OpenAI. https://openai.com/index/chatgpt/ (Accessed: August 29, 2024).

Prahl, A., & Goh, W. W. P. (2021). “Rogue machines” and crisis communication: When AI fails, how do companies publicly respond? Public Relations Review, 47(4), 102077-. https://doi.org/10.1016/j.pubrev.2021.102077

Raji, I. D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., & Denton, E. (2020). Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing. AIES 2020 – Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 145–151. https://doi.org/10.1145/3375627.3375820

Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020). A governance model for the application of AI in health care. Journal of the American Medical Informatics Association, 27(3), 491–497. https://doi.org/10.1093/jamia/ocz192

Roumeliotis, K. I., & Tselikas, N. D. (2023). ChatGPT and Open-AI Models: A Preliminary Review. Future Internet, 15(6), 192-. https://doi.org/10.3390/fi15060192Yang, K., Stoyanovich, J., Asudeh, A., Howe, B., Jagadish, H. V., & Miklau, G. (2018). A Nutritional Label for Rankings. arXiv.Org. https://doi.org/10.48550/arxiv.1804.07890