Huge datasets are the cornerstone of artificial intelligence (AI) systems,  including the large language models (LLMs) that power chatbots and other generative AI applications. These datasets are used to train AI systems to perform diverse tasks, such as analyzing text or images, or providing agricultural advice. However, AI developers often pay little attention to where the data originate—in most cases, from people and communities. Those data are frequently collected and used to train AI models without the permission or knowledge of the original sources. A related problem is datasets that underrepresent or exclude certain groups, such as women—AI models trained on such data can produce biased, discriminatory outcomes.

That is why there should be no AI without data ethics. The way data are collected, labeled, and used in AI apps should reflect values such as fairness, transparency, accountability, and respect for privacy. When these values are overlooked in AI development and deployment, people and communities may be harmed in various ways.

This approach is what guides the NDIZI project (Natural Language Processing to Develop and Innovate Zero-shot Intelligence), led by the Alliance of Bioversity International and the International Center of Tropical Agriculture (CIAT). (With “zero-shot learning,” AI models are trained to recognize objects or concepts they have not encountered before.) NDIZI aims to leverage AI tools to help smallholder farmers by collecting unstructured data about their crop preferences, analyzing it, and making information about their choices available to breeders for decision-making.

The importance of AI ethics

Each decision in the AI data lifecycle (from collection to cleaning, annotation, and structuring) matters (Figure 1). Any given pool of data is already a simplification of reality, and that simplification can create inequities if some individuals or groups are underrepresented or left out. How data are prepared, annotated, and structured for use in AI model training risks adding further problems. Finally, once an AI model is deployed, it interacts with people, sometimes reinforcing the inequalities present earlier in the data lifecycle or influencing decisions in unexpected ways.

Figure 1

Ethics, therefore, cannot be an afterthought. It must be built in from the beginning and revisited at every stage of development and deployment.

On a practical level, this means pausing at the very beginning of the development process to reflect on who may be affected by design choices, involving people with diverse perspectives, and being clear about what data are used, what assumptions are embedded, what risks may emerge, and who is accountable when harms occur. Ethical AI is not about chasing perfection, but about recognizing that technical decisions always have social consequences. When teams treat ethics as part of everyday practice, it becomes a habit that helps keep technology grounded in reality.

Practical case study: How AI can support farmers fairly

In a project in Tanzania, NDIZI collected audio comments in one-on-one interviews with 374 Swahili-speaking farmers across four regions in order to assess their needs. An AI speech recognition tool automatically transcribed the audio, and an LLM analyzed the transcripts to learn which crop traits matter most to the farmers—for instance, which varieties grow better in their regions or sell more easily in local markets. This information is made available to researchers and breeders, helping in variety release decisions and in establishing pipelines for new varieties.

But model performance can vary between different groups (e.g., men and women) and different locations (urban and rural)—here, for instance, depending on differences in speech patterns or pronunciation. NDIZI was created with this in mind, embedding ethics and human-centered design throughout the development process and in the field to identify and address these issues and make sure the technology works for everyone.

To achieve these important goals, we integrated AI design and data ethics into each part of the development process, following these principles:

• Listen first: Researchers were engaged early on to make sure the LLM extracts information from the transcribed audio that responds to real needs, and that an AI solution was making sense (as LLM-based outputs can sometimes be wrong or nonsensical).
• Plan for risks: The team identified possible harms, such as misuse of data or unintended consequences, and planned on how to prevent them. For example, we anticipated that the names of farmers might come up in the interview transcripts made by the AI tool, so we decided to carefully review them to remove any references to personal information.
• Include diversity: Farmers from different regions, genders, ages, and literacy levels were interviewed to ensure that the AI tool reflected their specific needs.
• Protect privacy: All recordings were collected with informed consent and securely stored.
• Reduce bias: Clear guidelines ensured that data labeling (e.g., farmer region, gender, age) and the interpretation of the LLM’s outputs were consistently applied in ways that did not lead to some groups being favored over others.
• Test for fairness: The AI tool was evaluated not just for accuracy, but also for whether it performed equally well across diverse groups.

Model results

Once the model had met these requirements and was up and running, what did it show about farmer preferences and AI performance itself in this context? Here are some takeaways:

• Accuracy (the ability of the model to extract the right crop traits out of the transcribed audio data) dropped the further away farmers were from urban centers, showing an urban-rural performance gap.
• The model had lower performance in extracting information from interviews with elderly farmers.
• Female respondents had lower transcription error rates, suggesting more intelligible recordings for men. Transcription error rates were higher for female respondents, suggesting that models recognized men’s speech more accurately.
• These findings are important because they show where improvements are needed in our process. If these AI tools—speech recognition and LLM—don’t work as well for certain groups, either individually or in combination, the solution is not to ignore the problem but, in many cases, to strengthen their representation in the data and refine the models. Some groups might be underrepresented in the dataset, which could affect the performance of the model for those specific groups.

The big lesson from NDIZI

Technology alone is not enough to meet the challenges of supporting smallholder farmers—and may end up creating more. AI tools created without attention to ethics and fairness risk reinforcing existing inequalities. But when designed carefully, AI can become a powerful tool to give farmers a stronger voice and ensure that agricultural innovations benefit everyone. If AI tools yield less accurate results for some groups, the solution is not to exclude them, but to find where the tools are erring and fix the problem. NDIZI shows that responsible design is not about slowing innovation but about making sure its benefits reach everyone.

Carolina Martins is a Digital Inclusion Consultant with the International Water Management Institute (IWMI); Berta Ortiz-Crespo is a User Experience Design Specialist with the Alliance of Bioversity International and CIAT. Opinions are the authors’.

Reference:
Lasdun, V.; Guerena, D.T.; Ortiz-Crespo, B.; Mutuvi, S.M.; Selvaraj, M.G.; Assefa, T. (2024) Participatory AI for inclusive crop improvement. Agricultural Systems 220: 104054. ISSN: 0308-521X. https://hdl.handle.net/10568/172411