Key takeaways

• Co-design involving farming communities improves the reliability of AI agricultural apps, ensuring they reflect local knowledge instead of external assumptions or biases.
• Critical AI literacy helps farmers question, evaluate, and shape AI tools.
• Development approaches vary, requiring different types of critical AI literacy.

Imagine two scenarios. In the first, an existing AI chatbot is integrated into a mobile agricultural extension app. Farmers ask questions and receive instant responses. In the second, something more ambitious occurs: a custom chatbot is built from the ground up using local soil data, climate patterns, and indigenous knowledge. Both rely on years of extension practice; however, the first is relatively resource-effective and prioritizes rapid deployment through adaptation of a general system, while the second, though more resource-intensive, emphasizes contextual relevance for users and long-term value through locally grounded design.

Both scenarios are unfolding  in existing projects now. They reflect decisions agritech teams are making to scale their work. In either case, it is essential to engage the farmers and communities that will be using these apps in participatory or co-design processes in which they collaborate with app developers and provide feedback to ensure the final product can meet real-world challenges.

To participate effectively in the design process, farmers and other stakeholders (e.g., extension officers, agronomists) need critical AI literacy: they should have a basic understanding of how AI apps work, what they can accomplish, and their limitations. But the two scenarios require different kinds of knowledge and AI literacy—an emerging challenge that agritech app designers and development practitioners must take into account as AI extension apps scale up in low- and middle-income countries (LMICs).

What is critical AI literacy?

First, it’s important to explain more exactly what we mean by critical AI literacy; is it the same thing as digital literacy? Or is there something more needed for stakeholders to successfully participate in designing AI applications for agriculture that they will ultimately be using?

Critical AI literacy has been defined as the set of competencies that enable individuals to recognize what AI is, understand how it works, critically evaluate its outputs, and communicate and collaborate effectively with AI systems. With AI tools increasingly used in agricultural production in LMICs, ensuring meaningful community engagement is essential for building trust, supporting adoption, and enabling farmers to participate in how these systems shape their work.

That requires the ability to question, critique, and seek changes as needed in AI systems, particularly those that shape daily practices. Critical AI literacy aligns with work in human–computer interaction (HCI)—the study of how technologies are designed, used, and experienced in everyday life—as it examines how users form mental models of opaque algorithmic systems in practice, as well as scholarship in AI ethics and algorithmic accountability that emphasizes the importance of transparency and user understanding of AI systems, and the negative social impacts of opaque systems.

Thus, for farming communities, critical AI literacy is not about learning how to code or to create specific algorithms. It is about being able to interrogate the system, in particular, identifying and addressing potential flaws and weaknesses. For an AI tool providing agricultural advice, this means having the confidence and conceptual tools to ask:

• How does the system generate its answers (explainability)?
• What data is this model relying on, and is it relevant to my farm (transparency about data sources)?
• Why is the AI confident about this recommendation (trustworthiness/reliability)?
• What assumptions might this system be making about my soil, pests, or rainfall (transparency about biases)?
• Does this align with what we know from our community practices (local knowledge preservation)?

Without the capacity or opportunity to ask these questions, a farming community risks becoming a passive recipient of AI advice. When stakeholders can question and contribute without barriers, they remain active participants in how AI operates in their fields.

In addition, research on agricultural technologies shows that farmers prefer clear, actionable advice rather than technical jargon. But when system responses are simplified to address this issue by removing background explanations, they can also become harder to question. The real risk is not complexity, it is opacity.

When farmers cannot see how or why a system produces a recommendation, they are left with only two options: blindly trust it or ignore it. They cannot meaningfully participate in a process of integrating the system into their work. That in turn depends on the role they play in the design of these systems. To address that issue, let us return to the two scenarios.

Scenario A: Plugging an existing AI into an agricultural extension app

Here, agritech company X is integrating an existing large language model (LLM) into a mobile app for the farming community. The vision is simple: farmers ask questions in their own words and receive answers instantly; the app is a kind of virtual extension officer. Similar approaches are emerging globally as organizations explore how generative AI might expand advisory services to rural communities.

In this scenario, the farming community does not shape the model itself, but still must decide how to use it. Rather than framing the design engagement around technical elements of the system, the discussion is grounded in the situations these stakeholders routinely encounter in practice. Through this feedback, designers can translate the farming communities’ experiences into concrete design decisions about how the system should behave.

In this context, the farming communities provide input on:

• Where and how the model would be built into the app user interface so that it naturally fits into existing practices (e.g., at what point in diagnosing a crop issue one would turn to it).
• How the system should communicate, using language, tone, and examples that align with how agricultural problems are already discussed in the field.
• The kinds of questions they would ask such a tool in real situations, expressed in their own words rather than technical vocabulary.
• The kinds of responses that would feel useful, trustworthy, confusing, or misleading when addressing typical farming challenges
• The situations in which they would ignore the system and seek human expertise.

Stakeholders are not expected to reason about prompts, guardrails, or workflows. Instead, designers work to incorporate these work-situated insights into system behavior. What farmers and extension officers require in this context is not knowledge of how the model is built, but critical literacy about how the model behaves. They need to understand that:

• The LLM system was not trained on their specific agricultural data.
• Its outputs may be incorrect or expressed with unwarranted confidence.
• Recommendations must be evaluated against local conditions and lived experiences.
• Global assumptions embedded in the model may not always hold in local contexts.

Their literacy, therefore, is focused on interpretation, evaluation, and critique of outputs and not on model development.

Scenario B: Building a custom agricultural extension app

Here, we imagined something more ambitious: an agritech company, Y, building its own custom app from scratch. Instead of relying on a general-purpose AI model trained on global data, this system would be built from the ground up using local data, including years of local agricultural knowledge, soil data, climate patterns, farmers’ lived experiences, and indigenous farming wisdom. Researchers in human-computer interaction have long argued that AI systems perform best when they meaningfully incorporate local knowledge and community-defined expertise.

Here, the stakes are higher, with greater opportunities and challenges. These stakeholders are not just interacting with an AI tool; they are shaping the foundation on which the model will be built.

This requires deeper forms of critical AI literacy, such as:

• Understanding how the data behind the model is relevant to their context.
• Defining which categories, labels, or distinctions are meaningful within the local farming context, such as what counts as “healthy” versus “unhealthy” crops, how pest severity should be classified, or which environmental conditions are most relevant for assessing farming outcomes.
• Identifying biases or gaps in the data collection process.
• Helping to determine what “accuracy” or “good performance” means in their agricultural reality.
• Articulating failure cases, when the model should not be trusted.
• Shaping evaluation frameworks grounded in the farming community’s values and local expertise.

This is literacy not only in the use of AI but in the process of its creation.

Why this matters

These scenarios are fictional only in the sense that they do not describe any particular project. But organizations are now exploring similar possibilities in many countries. Practitioners are eager to leverage the power of data-driven algorithmic systems, and rightly so. But as the participatory design community reminds us, the real question is not just how we can integrate AI, but who gets to shape it. Participatory design and participatory AI scholars warn that when communities are excluded from the design process, technology often encodes designers’ assumptions (i.e., biases) rather than users’ realities. This concern is even more pressing with AI systems, whose behavior can shift over time with data updates or model retraining. And this is where critical AI literacy becomes essential.

The real insight

Across both scenarios, one truth stands out: Meaningfully co-designing AI systems with farming communities requires investing in expanding the critical AI literacy of farmers and other stakeholders. Without critical AI literacy:

• They may trust AI too much or too little.
• They may disregard their own expertise because they assume “the AI must know better.”
• They may not recognize when a system is failing them.
• They may be unable to influence the assumptions, workflows, or data pipelines that eventually harden into design decisions.

However, with critical literacy:

• Stakeholders become partners, not subjects, in AI development.
• Their local knowledge becomes a legitimate design input, not an afterthought.
• They can advocate for their needs, question errors, and identify blind spots.
• The AI becomes grounded in real practices, not hypothetical abstractions.

Conclusion

In this post, we explored why critical AI literacy matters and what it looks like in practice,  showing how the kind of literacy the farming community needs depends on the role they play in shaping AI systems. In a future post, we will turn to the practical questions many agritech design teams might be facing:

• When should AI literacy scaffolding be introduced in the co-design process?
• How can AI literacy be taught in ways that are accessible, culturally grounded, and meaningful for these farming communities?

Because if AI is going to shape the future of agriculture, then the farming community should contribute to shaping the future of AI.

Chioma Chigozie-Okwum is PhD Candidate in the Department of Computer Science and Engineering, Lehigh University, Bethlehem, Pennsylvania; Ameen Jauhar is Data Governance Lead at CABI; Eliot Jones-Garcia is a Senior Research Analyst with IFPRI’s Agrifood Innovation and Resilience Unit. Opinions are the authors’.