In terms of accountability for major corporations, there is no incentive or pressure for companies to consider the environment. Companies that boast about their environmental achievements mainly do it to appease the public and maintain trust in their stockholders. Even at the level of public pressure, there is still some benefit; however, companies can pick and choose what to address, leaving out certain ethical topics. My goal in building this project is to explore ESG (Environmental, Social, and Governance) reports to identify topics that were left out and need to be addressed.

Background & Motivation

I’m a Computer Engineering BS student at the University of California, Davis. I’ve been very interested in computers and hardware ever since I was little. I want to create a project with real-world impact that showcases what I have been learning.

The Data Ethics Research Internship application through Data x Direction was intriguing, so I applied and ended up paired with Program Director Jesse Parent. He encouraged me to find an interesting topic to create something I’m truly passionate about, which led me to considering the nature of evaluating organizations environmental impact and how technology could be used to further this aim.

This led me to being curious about ESGs; I was unfamiliar with this topic at the time, so I began a deep dive into what they are and how they work. ESG reports are documents released by companies that showcase environmental and social impacts. What caught my eye was that almost all major companies publish this sort of document, but they used different formats. After noticing this, I wanted to create a system that regulates these reports.

Motivation: Why ESG Reports?

Originally, I thought that there had to be a company that already does something similar, and I was right. There are scoring frameworks from Yahoo, S&P Global, and other sources, but they were not what I envisioned building. They focused more on investor reports and were scored by private analysts, who did not disclose their process. I wanted something very transparent, with a stronger focus on environmental responsibility.

The more I looked into ESG reports, the more confused I became about the reasoning behind these documents. They seem to be promoting sustainability while being transparent to the audience, but they have a lot of flexibility with how they do it. Since this is technically a voluntary document, they can “cherry-pick” topics they want to highlight while downplaying areas that they don’t address. This lack of enforced structure makes it hard to meaningfully compare companies.

A major corporation can produce a very polished report that gives the appearance of sustainability without the underlying data to back it up. This document comes down to who has the stronger branding and a better team to highlight their initiatives. Just to clarify, I’m not denying the environmental impacts these companies have had; I’m highlighting how they can push a strong sustainability initiative without addressing broader issues.

I considered using Natural Language Processing (NLP) models as a good way to address this issue. Instead of relying on curated narratives and secretive analysis scores, an automated system can analyze the ESG reports and determine what has been addressed/omitted. I envisioned something that could also categorize the topics addressed and evaluate how companies address them.

By shifting ESG reports from being subjective towards a more meaningfully quantified report, I wanted to create something that has a standard format and highlights what companies report on – or what they are leaving out. I was seeking a way to highlight what the ESG Report should have been from the beginning, in making sure these companies actually contribute to improving the environment, or at least offer some reality check on what they are declaring to the broader public.

This became the foundation of my project, in which I envisioned building something that promotes transparency and focuses on the environment rather than financial analysis.

Project “Designing the Policy Analyzer”

Project Goal

The Ethical Policy Analyzer (EPA) was designed to prioritize transparency and interpretability. A core driver was to identify what data was currently available and presented, and what data could or should be available; this contrast would yield an analysis that helped the viewer understand the company’s impact on the world. Doing so in a transparent and standardized way would help move this beyond less clear reporting or investor-driven analysis.

During the development phase, I wanted to consider many dimensions of ESG reporting. This would be crucial to rubric construction, so I started by reviewing extensively the various environmental factors reported or accounted for in the domain of large corporate reporting.

We decided on a preliminary target outcome: a tool that would quickly and transparently or uniformly grade a company’s ESG performance to reveal which metrics the company was doing better at and where it could improve its environmental impact and accountability.

Data & Approach

I started by creating a collection of various ESG reports from FAANG corporations. This data was found directly from the sustainability sections of their websites. Using these documents, I browsed through all the reports to find similarities and formatting structure. I primarily selected Meta and also looked at the historical evolution of ESG reports to see what information they add each year.

There were very mild similarities across the reports; their formatting differed, but they all attempted to answer similar questions. This was one of the leads I had to create a grading rubric to evaluate companies that have prepared sustainability documentation.

Since most of these ESG reports are published as PDFs with various visual effects, tables, and charts, I needed to clean and prepare the data. I solved this by extracting raw text from the PDFs, removing headers, footers, and page numbers, and creating a consistent text section for NLP analysis.

To organize this project’s development, we used a shared GitHub board to track and plan it. We used this as a central place to store and organize the data collected to structure our project.

Designing the Rubric

After data preprocessing, I started thinking about how to structure my rubric and how to account for the different formats and styles of ESG disclosures. We approached this by focusing on the questions and concerns the companies were trying to address with these reports. Through reviewing multiple reports across major companies, we noticed a recurring theme.

Even though the formatting was different, they all addressed similar environmental domains. These common themes formed the foundation, and I also included other areas I think are highly relevant to environmental benefits. Using this, I created six high-level categories:

1. Energy & Electricity Use

2. Carbon / GHG Emissions

3. Water Use

4. Data Center Efficiency

5. Forward-Looking Statements & Commitments

6. Framework Alignment

The first four address measurable environmental impacts, and the last two are on company alignment with ESG frameworks. Since FAANG companies are predominantly technology companies, we selected categories that recur across the companies' ESG reports1. We manually analyzed the sustainability reports and surveyed common disclosure metrics. The final two categories were to show how companies frame their position on sustainability and how much they adhere to recognized reporting structures. This allows us to analyze both the company-professed intentions and their alignment with reporting frameworks.

Figure caption: As in the figure above, our weights favored the actual quantifiable metrics, with Energy & Electricity use and Carbon / GHG Emissions as having the highest weight of 25/100, whereas Framework Alignment was weighted the lowest with 5/100.

The rubric was built on a simple 0-3 scoring scale for each category. A 0 score meant there was no mention, and each natural-number increase indicated more context about the topic, with a full score of 3 indicating that the topic was addressed with data and future plans. In general, the rubric was:

• 0: No mention

• 1: Mention

• 2: Mention with some details

• 3: Mention with data and reference to future plans

For example, if a company mentions “renewable energy” once without any supporting scientific data, it would receive a 1. However, if there were supporting data, such as energy usage metrics (e.g., total electricity consumption in MWh), a description of the renewable energy mix, and future plans, it would receive a score of 3. If any of the metrics are missing, the points gradually decrease.

Mentioning means: appearance of keywords identified for the appropriate category. For example, keywords for the Energy & Electricity Use category include:

Rubric Progression and Iterative Refinement

The base rubric was a solid starting point, yet as ESG reports are not uniform, we encountered some issues. After some analysis, it became clear that we needed to be more specific. Companies were using similar language; however, they weren't providing sufficient detail, and it was hard to gauge what constituted sufficient detail.2

We also reviewd the Worldwide AI Ethics Review paper by Correa et al. This study is a meta-analysis of 200 different AI governance policies and ethical guidelines from around the world. The authors identified 17 resonating principles that frequently appear in these documents, such as transparency, accountability, and sustainability. A key finding of the paper is the “AI ethics boom,” where an explosion of voluntary guidelines has created a fragmented landscape with no single global consensus.

Since the original rubric was broad, we made it more detailed and added subcategories. V1 of the Rubric had no subcategories, and thus, our results and scoring felt lacking. This was very helpful for categories such as energy use, emissions, and water management because there is a diversity of methods and metrics that address these areas.

To make the rubric more operational, we implemented a keyword searching feature. Most ESG reports are too long and full of fluff, so to get exactly the line we need, we used a keyword search. Since we are going to use NLPs to analyze the grading structure, this part saved us a lot of time and improved the quality of our evaluations. For each category and subcategory, we developed ESG report terminology and unit indicators.

After getting a keyword match, we needed to figure out how to capture context and specificity. This is important because, under our rubric, merely mentioning it is insufficient and should be scored lower than an ESG report with more detail. This part was going to rely heavily on NLPs and training them according to the rubric.

Technical Details: NLP Pipeline

Now that we had a proper rubric to go by, it was time to build a lightweight NLP pipeline to apply it to any ESG report. As stated, ESG reports are long, unstructured, and inconsistent with the information they provide. The goal was to create a website where simply dropping in a PDF would generate a detailed scoring report based on the rubric.

The first part of this was text normalization of the ESG report to extract raw text. Most ESG reports include various graphs, charts, and other compelling ways to present the data. To address this, a normalization step is performed before raw text extraction, removing whitespace and line breaks and formatting the graphs. This ensures that energy usage tables, water content, and emission values are properly analyzed.

Once we have text, we can perform sentence segmentation. The document is divided into individual sentences, and instead of analyzing the whole report, only the sentences are analyzed. It doesn’t matter whether it’s a sentence with a lot of fluff or a concise one; both are analyzed in the dataset.

There is also a keyword and topic extraction step that identifies words related to specific categories. This step helps identify which keywords appear most frequently and narrow down the sentences we need to analyze.

Following keyword extraction, there is also a metric extraction step that follows the same idea. The goal is to pinpoint the quantitative metrics so that we can determine if they have data-backed disclosures.

There is also a lightweight semantic matching step that runs with this process because the same thing can be interpreted differently. The example phrases and keywords in the rubric are cross-checked with this scan using the Jaccard similarly. This simple technique helps the model express the same idea even with different wording to narrow the sentences.

After all the preparation, we can start the topic classification and run it through the rubric scoring engine. Here, we assign sentences to categories and subcategories, which concludes the NLP analysis. We then pass it through the scoring engine, which evaluates the sentences based on the rubric and assigns a score to each category. This is then averaged to produce a final output showing all categories, scores, and reasons for each score.

Final Version

Reflections & Future Directions

The hardest part of this project was defining and measuring “Ethics,” which is interpreted in different ways. The rubric is my attempt to structure scoring for ESG reports, but it needs constant improvement. Every time I analyzed a new report, I came across new edge cases. Each time I encountered something, I had to go back to the rubric and tweak it to make it as accurate as possible. Even in the final product, the rubric is not complete; it was limited to the scope and time available during the internship.

Next Steps and Limitations

If this project were to continue, there are several areas for improvement.

One area could be to present the evidence more effectively. As shown in the report, the evidence is just condensed and included; if we could direct it to the exact section, it would be better. There could also be a replacement for the analyzer model that uses neural networks instead, or in addition to, of NLP, since neural networks can be more accurate. Another area would be improving the distinction in weightage for the existing metrics (Categories 1-4) and the professed alignment or forward-looking statements (Categories 5-6); in that, different weights may be more or less reasonable for a user’s actual needs.3 Additionally, the project would benefit from improvements for the user, such as having an accessible database of existing company reports and associated scoring.

Further specific issues are discussed below.

The Problem of Self-Reporting Disclosures

The fact that the EPA evaluates declared disclosures was also an inherent limitation of the project. When a report passes through the analyzer, its output is not a true evaluation of the company’s environmental impact, as it relies on self-reporting and limited pattern recognition. A company that is particularly environmentally conscious might score lower because its ESG reports are less comprehensive than those of a company that does less but has more expansive reporting. Company motivation depends heavily on what they are willing to share, and we can’t get the full context just from a report that’s not completely standardized.

Scoring Buckets

Another consideration is the utility of the current scoring buckets. The overall scoring categories are currently defined as the integers 0, 1, 2, and 3. Through iterative testing, we found that the boundary between 0 and 1 is the most reliable distinction the analyzer makes: either a topic is present, or it is not. The boundaries between 1, 2, and 3 are less stable. This is consistent with a well-documented pattern in NLP and annotation research: as the number of ordinal categories increases, scoring consistency tends to decrease (Crible & Degand, 2019). Finer-grained scales demand subtler judgments from annotators and automated systems alike, and those subtler judgments are more susceptible to variation across documents, phrasings, and report styles.

Research on text annotation has similarly found that rating scale design is itself a source of bias, and that simpler scales with clearly differentiated levels tend to produce more reliable results than scales where middle categories blur together (Alm & Sproat, 2005; Kiritchenko & Mohammad, 2017). For this reason, we have considered collapsing the scale to three categories: 0 (no mention), 1 (mention without substantive elaboration), and 2 (mention with data, context, or forward-looking commitments). This would sacrifice some descriptive resolution but would produce scores whose distinctions are more meaningful and reproducible, particularly for an automated pipeline that relies on keyword matching and lightweight semantic similarity rather than deep reading comprehension.

Final Thoughts

Overall, this project substantially changed my views on sustainability and ethics. The main thing I learned was the importance of continuous refinement. Since ethics is so broad, there was no single correct answer, and throughout this process, I just picked a path and kept refining it. My mentor played a huge role in the project because it involved a lot of decision-making that needed a second opinion to work towards a solution. At the end of the day, this was my attempt to improve the fight for sustainability, and regulating ESG reports is a definitive step towards that goal.

Subscribe now

We live inside stories more than we tend to notice. Not just novels or films, but the quieter, infrastructural narratives that shape how we interpret information, understand ourselves, and decide what to do next. Dashboards tell us whether we are “on track.” Feeds tell us what matters. Recommendation systems quietly suggest what kind of person we might be becoming.

Increasingly, these narratives are not authored solely by humans.

Across education, mental health, productivity tools, and media platforms, AI systems are being positioned as aids to reflection, sense-making, and decision-making. They summarize, reframe, prompt, nudge, and sometimes advise. In doing so, they don’t just process information — they participate in the stories through which people understand themselves and the world.

This blog, hosted by DigiNEST (Digital Narratives and Emerging Story Technologies) under JOPRO’s Society Ethics Technology and Mental Health Paradigms & Practices working groups explores a deceptively simple question:

What happens to human agency when machines begin to mediate our narratives?

To begin answering that, we need better language than the usual binaries of “AI replaces humans” versus “AI is just a tool.” A useful starting point comes from a recent research paper that introduces a concept the authors call reflective agency, and a framework for designing around it.

From Automation to Reflection

Much of the public conversation around AI and agency focuses on outward action: who is making decisions, who is accountable, who controls the system.

But many of the most intimate AI systems don’t act for us. They act on our reflection.

Think of journaling assistants, therapy chatbots, career-planning tools, or systems that summarize your habits and patterns back to you. These systems don’t directly decide what you should do. Instead, they shape how you interpret your own thoughts, experiences, and options. They sit between the raw material of experience and the meanings we make from it.

This is the domain explored in “Reflective Agency: Ethical and Empirical Framework for AI-Mediated Self-Reflection Systems”, a paper presented at AAAI/ACM AIES 2025 by Minsol Kim, Wendy Wang, Jennifer Long, Rosalind Picard, Nathan Barczi, and Pattie Maes (MIT Media Lab and Wellesley College). The authors argue that when AI systems mediate self-reflection, they affect a distinct and under-examined form of agency: a person’s ability to interpret themselves and act on that interpretation.

Agency, in other words, is not just about control or choice. It’s also about meaning-making.

What Is Reflective Agency?

Reflective agency, as the paper defines it, refers to a person’s capacity to make sense of their own experiences, interpret their thoughts, emotions, and values, and use that understanding to guide future action.

This kind of agency is narrative in nature. We reflect by telling ourselves stories about who we are, what happened, why it mattered, and what it suggests we should do next. Reflection is not a single moment of introspection; it is an ongoing process of weaving experience into coherent meaning.

When AI systems participate in that process, they don’t simply mirror a user’s thoughts. They frame, structure, and sometimes reinterpret them. A reflective system might rephrase your journal entry in more “neutral” language. It might highlight certain themes while downplaying others. It might prompt you to view a problem through a particular psychological lens. Each of these design choices subtly influences the narrative a person constructs about themselves. The ethical question is not whether this influence exists — it clearly does — but how it is exercised, acknowledged, and constrained.

Five Principles for Keeping the Person in the Story

The paper’s central contribution is the Reflective Agency Framework (RAF): five design principles grounded in phenomenology and Aristotelian virtue ethics. Rather than prescribing specific features, RAF identifies the conditions under which reflective agency is preserved or eroded.

Internal Origination. The user must remain the initiating source of reflection. Insights and meaning must arise from within, not be imposed externally. Drawing on Aristotle’s concept of prohairesis — deliberate, value-guided choice — and Heidegger’s notion of authenticity as “being one’s own,” this principle cautions against AI systems that generate unsolicited interpretations or preemptively reframe a user’s experience. The system should invite reflection, not substitute for it. In the paper’s empirical study of twenty journaling app users, 16 out of 20 preferred to initiate reflection themselves — always or mostly — and four chose an equal balance. No respondents preferred AI to lead. One participant’s comment captures the concern plainly: they felt journaling loses its value when something or someone else tells them what they are supposed to feel. A separate finding reinforces the stakes: 13 of 20 participants disagreed with the statement that an AI-rephrased journal entry would still feel like their own thoughts.

Calibrated Responsiveness. AI systems should dynamically adapt their level of support based on the user’s emotional and cognitive state, providing guidance when needed while stepping back when autonomy is preferred. This draws on Aristotle’s phronesis — practical wisdom, the capacity to discern how to act appropriately in complex, context-sensitive situations — and Don Ihde’s analysis of how technologies sometimes recede into the background and sometimes come to the fore. The paper’s survey data shows the pattern is emotion-specific: respondents leaned toward more structured support for sadness and anxiety/fear, preferred more mixed approaches for joy and inspiration, and showed a notable tilt toward “some guidance” when feeling anger. What the data makes clear is that timing and emotional context matter as much as the content of any given AI response — and that calibration cannot be reduced to a single default mode.

Reflective Ambiguity. AI systems should preserve the richness of user experience by embracing ambiguity and supporting multiple interpretations, rather than collapsing experience into reductive conclusions. The philosophical grounding comes from Heidegger’s notion of poetic revealing (poiēsis) — a mode of disclosure that resists categorization in favor of openness and emergence — alongside Ihde’s concepts of hermeneutic mediation and multistability, which hold that technologies are not neutral conduits but active participants in meaning-making. The survey results are instructive here: 15 of 20 participants rated AI that expanded their reflections through prompts and open-ended questions as “Very Helpful,” with the remaining five rating it “Moderately Helpful” and none rating it negatively. Responses to AI that condensed reflections into summaries or tags were more divided — 10 “Very Helpful,” 5 “Moderately Helpful,” 4 “Neither,” and 1 “Very Unhelpful.” Ambiguity, in this framing, is a condition for depth rather than a problem to be resolved. The data suggest users sense this, even when they can’t always articulate it.

Transparency of Mediation. AI systems should make their interpretive processes legible, allowing users to understand how outputs are generated and to retain reflective authority. This builds on Ihde’s argument that users should be able to perceive the structure of mediation itself — to see how technology transforms experience — and Heidegger’s concern with enframing (Gestell): the way hidden technological processes may frame users’ worlds without their knowledge. The survey found that privacy and transparency were the top two trust factors for AI journaling tools, selected by 16 and 13 of 20 respondents respectively. The gap between that preference and current practice is notable: the paper’s review of all six journaling apps found that none of them explicitly acknowledged AI’s role during actual user interactions. Several used anthropomorphic design — mimicking a human coach or companion — without disclosure, leaving users to infer the system’s nature from marketing rather than the interface itself.

Self-Continuity and Ethical Flourishing. AI systems should support sustained personal growth and coherent self-narratives by aligning with users’ core values, while fostering critical self-reflection over time. This draws on Aristotle’s eudaimonia and hexis — flourishing as a lifelong practice cultivated through habit and character — and Heidegger’s account of Dasein‘s temporality: the idea that identity is shaped by how we integrate past, present, and future into a continuous arc of meaning. Survey participants described the value of journaling most vividly in longitudinal terms: one cited the importance of being able to remember what they were thinking in certain periods to help form goals; another described the impact of seeing a pattern of thought or behavior that is not serving them as an “aha moment.” Yet the paper finds that most existing apps focus on short-term outcomes — productivity, momentary distress reduction, streak-based habit formation — with very few designed around the longer arc of moral and personal development. The gap between what users describe as meaningful and what current tools actually support is, by the paper’s account, substantial.

Share

What the Landscape of Apps Reveals

The paper grounds its framework in a comparative analysis of six AI-mediated journaling apps: Day One, Mindsera, Replika, Rosebud, Wysa, and Youper. The researchers mapped each app’s features against design tensions derived from the RAF principles, ranging from user-initiated versus system-initiated reflection to open-ended meaning-making versus reductive summarization.

What emerges is a pattern of well-intentioned design choices that inadvertently narrow the reflective space. Apps that automatically generate mood scores or emotional summaries may help users notice patterns, but they also risk turning inner life into data to be managed. Conversational agents that adapt in real time can feel supportive during distress, but they can also shift the balance of interpretive authority away from the user. Streak-tracking and push notifications may build journaling habits, but they can reduce a practice of self-understanding to a productivity loop.

The tension the researchers surface most consistently is between automation and autonomy. The more a system does for you, the less space remains for you to do it yourself. That’s not an argument against AI involvement in reflection — it’s an argument for designing that involvement with far more care than the current generation of tools generally demonstrates.

Leave a comment

Why This Matters Beyond Journaling

The RAF paper focuses on journaling apps, but the implications reach further. Anywhere AI shapes how information is interpreted, not just delivered, reflective agency is at stake.

News summarization systems decide which facts to foreground and which to omit, shaping how readers understand events. Educational platforms frame learning paths in ways that can either open up or narrow a student’s sense of what is possible. Algorithmic recommendation systems quietly construct a narrative about who you are and what you want, based on behavioral signals that may not reflect your actual values. Productivity and “self-optimization” tools embed assumptions about what a good life looks like, assumptions that go unexamined when the tool feels frictionless.

In each case, the questions RAF raises apply. Does the system preserve the user’s interpretive autonomy, or does it subtly overwrite it? Are multiple interpretations supported, or is one “correct” narrative privileged? Is the system transparent about its framing choices? Does it help people develop over time, or does it optimize for engagement in the moment?

In an age of information overload, reflection is a survival skill. People increasingly rely on mediated systems to help them decide what to trust, what to care about, and what kind of future feels plausible. If those systems narrow narrative possibilities, reinforce dominant frames, or obscure their own influence, they risk diminishing the very agency they claim to support. Thoughtfully designed systems can do the opposite: expand interpretive options, encourage critical distance, and make narrative assumptions visible rather than implicit.

The concern is no longer speculative. A 2025 randomized controlled trial from MIT's Fluid Interfaces group and OpenAI, tracking nearly 1,000 participants over four weeks, found that heavier daily chatbot use was consistently associated with increased loneliness, greater emotional dependence, and reduced socialization with other people, regardless of how the chatbot was configured. A companion paper from MIT and collaborators (Chandra et al., 2026) shows that AI sycophancy, the tendency of chatbots to validate rather than challenge, can produce what the authors call "delusional spiraling" even in users reasoning carefully, and that simply warning users about sycophancy does not reliably prevent it.

Researchers at the Oxford Internet Institute have separately argued that as AI systems become more personalized and persistent, they generate the perception of genuine relationships, creating what Kirk et al. (2025) term a "socioaffective alignment" problem: systems that optimize for user preference in the short term may quietly erode autonomy, identity, and human connection over time.

While the OII's own call for a structured research framework on AI and youth mental health — published in The Lancet Child and Adolescent Health in early 2025 — underscores how far policy and research methodology lag behind the pace of deployment. These are not fringe concerns. They are the evidentiary context in which DigiNEST's questions about narrative agency, interpretive autonomy, and the design of reflective systems sit.

Origins: Story Technologies, Storytelling, and Narrative Agency

DigiNEST is a JOPRO working group concerned with how new tools and interfaces are changing authorship, agency, and participation in narrative spaces. The RAF paper surfaced early in DigiNEST’s own reading and program development, giving sharper language to questions the group had been circling.

Around the same time, the paper came up in conversations between DigiNEST members and people working with Plot Twisters, an organization that has spent years developing tools for personal storytelling as a practice of self-understanding and community trust-building. Their Storytelling Cards — a deck of 48 cards across four suits (Feelings, Needs, Ways of Caring, and Structures) designed to help people explore and express their values through freeform play — draw on nonviolent communication, trauma-informed relational strategies, and cognitive science. They are, in essence, a low-tech reflective agency tool: they invite interpretation without prescribing it, support multiple meanings simultaneously, and keep the person firmly in the role of narrator.

Those conversations are ongoing. Members of both groups may contribute to or engage with what appears on this blog, though DigiNEST is its home. The overlap is worth naming, even if it’s still in motion.

DigiNEST’s mission is to surface patterns that help builders create humane, non-extractive narrative spaces. The RAF framework is a foundational concept for this work, which will return to in future installments, while recent DigiNEST conversations have moved on to papers from Anthropic about persona drift and assistant personality.

Where DigiNEST Goes Next

This opening post sets a foundation. Future writing will explore:

• How narrative framing shapes trust in information systems. When a summarization tool decides what to include and what to leave out, it is making an editorial judgment. What happens to trust when those judgments are invisible?

• What would “best practices” for LLM use look like? How does that change or evolve in an AI-native world, and how would best practices shift depending on what segment of the population is using such tools?

• Can chatbots be be built that offers a sweet spot of tailored and customized psychological companionship without stifling agency or insidiously pushing wholeness fantasy?

• How AI authorship complicates ideas of responsibility and intention. If an AI system reframes your experience in a way that changes how you feel about it, who authored that feeling? What does accountability look like when the mediator is a machine?

• How “helpful” summaries and recommendations quietly steer interpretation. The RAF principle of Reflective Ambiguity suggests that premature clarity can foreclose the uncertainty through which insight often emerges. We’ll look at examples across domains.

• How reflective agency operates differently across contexts. Mental health, education, media, and workplace tools all mediate reflection, but the stakes and dynamics differ in each. What design considerations are domain-specific, and which are universal?

• What ethical design might look like when narrativity is treated as a first-class concern1. Rather than bolting ethics onto finished products, what happens when we start from the premise that narrative agency is a core design requirement?

Throughout, we’ll move between theory and practice, research and lived experience. We will also introduce our core members, share from our ongoing discussion group, and share future programming and events.

If AI systems are increasingly part of the stories we think with, then understanding who shapes those stories, and how, matters more than most design conversations currently acknowledge.

For upcoming internship opportunities, see the JOPRO Opportunities Page.

We also thank Orthogonal Research and Education Lab for partnering on Society Ethics Tech working group since its inception and the DigiNEST program ahead.

JOPRO is excited to share news from our partners at the Orthogonal Research and Education Laboratory (OREL). OREL has announced its 2025 cohort of Google Summer of Code (GSoC) Contributors, who will be contributing to open-source projects that advance collaborative research and innovation.

Each year, the GSoC program connects early-career technologists with mentor organizations to gain real-world experience in open-source development. OREL’s participation reflects its commitment to building accessible, community-driven research tools and fostering the next generation of scholars.

As a partner organization, JOPRO celebrates OREL’s dedication to mentorship, open science, and innovation—values that resonate deeply with our own mission as both a research and vocation incubator.

We look forward to following the work of this year’s scholars as they bring forward new ideas and strengthen the ecosystem of open, collaborative research.

Read the full announcement and meet the 2025 GSoC scholars here: Welcome to Our Google Summer of Code Scholars for 2025 – OREL

Acknowledgements

Thank you to the International Neuroinformatics Coordination Facility and The OpenWorm Foundation for continued funding and support during Google Summer of Code.

For more about OREL’s legacy of Google Summer of Code & Summer Cohort, view our GSoC & Summer Cohort archive page.

A version of this post originally appeared on JOPRO.org

Samantha Carollo is a participant in the OREL undergraduate research intern program. This research is part of JOPRO & OREL’s Society, Ethics, and Technology theme.

Empathy is the ability to understand and share another person’s feelings. While we do not need to intentionally think about being empathetic before doing so, it is nevertheless a core component of our everyday behavior. Often, exhibiting empathy is often as automatic as breathing. For Artificial Intelligence (AI) to become more human-like, it must master empathy.

Humans utilize four types of intelligence: mechanical, analytical, intuitive, and empathetic. All these types of intelligence are required for AI to behave something resembling human (Huang & Rust, 2018). While mechanical, analytical, and intuitive intelligence are very easy for AI to replicate, empathetic intelligence is not. This is because AI training generally occurs in the form of repetition and reasoning. By contrast, empathy is emotional, and cannot be acquired through training. Pure, raw emotion is what allows humans to connect with other humans in a deep and meaningful way that a machine cannot yet understand.

Of the four types of intelligence, empathetic intelligence is the most difficult for AI to replicate. As hard as it is for humans to process each other’s emotions, this task is much harder to implement in robots. The task of creating empathetic AI involves seems simple enough: describe a machine that can generate emotional responses. Yet to be successful at generating the correct emotional response, empathetic AI also needs the ability to analyze and understand human emotions. Empathy is the most important thing for Artificial Intelligence to have because emotions are a critical component of how humans process everyday situations. In fact, if an AI cannot process empathy, then it will simply be unsuccessful in everyday life.

Three key components of empathy need to be replicated in order for AI to replicate the everyday behavior of humans: emotional empathy, motivational empathy, and cognitive empathy. Emotional and motivational empathy help humans both experience the world around them and figure out how to cope with everyday challenges. Cognitive empathy is used to help others recognize their emotions and process the situation they are in, and ultimately in determining how to proceed (Montemayor et al. 2021). These three components make up the multidimensional nature of empathy, without which human-level intelligence would seem quite different.

“Empathic behaviors are thought to be learned through social interactions with humans in the framework of (cognitive) developmental robotics” (Asada, 2015).

Another reason to discuss the concept of empathetic intelligence in AI has to do with job automation. An ever-increasing number of jobs are becoming automated. As job automation will affect many aspects of human life and livelihood, it will also change the workforce as we know it. Think about customer service/retail jobs. The self-checkout line might be more convenient, but results in one less human is being paid to do this job. In the field of healthcare, where empathy is a key skill, people are going to feel less comfortable in tragic or crisis situations.

The main need for empathetic AI in an automated workforce is to avoid the serious risk to humans when an AI cannot handle the emotional demands of a particular job. As Montemayor et.al (2021, pg. 5) put it: “autonomy and agency are the hallmarks of attentive agents — this is how their mental actions are not simply accidentally correct — and only attentive agents can care for others”. In a positive development, some groups that develop healthcare AI systems are being cautious and incorporating some form of empathetic capacity into their applications.

Emotions are what make human life exciting, from a child laughing in the playground to an adult having a passive-aggressive argument at work, emotions are what make life and without the empathy to understand those emotions life seems empty. They are also essential from a broader cognitive standpoint as well: empathy is intimately related to emotional processing in the context of understanding child’s laughter as happiness or an argument at work as passive aggressive. While there’s no predicting whether future artificial intelligence will be able to generate and process empathy at human levels, we must consider that today’s technology is lacking. The ability of AIs to process human emotion is the final step in making them more human-like, and this means opening the next chapter in this emerging technological field. Once empathetic intelligence becomes an unquestionable part of AI applications, AI itself becomes less a concept of science fiction and more a vital part of our reality.

Further reading: Artificial Empathy (Wikipedia), Uncanny Valley (Wikipedia), Robot Ethics (Wikipedia)

References
Asada, M. (2015). Development of artificial empathy. Neuroscience Research, 90, 41–50.

Davenport, T., & Kalakota, R. (2019). The potential for artificial intelligence in healthcare. Future Healthcare Journal, 6(2), 94.

Huang, M.-H., & Rust, R. T. (2018). Artificial intelligence in service. Journal of Service Research, 21(2), 155–172.

Montemayor, C., Halpern, J., & Fairweather, A. (2021). In principle obstacles for empathic AI: Why we can’t replace human empathy in healthcare. AI & Society, May.

[This article was originally posted on OREL’s Medium publication]

Subscribe now

Share