Ask AI anything and it’ll almost always give you a plausible answer. But are those answers plausible because AI knows more, or is it because it is a system designed to respond to us? According to OpenAI’s report, AI systems are “rewarded” for responding, even if it is guessing, over “accepting uncertainty.” It’s safe to say that easy access to information comes with disclaimers.This is emblematic of a bigger concern around AI use: can we trust AI output? That question is particularly consequential in scholarly publishing, as AI becomes more embedded in publishing and editorial workflows. In some cases, it is even taking over certain tasks traditionally done by humans, and, in turn, influencing how research is communicated. This creates a unique problem for publishers because scholarly publishing is not just the means to getting scientific knowledge out there. As gatekeepers, publishers also evaluate the scientific merit of the research papers they publish, and therefore, play a role in determining which findings enter the scholarly record, and are trusted, applied, and built on. So, when AI enters the publishing process, we’re not just asking whether AI-assisted systems can be trusted. We’re also asking how publishers can maintain the reliability and transparency of publishing processes as they become increasingly influenced by technologies whose internal workings may not always fully visible to users.

In this AI era, information and support are more accessible, but confidence remains critical. As AI becomes more deeply integrated into research and publishing workflows, the value publishers provide may extend beyond providing access to scientific knowledge. They may also need to ensure that the systems supporting evaluation and communication remain transparent and trustworthy.

What does it mean for publishers to trust AI systems?

The scholarly publishing system is not without its challenges and limitations but there is an identifiable process behind how it works, one we understand and recognize. And as publishers incorporate AI assistance into their operations, a key consideration for them would be: how well do we know the AI system we intend to use?

Publishers may be well versed with what AI assistants are supposed to do, where they may lag, and their performance stats and reports. But that may not be enough to treat the AI output as reliable and actually allow it to influence editorial processes. It depends on: How well do publishers know these new systems they are introducing into the equation: Do they know what data the AI systems were trained on? Will the system treat every type of manuscript fairly? Publishers will continue to assess the research papers they receive and publish, but now they will also have to think about whether the new systems that are helping them assess those papers can be trusted. That is a crucial governance question that needs to be addressed.

Who really owns the decisions?

When publishers introduce AI into their processes, they are adding another layer into the chain of trust. The human-in-the-loop system, rooted in the idea that AI can assist with a task, but humans should be an integral part of the decision-making process, offers a way to keep the authority, responsibility, and accountability for making the final calls with humans. Human oversight does not guarantee that every uncertainty introduced by AI systems will be eliminated. However, it ensures that the responsibility for applying context to recommendations and making final decisions continues to rest with humans.

For instance, while determining if a manuscript fits a journal’s scope, an editor may rely on AI-generated recommendations along with their own assessment. So, the nature of human involvement may evolve, but editorial judgment will remain central to making final decisions.

So, even if humans are in charge of the final decisions, does it really settle the question of accountability? As AI becomes part of publishing workflows, accountability may depend not on separating human and AI contributions, but on clearly defining responsibilities and governance across the process. That’s where the role of publishers becomes more complex. Their responsibility extends beyond individual decisions and output to carefully evaluating the systems that influence those decisions.

How can publishers maintain trust when the rules change?

There’s no doubt anymore that AI technology is here to stay, is evolving fast, and is in one or more ways becoming part of research and publishing workflows as well. And transparency and disclosures are big parts of the conversation. For instance, many publishers and journals now require authors to disclose and explain AI use, regardless of the extent or purpose. Now if publishers are depending on AI systems to evaluate and process those very manuscripts, do they also owe transparency and disclosure about their own AI use, and to what extent? Questions like these are likely to become important as AI adoption grows.

Once AI becomes part of the publishing infrastructure, it is not just an internal operational matter anymore; it changes the systems as we know them and alters the relationship between publishers and the stakeholders within the larger scholarly ecosystem, including authors, readers, editors, reviewers, and institutions. We know that the system is changing and will continue to evolve, and a fundamental part of continuing to trust the system will require transparency and communication.

With AI entering the picture, the responsibility of publishers has only grown, and so has the importance of their role. The challenge is not whether AI should be part of publishing workflows. In many ways, it already is. The challenge publishers now face is ensuring that AI enhances efficiency while providing the oversight that underpins trust in scholarly communication.