Audio transcript
===
[00:00:00] Anna: it is really important to make sure that we have standards around things such as peer review. And in this case, the working group was concerned with, four real pillars that we had, which were identity transparency. So who's involved in this particular peer review process. specifically also who does the reviewer interact with? So there are some really interesting models out there where reviewers interact with each other or even interact with authors as well.
[00:00:32] Jude Perera: Welcome back everyone. I'm your host Jude Pereira and today we'll be sitting down with guests Anna Jester. and Annette Flanagan to discuss peer review in relation to research integrity and AI. As experts in the field, Anna and Annette will be giving us great insights as we explore how scholarly publishing is adapting to a constantly changing technological environment when it comes to peer review and diversity initiatives. Okay, now on to the good stuff. Anna and Annette, thanks for joining us on the Research Impact Podcast.
[00:01:09] Anna: Thank you so much for having me, Jude, as well, join the podcast. I'm honored to be here with Annette as well. I am, sales and marketing really for e journal press, which has provided submission and peer review systems, production tracking systems, and billing payment systems for surprisingly more than 20 years. And we have been acquired by Wiley plus almost two years ago now. So glad to be part of a greater network and have some fun colleagues, some of whom I've also known for many years. And that's really how my opinions about submission and peer review got formed was by really seeing what it is that journals need and hearing the questions that they ask and aiming to help them over the years.
[0001:51] Annette: Thank you, Jude. It's a pleasure, to chat with you and to Anna, who I've known for, I think, several years now,in the scholarly publishing space. I am Executive Managing Editor and Vice President for Editorial Operations for JAMA and the JAMA Network. We publish 13 journals. I like to think of us as a middle sized publisher.
[00:02:13] Annette: With a big footprint, I'm a cardiology nurse by training, and I have been in this field for more decades than I'd like to say, and I'm very excited to be talking with you about, one of my longest, favorite subjects, peer review, and now my new favorite subject, AI,
[00:02:30] Jude Perera: I'm super excited to have you both, in this podcast, or I mean, even to have a conversation in general with both of you, because I've had the pleasure of knowing you before this podcast as well. but, jumping right in because we really need to get down to our, you know, hot topic. So, views on using AI in peer review vary widely throughout the industry.
[00:02:52] Jude Perera: From completely restricting the use of it, seeing the potential for it to transform the peer review space without relying on it completely, to eventually relying on it completely, essentially. So how do you see the scholarly publishing space adapting to incorporate or restrict this ever changing technological environment, now and of course in the future?
[00:03:17] Anna: so certainly my feelings about it are definitely a little bit more about the technology and how the technology can help the process. I will certainly let Annette speak to how the journal feels about it and which roles should be included. But from my viewpoint. It's long been a concern that authors, or far worse, paper mills, might actually submit the same paper to multiple journals.
[00:03:41] Anna: And there have certainly been ways to try and help make sure that wasn't happening. Policies for the journals would certainly, would clearly state that isn't something we're going to allow. But I think collaboration amongst organizations, including the STM Integrity Hub, allowing journals to know if a paper has been submitted to more than one journal are really key.
[00:04:03] Anna: Those types of things help make sure that we're not wasting peer-reviewer and editor or even staff time any longer than necessary on manuscripts that are either breaking the rules, having been submitted to multiple places, or are from nefarious places such as paper mills.
[00:04:20] Anna: I also think that there should be other tools in place as well. So we've had plagiarism detection as part of a standard offering, Integrations with peer review systems for many years to try and help editors make or staff make an, informed decision about whether a paper had any kind of plagiarism issues. And I think that image manipulation is another space where AI is really going to be helpful. If a human is trying to look at an image, we might not immediately notice things like splicing or items that have been removed or something that hilariously to me is called beautification. but things like that, that can really, if a. If I can help alert our reviewers or editors to take a look at something and make that final decision, that can be more helpful by far than just asking someone to do all the legwork themselves.
[00:05:09] Annette: think, one thing that's important is the connection between the technology and the output, in this case, The published record of the scholarly article, I think that we are seeing a pace of development. Your words, Jude, ever changing? It's rapidly accelerating. If you think, you know, the first digital computers were developed in the 1940s or fifties, and then first microprocessor in 1970.
[00:05:35] Annette: It took another 60 years till we had this little mini computer show up the iPhone in 2007. And then fast forward to 2015. When OpenAI was launched and GPT
[00:05:49] Annette: Generative pre trained transformer was launched and our worlds were rocked in last November, right? With 5, and since then, multiple versions have come out with advances allowing for integration with ChatGPT, which is basically a language model to generate graphics, images, data visualization, statistical analysis.
[00:06:14] Annette: So it's here, it's moving fast, and from my perspective, the technology, which is moving so fast, requires policies and procedures to adapt equally, as rapidly. And, I can talk a little bit more about that maybe how we respond and how we reacted. to that, at JAMA and the JAMA Network Journals.
[00:06:34] Annette: So, in January of 2023, again, reactively in response to the November release of ChatGPT, we published an editorial in JAMA announcing our guidelines for authors and authorship with regard to use of ChatGPT in the creation or editing of content. And we declared, as did many other organizations at the time, That non human ChatGPT language models can not be named as authors.
[00:07:03] Annette: At that time, nature had published a new story that had identified two preprints and two journal articles that had CHAT GPT literally in the byline, and those were indexed in PubMed. One of those has subsequently been corrected to remove the author, and one of the preprints was subsequently published in a journal, a peer reviewed journal, without CHAT GPT as an author.
[00:07:24] Annette: So, these guidelines on authorship are developing, in response to that. the key is, I don't think we can determine everything that's going to happen in this space, because they are being widely used. Is to figure out how to best guide authors, reviewers, editors, and eventually readers, with how to report transparently on the use of these tools and understand what their pros and cons are. And right now, they are fairly equal.
[00:07:51] Jude Perera: Wow, that seems like Two very, very beneficial and non beneficial things going against each other. So now I'm going to go into a bit of a different topic. and I'm going to ask this from you, Annette. So can you tell us a bit about the joint committee for action on inclusion and diversity in publishing? And also, do you feel that the joint commitment has had an impact on DEI initiatives in the industry yet?
[00:08:18] Annette: Joint Commitment for Action on Inclusion and Diversity in Scholarly Publishing was launched by the Royal Society of Chemistry.
[00:08:26] Annette: it currently has 56 publishing organizations representing more than 15, 000 journals. So, I think what I want to say that's most important is the sharing and collaborative effort,that initiative has brought together.
[00:08:39] Annette: I think, a number of things have happened that have improved how we identify problems and approaches to improve equity, diversity, inclusion in peer review and scholarly publishing. There have been a set of standardized questions for self-reported diversity of data collection of everyone in the ecosystem, whether it's authors, reviewers, editors, board members.
[00:09:03] Annette: There are minimum standards for inclusion and diversity in scholarly publishing that this group has put together. many of us helped review and develop some of those standards, and I think that has been a key to this initiative, and it's not stopping, it's doing what many EDI efforts need to do, it's staying focused.
[00:09:24] Annette: And continuing to move. It's not just something you,you know, you flip a switch or you make a fix and then you move on. It is a continual important effort, that we need to pay attention to. I will say that we learned, as a participant, a few things from the early part of that initiative. we learned from one of our colleagues about how they were handling post publication name changes of authors.
[00:09:46] Annette: And we were able to, release and develop a policy and process for that that allows authors to change their names for any reason through a formal correction process if they wish, or through something we call a silent correction. we also added a question to our reviewer form, asking reviewers to identify any potential EDI concerns in a manuscript to alert editors to that so that we can have a dialogue with the authors on how to improve the reporting of a particular study.
[00:10:14] Jude Perera: Brilliant. And now, for your perspective, Hannah, how are these new policies helping to improve transparency and quality throughout the process?
[00:10:25] Anna: Yeah, so from my viewpoint, and I fully agree with what Annette has said, and I am so glad some standards have been put in place, I think the first thing to keep in mind is there have been journals who just hadn't, for whichever reason, started down that path, and that is somewhat important in that we really need to make sure organizations are thinking through what they're about to do, truly planning it, defining it, and expecting to refine it, all before they start collecting data. So this data is very important. A lot of us had to react when GDPR went into play, you know, somewhat quickly, but we really need to make sure that we're following standards whenever we can and actually telling people what we're going to do with this data. the other thing is that, when you're going to collect data.
[00:11:11] Anna: Make sure you're doing it in useful ways. So not only in ways where people can actually self identify based on the options that are presented to them, but they need to self report.
[00:11:22] Anna: You know, so, I've known Annette for many years, as we said. Jude, I've known you for some time as well. But we all know each other based on the spaces and times that we've been in the same place, right? So you don't necessarily know me in every possible way. And,it's really something to keep in mind. So if you saw me with straight hair, instead of with curly hair, you might say, she looks familiar, but I'm not sure.
[00:11:46] Anna: Right. And. One of the things I've said to people about self reporting, it's so important.
[00:11:51] Anna: So if you look at all of us, we all care about peer review. We all care about AI . We all care about ethics. But does that mean that I'm willing to let you put into a database what race or ethnicity or gender I have? I'd really prefer I get to enter that information myself.
[00:12:08] Anna: So self reporting is also key.
[00:12:10] Anna: In addition to that, I would just note, there are so many ways information can be entered. So it might be gender and race and ethnicity. And there are now standard lists from the joint commitment for those. And I greatly appreciate that as a software vendor, we've had many people say, well, what terms should we use? And that's one of those spaces where I say.
[00:12:31] Anna: Let me be clear. There are standards, but I cannot tell you what is best for your journal or your society or your organization or what your goal should be. So people need to know that for sure. what I can say is that absolutely 100 percent before you're collecting that information, you need to tell people exactly how you're going to use that data. In my view, you should be publishing information about that data, perhaps in the Peer Review Congress and other places. I highly recommend the Peer Review Congress as a place to hear what other people are doing as well. and then you also need to make sure we're giving people an option to say, I prefer not to disclose this data.
[00:13:07] Anna: So if we're going to require this question be answered, we also need to make sure we're saying, I, you know, Prefer not to disclose is an option and really paying attention to how many people said, I prefer not to disclose. So if people don't trust you with this data, it's important that you realize that as well. So all of that though, really starting the conversation and making sure that these types of discussions are being part of scholarly publishing, not just in 2023, but in the years to come have been really important to me and I'm very glad to see it.
[00:13:40] Jude Perera: Thanks, Anna.
[00:14:13] Jude Perera: Now, coming back to one of our key focuses, of this podcast episode, which is peer review, Anna, so you are part of the NISO working group to create the peer review standards, but can you tell me why was it important for you to make sure that software providers are included in the discussion to create these standards? And how do these new standards help publishers fulfill their commitment to diversity and inclusion?
[00:14:41] Anna: Yeah. So thanks for that. Definitely. I have to just say as well, being part of the NYSO working group was one of the most painless committees I've ever been part of. Highly recommend it. Please do join other working groups as well for topics that are in scholarly publishing. They need all of our input and definitely from the software companies as well.
[00:14:59] Anna: I think it's important to note, we were not the only software company that was represented as well, right? So it's good for us to be involved because we need to be aware of how people are going to want to change terminology in this case. We started the working group, it was the peer review taxonomy.
[00:15:15] Anna: We even edited that to be terminology, right? So think about how much we've cared about every word that's in place here. but it is really important to make sure that we have standards around things such as peer review. And in this case, the working group was concerned with, four real pillars that we had, which were identity transparency. So who's involved in this particular peer review process. specifically also who does the reviewer interact with? So there are some really interesting models out there where reviewers interact with each other or even interact with authors as well.
[00:15:47] Anna: Far more than just, here's the text of my review, please do everything I've asked you to. Um, and then also what review information is published. So that one varies widely, right?
[00:16:00] Anna: So generally speaking, someone might say, Oh, yes, we've put information about the review process and they maybe said three reviewers were included. Well, sure, that's information about the peer review process, but it's not exactly a copy of the decision letter or actually the reviews themselves, right? So there's just different layers to that. And then also whether or not post publication commenting takes place.
[00:16:22] Anna: And I've joked for a long time, there are two types of people. Those who read the comments and those who do not. Right. So we need to make sure those comments, if they are collected, ideally are collected in a way where we can actually know if they're anonymous or if they're actual people, you know, sometimes people don't feel comfortable saying who they are.
[00:16:39] Anna: And that's always something to consider as well. But if we really want to get information about this paper that was really constructive and helpful to all of us as part of learning more about the topic, it's important in my view to know who is actually commenting as well. So those types of input are really kind of crucial, not only to make sure that software companies are aware, these are the conversations that are going on, will we be able to accommodate these in our current workflows already?
[00:17:04] Anna: Will we need to code other things? but also just making sure we're really part of the discussion and not in a silo out far past the Land of Useful.
[00:17:13] Jude Perera: So, Nat,, do you want to add something to that?
[00:17:16] Annette: Absolutely. I think what Anne is talking about, these initiatives, the NYSA Working Group, the Joint Commitment on Action for , Inclusion and Diversity in New Scholarly Publishing, and many others, are addressing a really important problem.
[00:17:30] Annette: Bias. Bias is important. It's pervasive in the research and scholarly publishing ecosystems.
[00:17:38] Annette: And we have to go beyond recognizing it. We have to deal with
[00:17:41] Annette: it. One of the ways we deal with it is,trying to be more inclusive and also trying to find ways that everyone that's involved can self report who they are, so that we can, as scholarly publishers. Benchmark what we're doing and where we want to make improvements. How can we bring more people in to scholarly publishing? Whether we're talking things around gender or we're talking about, people who are represented by certain or ethnic groups, early career folks,geography, how we control for what we call the celebrity bias, or the VIP bias. and I think this is really important. And I would say that this is fairly new, you know, certainly within the last five years and it's spurred by these types of initiatives that are going on.
[00:18:33] Jude Perera: Yeah, and that certainly sounds, I mean, last five years, that's not that long of a period. But I'm glad it's now rather than, later. just for the benefit of our listeners, as well. Can you let us know what,N I S O stands for?
[00:18:50] Anna: Yeah, it's the National Institute for Standards Organization, and it's in fact, the standard that was actually published is actually nice. So, and C standard, so I will plunk in the chat.
[00:19:00] Anna: I actually have in the bio that I had last week during 1 of our meetings as well. I actually included the fact that I actually have a favorite standard, which is pretty nerdy, I know, but I do.
[00:19:12] Annette: What is it?
[00:19:13] Anna: oh, it is this one that I helped work on. Hands down, the ANSI NISO Z39. 106 2023 Standard Terminology for Peer Review
[00:19:23] Annette: Goes right off your tongue, doesn't it?
[00:19:25] Anna: I, you know, you gotta practice a couple times, but as long as you only have one standard that you're having a favorite of, It's not like nieces or nephews or parents. You can, you stick to one.
[00:19:34] Jude Perera: You always have your favorite, of course. so how would one essentially look at, you know, being a part of this? What would you say the best way to enter into this would be?
[00:19:45] Anna: Definitely. So, on the NYISO website, there's actually a join NYISO. as well as standards and committees and other kinds of ways to join and be part of it.
[00:19:55] Anna: I would also say other organizations, so I believe, if I recall correct, the reason I joined the working group was that Nettie Lagasse, who is the kind woman who herded all of us far more than cats, as part of that standard, working group, she actually had put out a call for people to join the working group, and it went through the SSP.
[00:20:16] Anna: center where we all go and kind of get notifications about various things that are happening in the organization. So that's actually how I ended up joining the working group was seeing that type of notification, but I know that they post them on things like LinkedIn as well as not just a site for scholarly publishing or. Council of Science Editors or International Society of Managing and Technical Editors, why, you know, websites and notification platforms as well.
[00:20:37] Annette: I had the pleasure to participate in a recent working group, NISO working group called, CORRECT, communication of, retractions, removals and expressions of concern. And, that has just been released for public comment. So that's one of the things that NISO does. It has a group of so called experts who talk among themselves and develop something, and then it's released for public comment so that it can be improved.
[00:21:03] Annette: and I think it's one of the great services that NYSO provides.
[00:21:07] Jude Perera: Brilliant. So you heard it right there. There's your sources to anyone who's listening and. Wants to get involved with NYISO, there you go.
[00:21:15] Jude Perera: So now, jumping on to the next topic, Annette, so this one's for you. How has being the Executive Director for the Peer Review Congress influenced your perspective of peer review?
[00:21:28] Annette: Oh, wow. , so fortunate to be able to, help. Lee the International Congress on Peer Review and Scientific Publication. It was founded more than 40 years ago by Drummond Rennie, a former deputy editor,at JAMA. it is a unique, one of a kind, research only meeting that is held on an Olympic schedule every four years.
[00:21:51] Annette: and we're all in the same room,all of us, so there's no breakout sessions, there are no panels. and, it is a time for researchers to present their research into peer review and scientific publication. We're essentially flipping. We're taking, and putting peer review under the same lens or microscope that it puts science under itself.
[00:22:13] Annette: And we're flipping that. it's three days. we're going to have the 10th Peer Review Congress in September 2025.
[00:22:22] Annette: It will be hybrid. So in terms of inclusivity, we want people who are unable to travel or do not wish to travel to be able to participate. It'll be three days of back to back run like a train research meeting aim is to improve.
[00:22:38] Annette: The quality, the credibility, the transparency of peer review through publication, post publication to actually improve the entire ecosystem. And we're really very excited. We launched a call for research was published in JAMA and the BMJ in September. We have this announcement, a long list of ideas of topics for research.
[00:23:03] Annette: and anyone who's interested in this is welcome to participate, and we encourage that, the, abstracts will be due in January of 2025, for a September 2025 meeting, which will be held in Chicago, which is right out my window, I'm actually probably pointing somewhere near to it.
[00:23:20] Anna: Yeah. Can I just add to that and say, it is one of the best conferences on the scholarly publishing circuit. Highly recommend. I've been to more than I should admit to at this point. It's clearly date me, but, The very first time I went, I just sat and watched presentations and thought to myself how wonderful it is that people truly are taking a look at exactly what they're doing as part of the peer review process. And willingly talking about what they tried, if it worked or if it didn't work. There's plenty of people who will tell you we found this did not work. And the just wonderful honesty of hearing that and keeping other people from having to try that in their peer review process, has just been a wonderful part of science itself, but truly the papers, the presenters cover absolutely every part of scholarly publishing. It's not. It's not an individual discipline, it's not a couple of different continents, it is truly universal and for that it is always going to be one of my favorite meetings, so thank you so much Annette for all of the work that goes into that because I know it is an enormous task and please know we really appreciate it.
[00:24:32] Annette: Oh, Anna, that's really nice of you. Thank you. I have to say, just to bring this back to AI, I expect that we're going to get a lot of research abstracts on AI and how it's being used, and tested and improved , in, in scholarly publishing,in the last peer review Congress was held in 2022. And we had two new areas of focus of new research.
[00:24:56] Annette: One of them was EDI, and there was a, about five or six plenary research presentations on what, authors, researchers, and journals were doing to improve EDI and scholarly publishing. The second one was pandemic science. You know, we didn't have a pandemic before that, and there were tons of abstracts looking at what happened during the pandemic, not just the huge surge and research on the pandemic, but how research was managed during the pandemic.
[00:25:23] Annette: And, so I expect there'll be AI and there will be, I don't know, the new EDI, the new pandemic science, focus that will come in. I'm sure we're going to have a lot more on EDI as well, because as I said earlier, it is. Bye. An important focus of how we're moving and maybe we'll hear about how AI can help us improve our EDI initiatives, which would be for all of us.
[00:25:47] Jude Perera: . So now, on that thought, actually, before we close, would you like to tell us, both of you, of course, would you like to tell us about what, is the thing that really excites you about peer review and AI. You know, on a positive note, that is, in the, let's just say near future. We'll start off with you, Anna.
[00:26:08] Anna: Yeah, I would say the potential for streamlining the process. we've constantly heard people saying they'd like to do it faster and with that comes the concern. That it's been too fast, right? We haven't looked at quite everything. So being able to provide tools that will help people be as efficient as possible and help us make sure that we're publishing all the good science that is available to us.
[00:26:32] Anna: Those are really the things that I'm looking for from it. I certainly agree. There are things we need to be careful of as well with every new technology there is, and there is. Absolutely the law of unintended consequences. So we'll need to also be part of helping make sure the community as a whole of scholarly publishing can react to those instances in a meaningful way and not necessarily slow down progress.
[00:26:58] Annette: Yeah, I'll agree. I've been thinking about something. I framed it as the three E's. This is the positive part of AI can go on and on about the current negative parts and the threats. I'm going to stay focused on the positive part. The three E's are Ease. Efficiency and equity ease, making mundane tasks easier, right? That's what AI is going to help us do. Efficiency, allowing for everything from faster, literature searches to, faster submission of manuscripts into peer review systems and equity, helping everyone get a chance to get into the community. AI should help that, that if that's translation for people who have English as a second or non language, or for those in one scientific discipline to be able to communicate with others, to help summarize scientific research, for the rest of us who consume it.
[00:27:55] Annette:
[00:27:55] Annette: and all of those probably go into a fourth E, which we'll call excitement.
[00:27:59] Jude Perera: I love that, the three E's, and now maybe four E's, ending with excitement, but, of course, I wanted to focus on the positive, but there are lots to look out for as well, and I think that's a conversation in itself, and we've touched on that. in a couple of our previous episodes and also the episodes that are coming up in the future. So since we have to, I mean, this is because we have to draw to a close today. I want to say thank you so much, Anna and Annette for joining us. and I have learned so much and it's always a pleasure to talk to you. so before we go away, can you tell us how is the best way that our listeners can reach you? I mean, is LinkedIn a good way, with any questions and thoughts following up from this?
[00:28:46] Anna: For me, definitely, I would say LinkedIn is the best way that people can follow up. definitely Anna Jester, it will note that I am from e journal press and it will also tell you that I love peer review if you pay attention. So yeah, definitely an easy way to find me.
[00:29:01] Annette: Likewise, I'm available on LinkedIn. email's fine as well. and then maybe at a meeting, the spring publishing conferences are coming up, if I haven't met someone who's a listener, I'm happy to meet them, share a coffee or tea, and actually, June and Anna, I hope to see both of you as well soon. This was really fun. Thank you.
[00:29:21] Jude Perera: Thank you, and you heard it, an invitation to go say hi! And to strike of a con conversation for sure. So, and then for our listeners, please feel free to share any suggestions for topics and guests via the podcast page on our Wally Partner Solutions website. There will also be links to any resources mentioned in today's show in the show notes. And once again, thank you so much for listening and keep an eye out for the next episode.
[00:29:46] Jude Perera: So till next time. time.