At the bend in the curve

A generation of physicists has never done research without arXiv. Where did it come from?

Steinn Sigurðsson arXiv came out of the custom of circulating paper preprints. In the 1980s, the major institutes would post the week’s preprints in an envelope, and because that cost money, they only sent them to their friends (see “Paper trail” figure). Harvard sent a single copy of each to Princeton or Berkeley, but not to every university, so there were literally fights over them. As a graduate student, I remember sitting in the hallway, waiting for a professor to finish with the preprints and toss them out.

Then string theory became extremely hot and people did not want to wait for results. Joanne Cohn, then a postdoc at the Institute for Advanced Study, was sending the TeX source of new papers around by email, but the list had grown cumbersome. In the summer of 1991, we were at the Aspen Center for Physics, and somebody said, “This is stupid. We have computers. Why don’t you automate it?” Paul Ginsparg, then at Los Alamos, was intrigued, and by the next day he had written the scripts. All of this happened before the web.

Peer review is often presented as a guarantee of scientific reliability. In practice, physicists read an article on arXiv first, and often only there. What responsibility does that place on the platform?

Steinn The early adopters were professional researchers, so arXiv ran on reputational risk. You put a paper on arXiv when you were ready, on the assumption that if you screwed up we would know, and if you screwed up a lot you were a screw-up.

Multiple studies over the years have found that most papers do not change much through peer review. Although, as I like to point out, adding the word “no” is a tiny change that is very substantive. On the theory side, it may also matter more to be interesting than correct. Many of the big breakthrough papers were concept papers with flaws, and the ones that survived are the ones whose flaws got cleaned up. That distinction is going to matter in the artificial intelligence (AI) world, where there will be a lot of papers that are correct but that nobody cares about.

Most researchers click submit, go to bed and find the paper online in the morning. What happens in between?

Steinn If you are an established author and your papers have gone through without problems, your next paper usually passes unless something raises a concern. We rely on trust. New authors get a closer look. Most of the time the moderator glances at the title and abstract, sees what it is about, often knows the people involved, and it is fine.

A smaller fraction needs follow-up and is put on hold. The paper may belong in a different category, fall outside arXiv’s scope, be completely deranged rubbish, or simply contain something weird that needs checking. And taking a look is not fun. It can be fascinating, but it is not enjoyable. The hold queue has to be cleared as fast as it fills, and that is where arXiv is breaking down, because the holds are building up faster than we can clear them.

Greg Morrisett If I can add to that, there are also lots of well-meaning people submitting things that do not belong on arXiv. Students, for instance, post final projects that may be fascinating documents but are not research for the audience arXiv serves. Many people think arXiv is just a website, when it is really a community of some 270 volunteer moderators and 25 staff. Paul Ginsparg recognised early that the attention of the readers is what is valuable, and moderation protects that attention.

Steinn Exactly. If every introductory computer-science instructor told students to put their term projects on arXiv to learn how submitting to arXiv works, then arXiv would die.

On 1 July, arXiv left Cornell University to become an independent non-profit. Why?

Greg Paul moved from Los Alamos to Cornell in 2001 and brought arXiv with him. The university was never quite sure where to put it, because it is neither an institute nor a research centre. It started in the library, and eventually the provost handed it to me when I was dean of computing and information science. As a computer scientist, I valued arXiv immensely, so I was happy to take it on.

Meeting arXiv’s funding needs was always difficult, and we were limited in our ability to hire developers. I love universities, but they are not set up to run a production web service, an archive and a volunteer community. There was a good argument for staying. Universities endure, and that matters to something that is supposed to last. But there was also a risk that a future dean might not value arXiv as much as I did.

So we created an independent non-profit and moved the staff across from Cornell, with the Simons Foundation and others stepping up to fund it for its first three years. The aim is to give arXiv clearer leadership, more flexibility to hire and a structure built around its own mission, rather than leaving it as an unusual service inside a university.

What will users notice?

Greg Everybody worries about this. On 1 July, hopefully nobody noticed anything more than a change of colour, because we took the Cornell seal off the webpage. That was intentional, because the core mission is exactly the same. There were concerns that arXiv would suddenly be supported through advertising. Not at all.

There will be small changes that add capabilities. The HTML renderer we developed for accessibility turned out to be useful to everybody, and we want better tools for moderation and discovery. But we are not expecting the experience of submitting or reading a paper to change substantially.

Could somebody one day buy arXiv and turn it into something else entirely?

Greg A famous billionaire, who has since passed away, once tried to buy arXiv. We said no. The new organisation has a board drawn from the arXiv community, so that it understands the core mission and ethos. Who knows what the future holds, but no, I do not expect arXiv to sell out. Like any non-profit, it will have to live within its means.

Submissions are surging (see “Overtaken” figure), and AI now writes or assists some of them. From the inside, how much has that changed arXiv?

Steinn The short answer is that things are changing faster than we can deal with them, week by week. We went very quickly from many submissions that were mostly garbage slop to highly polished fluff, a mix of correct although trivial results, but recently including some increasingly substantive and even novel research results (see “Without genius or study” figure). My canonical example is a high-schooler who grabs a public dataset of X-ray images of cancer patients, runs a slightly modified open-source model and concludes it does 1% better at diagnosing cancer. It looks like a paper. It has a claim and a result, but it is done by someone who does not understand what they are doing.

At the same time, genuinely interesting results are arriving at a higher rate. Projects people had set aside, what I call graduate-student test projects, interesting and worth doing but not top of the queue, are being wrapped up in days by teams of AI agents. In modelling or theory, something that might once have been a chapter of a PhD thesis can now take anything from hours to a few days. You can do it live. I was sitting in a meeting the other day and asked Claude, “How about this as an alternative explanation of what the speaker just said?” Claude said, “No, there is this one problem.” So I asked Gemini, “Is Claude right?” Gemini said, “Yes, but there is a way around it.” There was a concept for a paper in 40 minutes.

Disruptive results from AI-assisted work are near. We have hit the bend in the curve. Our current mantra inside arXiv is that the only thing that scales like AI is AI.

AI is also breaking the alignment of incentives. Counting papers is going to become meaningless. If I took this afternoon off, I could probably send out five papers today. So we need to identify the papers on which other researchers actually build. We are going to use knowledge graphs and semantic analysis to see where ideas are going and what later work depends on.

The difficult part is that some of the mechanisms that work can also be retrogressive. Some journals are already moving towards systems where you need people to vouch for you before you can submit. That takes us back towards the 19th century and the idea that somebody has to say you belong to the club.

Greg, what can arXiv itself do?

Greg We were already writing more than people were reading, and that has now exploded. The scarce resource is the attention of readers, their eyeballs and their time. I do not think anybody yet has an answer to how readers should deal with that onslaught.

One change we have made goes back to trust. An academic email address used to be a useful heuristic. Whoever had one was probably a researcher, so we let it fly. Since January, we have asked for endorsement of new authors, including those with academic addresses, to put some of that reputational pressure back into the system. It seems to have helped a little.

There are also things we can automate. We can scan for dangling references and other signs that something is wrong, then give moderators better evidence about what deserves a closer look. But the social incentives matter just as much. For 30 years, if you posted crap on arXiv, your reputation suffered and people trusted you less.

Endorsement works, but it is antithetical to one of our principles, which is to be open and accessible to the whole world. That tension will be a challenge no matter what.

This year, arXiv turns 35. How would you like to see it at 50?

Greg Still serving the communities and surfacing the research people want to read. I would love to see arXiv expand into adjacent areas, but that takes a willing community with the preprint ethos. You cannot simply add a category to a menu.

I would also like moderators to enjoy the role again. Tom Dietterich used to describe the excitement of having the great papers cross your desk first. We now get around 100 machine-learning papers a day, and no human can read them all and keep a sense of the field. Moderators need sophisticated automation, and readers need better tools to find papers, make connections, summarise and translate.

Paul’s taste in setting up arXiv also matters. LaTeX was not an obvious choice, but it means that papers from 30 years ago can still be rendered in new ways. He was also wise not to compete with journals. I would welcome more open journals overlaid on arXiv, with arXiv remaining the distribution layer and peer review sitting above it.

When you show the site to a non-scientist, they say, “This looks like 1990s HTML,” and we say, “Yeah, exactly.” We are not trying to be flashy. It serves the community well, and we do not want to mess that up.

Steinn I agree with Greg. I would add that arXiv is now a large repository of knowledge, but its primary purpose is to serve up new science. Somebody is going to be doing science, somebody is going to be parsing that science and somebody is going to use it to do more science. arXiv should be there in the middle, enabling that at speed and scale.