Doing science in the open
I am a gravitational-wave astrophysicist at the Institute for Advanced Study (IAS) in Princeton. I have had the pleasure of interacting with IAS members and visitors who hold vastly different views on AI, from conscientious objectors who refuse to use any kind of AI to those who believe that we should replace ourselves with AI scientist agents. Most agree on one thing: we urgently need to adapt the way we do science, whether or not we use AI.
In response, I have decided to conduct the following experiment: I will open-source my research projects, in an organized manner, while they are still in progress rather than after publication. This includes not just the results, but the entire development process: research plans, development logs, routes taken (including dead ends), software, and data (the tech-bro buzzword for this seems to be “glass-box”). This means that anyone can see the progress of my projects in real time, reproduce and verify my results as they are produced, comment on, or build on my projects at any time. All of the materials will be made public on GitHub and Zenodo with the open-science framework I am developing at the IAS.
Reasons why I am doing this (loosely in descending order of importance):
- Science should be open. I have heard multiple scientists say that they will no longer share their ongoing work with colleagues because scooping has become so easy with AI. I believe that this is sad and unhealthy. We seem to have placed so much emphasis on priority instead of on doing our work properly and fostering the free exchange of ideas. If this does not change, our beloved fields will be killed not by AI, but by suffocating ourselves in silos. I hope that I can change this just a little bit by being completely transparent about my ongoing research as an early-career scientist (whose career is still at stake).
- The route taken to obtain a result might be as important as the result itself. This is especially true when generating results becomes cheap but understanding them becomes the bottleneck. For projects with significant AI assistance, publishing all the inner workings will also make it much easier to understand and verify what has been done.
- The arXiv and journals are flooded with papers. There were already too many papers before the AI boom, and there are even more now, causing a huge decline in the quality of life of arXiv moderators, editors, reviewers, and the poor grad students who are asked to summarize their subfield’s arXiv papers of the week. In fact, on October 1, 2026, arXiv announced a rate limit on paper submissions. During the experiment, I will only submit important results to the arXiv, and of these, only send those that I believe contain useful insights to journals. All other results will simply sit in the GitHub and Zenodo repositories built from the open-science framework.
- We need to understand what LLMs can’t do, so that we know where to spend our effort. That requires publicizing failed projects and dead ends. There seems to be a prevailing sense of existential dread among academics, especially physicists and mathematicians. This is often based on the belief that if LLM agents can solve a couple of important problems, they will be able to solve most, if not all, important problems in the immediate future. This is an extraordinary claim, so it requires extraordinary evidence. I do not think that anyone can say that it will certainly happen, at least not at >5σ (or >2σ, if you are an astrophysicist); word on the street is that not even the AI labs understand the limits of their own frontier LLMs. Personally, I think that such claims are based on naive extrapolations, not on any solid evidence or well-tested theory (here is a reminder of what happened the other time a certain smart physicist tried to extrapolate). Maybe in a year or two, we will start to realize that there is a subset of problems that AI might fundamentally struggle with, where humans can still contribute meaningfully. By publishing the projects that LLM agents tried to attack but failed, we might be able to understand their limits better.
- We do not need to waste effort trying the same approaches on the same problems. If you see that I am working on the same thing as you, just let me know, and we can figure things out, e.g. whether I should stop and let you proceed, whether we should join forces, or whether we should work independently with different methods and cross-check our results. I do not see the point in wasting time (and tokens / electricity / water) on overlapping work.
- If, for some reason, I decide to care about priority in the future, the public records will support me in a priority dispute. Yes, even if a certain AI lab decides to scoop my Nobel-Prize-worthy idea by spawning teams of 10,000+ frontier agents.
Q&A
“Will you open-source all of your projects from day one?”
I cannot guarantee that. For projects that I am working on alone, most likely. For those with coauthors, it will depend on the nature of the project. Also, it is not like you are going to be able to peek over my shoulder while I am working. I still control when I release the material publicly, although I will try to do it as frequently as possible. I might still flag certain parts of a project as private, e.g. private communication, notes that are too messy or irrelevant, or tasks that are too embarrassing (e.g. where I spent three days figuring out why a minus sign is missing). If I have a Nobel-Prize-worthy idea, I might also balk and decide to keep it to myself. Luckily, I do not have this problem right now.
“What if someone scoops you?”
I do not expect this to hurt my career much. In my field, independent papers generally receive comparable credit, even if one comes out months earlier than the other. My colleagues are not dumb; they can tell if my contribution is meaningful even if someone else published first. Also, I expect that hiring committees will have to move away from using “number of publications” or “priority in results” as a figure of merit, so I am not too worried about that either.
“What if I find that you are working on the same thing as me?”
Send me an email; I’ll be happy to discuss. You can also build on my work directly; that is the point of the open-science framework. But please do cite me (e.g. the DOI of the Zenodo repository) if you decide to use my results.
“But the work you make public is not peer reviewed if it has not been or will not be published.”
That is correct. Please keep this in mind if you are using my results. Most results will be marked “unverified” in the repository until I am confident in them. If you find a mistake, please open an issue in the repository. You could even create a pull request if you want to fix it for me.
“How would you advertise the results you don’t post on the arXiv?”
This is a good question. Firstly, if I decide not to post something on the arXiv, it means that I do not think that the results are that interesting anyway. Moreover, I am blessed enough that some people in my field know who I am and might follow my research even if it is not on the arXiv. I am also fortunate enough to have opportunities to attend conferences and present my work, so I would have many chances to advertise it. However, I understand that someone less privileged than me might not have these opportunities, so I am not advocating that everyone should stop posting all of their work on the arXiv.
“Are you pro-AI?”
I don’t consider myself “pro-AI”, at least not unconditionally. I think some use cases of AI are wasteful, some are questionable, and some are outright evil. However, I don’t see a fundamental problem with using AI to help with scientific research, as long as the results are verified to be correct and you don’t flood the arXiv with slop papers. I am paid to do what I do because society thinks that it is important to have someone working professionally on astrophysics, and AI helps me do that job better. Personally, I use AI to varying degrees for different projects. I do most coding tasks with AI, but I like to write papers with much less help from AI because I’d like readers to feel that I am talking to them directly (e.g. see my recent paper).
There are many societal issues stemming from AI (P(doom), social equity, data centers, etc.). I have my own opinions on these, but they are outside the scope of this page. Nonetheless, I do not think that we should be morally required to abstain from using AI for scientific discovery. As long as AI use is responsible, it is okay. Of course, scientists should also have the academic freedom to choose to avoid AI in their research.