Apr 10, 2025

CSPubs & ICLR Points FAQ

You can find a preprint paper of the methodology on arXiv. The result is also on an interactive website here. The scripts used to generate the result can be found on GitHub.

History

This project was intended as a light-hearted joke poking fun at the (perceived) effort to produce an ML paper, developed by me during the 2025 spring break. However, it has gained significant traction since then, and I have heard that people started to use it in annual reports and tenure cases. Therefore, I think it deserves a more serious treatment, so I rewrote this FAQ with some of my deeper thoughts.

Why Do We Measure Effort?

Researchers found that people from outside of academia often have limited knowledge of the relative effort needed to produce a paper in different areas and/or different venues. The measurement of relative effort across different areas is designed to combat the perception that every paper is "worth" the same by comparing publication norms across areas.

Why Top-Tier Conferences?

According to CSRankings:

...(Measuring publications in top-tier conferences) is intended to be difficult to game, since publishing in such conferences is generally difficult: contrast this with other approaches like citation-based metrics, which have been repeatedly shown to be easy to manipulate.

How Do We Measure Effort?

Obviously, it is impossible to tell how much effort is being spent in each area. Instead, I am going to make the (bold) assumption that each faculty spends the same amount of effort in publishing papers. Therefore, the total amount of effort in one area is in proportion to the total number of faculties in that area (multiplied by the number of years we are measuring). I will use the CSRankings faculty data for the purpose. If a faculty appears in multiple areas, I will (boldly) assume that he splits his effort equally in each area.

How Do We Measure Publications?

I use data from CSRankings, which comes from DBLP. By default, I include papers published in 2019–2023.

Why ICLR?

When the project was intended as a joke, I found some outliers of the ICLR reviews/rebuttals to be quite entertaining:

• Flight recorders are painted orange
• If the reviewers do NOT agree with these two claims, please provides (sic) the Python code to prove their opinions.
• your area chair does not read Twitter, Reddit/ML, etc.
• The document is impeccably articulated, boasting a lucid narrative complemented by logically sequenced sections and sub-sections.

Limitations

I think this study is by no means a perfect one, especially in these aspects.

1. Subjectivity of Areas and Conferences. CSRankings' selection of areas and conferences is not without debates. People have also argued that interdisciplinary areas, such as bioinformatics and education, should take prestigious journals into consideration as well.
2. Individual Conference Comparison. People have requested that individual conferences be measured as well, which I find difficult to do. While it is probably true that in my field of security, most research effort is (at least initially) directed at a publication in top-tier conferences such as S&P and CCS, it is kind of difficult to say how much effort is spent in S&P, compared to, say, CCS. Therefore, I have no idea how this should be done, and would welcome insights.
3. Potential for Misuse. I will cite Goodhart's law: When a measure becomes a target, it ceases to be a good measure. I personally feel that evaluation by any sole criteria is dangerous, as the criteria becomes twisted when people start to optimize for it. The birth of this project is to combat the perception that every paper is "worth" the same, not to create another artificial hurdle that is highly sensitive to some specific selection of areas and conferences. Let us hope that I don't have to live in the dystopia I created...

Some Discussions & Mentions of ICLR Points

• My X post
• The Structure and Interpretation of Computer Science Academic Metrics, by Prof. William J. Bowman