astro-ph Rationalized

By Sean Carroll | January 20, 2009 9:02 pm

Here is probably the single most helpful thing I have ever done for the world. Last month Paul Ginsparg, who did a world-changing thing by inventing the arxiv system for sharing scientific preprints, was visiting Pasadena, and dropped by Caltech. We chatted a bit about blogs, the internet, the preprint server, ways one might incorporate links to blogs and talks and newspaper articles and all that (some of which already exists in the form of trackbacks). And he told me a fun math problem I will blog about at some point.

And then he asked, “Is there any other obvious way the arxiv could be improved?” To which I naturally responded, “You mean in addition to subdividing astro-ph into categories?”

The problem with science is that there’s just too damn much of it. Every weekday, when one peeks at the new listings on astro-ph, one is faced with 40 to 50 new abstracts to read. That’s a lot of science to wade through, and it’s especially bad for people who work on the boundaries and might also be interested in hep-th, gr-qc, hep-ph, and/or other categories. (I haven’t yet broken down and started reading quant-ph.) Especially since, just because you are interested in issues at the interfaces of conventionally-defined disciplinary boundaries, it doesn’t follow that you are interested in every single kind of research that is carried out in every one of those disciplines. An early-universe cosmologist, for example, might not be interested in star formation or the interstellar medium. Or they might be; but perhaps not.

Nevertheless, everything astronomy-related on the arxiv gets put into astro-ph, from models of inflation to light curves of W UMa contact binaries. And if one was interested only in some subset, one needed to sift through the 50 abstracts to search for the few that struck a chord.

Until now! Paul and Mark Wise and I chatted for ten minutes and came up with a perfectly sensible (I like to think) set of categories into which astro-oriented papers would mostly fall, and Paul went away promising to implement such a scheme. After chatting around with a few actual astrophysicists and fine-tuning the system, it’s now done! That wasn’t so hard, was it? (Part of the reason this hadn’t happened much earlier is that certain astrophysicists who will remain nameless took a “eat your vegetables” approach to the problem, insisting that it was good for anyone to look at every single astro-ph abstract if they were possibly interested in any of them.)

Here is what I was happy to find in my email just now:

By popular request, the Astrophysics (astro-ph) archive has been split into six subcategories:

CO Cosmology and Extra-Galactic Astrophysics
EP Earth and Planetary Astrophysics
GA Galactic Astrophysics
HE High Energy Astrophysical Phenomena
IM Instrumentation and Methods for Astrophysics
SR Solar and Stellar Astrophysics

For more information, see the subcategory descriptions at (including links to the subdivided new and recent listings). This split should make announcements of new papers more manageable for those interested only in subsets of astro-ph. New astro-ph submissions must assigned one or more sub-categories. (Existing astro-ph articles will be machine-classified according to the new scheme when enough training data has been collected.)

To subscribe to the daily e-mail notifications for only a set of subcategories, you should first cancel your existing subscription, and then subscribe only to the subcategories of interest via physics. See For example, you could send two emails

Subject: can

Subject: subscribe [Your Name]

add CO
add GA

O frabjous day! Callooh! Callay! Undoubtedly some curmudgeons will gripe that their particular kind of research doesn’t fit snugly into any one of the categories. Fair enough; let the powers that be know, and they’ll do whatever is reasonable to make sure the system evolves appropriately. But for right now, my early evenings (abstracts appear at 5 p.m. Pacific time) just got a little brighter.

  • DrEvil

    Great, now we know where to re-direct packages from Ted Kaczynski.

  • Eugene

    yay. now the goal is to write a paper that can both be classifed under astro-ph.CO and astro-ph.EP

  • Paul

    Great to see, I’ve pined for a more finely divided astro-ph for a long time too.

    What I’ve always done before this was to use Yahoo Pipes to filter the astro-ph RSS feed into two main sub-feeds & several smaller ones. Of the two main sub-feeds, one is filtered to only contain abstracts with keywords I’m particularly interested in seeing & the other for everything else in the feed. That way I could always go over the “everything else” at a more leisurely pace than the keyworded abstracts, which get higher priority, & also keep the two feeds seperate in my Google Reader interface for ease of searching later.

  • Jonathan Dursi

    Bah. Why on earth would they just not allow tagging of articles with user-created tags? Or even just fixed tags with (say) the ApJ categories? If you are going to implement some sort of categorization, why so limited and useless?

  • Eric

    Thank you! I was just looking at the number next to astro-ph in my Google Reader and thinking it would soon be time once again to give up and hit “mark all as read”. Hopefully less of a problem now.

  • nav

    omg. this is the best news i’ve had since, well, itzykson and zuber was reissued by dover.

    though, i think providing an early blogging voice was probably equally helpful to the world. well, the world of procrastinating grad students anyway.

  • Ben Button

    “Is there any other obvious way the arxiv could be improved?”

    You missed out on a golden opportunity to say, “Please go back to the old hep-th/97123456 names and drop the idiotic and uninformative naming system that you foisted on us a couple of years back. Oh, and tell your lackeys to be slightly more polite.”

  • Robert Cumming

    Now that’s weird: I sort of thought ‘Galactic’ was about The Galaxy and that our very much extragalactic paper (from yesterday!) would land among the cosmology. Surely ‘Galaxy astrophysics’ would have been more obvious?

    Anyway, nothing wrong with our bedfellows in today’s listing, so I’m not complaining.

  • Martin

    Robert: yes, it’s particularly odd to have ‘galactic’ and ‘extragalactic’ and then put most (but not all) papers about galaxies, quasars and AGN in the former category. I take it this is just an error by whoever manually classified all the recent papers, rather than reflecting some very obscure policy that I can’t figure out before breakfast.

  • Peter Coles


    Great idea. But “extragalactic” should definitely not be hyphenated. Not that I’m pedantic or anything.


  • Sam

    I think scientists need other tools to deal with the proliferation of preprints each day and shouldn’t rely on the arxiv for everything.

    Another way is to sort the preprints, based on a network of co-authorship i.e. based your list of the best authors and their co-authors. This is the approach implemented by Jean-Philippe Magué & Brice Ménard over at

    I recommend it- it works.

  • Dr M

    Thank you, Sean and everyone else involved! Being one of those people who is trying to work at the interface of several arXiv categories, in addition to wanting to keep reasonably up to date on some related areas, I really appreciate this subdivision. I’m currently subscribed to several of the astro-ph subcategories, but I hope that when everyone is familiar with the system, I’ll be able to narrow it down a bit more when I see exactly where the interesting (to me) papers appear.

  • Mike

    So it was you and Mark?!? Good for you!

    We’re all very happy about this.

  • Bee

    Didn’t they do a survey last year? I am sure this must have been repeatedly mentioned there as a suggestion for improvement. And an improvement it is indeed!

  • blanton

    Hmm. Eat your vegetables.

  • Robert Cumming

    That was fast: astro-ph.GA is now called Galaxy Astrophysics.

  • change

    When I asked them about this a decade ago, they told me that “An african american will be president of the United States before we get around to subdividing astro-ph.”

  • yet another astronomer

    Hmm — I guess I’m one of the curmudgeons who isn’t necessarily crazy about this idea. I’m perfectly fine with it as long as you can still get the full astro-ph listing every day, which apparently you can, so to each his/her own.

    Why am I not instantly thrilled? I don’t think it’s just “change is scary!” but maybe I’m fooling myself. I do occasionally read papers in every one of these categories, not because I work in all of them but because the basic background required to read papers in most of them is similar — and interesting stuff is happening in all of them! And we talk about papers in all these topics in weekly discussions; it’s a lot less fun if each person has only read the papers that are directly relevant to what he/she is working on.

    The other complaint you mention — that some work doesn’t fit neatly into one of these categories — is I think a real one, too. This setup isn’t transparently great for any basic physics development that is intended to apply equally well to many types of objects. (E.g., last week, I read a paper arguing for a common scaling law giving magnetic field strengths in generic astrophysical convection — so planets, stars, etc. Cross-listing is fine in this case, but seems a little like a patch — maybe it’d be better to have a “physics of astrophysics” type subcategory instead?)

    None of these are deal breakers, of course, and it sounds like many other people were clamoring for the change. So enjoy! (Just don’t take away my full astro-ph mailing, please.)

  • Bo Milvang-Jensen

    I noted that in the current description at clusters of galaxies and galaxy evolution are listed both under astro-ph.CO – Cosmology and Extragalactic Astrophysics and astro-ph.GA – Galaxy Astrophysics. I wonder if that is intentional.

  • BG

    This has been under discussion for over ten years but somehow all it takes is a ten-minute discussion with Sean Carroll and it’s suddenly implemented?

    I mean, I happen to mostly agree with Sean and like the change. But this has a significant effect on the research community, and it’s quite disturbing that after ten years of hemming and hawing that the whole thing seems to have been done on a whim, after consulting all of two people. Is this really how we want our research archives to be managed? I guess (for now) we don’t have much of a choice.

  • Neil B

    I guess speculative astrophysics, like what the universe would be like if had more dimensions or compact ones failed to unfurl just right, etc, would be in “CO”? How about the really speculative ones, like whether everything is just math and whether Bayesian selection deduction can help us about which range of possible worlds we’re in, etc? Some people like to write about that.

  • Doug A

    I’d have to agree with Sam, there must be a better way. Though these categories are useful additional meta-data. is a start, though you have to seed it.

    I’ve seen a version of digg for the arxiv, but nobody uses it.

    I’d like to see something like for the arxiv, where it learns your preferences. Sadly that service trains on all the wrong characteristics to be useful.

    Anybody else have helpful hints on ways to sort the arxiv, preferably adaptively, without filtering out content?

  • Rob

    Yet one more example of the particle physics community imposing more specialization on the astrophysics community. They split the whole thing into six subjects rather than just tag the cosmology ones they are interested in. Extremely stupid.

  • mollishka

    This is devastating. 40 or 50 titles to skim every day is not too much to catch the ones slightly outside your field of study to find the ones that might actually be relevant to you. (The ongoing argument over the Solar oxygen abundance leaps to mind.) But people are lazy and willing to take the “easy” route at the cost of science.

    And ditto to BG and Rob.

  • Sean

    Seriously, people are being offered an option that they can choose to take advantage of, but are not being forced to, and they’re complaining about it?

    I love the internet.

  • onymous

    It’s clearly part of a devious plot: applicants to graduate school will have to specify which sub-arxivs they like to read, and if they pick the wrong one their applications will be thrown in the trash without a second glance!

  • Brian Lacki

    I’m not really against this, and I don’t think we should force everyone to see all the astrophysics papers (as long as the option to see all of them remains). Though, like others I am concerned about how this was decided.

    Isn’t Cosmology and Extragalactic Astrophysics a rather big category? Even given how many extragalactic papers are classified under Galaxy Astrophysics, it’s still a vast range (under the recent submissions, you’ll see that it has 15 papers; the rest have 1,3, and 7-9). In particular, I think there’s a rather big difference between the very-high-energy very-very-early very-speculative cosmology papers and the more mundane z = 0 clusters papers. To use your example, I would guess that there is as big a gulf between the people who read the quantum-gravity cosmology papers and papers on galaxy clusters as there is between the people who read galaxy cluster papers and W UMa light curve papers. In fact, a lot of people who work in galaxy clusters probably have been or are astronomy students rather than physics students, in which case they’ll have some idea of what the variable star people are talking about but no grasp of the quantum gravity papers.

    So, I think there would be some merit to splitting cosmology into an Observational Cosmology and Extragalactic section and something like a High Energy Cosmology section — papers that are cross-listed under gr-qc and the hep categories would tend to fall in the latter. There would be considerable overlap, of course — dark energy, astrophysical detection limits on dark matter, for example — but there’s also overlap between the other categories in astro-ph.

    Also, I wouldn’t want High Energy Cosmology to be sloughed into the current High Energy Astrophysics section. Again, while there’s considerable overlap (especially with dark matter detection), most of the High Energy Astrophysics section deals with the high energy tail of non-cosmological things that don’t require quantum field theory or general relativity to understand. Most High Energy Astrophysics papers have strong connections with several branches/sections of astrophysics (supernovae, the ISM, pulsars, AGNs, even planets). The High Energy Cosmology, on the other hand, tends not to make much contact with most of astronomy, and I think is largely a different community.

    A few other gripes:
    *Gravitational wave production and detection is listed under “Solar and Stellar”. What about gravity waves from merging supermassive black holes (detected with LISA) or inflation?
    *A lot of nebulae might also fall under “Solar and Stellar” besides “Galaxy”, but there’s some overlap.
    *I’ll echo yet another astronomer’s suggestion of a Physics of Astrophysics subcategory. Hydrodynamic instabilities, turbulence, and magnetic dynamos would fall into it naturally, for a start.

    Seriously, people are being offered an option that they can choose to take advantage of, but are not being forced to, and they’re complaining about it?

    “By the way, honey, I took out another mortgage for a summer home. Why are you so upset? It’s not like you have to live there, you know.”

  • Jonathan Dursi

    Sean @ 1:33pm:

    People aren’t complaining that there’s a new option; people are complaining that after 10+ years this finally happened and it’s useless to so many of us. A nice, general purpose, well-thought out system could have been used; instead, a few guys bullshitting over lunch decided that the entire discipline could be most usefully divided up into 6 non-overlapping chunks, and so that’s how it was done.

    A *lot* of thought has gone into categorization of astrophysics, and into tagging of large online databases. Ideas could have been taken from either or both of those, but were not.

    For someone interested in methods (say, different computational or observational techniques) or underlying bits of physics (hydro instabilities and dynamo theory, as pointed up above, or CR acceleration, or turbulence, or…) with wide application to lots of different sorts of objects, the above categorization is *exactly* orthogonal to what would be useful.

  • Jonathan Dursi

    The underlying problem being that this wasn’t done as a useful, interesting way of categorizing astrophysics papers; it was clearly done as a response to “man, am I ever tired of seeing the abstracts for those boring stellar astronomy [or whatever] papers”

  • Troublemaker

    Seriously, people are being offered an option that they can choose to take advantage of, but are not being forced to, and they’re complaining about it?

    Probably because it means that a different, better option will not be soon in coming.

  • Sean

    Look, for purposes of an entertaining post I focused on the conversation I had with Paul and Mark. Which I do believe was important in actually sparking something to get done. But, as others have noted, something like this has been discussed for a very long time, and a lot of input was solicited, including after our conversation. (And, more importantly, it was checked that the proposed scheme would fit as many as papers already on astro-ph as possible.) It had been held up because of people who thought that any sub-division was bad, and the fact that the tiny arxiv organization had other priorities to get to first.

    There are a large number of ways to categorize research in astrophysics: by type of objects being observed, by methods being used, by wavelengths observed or instrumentation employed, etc. No system will ever keep everyone happy. One could have formed committees, and solicited white papers, and held town hall meetings, and something might have been done within the next twenty years. Or one could just do it, and then tweak the system in response to community input. Which will be listened to, especially once the system has had a chance to be tried out, and especially if it’s of the constructive type (“these should be the categories” rather than “your categories suck”).

  • Jonathan Dursi

    “There are a large number of ways to categorize research in astrophysics: by type of objects being observed, by methods being used, by wavelengths observed or instrumentation employed, etc.”

    Yes, rather. Wouldn’t it be lovely if someone had noticed that before the implementation of this one-dimensional system?

    Many such categorization systems which are well tested *already exist*. ApJ has a multi-`dimension’ hierarchy, for instance. A stripped down system with categories for both scale of object (as now), type of object, type of physics, and type of method would have worked. Simple tagging with arbitrary tags would have sufficed. The web has a lot of that already.

  • Troublemaker

    Look, for purposes of an entertaining post I focused on the conversation I had with Paul and Mark.

    Except that the post isn’t entertaining. It’s just a smug “YOU’RE WELCOME” to a community that didn’t ask for your help. You can’t be surprised when it comes off as off-putting.

  • Bo Milvang-Jensen

    I also think that the wider community of astro-ph users should have been consulted before introducing this new system.

  • onymous

    These complaints are, largely, silly. The arxiv always already relied on relatively broad, overlapping, loosely-defined categories (astro-ph, gr-qc, hep-ph, hep-th, ….). They’re a crude sorting mechanism, not a detailed hierarchy that classifies every paper. Anyone is free to go design their own classification scheme where papers can get multiple tags and one can search for papers in the intersection of “gamma rays” and “dark matter”, or whatever else you like. Spires, for instance, gives many different ways of indexing and searching papers related to high-energy physics, and it’s frequently far more useful than the arxiv itself for finding a paper. But that isn’t what arxiv does. Arxiv gives a place to store the papers, and a convenient daily listing. Exhaustive categorizing and tagging is not a useful way to get a daily listing. The most useful scheme for the arxiv is one where the categories are relatively big, so you don’t miss anything, but not so big that you have difficulty getting through the daily list. The new astro-ph scheme looks like a nice compromise.

  • Ben

    Sean, there are a couple of reasons why people might bitch, even if it is unfair to you. Yes, I know that subcategorization has been debated for a long time. In fact, the 2005 reorg plan, see for ex., was significantly worse – it didn’t even have an instrumentation category. I will refrain from speculating on why.

    Reason 1: Astronomers do have to know some things outside their little subfield. You can’t understand galaxies without knowing basics about stellar evolution, and a result on globular clusters or Galactic star formation can influence extragalactic or cosmology papers. This is less true of, say, condensed matter vs. particle physics. Many astronomers like this, and want to encourage it in students, aka “Eat your vegetables.” (Just like they want both faculty and students to attend every week’s colloquia and not skip the extragalactic/galactic/planetary ones.) Now, I recognize that we can still read all of astro-ph if we want. But this is a symptom of the field getting bigger and less manageable and people devoting their time to the endless grant and teaching cycle rather than listening to their colleagues. So lots of people will resent it even if it’s inevitable.

    Reason 2: It activates suspicions that the subdivision is driven by physics and cosmology types who don’t want to see the titles of, much less read, papers about globular clusters or A stars. Of course, there are, in fact, people out there who work on A stars and don’t want to read yet another speculative paper about the microwave background. But they tend not to be vocal about it. Again, this is partly a symptom of a shift in the profession: cosmological tests, dark energy, and the like are attracting a lot of interest, and money, from the physics side and many astronomers are a little worried that this will re-order priorities or make them small parts of large machines. This is another thing that we resent because it’s inevitable. It’s like resenting globalization, or when your neighborhood bank got bought by Citibank. You have to deal with it, but you don’t have to like it.

    So we’re just shooting you, and the arXiv, because you’re the messenger. I hope that makes the holes easier to patch.

  • Ben

    Link in my previous comment didn’t work because of a trailing comma: the 2005 categories are at
    They were: Cosmology; Galaxies; Stars; Earth & Planetary. So it was possible to have a worse system than the 2009 system.

  • Thor

    Hmmm… I wonder how large the training set needs to be to categorize past entries. I’m guessing – I’m hoping, that the threshold is set low enough that past entries are tagged into multiple groups if they contain enough keywords from various categories. Does automatic categorization dump the entries into separate folders or are the entries just tagged? If the system can learn from individual subscribers interests, then it should be that much better. This shouldn’t be that big a deal in the end. Good improvement.

  • Pope Maledict XVI

    The funny thing here is all the glee at not having to look at the titles of papers on the arXiv. I suspect that, for many people, reading the arXiv is a painful duty, and *anything* that reduces the time spent on it is welcome. As somebody said on another blog, the best thing about Christmas is that the arXiv gets turned off for a day. It tells us something about our true feelings about the kind of research we are all doing and [reluctantly] reading about these days.

  • Questioner

    I, for one, appreciate this attempt. For anyone who doesn’t like it, the old option is still available.

    But is there any hope of something like this being done for hep-th, hep-ph, and gr-qc? Why only astro-ph?

    Is there a place where suggestions for categories are being accepted?

  • Questioner

    In my dream world (and this is really dreaming) I could go to arXiv, select from a small collection of keywords, and have a custom RSS feed appear.

    Again, that’s in my dream world; although I can’t really think of a reason why they can’t do it!

  • A. Gradstudent

    Bitch bitch bitch bitch….

    Just kidding.

    Thanks, Sean.

  • Hiranya

    Questioner, your dream world exists :)
    I think I will keep using it rather than this categorization.

  • Omar

    Questioner – you can also use myADS to get a customized astro-ph mailing (or RSS feed), sorted by your own keywords:

  • Hiranya

    Omar, didn’t know about the myADS, I should check that out! The cosmocoffee app (written by Antony Lewis) allows you to filter all the arxiv sections (not just astro-ph) you want to read with your personal selection of keywords, bookmark papers you like with the option to add notes, and also to add them to your local journal club (there is an extra journal club functionality you have to set up first). It’s a neat tool.

  • Mark

    Hi Hiranya. I’d missed the app you mention and it does look nice – thanks!

  • Questioner


  • Mike Hudson

    @Hiranya, @Questioner:

    Almost all of Questioner’s requests are satisfied by cosmo coffee, EXCEPT that while you can get customised arXiv results on a web page, you cannot get an RSS feed of these results (AFAIK)

    Overall, I prefer cosmo-coffee to myADS because I can add author’s names into mix as well as abstract keywords.

  • Hiranya

    Hi Mike, I am afraid I am one of those people who never warmed to RSS feeds. I like sitting down with the day’s “catch” from the cosmocoffee filter with a cup of coffee, reading through and bookmarking the papers I want to read more carefully. I guess the RSS feed is useful if you don’t read the arxiv every day (e.g. while you are travelling)? I can ask Antony whether he can add it. I also get the full arxiv abstract email and if I have time I quickly page through it to see if the filter missed something interesting, but it only does so very rarely once a good set of keywords is in place.

  • Hiranya

    Actually it looks like there is some kind of “feed” here.
    I don’t know enough about RSS to know whether this is what you want but it looks promising.

  • Mike Hudson


    I think the feed above is for Cosmo Coffee forum discussions, not “arXiv new filter”.

    I seem to recall asking Anthony for a feed for the latter, but he said it was not available.

    As you say RSS can be useful to keep track of which ones yo have read and which ones you have not.

  • BG

    In an ideal world, this -would- just be another option for how to filter through papers. But like it or not, the categories the arXiv chooses really do affect who reads what papers. Perception shapes reality and all that.

    What I don’t understand is that I thought there -was- a panel formed to come up with categories, and they even had a proposal put forth… what happened to that? Why was it put aside in favor of the current implementation? Did Andrew Connolly (who’s supposed to represent astrophysics to the arXiv advisory board) have any input? Why is this all so opaque?

    Like I said previously: I support the change. It’s not the “what” that makes me feel unsettled — it’s the “how”.

  • Pingback: Scholar Blog » Astronomipreprintarkivet astro-ph opdelt i underkategorier()

  • Shantanu

    maybe gr-qc could be split up into gr-qc-ex (which focusses on experimental aspects
    of gravity) and gr-qc-th which deals with everything else in gravity.

  • Bruce Macintosh

    So extrasolar planets go where – “Earth and Planetary Astrophysics”? or “Galactic
    Astrophysics”? I’m not strongly opposed, but I think the categories could have been defined differently.

  • Bruce Macintosh

    OK, from the descriptions, extrasolar planets go in Earth&Planetary.

    Not sure where planet formation goes – E&P includes “structure and formation of the solar system” , solar and stellar includes “star formation and protostellar systems”‘; cross-listings might take care of that but that seems spotty so far.

    I’m deeply curious to find out what “stellar astrobiology” is.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Cosmic Variance

Random samplings from a universe of ideas.

About Sean Carroll

Sean Carroll is a Senior Research Associate in the Department of Physics at the California Institute of Technology. His research interests include theoretical aspects of cosmology, field theory, and gravitation. His most recent book is The Particle at the End of the Universe, about the Large Hadron Collider and the search for the Higgs boson. Here are some of his favorite blog posts, home page, and email: carroll [at] .


See More

Collapse bottom bar