Today Steinn followed up a nice little post on the role of luck in science (is “lucky” an insult?) with a spot-on description of how being ahead of the curve can damn you in grant proposals:
selected referee comments on “A bold proposal to do something new and interesting”, years 1-3, with added bonus translation
- very speculative, no track record in this area, would be helped by showing preliminary results proving methodology and showing that results will be forthcoming [trans: come on, first do the research then ask for funding, don't you know anything?]
trans: – huh, I never heard of this! Some new stuff. Speculative.
Oy! He wants full postdoc for three years?
Who does this guy think he is?
(Years 2 and 3 are equally good).
I think the reaction that Steinn describes can be summarized in Dalcanton’s Lemma of Proposal Writing: “It is nearly impossible to change a referee’s mind about something they think they already know“. If the reviewer comes to the problem with no preconceived notions (i.e. they’ve never read a single paper in your field), you can really make progress in educating them. Same deal if you’re moving forward on well-trodden ground (say, pushing SN Type Ia surveys to larger distances). However, if the reviewer knows something about your topic and thinks the problem is already solved, or uninteresting, or technically unfeasible, or crazy, 10 pages of perfectly formatted prose and elegant figures may still not be enough to change their minds, even if you are 100% completely and totally correct.
Shifting a referee’s frame requires that you first realize that most readers aren’t going to believe you if you’re talking about something that no one else is. Jumping into a bunch of details that seem sensible to you gets you nowhere, when the referee still can’t figure out why you’d bother even thinking about the question. You’ve got to knock down their frame before you stand a chance of getting anywhere. For example: “Although everyone assumes that stars form from gas, here’s a series of three plots demonstrating that that’s completely false when looked at in detail.” Or, “It may seem that the velocity requirements for measuring doppler shifts to detect extrasolar planets are beyond current technical capabilities, but here’s a series of plots where I show that current detection limits are indeed at a level where a monitoring campaign could detect shifts due to Jupiter-mass planets.” Or, “While the theoretical idea that the moon could indeed be made of cheese does not initially seem compelling, here are three analytical calculations suggesting that the properties of cheese could indeed be superior to rock in explaining the observed lunar properties, and thus that further work on the lunar cheese model (LCM) is warranted.” The frame breaking can’t be just a throwaway line, but must be direct acknowldegement of and attack on the paradigm that a likely reviewer would bring to the proposal.
Under the above Lemma, Steinn got shafted in Year 1 because the reviewer came with a frame that said “This can’t possibly work”, and, by not completing enough of the work before submitting the proposal, Steinn didn’t have enough ammo to break the frame. He’s also right in the “Who does this guy think he is?” comment, since the other way you can break a frame is to have enough of a rep that people know never to bet against you. The whole business is another example of Aspects of the Running of Science That Are True, Probably Unavoidable, But Not Necessarily Fair or Optimal.
May 5th, 2008 at 6:23 pm
Tis very true, and unavoidable really (the only solution would be having a host of Edward Wittens who only work in grant departments, which of course is not the case).
This is why the usual strategy for budding physics is publishing with big names to jumpstart their reputation. Sadly, this is a priviledge of the best schools and unfairly penalizes those at less well known places with less well known faculty, but c’est la vie.
May 6th, 2008 at 1:04 am
“Oy! He wants full postdoc for three years?”
A small suggestion, which will no doubt be hugely unpopular with all faculty everywhere:
Instead of dishing out grants to faculty to hire postdoc slaves, how about instead using the money to fund postdocs directly through fellowships based on open competition, letting them choose what they want to work on and with whom. If the projects that various faculty wish to pursue are good, postdocs will be interested to join and work on them. Letting the fate of project proposals be decided by the interest they generate from postdocs in this way would be at least as reasonable as basing it on referee reports of the type mentioned in the post.
A hundered years from now historians of academia are going to be amazed that something as medieval as postdoc serfdom was still continuing in the 21st century.
May 6th, 2008 at 8:15 am
Amused:
“Instead of dishing out grants to faculty to hire postdoc slaves, how about instead using the money to fund postdocs directly through fellowships based on open competition”
I can see the appeal of that. However, as a junior faculty member one thing I appreciate about my postdoc years is that I was able to devote full time to actually doing science (as opposed to obtaining funding). Writing grants takes time, and as things get more competitive, the funding chase tends to become more time consuming.
Postdocs have a narrow time window in which to produce impressive science. their productivity would drop significantly if they had to devote much of their effort to grant writing (although, perversely, it might be better preparation for being a faculty member).
May 6th, 2008 at 9:31 am
Why not apply for a grant on some other topic and use the money to do the research you want to do? This is what we usually do in theoretical physics.
May 6th, 2008 at 9:44 am
Independent postdoctoral fellowships are a good thing, and lead to good science.
Having more of independent postdocs would also be good.
But, the mistake is thinking that all science funding agencies are focused on curiousity driven blue sky science, most are not.
In particular the major funding agencies for the physical sciences are mission driven to a large extent. This generally implies there are sequences of tasks that need to be carried out by people with particular expertise – for example there may be a need to build some equipment, or to analyse a particular data set, or to model a particular scenario, or to attempt to calculate a known unknown.
Often such tasks are beyond the capabilities of graduate students, at least on the time scale the problem is to be done on, the expectation is that the research requires someone with a PhD. Arguably the faculty involved should do some of these tasks themselves, rather than hire postdocs to do so, but often the labour involved in doing the task on the time scale required is more than is available from the faculty PI – some things require a 50-100% time committment to be done in a timely manner. (Buying out faculty from teaching and service duties is possible but discouraged in many cases and cost-ineffective in other cases).
So… there are grants for postdocs to work on specific pre-defined tasks.
Typically they require 50-80% time commitments, notionally, the rest of the time being available for exploratory independent research.
We can argue about whether this is a good system, but right now that is the system the US uses. It is relatively cost effective and flexible, it is arguably damaging in the long run.
In the short run, faculty apply for grants to hire postdocs, and soak up a large part of the supply of recent PhDs in the process.
There are worse things than being a postdoc.
May 6th, 2008 at 6:19 pm
“Typically they require 50-80% time commitments”
translation: 12 to 19.2 hours a day, 7 days a week.
May 6th, 2008 at 8:17 pm
Working as a postdoc on a somebody’s project isn’t necessarily bad. It can mean that you get access to resources that you would never be able to attract as a young non-established independent postdoc. One could argue that PIs with resources should attract the independent postdocs, and this happens a little, but the agencies that are allocating the resources (telescope time, laboratories money to build instrumentation, theory centers with supercomputers, whatever) have an interest in allocating money to PIs to hire good people. It does no good to cut metal for a fancy facility if there’s not people to help build it and to use it.
Postdocing does not equal serfdom. When it does, this is a problem with either the standards of treatment in the field (for example, biology postdocs have less independence than astronomy postdocs) or the behavior of individual PIs. It is difficult to change people’s behavior and there are relatively few consequences to people who behave badly though. I fantasize that granting agencies should somehow compile and anonymize followup interviews for grad students and postdocs that they have paid for.
May 7th, 2008 at 2:14 am
I disagree a bit with some of what has been said above, but also feel a bit guilty for taking the discussion in a different direction from what Julianne probably intended (sorry, couldn’t resist). So, to avoid hijacking the thread, I’ll wait for a more on-topic occasion to say more about this.
May 7th, 2008 at 7:46 am
I guess the frame-breaking ammo needn’t be “home made” though. Might it not be enough to point out some recent related work by others which had the same effect?
For example, a black hole cosmology proposal related, or relatable, to quasinormal modes could mention how Shahar Hod’s work has (AIUI) overturned earlier studies concluding that these always evolve to the same form.
May 7th, 2008 at 9:11 am
John — agreed completely. But, you have to do more than just toss in a few cites. There are lots of silly papers in the literature, and a non-expert (but generally informed) referee will not necessarily see the citation and get “Oh yeah! Hod 2007! That changes everything”.
And amused — carry on with the discussion as you wish!
May 7th, 2008 at 11:59 pm
Having recently served on an NSF review panel, it seems to me that both Julianne and Stein are pretty far off the mark in this discussion. They seem to think that these proposals would be funded, but for the reaction of some unreasonably hostile referee.
This is not the situation at all in any of the review panels that I’ve been on. It is invariably the case that there are many proposals that deserve funding that can’t be funded because of the ~15% NSF funding rate. With such a small fraction of proposals to be funded, almost any tiny error in a proposal will get it rejected. Plus, many good proposals with no errors still won’t get funded because the acceptance rate is so low. Basically, your proposal has to be judged to be really exceptional to get funded, and many very good proposals are rejected without anyone necessarily being shafted.
May 8th, 2008 at 3:00 pm
Er, I do have some experience on both side of the proposal game…
The post Julianne kindly linked to is very much intended to be tongue in cheek – that time of year, y’know. But, as with most attempt at humour, there is also some truth to it.
For what it is worth, the comments I chose are illustrative but not verbatim examples of some actual extremes. The translations are also not all that far off the mark…
The refereeing doesn’t get to be unreasonable until you realise that “referee 1″ was actually “Prof Joe Doe” in the second example.
There are agencies that are not the NSF where a 15% funding rate would be wonderfully high. And, yes, most good proposals can not be funded.
May 8th, 2008 at 3:25 pm
I also did not mean to imply that following my advice guaranteed success. But, failure to follow it guarantees that you won’t make it into the 15% region. When review committees have to make those nearly impossible choices (as you just had to — and as they say to returning vets, “thank you for your service”!), they’re rarely going to go out on a limb vs funding something that seems like a sure thing.
And OUCH on “referee 1″ = “Prof Joe Doe”. That kind of behavior just sucks ass.