Funnily enough, I’ve slowly been working on a post on what it is like to write letters of recommendation for people at different stages of their careers, as part of a series of posts I’m hoping to write on aspects of being a Professor. Then along comes Julianne’s (rather innocuous and humorous in my opinion) post, and all kinds of craziness breaks out in the comments. So I’m still working on the other post, but I thought it might be useful to provide some unvarnished facts about letters that I’ve picked up over the years. Most certainly everything that I’m about to write is anecdotal, based on my own direct experiences, and acquired through conversations with colleagues. Nevertheless, I have carefully read literally many thousands of letters of recommendation for graduate students, and still more thousands for postdoctoral and faculty positions combined. So feel free to weigh that any way you see fit.
So, some observations:
1. I have never, ever, seen, nor heard of, a straight out negative letter for a graduate student. This applies to letters coming from the rest of the world just as it applies to US letter writers.
2. It is true that letters from European letter writers tend to be more reserved than those from their US counterparts. However, this doesn’t make them any more useful, since what matters is calibration. If a given letter writer is equally reserved when writing about a mediocre candidate as they are when writing about a tremendous one, then that’s just as much a problem as effusive letters for them both.
3. Speaking of calibration of letters, one way committees can establish that is through consistency between the strength of the letter and the student’s grades. Just to be clear, it is, of course, possible that a letter writer believes that a student will be very successful despite unremarkable grades. But if this is the case, then the writer needs to explain why in the letter, so that the committee can make an informed decision about how to rank the file.
By the way, it is also perfectly possible that a person with poor grades and who hasn’t convinced letter writers of their excellence in some other way, nevertheless secretly has the potential to do wonderfully in graduate school and in research. Nobody denies this, but an admissions committee faced with a file like that has very little choice but to deny entry. How would they know to do otherwise? One can’t expect the committee members to be mind readers.
4. When writing for a graduate student, it is quite possible to write a letter that only emphasizes their positive aspects, but still won’t help that much to get them in. In part, this is because one is usually asked to numerically compare a graduate applicant with their peers: something along the lines of “How do you compare this student’s potential for success in research with others at the same level you have encountered? Top 5%, 10%, 20%, …?” There are usually other questions, replacing “research” with other qualities. No matter how much one likes a student, one is most certainly supposed to be honest here.
5. Rather frequently, one reads something in the personal statement that makes the reader wince. It may be immature, it may be a little unprofessional, it may be a little arrogant, or it may show that the student is entirely unfamiliar with the program that he or she is applying to. The last example may rightly sway the committee somewhat, although the truth is that if the candidate is very good, they’ll overlook it immediately. The others are extremely unlikely to have any effect on the candidate’s chances of admission, although the committee may giggle over them a little (the members are, in fact, human).
6. Most letter writers are writing for more than one person in a given year, or to the same institutions in multiple years. If one’s letters are to be trusted, it is crucial for readers to be able to – you guessed it – calibrate one’s letters. This means that even if you are writing letters that truly recommend each candidate, if you feel that there is an honest to goodness hierarchy among the people you’re writing for, then you owe it to both the candidates and the institutions to express that, although how strongly one chooses to do this will vary.
I do not think that these are controversial statements, nor do I think many Professors will quibble with them too much, even though some may have seen a few negative letters, or perhaps find one type of letter a little more useful than another.
Perhaps the most important thing for prospective graduate students in particular to keep in mind is that admissions committees, while certainly holding great power over individuals’ futures, are in fact desperately seeking good candidates, and are willing to overlook all kinds of blemishes, indiscretions, and specific weaknesses if they feel that they’re getting a fundamentally good candidate. A single specific fact about an application is very unlikely to ruin a person’s chances (you’d be amazed at the GRE scores of some students admitted to even the top programs). Rather, the committee tries to get an overall picture of the candidate, and then to rank them relative to other candidates (also taking into account the department’s research needs at a given time). Only then are admissions decision taken.
I have certainly missed some issues and subtleties here. But the basic idea should be clear and, if my own experience is anything like typical, then it should help some of you, particularly prospective graduate students, to understand what really goes on with letters. It is quite terrifying to ask people for letters and not to know precisely what’s said in them. Hopefully it helps to know that mostly, by far, you can rely on people to do what they can for you, without being dishonest (and this is important – you can’t expect them to write that you’re one of the best students they’ve ever seen if they don’t think that is the case).
For the record I personally don’t write negative letters for people. For those that I can recommend at some level, I tell them what kind of letter (generally, no specifics, of course) they can expect from me for the kinds of positions they’re applying for. They can then decide whether they still want me to write.
January 19th, 2009 at 9:34 pm
It’s also worth noting that the best kinds of recommendation letter will be from someone who knows you well enough to write one. I’ve heard stories about people trying to get LoR from Professors of classes they did well in and ended up screwing themselves because they prof didn’t remember, and likely never knew, who the student was. This is mainly a problem with huge lecture classes that premeds and biologists have to take early on.
I’ve also heard that among those that know you well the most impactful letters of recommendation come from people with whom you have directly worked. So, for instance, instead getting a letter from the prestigious professor in whose group you worked but you rarely saw, you’re better off with a letter from the postdoc you saw every day, or etc.
January 19th, 2009 at 10:17 pm
Lots of things come to mind, here is one. It’s not a good idea to get a letter from someone who doesn’t really know you, therefore writing a generally sympathetic letter, but not precisely one full of information distinguishing you from all other candidates. If you are thinking about the next step and you cannot come up with 3 profs that know you well enough, think about ways to change that situation.
January 19th, 2009 at 11:13 pm
Why is this acceptable?
Seriously, why is it acceptable to rank people by some arbitrary, subjective criteria instead of just honestly recommending someone based on the traits, qualities, and skills that you’ve observed? Why do you need a letter that says, “Mark Trodden is better than Bharat Ratra, but not as good as Michael Dine and perhaps the equal of someone like Keith Dienes,” instead of a letter that tells more about who you are, what you were like to work with, what type of environment you create, how you get along with students and faculty, what type of experience they’ve had working with/alongside/under/for you?
Why could I not have a student who excels at one thing and a second student who excels at something completely different, and recommend both at the highest level for their job? It’s abhorrent to me that someone’s opinion and estimation, with no accountability for the veracity of their statements (they don’t even have to write a good letter or show the letter to the person who requested it), can determine the entire success or failure of someone’s academic career.
This doesn’t fly anywhere outside of academia, and I’m really surprised that there haven’t been more lawsuits to force more transparency in this matter.
January 19th, 2009 at 11:30 pm
Why is it that people seem to skim over blog posts and then get all hot and bothered without reading carefully?
I assume your little rant is aimed at ” if you feel that there is an honest to goodness hierarchy among the people you’re writing for, then you owe it to both the candidates and the institutions to express that”. but if you read that carefully, you’ll see that it begins with the word “if”. It may well be, as you point out, that one doesn’t feel that there is an honest to goodness hierarchy among students one is recommending at the same time. Sometimes this is precisely for the reason you say, that they excel at different things – in which case one doesn’t have to rank, and can point out these differences. I’ve done it myself, in fact. But if you do feel that there is a hierarchy, then this is precisely what institutions would like to know
Furthermore, there is absolutely nothing in what I wrote that indicates that providing a ranking (when one chooses to do it) is supposed to take the place of ” a letter that tells more about who you are, what you were like to work with, what type of environment you create, how you get along with students and faculty, what type of experience they’ve had working with/alongside/under/for you?” That would be a silly letter indeed without such information. Which is why I never write them, nor receive them.
Happier now?
January 19th, 2009 at 11:36 pm
I should add, just in case more indignant confusion should arise, that most people I know only use ranking (when at all) when writing for faculty applicants, or occasionally for postdocs. I haven’t seen it for grad students.
January 19th, 2009 at 11:56 pm
I’ve only seen it for grad students when someone has written letters for two students applying to the same program. However, it’s usually handled not as “Susan is better than Bob”, but “Susan is probably has the better developed analytical skills, and Bob shows stronger skills in the lab. However, both would be a tremendous asset to your program.” The comparison is also frequently done when someone has previously recommended a student who enrolled in a previous year, but again avoiding a blanket “better” or “worse”, and instead pointing out areas where the student is particularly strong compared with the former student we’re very familiar with.
January 20th, 2009 at 2:02 am
I write letters with rankings for faculty and very strong postdoc candidates only. I started doing this after being on a few faculty search committees where other committee members repeatedly complained that letters without rankings were unusable. I actually don’t share that opinion, and find certain types of ‘ranking’ letters distasteful. But the fact is that this seems to be a common sentiment, so I have adjusted my letters accordingly to avoid disadvantaging job candidates.
I do understand the deep frustration of being on a committee trying to narrow 120 applicants, 50 of whom are plausible candidates, down to 5 for interviews. Having letters from Spokesman X that say “Jack is better than Carla, and she is better than Mike and Dave” makes that job easier than if one has to play cryptic language-interpretation games. But there’s a big downside when this happens in a field with large collaborations: interview slots and then offers are ridiculously concentrated on two or three people who become the “stars” of that year in the rankings. This leads to a few people getting ten (seriously!) faculty offers and lots of searches failing. With some years’ perspective, I think these superstar postdocs rarely end up living up to the extreme hype they get.
My personal to the dilemma is to rank candidates not against one another, but rank their promise against well-known people one career step higher than the position. So postdoc candidates can be ranked against current junior faculty, and assistant professor candidates against recently tenured scientists. This seems less distasteful to me for some reason, and it also introduces just a little more noise into the evaluation process.
January 20th, 2009 at 2:08 am
Thanks Mark and Julianne for sharing your experiences. And I am looking forward to Mark’s other posts. (I hope you don’t mind me calling you by your first names. I thought it would be better to be informal on a blog.) Here I would like to add an applicant’s perspective. Perhaps, I should have just kept my mouth shut, since I have not yet been admitted anywhere. But I feel compelled to express my thoughts. So here I go…
First of all, I understand the dilemma of the admission committee and living the pain of being a prospective graduate student with an imperfect score card. However, I would like to blame the system for the suffering of the both parties. I believe, a little modification to the system can ameliorate the situation.
From what I understand, the admission process is inherently subjective. And I believe, this is how it should be. Faculty members are looking for someone young( not in age but in experience, I suppose) and smart whom they would like to work with, and the students are looking for someone experienced and supportive whom they can learn from and work with. This is sort of like a match-making process. Thus subjective. However, the admission committee’s effort to convert this subjective process into an objective one results in a semi-objective semi-subjective mess. It becomes like an arrange marriage which arguably works but lacks all the fun and enthusiasm – by the way I know this because of growing up in Bangladesh.
I believe, interviewing students before accepting them would solve this problem. For example, in current system about 400 students apply, 50 of them are offered admissions and 25 accept the offer. What I propose is this: instead of cutting down to 50 all on a sudden, identify 150 applicants whom few of the professors are interested in; let these 150 lucky students know that so and so professor would like to interview them over the phone or in person. For example, Professor A & B interview 6 students and pick out 2 of them. This way, you have 50 students.
The above mentioned system is much more effective. Those 150 students would have a chance to talk to people they wanted to work with and the professors had a chance to talk to the student they were interested in. At the end of the interview or the communication both parties would know whether it is going to work out or not – nothing need to be said explicitly; just like a date. I agree that it is a little more work, but this extra effort would be beneficial for both the university and the student.
January 20th, 2009 at 2:29 am
Letters of recommendation… It makes me feel bitter and brings tears to my eyes remembering my own past struggles as well as insecurities surrounding obtaining them…
Can you scientists not come up with a better way to evaluate the quality of a candidate than such subjective and incomplete measures as just three or four letters of recommendation? I especially found it appalling many fellowship/grant programs for grad students/postdocs required several letters of recommendation. If an advanced grad student or postdoc’s research proposal is decent, why should you care who is conducting the research? Shouldn’t a Ph.D. enough to demonstrate your qualification?
Fear of litigation has already made the “job reference” practice obsolete in many other sectors, as Ethan mentioned. Why does this practice continue in academia?
January 20th, 2009 at 3:11 am
[...] Funnily enough, I’ve slowly been working on a post on what it is … Go to Source [...]
January 20th, 2009 at 4:29 am
ts: you seem to think that the letter of recommendation is the only basis on which the choice is made. That’s not the case. Interviews, past publication record and, where appropriate, research proposal all have a role. In fact in my institution we don’t even see the letters of recommendation until after the interview stage, though I think that’s unusual.
As for ‘Shouldn’t a Ph.D. enough to demonstrate your qualification?’: the short answer is ‘no’. When we have 50 applicants for one post, all with PhDs, we need some other way of distinguishing between them.
January 20th, 2009 at 4:47 am
I think Milton has a point here. An important criterion for a successful PhD is the sympathy between the PhD student and his supervisor. What good is it if an applicant has perfect grades and maybe his former profs and the committee members like him, if the supervisor (prof or postdoc) he is supposed to work with for several years does not like him? These things are very subjective and only the designated supervisor himself can decide whether he wants to work with someone or not. No admission committee can take this decision for him. Therefore PhD students should be chosen by the potential supervisors and not by a committee (the committee could take a preselection though, but a rather loose one).
January 20th, 2009 at 5:29 am
Tobi, good point. It seems to me though what you suggest is grosso modo what happened when I was admitted in engineering (M. Sc.A., a French acronym) and what happened when I was admitted in humanities departments (South Asian Studies and Religious Studies, M.A and Ph.D., respectively). Are science departments different?
The way I understand it (from explanations from people who know), one or more faculty member has to express interest in actually working with specific candidates. So a candidate who does not generate interest from any faculty member (despite a stellar application file) will not be given an offer. This is followed by negotiations with the whole committee when there are more candidates picked by faculty members than spots available.
Milton does have a point but I think Milton’s suggestion can already be implemented, no? For my application in humanities departments, I asked to talk to faculty members before sending in my applications. I specifically wanted to speak to people I would be working with. In all cases, setting up a meeting was possible. Being there and talking to faculty helped in two respects. 1) I eliminated a potential department from the running after talking to their faculty. (Not a bad department but I found that my interests and theirs diverged.) 2) I established myself as a real human being, genuinely interested in the departments I was visiting. Do science departments not allow potential applicants to visit?
(A quick Google search tells me that the physics department at MIT allows and even encourages visits:
http://web.mit.edu/physics/graduate/applicants/gradvisit.html
Is this an exception?)
January 20th, 2009 at 6:19 am
I heard from several older physicists when I was a grad student that Albert Einstein was so free with his letters of recommendation that they eventually became almost worthless (as recommendations – they would probably have some monetary value now).
January 20th, 2009 at 6:22 am
Martin: No, of course I didn’t mean that the letters were everything, though my personal experiences seemed to indicate that I was much more likely to get serious considerations by those who apparently knew my letter writers (a matter of course).
Also, I didn’t mean at all that a Ph.D. is quite enough to judge candidates. What I meant is that I felt it was not very fair to weigh letters of recommendations in distributing (monetary) resources such as fellowships and grants when the merit of candidates’ research programs should be the most important consideration. If you need to break a tie among candidates who are of similar quality in terms of their research programs (demonstrated in proposals) and ability (Ph.D. minimally qualifies I hope), why don’t you just roll a real dice rather than rewarding the already rewarded, in terms of human resources (i.e., matching with advisor, ease of finding “good” collaborators, etc.)? That was what I meant.
January 20th, 2009 at 6:47 am
I’m concerned about the flak these posts are generating, which demonstrates a complete misunderstanding of how the system ought to work. My view, which I expressed on the other channel, is that letters of recommendation are not just a service for the candidate but also to the selection panel. All graduate schools receive many more applications than they can accommodate so they have to choose those who will fit in best. This involves all sorts of quantitative information as input but is still an enormously difficult task. Letters from referees can be useful in that they may bring out positive aspects of a given application that the selectors may have missed and/or they may point out the students weaknesses. The latter is not with the intent of sabotaging the application, but in order to give as candid as possible an assessment of the strengths. In the vast majority of cases, those students that apply to graduate schools have very many more strengths than weaknesses. Certainly by far the majority of reference letters I have written have been overwhelmingly positive. I feel obliged to paint as positive picture as I can without being misleading. But it is not acceptable to omit information that might be relevant in a negative sense, especially if it is serious (such as a disciplinary offence). Indeed, to omit such information from a formal reference is considered illegal in the UK.
January 20th, 2009 at 6:53 am
Makes sense to me Peter.
January 20th, 2009 at 6:55 am
ts: to be blunt, because institutions aren’t in the business of being ‘fair’ in the sense of deliberately blinding themselves to information that might help them to discriminate between candidates. They’re interested in getting the best possible value for the considerable investment in time and money that they are proposing to make when they hire someone. Do you think it’s ‘fair’ to ask them to behave any differently? Does this duty to be ‘fair’ exist for all employers, or is it just academia, and if so, why?
I agree with you to the extent that I think letters of reference should be given relatively low weight in this process, but if it comes to breaking a tie, I would choose a candidate with good references over a candidate with lukewarm ones every time (given apppropriate referees), simply because it would increase the a priori probability that that person would work well with me and my colleagues, which is the key unknown factor for both postdocs and PhD students.
January 20th, 2009 at 7:28 am
When I evaluate postdoc and junior faculty candidates, I look ONLY at their research papers. It doesn’t take long to look through some of their papers and make a decision. One can get a good idea in only a few minutes, and read more carefully later in the selection process. I completely ignore letters of recommendation.
When I write a letter, I write–”so and so has done xxx and this is important IMHO because yyy and difficult because of zzz.”
This procedure is based on the assumption that their research is theirs and not done by their Phd advisor–which, sadly happens more and more.
My Phd students are told ( as I was): ” You choose the research problem; you solve it; I will review your solution and decide it it is correct and worthy of being submitted to the committee in consideration for a Phd.”
Academia a a rigged system:
Academia is a rigged system–students who go the right schools, and get good ( and completely subjective) letters from the right people go to the next stage. Often, the right schools have easier courses, and easier grading than the wrong schools.
This process starts in PREP school–which makes admission to the right universities
much easier–even if the prep school education ( as I have observed in many years of university freshman classes) is no better than a public school.
Want to win an Intel Science Scholarship in High School–then get into a “research relationship” with some uni
prof–for whom you are cheap hands–and submit that as a project. Essentially all winners do that. I view this as
cheating. They then have their pick of the right schools etc.
Advisors
give so much help with the Phd that they often write it for the student. GRE’s are taken, but so are prep courses that are based on answer keys to previous exams.
Grading at all levels in courses is unfair, because students who belong to the right scoial networks ( like frats that tutor their members and have a file of old exams with answers) get an easier ride than truly independent
students who go it alone.
At the postdoc level, there are fancy prizes, and these are given by committees of the right people–and their students ( as well as students from their network or uni) tend to win. So, if your applicant has a Sloan grant or a Macarthur or a Presidental Research Award, does that really mean they are talented at research? No, but, they will get the job.
And , of course, connected Postdocs have a network to help them with their research ( without attribution, or with attribution–in the latter case, the student puts a very impressive list of names as “useful conversations” in the paper). I read that list as ” So-and So did this paper for me,
When the time comes to get a job, letters are written ( almost always not to be seen by the student, postdoc or junior faculty member) and are
totally subjective. People from the right schools get the good jobs–the ones at the right schools, which come with early tenure, easy acceptance of grants ( based on the schools prestige) and lots of resources.
Nobody has ever done any study to see if students from the wrong schools, or with poor letters–given the same advantages above would perform as well.
And, no , this is not sour grapes, as I am a tenured professor at a decent university ( in the top tier). But,
I recognise the rigged system. I benefited slightly from it, at the letters level, probably.
January 20th, 2009 at 7:44 am
Gaming the system:
If you are in a theoretical science ( physics or math etc.):
Aquire a group of friends in your early career, and make this arrangement: ” I will put my name as a co-author on your papers,” You can get friends in several different fields of your speciality–and use as many as two on any one paper.
Soon, you will have lots of papers, and your well connected advisors will push these papers–as will your co-conspirators advisors.
That is how young people can have seventy five or eighty papers in fields where a person usually retired with twenty or thirty papers. The pressure for having lots of papers is now far greater, and students have adapted to it.
Sadly, hiring committees have not understood how easy it is to game the system.
Or perhaps they just don’t care–if you hire that guy, you will have lots of papers coming out of your group.
In fact, if you hire the people who gammed the system in the ways described in my previous post, you will get all the advantages of connection for your group.
It’s all very sick.
January 20th, 2009 at 7:47 am
Even better, get friends to do this in quite DIFFERENT areas in your field. Then, you will be praised for the amazing
width of your research.
“wow, this guy has done important work in cosmology and also in theoretical optics, and in nonlinear systems…”
HIRE HIM.
January 20th, 2009 at 10:35 am
All this is very scary.
January 20th, 2009 at 10:48 am
[...] Letters of Recommendation – Assorted Observations [...]
January 20th, 2009 at 11:59 am
Over the past 12 years I have written 40 or 50 letters of recommendation for undergraduates applying to grad school, or for scholarships, or for jobs. I’ve always resisted ranking them against a peer group (e.g., all math students at my institution who have gone on to do graduate work) partly because I do not believe it is possible, or even desirable, to do so, but also because the peer group at any institution is unique in ways that the people reading the letter are not likely to understand.
Therefore I find it disturbing that in the past two years, most of letters I’ve written must be submitted as part of a web-based recommendation form that requires you to fill in every field, including a ranking of the applicant.
January 20th, 2009 at 12:17 pm
Mark,
I read your “if”. I disagree with you that ranking candidates against each other is ever acceptable. I don’t think it’s ever tasteful to say, “Carla is worse than Stan, but better than Mike and Dave,” and I don’t think it’s ever useful. If I were Carla, and I didn’t get a job and you wrote me that letter, there is legal precedent and good cause for me to sue you.
That said, I know what goes on in departments, and I know what letters of recommendation often look like for faculty, postdocs, and grad students. This type of ranking is rampant, and as far as I can tell, never objective.
The major point is that I think transparency is sorely needed in academia, especially when it comes to letters of recommendation and the hiring/acceptance process. You want to get rid of things like gender and race biases in hiring across the board? This is, I believe, a necessary step.
I think you’ve detailed pretty well how the system works, and I’m not attacking you for it. But I’m pretty dismayed at the archaic system, and think it needs to be challenged, not just expounded and accepted.
January 20th, 2009 at 1:06 pm
Yo Penny, shhhhhh! Don’t let the cat out of the bag.
January 20th, 2009 at 4:52 pm
[...] the current DEFCON 1 level of anxiety about recommendation letters, and my belief that more transparency is usually a good thing, I thought it would be informative to [...]
January 20th, 2009 at 4:55 pm
Martin: I think you keep misunderstanding me. Though I personally do not like the practice as a job seeker, I cannot really be against those who use letters of recommendations when individuals (or institutions with their own characters and goals) are hiring somebody. There is no incentive to be fair when their premise is not build on being fair to everyone; society after all is made of personal connections. What I felt is unnecessary is to use those letters when research programs, funded by public money, are evaluated. I’d rather have my tax money used for good science than anything else. Who specifically does that good science is not that relevant for society.
January 20th, 2009 at 6:50 pm
[...] follow-up post tempted to defuse the issue with an injection of common sense, but it remains to be seen whether [...]
January 20th, 2009 at 7:07 pm
(the members are, in fact, human)
LIIIIIEEEESSSSSSSSSSSSSSSSSSSS!!
January 20th, 2009 at 10:58 pm
Hey all. Whew! Hot in here! I am a co-contributor to this blog, I serve on the Physics GRE Committee (we write the actual exam, which I realize you detest), I serve on our department’s Graduate Admissions Committee, and I write letters for undergraduate students heading to grad school. This afternoon I read about 35 applications to our program, from all over the world. I think I know and understand the system.
The system is imperfect, there is no question. We want to admit students who have a very good chance of succeeding in grad school, and not all of the people we admit do succeed. The information we have to evaluate whether they will succeed or not is their academic record, their test scores, their own statement about their experience and interests, and letters of recommendation. For students on the edge of admission, all of these are in play. For truly borderline cases (on paper), I do sometimes talk to the candidates themselves, and it can make a big difference.
But all of you applying to grad school really have to understand that there is a range of abilities, and there are a number of dimensions to what we even call “ability”. Everyone is different. And some students are really excellent in several dimensions, and it simply shines through in their applications. I really don’t think we are missing too many truly excellent people in this process.
What I worry about is that we fail to admit students who have the *potential* to become excellent, and people can and do grow and change. And there are some students who look great in their application and then fade out in grad school.
The bottom line: the system is not perfect, but neither are the students the system evaluates. Rest assured, though, that the people on admissions committees at all the big research universities are doing their best to give you, the student applicant, the best chance possible to show what you’ve got.
One last piece of advice: don’t hesitate to send personal email to professors that you want to work with. It almost certainly won’t hurt, and can make a big difference in your chances. Don’t be surprised if you don’t get an answer, because professors really are very busy. But don’t be afraid…
Good luck!
January 20th, 2009 at 11:00 pm
[...] One of the less glamorous things also less appreciated (by the public anyway) that professors do is to write letters of recommendation. I do this for students trying to go to grad school, grad students trying to get post-docs, and for post-docs trying to get faculty positions. Here’s a look at the process from Cosmic Variance. [...]
January 20th, 2009 at 11:04 pm
P.S. Doing research as an undergrad is nice, but not essential to getting admitted…
January 20th, 2009 at 11:10 pm
It’s becoming almost essential in astro. There are so many REU programs compared to the number of astronomy majors that very few applicants have not had some exposure to research. We still will admit generically smart people with little research experience from time to time, but so few actually cross our paths that I can’t accurately assess if we’re biased against them or not.
It’s probably less necessary if you’re applying to a physics department intending to concentrate in an astro-related subfield (neutrino detectors, dark matter detection, etc), however.
January 20th, 2009 at 11:24 pm
hmm….i never did an REU but I did spend a couple years on a project w/a professor at my university (though I can understand how some students may not have that option depending on their undergraduate institution)
has the # of reu’s exploded that much in the past couple years in astronomy/physics?
January 21st, 2009 at 12:28 am
Doing research with a professor in your home department is fine too.
My impression is that a lot of departments have hired faculty in astronomy over the last ten years or so. It’s become more interesting to physicists, so even colleges without a separate astronomy degree now have an astronomer or two lurking the halls. Moreover, I think with the wealth of publicly released data from large surveys and space-based telescopes, it’s much easier for faculty to have interesting research programs even at a smaller college. Back in the day, if you didn’t own a 4m piece of glass somewhere, you were pretty much screwed trying to run a cutting edge program through only allocations from the national observatories. Now, you can do hot stuff no matter where you are. This has greatly expanded the research opportunities for students, beyond just what’s available through REU’s (which I think have expanded as well, but I’m less sure).
January 21st, 2009 at 2:45 am
ts: Whether the money is ultimately public or private, the institution has a duty to spend it effectively: that means choosing the best person to do the research: and that means taking into account all the information that one has about the candidates.
January 21st, 2009 at 9:26 am
As a possible antidote to the view that letters are a tool of the Man to keep good scientists down (and its corollary: that you’ll never succeed in academia if your advisor doesn’t like you or isn’t famous), let me offer two anecdotes.
1) I served once on a panel to evaluate applicants to a postdoc fellowship which has the unusual feature that it doesn’t take recommendation letters. The application is purely proposal + CV based. On the one hand, those of you who hate the idea of letters could probably use this as slight ammunition for the idea that they could be done away with (though I have no great ideas about how you’d do that in fields where the norm is to be one of many authors on a huge collaborative team). But on the other, I can tell you that a surprising number of the candidates we picked purely on the basis of their research proposal (plus, secondarily, publication record etc) ended up being the folks who also got offered lots of other positions that _do_ take letters into account. What people say in letters, and what shows up in other — perhaps more objective — measures are not orthogonal.
2) I know someone — now a professor at a good research program, much honored by many people — who, while still a grad student, had to testify against his PhD advisor in court. So let’s just say they weren’t on the greatest terms. One bad letter didn’t kill him; it won’t kill you either.
January 21st, 2009 at 12:44 pm
Yet another astronomer:
Your anecdotes are enlightening! But at the level of anecdotes I can also show nearly the opposite, at least at the level of grad school admissions (where this recent discussion started).
I attended Caltech as an undergrad back in the 1980’s, and I can say that we had more than our share of folks who were undeniably brilliant but weren’t necessarily recognized within the academic system. I can discuss one of these cases whose numbers I happened to know. This fellow, quite a bright light and much smarter than me, graduated in physics with (if I recall correctly) a 3.5 in-major GPA and a 960 physics subject GRE. No one had any doubts that he knew his stuff! and he, along with the rest of us, applied to the usual first-tier graduate programs. But it came as a surprise to many — would it surprise you? — that he was rejected nearly everywhere he applied, including Berkeley, Chicago, Princeton and MIT. I don’t know any more details, but it’s only reasonable to infer that his recommendation letters were somehow poisoned and that’s what did him in.
This is not to say that any injustice was done, necessarily; maybe there was a good reason he couldn’t get good letters. But it is a clear case of where a bad letter, while it might not absolutely kill you, can definitely keep you out of the first tier. So I think Mark’s advice is good: one should take great care in whom to ask for recommendations, and be as sure as you can that they’ll make you look good. After all, when you apply to grad school this is pretty much the only degree of freedom you have and so you might as well use it.
January 21st, 2009 at 6:21 pm
Martin: I don’t disagree with you on the usefulness of reference. My premise is that letters of recommendation are very useful information when used properly, and I do of course tend to value the words from those whom I trust myself. The only problem is their vulnerability to all sorts of calibration/human biases and our tendency to succumb to authority.
Given that possibility of misuse, I just think that letters of reference should not be essential at all when ranking research proposals and programs for distributing resources specifically made available for them. Having more information doesn’t always help in a good way. People have been struggling in finding proper ways to evaluate others, hiding and unhiding parts of identities like name, gender, sex, race, sexual orientation, etc. for so long in order to fight innate prejudices that having access to more information introduce.
As yet another astronomer’s anecdote suggests, in the end I don’t even think abolishing recommendation for grant/fellowship programs will change anything. It will simply keep the door open to more ideas and people when something goes terribly bad for few unlucky ones. The current system is so inbred that the cost of failure in establishing personal relationships seems just too high.
January 21st, 2009 at 8:15 pm
The sad, but perfectly understandable, truth is that if a letter writer ranks
candidates then that is ALL that the reader will see. Nothing else in the letter
will matter (oh, I’ve probably read and written as many letters than the youngish Cosmic Variance folks). Milton has it exactly right that trying to quantify an inherently subjective process is destined to fail (and isn’t very smart, to be blunt). As scientists, we like to
think that EVERYTHING is quantifiable, when wise people realize that is just not so. Oh, you’ll feel better if you rank candidates, but does that really outweigh the harm you can do? The medical profession does (very) occasionally get things right in their training…
January 23rd, 2009 at 2:57 am
ts: We started off by talking about hiring PhD students and postdocs. I think if these people have ‘failed to establish personal relationships’ that’s an important piece of information.Of course at that level one cares very little about who writes the reference.
If we’re talking about grants, that’s an entirely different situation: the funding body doesn’t have to live with the awardee for the next three to six years, so it can decide based on the science. And in my experience it does. Of course an element of ‘can this person, or these people, really deliver what they are promising?’ enters into the assessment of a grant proposal — but that can be taken into account by the referees. I’ve never been asked to referee a grant proposal from a person or a group that I didn’t know, and my knowledge about what they’ve done in the past feeds in to what I expect them to be able to do in the future.
For fellowships I admit it’s a bit tougher. The fellowship selection panels will not know the candidates, especially for a fellowship over a broad discipline, so they obviously need some input from others. At a junior level they may find they can’t find external referees who know the candidate either. I am told by people who’ve been on those panels in the UK that the external referees, who go basically on science and standing in the community, get much more weight than the referees chosen by the candidates, but it may be different elsewhere, and if you wanted to get rid of the letters of reference from *that* system, I wouldn’t object.
January 24th, 2009 at 9:34 pm
One
I think it is ugly and arrogant to describe someone as excellent or average. Obviously, you don’t know that. It is an unmeasurable. I do think it advantageous to describe someone’s output in those terms. A subtle difference? I don’t think so.
Two
Different disciplines have different weighings but isn’t a person’s demonstrated ability to 1) ask the right questions 2) answer the questions right 3) work well with others – defining what you want in a applicant?
Three
Any perfect system must be past-looking. Isn’t it as likely as not that as you aging adepts “perfect” your system that you’ll come to regret that you got what you designed it for? Generally clones express no new phenotype. There is a reason why (general) AI has made little progress. Something about any sufficiently complex system containing unprovable truths and unprovable falsehoods. Point is, perfecting the system almost surely is counterproductive (except in making the rule maker feel good about spending less time and energy with this very difficult job).
Four
Any of you “rankers” keep statistics to determine how well your decisions actually predict outcome? Why not? Remember Harry Potter was rejected by the big publishing houses – by the people who get PAID to be right.
Five
Too many candidates? Wonder if there is an under used resource somewhere that could hugely streamline the process of communicating with them. Letters of Application? Lol. Why not clay tablets and an oven? E-Mail, Blog, Twitter, Chat, IM … anyone?
Six
Is it the case that in a “perfect world” all matriculates should graduate? Does failure generate any value? Also obvious. The whole buffalo has a function and purpose – in a perfect world – but are we wise enough to understand?
May 1st, 2009 at 12:46 pm
[...] post on the subject). As for the letters of recommendation for students, see also here, here and [...]