#KeyLIMEPodcast 245: Speaking in Code?

Reference

Saudek et. al. 2019. Pediatric, Surgery, and Internal Medicine Program Director Interpretations of Letters of Recommendation.Academic Medicine. 94

Reviewer

Linda Snell (@LindaSMedEd)

Background

Letters of reference (LOR) are important for resident selection; they provide valuable information about candidate’s work habits, interpersonal skills, teamwork and other attributes which might be difficult to get from other sources. There are many publications going back decades about how to optimize the letters, training the letter writers, whether LORs should be standardized, and mentioning the challenge of providing honest information in this ‘high stakes’ document, particularly when the trainee may have requested the LOR.
One aspect of LOR is the language used to describe applicants, and how these descriptors have come to mean more than they suggest – e.g. ‘imminently about to blossom’ = lowest quartile.
The authors did a survey study in pediatrics showing that letter phrases convey code, but it was unknown if these results were generalizable in other disciplines.
Understanding the ‘codes’ is important and may help faculty write more understandable LORs

Purpose

To describe interpretations of commonly used phrases in LORs by PDs other than in pediatrics.
To identify areas of agreement or variation between pediatrics, surgery, and IM LORs.

Key Points on the Methods

National cross-sectional survey of PDs in surgery and IM and compared results with previously published responses from pediatric PDs. Surveys were sent through IM and PD national associations
Used a similar survey to the previously published study in peds however added questions asking where they would rank a trainee described with various phrases, and asked “Have you ever written a letter of recommendation using
code words or phrases to describe a student that you are not explicitly stating?”
PDs rated

• 13 letter features (e.g. Description of depth of interaction with applicant , specific behavior traits of the applicant, applicant’s abilities, Summative statement on strength of recommendation, personal stories about the applicant , Involvement in program/hospital activities or community service activities, Participation in research) on a 5-point Likert scale (1 = not at all important to 5 = very important)
• 10 applicant abilities(e.g. Work ethic, Trustworthy, Team player, Professionalism, Compassionate, Maturity, leadership, Resilience, Resourcefulness , inquisitiveness) on a 5-point Likert scale (1 = not at all important to 5 = very important)
• 14 phrases/ summative statements(e.g. performed as expected, improved, I would rate in top 5%, would like to have them on my service again) on a 5-point Likert scale (1 = very negative to 5 = very positive).

The modified survey was not validated.
For purposes of analysis 1&2 were grouped together as not important/negative, and 4&5 were grouped together as important/positive
Appropriate stats used to look at differences between items and specialties (Kruskal-Wallace anova, Mann-Whitney, Cliff D, Bonferroni, principal component analysis)

Key Outcomes

Response rates: 43% peds, 55% surgery, 42% IM
Most (73-85%) thought LORs an important component of selection, and that a poorly or well crafted letter may have influenced their opinion and rating. (61-70%; 86-87%)

1. Importance of letter features – differences between disciplines:
   • letter ≥ 4 paragraphs, (IM> peds)
   • academic rank of letter writer, (surg>peds)
   • including a summative statement, (surg, IM > peds)
   • involvement in community service, and involvement in program/hospital activities (IM, peds > surg)
2. Importance of abilities – difference between disciplines:
   • leadership , (Ped, sugery > IM)
   • compassion (peds, IM > surg) more important than did
   • resilience (surg > IM & peds)
3. Letter phrases could be grouped into positive, neutral or negative; with some differences between specialties as in table. All 3 specialties considered “exceeded expectations” a positive phrase; “I recommend” a neutral phrase; and “showed improvement,” “performed at expected level,” and “overcame personal setbacks” negative phrases.

Key Conclusions

I conclude that there are important features of a good LOR, and these vary with discipline.
As well, the abilities viewed as important vary with specialty, although work ethic, teamwork, trustworthy and professional were highly rated by all.
The authors conclude “PDs use code language to describe applicants, especially those who are below average. These results have implications for faculty development in letter writing and advising students re selecting LOR writers.”

Spare Keys – other take home points for clinician educators

Not comparing identical surveys between specialties
Survey not validated
However …
The RR <60%, but that is quite good for a med ed mailed survey
The authors ask an important practical question

Access KeyLIME podcast archives here