For the fourth year, we are collaborating with the ALiEM Faculty Incubator Program to serialize another volume of Educational Theory made Practical. The Faculty Incubator program a year-long professional development program for educators, which enrolls members into a small, 30-person, mentored digital community of practice (you can learn more here); and, as part of the program, teams of 2-3 participants author a primer on a key education theory, practically linking the abstract to practical scenarios.
They have have published their first and second e-book compendium of this blog series and you can find the Volume 3 posts here (the e-book is in progress!) As with the previous iterations, final versions of each primer will be complied into a free eBook to be shared with the health professions education community.
Your Mission if you Choose to Accept it:
The ALiEM Faculty Incubator Program would like to invite you to peer review each post. Using your comments, they will refine each primer. No suggestion is too big or small – they want to know what was missed or misrepresented. Whether you notice a spelling or grammatical mistake, or want to suggest a preferred case scenario that better demonstrates the theory, they welcome all feedback! (Note: The blog posts themselves will remain unchanged.)
This is the ninth post of Volume 4! You can find the previous posts here: Cognitive Load Theory; Epstein’s Mindful Practitioner; Joplin’s Five-stage Model of Experiential Learning; Maslow’s Hierarchy of Needs; Miller’s Pyramid; Multiple Resource Theory; Prototype Theory; Self Regulated Learning Theory and Siu & Reiter’s Tau Approach.
Siu & Reiter’s Tau Approach
Authors: Sugeet Jagpal; Leon Melnitsky; Sam Zidovetzki (@sameddoc)
Main Authors or Originators: Eric Siu, Harold I. Reiter
Part 1: The Hook
Irena, a junior faculty member at a medical school, is asked to consult on the development of a brand new medical school in her state. In particular, she is asked to comment on what should be taken into account in the admissions process. Irena remembers feeling quite confused by the components of the application process, and wonders if she could possibly suggest new angles that would make it easier for future applicants to apply to medical school. Excited about the prospect of being able to think through this de novo, Irena accepts the consulting offer.
As she reflects, negative memories of the application process come flooding back. She recalls having a letter of recommendation uploaded late, and the sheer panic that she felt at the time. She wondered back then, and still wonders to this day, if that missing letter of recommendation would have cost her her medical career. She also remembers working for days on her personal statement, only to find time and time again that interviewers had not taken the time to read it. One interviewer even told her that he did not look at personal statements prior to meeting the applicant – he trusted his ability to assess the applicants himself. He mentioned that in most cases he did not trust their ability to self report.
In addition, Irena was a biology major in college, and seriously thought about adding a minor in the liberal arts to make her application more well rounded for medical school. The added credits would have cost her additional money, and she chose against this as a preparatory course for the MCAT had already depleted her bank account. She ended up doing quite well on the MCAT, but would the money have been better spent on the minor in liberal arts? Would either of those have predicted success in medical school or a medical career?
Irena sincerely hoped that she would she be able to suggest an admissions process that would allow medical schools to select excellent candidates AND prevent unnecessary angst in thousands of college students. She enthusiastically dives into the data behind the admissions process. She is pleasantly surprised to learn that this is a debated topic, and her concerns have been echoed by others. She zeroes in on Siu and Reiter’s TAU approach, drawn to their separation of the cognitive and non-cognitive domains of the admissions process.
Part 2: The Meat
Overview
The TAU approach to non-cognitive evaluation provides guidance to medical school admissions committees on predictive validity of assessment tools currently used for medical school admissions that describes three main principles.
- Trust no one: applicants are not trustworthy because the stakes are too high.
- Avoid self reporting: the weaker applicants are the worst self assessors.
- Use repeated measures: A single medical school interview is not reliable. It is recommended to implement a repeated measures approach to the selection process.
Background
In the highly competitive market of medical school applicants, it is essential for the admissions committee to use reliable and valid selection tools.
Most applicants to medical school are highly motivated individuals who will stop at very little in order to gain acceptance. Currently there are multiple tools available at the disposal of the medical school faculty.
Applicants’ grade point average, Medical College Admissions Test score and the multiple mini interview are among the most practically relevant and statistically significant in correlation to medical school performance.
The TAU perspective recommends having little faith in the validity of the current application process. In particular the “Trust no-one” approach is suggested when it comes to evaluating an applicant’s references, file reviewers or interviewers.
The applicants’ inability to objectively self report is highlighted in this paper. It is unreasonable to expect medical school candidates under tremendous pressure to honestly and reliably self assess on the application.
Using repeated measures to judge the validity of the interview process is deemed highly valuable. For example, multiple interviews are recommended instead of the single interaction in order to effectively predict candidates’ potential performance.
Modern takes or advances
Letters of reference are a common part of an application, both for medical school and residency programs. Letters have been criticized as having ‘poor predictive ability’ and poor ‘inter-rater reliability’ (1). Since the TAU approach was published, a trend toward standardized letters of evaluation (SLOEs) has emerged in many medical schools. The SLOE allows for more of an objective measurement and is not just a personal narrative, which was the primary mode of assessment in previous letters of reference (4). In addition to the narrative portion, a typical SLOE will contain ‘seven competency-focused questions’ which require the evaluator to rank the applicant in one of several tiers (4). This ranking is to be done by multiple evaluators relative to the applicants’ peers which brings the SLOE well within the TAU framework of repeated measurements.
Scholarly activity in the form of publications and presentations has also been cited as important in the application process. A survey of Emergency Medicine Residency Directors listed scholarly activity as being an important factor in assessing an applicant (5). Other specialties such as general surgery and orthopedics have also had scholarly activity mentioned as an important component of an application (6). Scholarly activity can be assessed in an objective way with different weight being given to the strength of the study, publication or presentation in which the applicant participated. If objective criteria are used, scholarly activity can be used quantitatively to add to the global assessment of the applicant. This quantitative measurement fits with the TAU framework by avoiding self report.
The TAU frameworks are becoming more and more important as an increasing number of applications to health professions are ‘non traditional’, providing significant diversity in the components of the application. This has increased the difficulty of comparing applications relative to each other. Using the TAU approach for admissions allows admissions committees to appreciate the uniqueness of each individual application without compromising the validity of predicting future success.
Other examples of where this theory might apply in both the classroom & clinical setting
Other examples of where this theory might apply in both the classroom & clinical setting
The TAU approach has far reaching applications to academic assessments beyond the application process itself. Clinical evaluations of residents and faculty are important indicators of performance and similarly should avoid self report. Classically Emergency Medicine resident assessment is done with one-on-one direct observation (7). This method of assessment is currently limited by evaluator bias and subjectivity. Although there are attempts to decrease this objectivity via interventions such as multiple choice testing or OSCEs, these are not repeated often enough in most training programs to be reliable.
Increasingly, evaluations of residents are milestone based, providing a framework for all faculty to assess residents objectively. These competency based evaluations are completed on a more frequent basis, and compared to other residents within the same program and across other programs to give a more accurate assessment of a residents’ performance (7). This is more consistent with the TAU approach.
Early career faculty new to their role would also benefit from a TAU approach to assessment of their administrative skills, teaching skills, bedside diagnostic acumen, and other roles they have been assigned. Colleagues of new faculty, support staff as well as trainees should assess early faculty in several standardized parameters including teaching ability and bedside clinical skills. Those in existing administrative leadership roles should also give feedback to early faculty on their leadership skills and organizational abilities, instead of relying on their self assessment of these abilities. The principle of using objective data as part of a TAU approach at assessment is applicable to many clinical and educational realms.
Annotated Bibliography of Key Papers
The original paper: Siu and Reiter (1):
This paper explains the TAU approach, stating tools that show predictive validity of future performance (such as the MCAT and multiple mini-interview) are perhaps more useful to the medical school admissions process than the tools that do not show validity (such as personal statement and letters of reference). This paper allows the reader to focus on each component of the admissions process individually.
Looking further into cognitive vs. non-cognitive admissions procedures: de Visser et al. 2017 (2):
Discuss two different procedures for medical school admission (cognitive vs. noncognitive) and try to correlate them to outcomes in medical school completion. The cognitive procedure relied on scores on testing modalities, and the noncognitive procedure relied on noncognitive skills such as behavioral interviews. In their study, they used an algorithm to automate selective into medical school in two arms (cognitive procedure vs. noncognitive procedure) and compared students in each arm for dropout rates, grade in nursing attachment, and practical clinical course in year 3, along with a few other measurements. They demonstrated that students selected through cognitive procedures were less likely to drop out, and that students selected through the non-cognitive procedure had higher grades in nursing attachment and practical clinical course in year 3. They suggest that in order to avoid the drop out of potentially good clinical candidates early in the curriculum that selection at each medical school should resemble the early medical school curriculum.
Applying this to an actual medical school class: de Visser et al. 2016 (3):
Describe using a selection process for medical school in which applicants were tested on tasks (such as taking online exams) that resembled early medical school. They discuss that they designed this to select applicants that would do well in their curriculum, and not necessarily the best doctors. They compared students selected in this manner to students selected via a lottery system. The selected students performed better, but the study did not follow long term outcomes so the long term effect is unclear.
Limitations
The TAU approach has limitations as it allows for the possibility that future generations of applicants will go as far as modifying their applications solely in order to increase chances of admission thereby “cheating the system”. The “trust no one” theory can be interpreted as disrespectful and demeaning to the applicants and those that spend a significant amount of time writing required reference letters and various evaluations.
The TAU approach relies heavily on the MCAT as a predictor of success in medical school. It should be noted that in recent years, MCAT consulting and coaching has become increasingly popular. Those that can afford to pay $200/hour for a professional MCAT coach will have a higher chance at a better score, without having a higher chance of success in medical school itself. This also inherently increases the inequity of the admission process itself.
As far as mini interviews and use of “repeated measures,” there is a gray zone of uncertainty particularly when it comes to how defining how many mini interviews are required in order to conduct a reliable interview process. There is also no clear definition offered of the exact number of so called “repeated measures” regarding the number of interviewers that would satisfy criteria for multiple evaluations as suggested by the Siu and Reiter approach.
Part 3: The Denouement
After a deep dive into the TAU approach, Irena submits her report to the medical school admissions committee. She recommends that they focus on the candidates’ GPAs, MCAT, and also rely on multiple mini interviews. She also recommends a de-emphasis on subjective measures such as personal statements and letters of reference. Given the higher likelihood of a better matched candidate with repeated measurements, she recommends scheduling multiple mini interviews for each potential candidate. She does clearly states in her report that the committee should look at all subjective data with caution.
She goes on to suggest an ideal interview process would involve initial screening of applications based on objective measurements such as GPA and MCAT scores. This would be followed by an interview day that involved multiple mini interviews, with only a brief review of personal statements and letters of recommendation as these were least likely to correlate to success.
Irena strongly suggests that in order to be as objective as possible, the medical school should undertake a training process for their interviewees as well as the admissions committee. By having people meet and determine beforehand what to look for in a letter of reference, for example, there would be an opportunity to create repeatable measurements to create some objectivity in this data. Another example would be to determine how to quantify scholarly activity – this would allow the committee to speak more objectively about the candidates, in line with the TAU approach. She suggests a debriefing meeting after the admissions process so that all of those involved in the process can give and receive feedback, allowing the interviewers and committee to function as a team and to become a reliable resource for each other in the future.
In addition, she suggests that they track their enrolled students and adjust their admission screening process accordingly. By having an iterative process, she expects that they will be able to calibrate their admissions process with time and increase their satisfaction with their medical school class. In an ideal world, they would be able to track their admissions process to success in medical school and then success in the graduate medical education world, allowing them to fulfill their medical school’s mission. If a deficiency or need was identified, for example, in the residency milestones of their trainees they would be able to incorporate that both in their medical school curriculum as well as calibrate, if needed, their admissions process. This would be a very valuable resource for the medical school.
The admissions committee is grateful for Irena’s report. They inform her that they will be asking for her help moving forward as they calibrate their process.
Don’t miss the tenth (and final) post in the series, coming out Tuesday, April 21, 2020!
PLEASE ADD YOUR PEER REVIEW IN THE COMMENTS SECTION BELOW
References
1.Siu, E. and Reiter, H. (2009). Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Advances in Health Sciences Education, 14(5), pp.759-775.
2. de Visser, M., Fluit, C., Cohen-Schotanus, J. and Laan, R. (2017). The effects of a non-cognitive versus cognitive admission procedure within cohorts in one medical school. Advances in Health Sciences Education, 23(1), pp.187-200.
3. de Visser, M., Fluit, C., Fransen, J., Latijnhouwers, M., Cohen-Schotanus, J. and Laan, R. (2016). The effect of curriculum sample selection for medical school. Advances in Health Sciences Education, 22(1), pp.43-56.
4. Jackson JS, Bond M, Love JN, Hegarty C. Emergency Medicine Standardized Letter of Evaluation (SLOE): Findings From the New Electronic SLOE Format. J Grad Med Educ. 2019 Apr;11(2):182–6.
5. Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000 Jan;7(1):54–60.
6. Melendez MM, Xu X, Sexton TR, Shapiro MJ, Mohan EP. The importance of basic science and clinical research as a selection criterion for general surgery residency programs. J Surg Educ. 2008 Apr;65(2):151–4.
7. Colmers-Gray IN, Walsh K, Chan TM. Assessment of emergency medicine residents: a systematic review. Can Med Educ J [Internet]. 2017 Feb 24 [cited 2019 Jun 5];8(1):e106–22.
The views and opinions expressed in this post are those of the author(s) and do not necessarily reflect the official policy or position of The Royal College of Physicians and Surgeons of Canada. For more details on our site disclaimers, please see our ‘About’ page