Future doctors are tested on critical thinking and problem-solving in admissions exams.
Applicants with strong critical thinking and analytical reasoning skills on the Medical College Admissions Test (MCAT) may perform better on the Acuity Insights Casper situational judgment test (SJT).
The 2023 Health Professional Student Association/Student Doctor Network (HPSA/SDN) Situational Judgment Test Experience Survey revealed an association between those who self-disclosed high Casper results (in the 4th quartile) and high scores on the MCAT Critical and Analytical Reasoning Skills (CARS) section.
Dr. Kelly Dore, Acuity Insights’s Co-Founder and Vice President of Science & Innovation, was enthusiastic about this finding.
“Our research, and that of our partners, highlights Casper’s ability to bring forward non-academic aspects of an applicant and its positive impact on leveling the playing field in admissions. By complementing academic assessments with a nuanced evaluation of personal and professional competencies at the beginning of the admissions process, Casper enables a wider range of applicants to be considered. We partner with hundreds of programs committed to embracing fairer, holistic admissions and opening doors for applicants to showcase their full potential. SJTs like Casper offer opportunities for all candidates to authentically demonstrate their strengths and life experiences, ultimately contributing to their success in programs and future careers. We welcome and appreciate the feedback we receive from applicants and advisors, like what is presented in HPSA’s report.”
Testing for interpersonal skills with situational judgment tests
Students applying to medical or health professional schools demonstrate academic mastery through grades or content-based exams such as the MCAT. However, medical schools and healthcare workers also want applicants with the highly developed interpersonal skills and ethical values necessary for effective patient or community care. SJTs are standardized assessments that evaluate a candidate’s “soft skills” essential for working in teams, adapting to new challenges, and becoming effective leaders.
Over a decade ago, professors at McMaster University developed an online assessment called Casper (Computer-based Assessment for Sampling Personal Characteristics). By focusing on a candidate’s social intelligence and professionalism in a standardized manner, admissions committees gain information about a candidate’s shared professional values as a future colleague-in-training.
Examinees taking the Casper assessment are evaluated on ten competencies based on 14 situations in a two-hour computer-based test. Each situation is judged by a calibrated, independent judge who assigns the response a score between one (poor) and nine (excellent). The scores from these scenarios are combined to produce a statistical score based on the test-taker cohort. After a few weeks, applicants are given their quartile, while individual schools are also provided with the candidate’s percentile rank.
While many companies use SJTs to develop a shortlist of candidates to interview for job hires, Acuity Insights encourages programs to incorporate the Casper results into a more holistic application review process.
According to the Acuity Insights website, Casper measures 10 competencies: collaboration, communication, empathy, equity, ethics, motivation, problem-solving, professionalism, resilience, and self-awareness.
Focus on analysis and problem-solving skills: Casper scores may correlate with high MCAT CARS performance
In contrast, the Medical College Admissions Test (MCAT) administered by the Association of American Medical Colleges (AAMC) is the gateway entrance examination for US and Canadian medical students. While designed to test an applicant’s foundational knowledge for medical school, it also evaluates higher-ordered critical thinking and problem-solving skills. Specifically, “critical thinking” refers to using logic and reasoning to identify strengths and weaknesses of alternative solutions, conclusions, or approaches to problems (AAMC Premed Competencies for Entering Medical Students, accessed March 3, 2023).
The Critical Analysis and Reasoning Section (CARS) consists of 53 questions based on nine non-science passages. Applicants often score lower in the CARS section than in the other three science-based sections.
For the first time, the HPSA/SDN survey sample suggests a relationship between high performance in Casper and the MCAT CARS.
Emil Chuck, Ph.D., Director of Advising Services for HPSA, authored the independent study. “The MCAT CARS section tests decision-making and the ability to consider alternatives and consequences, even though it is a multiple choice exam based on esoteric, academically-written passages. Casper also does this, but its open-response format allows examinees to explain their reasoning and bring their life experiences. Seeing a statistically significant relationship between these scores with this cohort was a pleasant surprise as no one had reported a connection before.”
“This is especially important because our sample captured a diverse group of qualified applicants who can score well on these tests. While such a link seems intuitive, I don’t know if anyone has shown a correlation previously.”
The Casper exam has been extensively used for medical school admissions throughout Canada for the last decade. Many US health professions programs, including medical residency programs, have also used the Casper exam to gauge applicants’ communication skills, interpersonal management, and ethical/moral reasoning. As shown on its website, Acuity Insights research with partner institutions shows Casper reliably identifies candidates with fewer professionalism issues and that it can predict clinical performance. Many business, nursing, and engineering schools are adopting the Acuity Insights assessments as part of their admissions processes.
How test-takers got higher scores
The survey showed that the most successful applicants took a few hours to become familiar with the technology and question format for Casper. Most relied on the Acuity Insights website to guide their preparation, and most did not use any commercially available preparation packages from admissions counseling consultants. While some relied on influencer videos found on social media, many used free resources such as the SDN articles on Casper and the SDN Casper/SJT/PrepMatch forum for simulated scenarios to craft responses.
Survey respondents also identified important life experiences and coursework that helped them with SJT preparation. “Most of our respondents leaned on their past experiences working in a diverse team environment such as a job, with community or campus organizations, or as part of a study group.
“SJTs favor self-aware test-takers who engage with their communities and not necessarily those who superficially complete activities to get a competitive medical school profile.”
How SJT scores are used in applicant reviews remains secretive
Since these situational judgment tests are becoming a permanent fixture in the admissions or residency selection process, survey respondents seek information to develop a more even playing field and further reduce inequitable barriers to entry.
“Even among those with high scores, the examinees wanted more transparency, including the general rubric scheme that results in strong Casper performance, to confirm their preparation strategies have validity.
Furthermore, programs should clearly describe how SJT scores contribute to shortlisting applicants for interviews and offers; students do not have a positive perspective of programs that want applicant SJT scores and see them as an unnecessary hoop to jump through.”
“I hope these findings will inspire additional research on developing specific problem-solving skills critical in healthcare and other professional disciplines. Considering alternate perspectives or approaches is rewarded in these assessments and real-life situations. Ideally, professional-level critical thinking and analysis can be triangulated based on results from Casper, other SJTs (like objective structured clinical examinations), and professional exams (like the Law School Admissions Test). ”
About the Survey
The 2023 HPSA/SDN Situational Judgment Test Experience Survey aims to explore applicants’ encounters with SJTs, reflecting diverse perspectives in the medical school admissions and residency selection process. The executive summary is available for download, and the full report is available upon request.