Designing a contemporary SJT to measure multiple constructs across various levels of complexity
Presented at the 2023 Virtual ACSG Conference on 17 March 2023 from 11:15 - 12:00
Situational judgement tests (SJT), as an assessment centre method, have been a progressively prevalent and useful measure designed to enhance talent decisions by evaluating how effectively candidates would respond to a typical work-related situation (McDaniel, Hartman, Whetzel, & Grubb, 2007; Whetzel, McDaniel, & Nguyen, 2008). Although SJTs had typically functioned as a lower fidelity means of assessment, SJTs are an assessment method that can be used to assess various constructs (McDaniel, Morgeson, Finnegan, Campion, & Braverman, 2001). More recently, there has been increasing interest in construct-focused SJTs for higher fidelity, where scenarios of the SJT and corresponding response options are designed to measure pre-determined constructs (Campion & Ployhart, 2013; Lievens, 2017; Lievens & Motowidlo, 2016). This presentation will share a practical case study and the lessons learned from the design of a contemporary construct-focused SJT for a leading financial institution to assess specific constructs across various levels of complexity.
The assessment need of the financial institution was to better understand how their people measure against the constructs of their Brand & Marketing Model. This would aid in identifying the strengths and development areas for a future-fit team and support the delivery of the strategic ambitions. The Model comprised nine pre-determined constructs as the capabilities identified for success across three levels of complexity (i.e., Operational, Tactical, and Strategic). Through consultation, a customised SJT assessment solution was found to be fit for purpose. The SJT was designed to present candidates with scenarios similar to the situations they would confront at work to which they are required to respond by making use of corresponding response options. These situations and response options were designed to measure the pre-determined constructs of the Model across the levels of complexity.
The design of this SJT required considerable project planning and management and a resolute group of Subject Matter Experts (SMEs). Suitable third-party SMEs were sourced and fairly incentivised to participate and contribute to the development of the SJT. Experienced assessment centre practitioners worked closely with the SMEs as they were guided during focus groups through the design stages of the SJT. This approach led to the identification of specific scenarios that were unique and critical to form the foundation of the SJT. Then, by combining psychometric know-how and efficacious scoring methods with input from the SMEs, the SJT was developed.
For the purpose of this presentation, we will demonstrate how conventional approaches for SJT design were modernised to develop a construct-focused SJT as an assessment centre method for the pace and dynamism of the contemporary world of work. Practically, we will work through the stages for the development of this SJT using a process map, with particular focus on developing a customised, automated and cost-effective SJT for pre-determined constructs and across various levels of complexity. Lessons that were learnt of what worked (and what could have worked better) will also be shared. These will include amongst others, positioning and contracting with the client and third-party SMEs, planning and managing a project of this scope, facilitating focus groups in a digital workplace, the value and extent of client involvement, and means to enhance client and candidate buy-in.
Sebastian Clifton is a registered Industrial/Organisational Psychologist and Management Consultant (Research and Development) at TTS-Top Talent Solutions. He holds a PhD in Industrial Psychology from the University of Johannesburg, which focussed on personality assessment in the South African context. He is the Chair of the Society for Industrial and Organisational Psychology of South Africa (SIOPSA) Academy. He is also a member of the Assessment Centre Study Group of South Africa (ACSG), Golden Key International Honour Society, and International Test Commission (ITC). He has presented at national and international conferences on the topics of psychological assessments, assessment centres, measurement validation, and cross-cultural assessment.
Eltroné Wilhelm is a registered Industrial/Organisational Psychologist and Principal Consultant at TTS-Top Talent Solutions. She has worked within a range of various industries and companies, focusing on the Mining Sector, Financial & Insurance, Aviation & Transportation, and Fast-Moving Consumer Goods (FMCG) industries. Her 21 years of experience consists of 12 years in consulting; 5 years in managerial roles; and 4 years in a Human Resources generalist capacity.
As a consultant, she implemented, managed, and delivered professional consulting services and projects in areas ranging from realistic job previews, situational judgement tests and questionnaires, job analysis, executive assessment, and development centres, talent management interventions, competency design, team sensitization and development, performance, and talent management.