Advances in artificial intelligence (AI) are rapidly reshaping the field of assessment and selection (Gonzalez et al., 2022), prompting renewed scrutiny of how these technologies are integrated into assessment centres and professional practice. The increasing accessibility of generative AI (GenAI) has made it possible for trained (or untrained) individuals to create assessment content and abridge assessment centre tasks, but at the potential cost of scientific rigour and best assessment practices. Examples could include using Large Language Models (LLMs) to develop a simulation exercise without the input of subject matter experts (SMEs) or generating a job profile from merely a job title. These developments risk diluting assessment integrity, compromising reliability and validity, and could be potentially harmful to test-users. As such, assessment centre practitioners will need to champion the transformative impact of GenAI in the assessment landscape while simultaneously adopting a tactical approach that remains grounded in core psychometric principles.
A further challenge is that GenAI systems are not static models. Their internal structures, training data, and embedded biases evolve rapidly and often without user visibility. An assessment centre practice or process that has been proven reliable and valid in one model version cannot be assumed to generalise to another. As a result, responsible adoption of GenAI requires rigorous governance, transparent methodologies, and a commitment to ongoing validation. Without these safeguards, practitioners’ risk unintentional drift in measurement precision, construct coverage, and fairness, and ultimately compromising the defensibility of their tools.
The primary aim of this paper is to present a real-life case study outlining the reliable and valid augmentation of GenAI in the development of an AI-Assisted Job Profiler. Leveraging experiences garnered from the development of the AI-Assisted Job Profile tool, a secondary aim is to highlight a governance framework, outlining responsible and ethical best practice approaches to mitigate, augment and embrace GenAI within assessment centres specifically, and within psychometrics in general.
Initial research will be presented pertaining to testing the reliability and validity with a LLM (i.e., ChatGPT) that produced consistent and theoretically aligned outputs supporting the development of the AI-Assisted Job Profiler tool. This will be followed by sharing the results from the development and testing of the AI-Assisted Job Profiler tool and will highlight important considerations for similar assessment centre endeavours. One critical finding was the importance of practitioners developing a thorough understanding of the specific GenAI tools and model architectures they employ, as well as the necessity of continuous validation of the platforms on which assessment practices would depend.
The paper reasons for a balanced, evidence-driven approach to integrating GenAI within assessment centres. A framework will be proposed for the ethical adoption of GenAI, that includes model-specific validation, transparency about GenAI involvement, and alignment with professional standards and best practices. The journey to responsible GenAI integration is ongoing, requiring vigilance, reflection, and proactive stewardship to ensure that innovation enhances rather than undermines the quality of assessment centre practices.
References
Gonzalez, M. F., Liu, W., Shirase, L., Tomczak, D. L., Lobbe, C. E., Justenhoven, R., & Martin, N. R. (2022). Allying with AI? Reactions toward human-based, AI/ML-based, and augmented hiring processes. Computers in Human Behavior, 130, 107179. https://doi.org/10.1016/j.chb.2022.107179
Dr Sebastian Clifton is a registered Industrial and Organisational Psychologist and Management Consultant: Research & Development at TTS-Top Talent Solutions. He holds a PhD in Industrial Psychology from the University of Johannesburg. His postgraduate research focused on validating and operationalising personality models in the South African context. He was the recipient of The Professor Johann Schepers Award for his contribution to the field of psychometrics. Dr Clifton is an Executive Committee member of the Society for Industrial and Organisational Psychology of South Africa (SIOPSA) and the Chair of the SIOPSA Academy. He is a member of the Assessment Centre Study Group of South Africa (ACSG), Golden Key International Honour Society, and has been a member of the International Test Commission (ITC). He has presented at national and international conferences and consults to companies locally and globally in areas that include the development and use of psychological assessments, data analytics, validation and other similar research, cross-cultural assessment, and assessment best practices.
%202.jpg)
