Insights from a Validation Case Study in a South African Call Centre
Presented at the 2023 Virtual ACSG Conference on 17 March 2023 from 10:15 - 11:00
Research has long shown how hiring the right people for the right roles can significantly enhance not only individual, but also organisational performance. Continually improving the efficiency and effectiveness of the hiring process is a challenge and priority for many organisations, and psychometric assessment can be a valuable resource in this. When using a science-based approach, research insights like Hunter & Schmidt's 1998, 2004, and 2016 meta-analysis provide critical foundational support for using assessment methods to predict job performance. However, fuelled by our efforts to add predictive power to our hiring processes, it is common practice to use the same assessment(s), with particular emphasis on using the same personality questionnaires (PQs), for different roles. This approach may be, unintentionally, providing blurry insights and reducing predictive power. It is therefore important to carefully consider the design of psychometric assessments to maximize their value.
This case study investigated the value of assessments in a call centre selection process and explored the value of a tailored approach. The goal was to demonstrate that the assessments could not only differentiate effectively, but to also add value over and above an already well established and proven selection process. Furthermore, we aimed to explore the potential value of an approach based on specificity. In essence, our objective was to create a bespoke battery including a PQ that is as short, sharp, and punchy as possible – all without compromising on validity and instead, supercharging it.
Following best practice, an initial battery was designed in collaboration with the client. This included insights from an initial trial of the tools with a cross section of current call centre agents, a review of the already clearly defined job description and associated competencies, and interviews with role SMEs in the business. Due to the intention of the project, a broad range of personality traits were included to allow for deeper investigation (36 traits). The battery was introduced into the client’s existing selection process so that candidates who were successful based on this process (n = 61) were then assessed. The successful candidates, who met the established minimum abilities criteria (n = 43), progressed through a standard training programme, which includes evaluations of performance. A correlation analysis was conducted to examine the relationship between the assessments and performance, focusing on the trait level correlations of the PQ.
The analysis provided clear evidence to motivate that ‘less can really be more’ when it comes to our approach to assessment. Despite the range restriction caused by the exclusion of candidates (based on their performance in the standard selection process and the abilities assessments) the results indicated strong potential. The below effects were observed:
A .40 correlation between candidates’ score in the standard selection process and their performance in training indicated that the existing process is successfully supporting the identification of top talent.
A .26 correlation between the full-length battery and training scores indicated that while the original battery may have the potential to support effective selection decision-making, there was notable room for improvement.
A bespoke PQ was suggested based on the correlations observed. The new correlations of between .49 (9 trait PQ) and .62 (15 trait PQ) demonstrate how superfluous traits can risk confusing decision-making.
A .60 correlation between performance and a total score comprised of the selection process score, abilities assessments, and a tailored 15 trait PQ demonstrates how customised design can add significant value.
In summary, shortening a PQ has the potential to significantly improve its effectiveness and efficiency in identifying top talent. The ripple effect of this is an improved, quicker candidate journey, clearer assessment reports, and easy, early integration into the selection process. This could lead to a more positive brand experience, strengthen decision making at pace, easier communication and understanding of results across a range of stakeholders, and the ability to manage the volume of in-person interviews and other time-intensive qualification steps required.
Jenna Smit has been a registered Psychometrist since 2018, with experience in private practice, banking, insurance, call centres, and consulting environments. This experience includes two internships – one as an Intern Psychometrist, and the other as an Intern Industrial Psychologist. My academic qualifications include a BCom in Industrial Psychology and Human Resource Management with an additional major in Psychology, a BA Honours in Psychology (cum Laude) and a BCom Honours in Industrial Psychology (cum laude) where I achieved the top overall and research results in the class. Lastly, I am in the process of finalising my BCom Industrial Psychology Master’s thesis through Stellenbosch University
My current role is a Consultant and Account Manager at CHC Consulting, where I have a specific focus on psychometric assessment, for both selection and development purposes.