A key challenge faced by every hiring manager is how to determine, based on limited information, whether any given applicant would be a good addition to their team. In making this determination, two key pieces of information are required. First is the answer to the question, “Does this applicant possess the knowledge, skills and abilities to successfully perform the job?” The second question is similar, but has an important difference, “Given that the applicant possesses the knowledge, skills and abilities to perform the job, do they possess the ‘inner characteristics’ that would lead them to be a particularly effective member of the team?”
The reason that the second question is important lies with the fact that “What people can do is not necessarily what they will do.” I can remember a time early in my career when I worked for an organization who hired a Ph.D. organizational psychologist to help them in their consulting division. Excitement ran high because this was the first doctoral-level person hired. So convinced was the company about this individual based on his qualifications, that they offered him a high salary and a senior-level position.
The surprising thing was that, in spite of his education and experience, he actually seemed to lack—something. His productivity was low. He was overly rigid and demonstrated an inability to adapt to what some may call the “real life world of limited budgets and abbreviated schedules.” The result was that after a relatively short period, this highly qualified individual was fired.
This memory has always stood out to me and highlighted the very real challenges faced by those responsible for hiring.
The most common approach to personnel assessment involves assessing whether applicants possess the knowledge, skills and abilities that are required at entry to the position in order to perform those job tasks deemed important or critical to job success. The process for doing this is well worn territory in science, practice, and in the courts.
Most commonly, the process for doing the above is called “content validation.” It involves studying the job, identifying important or critical job tasks, and the knowledge, skills and abilities required in order to perform those tasks. Assessment content is then developed to measure an applicant’s possession of those knowledge, skills and abilities.
Assessments can be crafted in a variety of formats including but not limited to interviews, writing exercises, multiple choice tests, job simulations and skills tests. Each of these, and others not listed, can be powerful tools for ensuring that job applicants have the capacity to do what it takes to successfully perform a given job.
The challenge essentially then becomes, how can one determine whether the qualified applicant will actually perform to the level that they have the capacity to or, even more challenging, do they possess the inner characteristics that match the more subtle characteristics that identify the highest performers?
An example that I like to share involves the job of manager. Clearly there are specific knowledge, skills and abilities that a person must possess to perform the work of a manager. They must have the ability to organize information, solve problems, be able to write clearly and effectively, etc. However, clearly, there is more to being an effective leader than that.
So. what are the characteristics of effective leadership? Well, integrity is important. So is an attitude of mutual respect. A tendency to stay on task in meetings will be greatly appreciated by their staff. The propensity to listen to the ideas of others, ask questions, and integrate those that can further achievement of the stated objectives is finally important.
While this is not an exhaustive list, a characteristic shared by each of them is that they do not lend themselves to traditional assessments used by hiring managers—they have the appearance of being un-assessable.
And yet, they are not. It is true that they can involve thinking outside the box but measuring these important applicant characteristics, and many more is possible.
There are several options. For assessing integrity, it is possible to develop or even purchase integrity tests that promise to assess an applicant’s honesty or their likelihood of stealing from the company. The challenges with integrity tests are that first you must determine that integrity is of such a high value that it warrants its own assessment. If you do not have issues with new hire honesty or your employees do not have access to large amounts of cash or confidential information, then an integrity test may be overkill. Another challenge is that unless you build your own test, there is a high likelihood that it may not have face validity to your specific job.
One of the most powerful options for assessing the un-assessable is the job simulation. Allowing people to actually demonstrate complex work that mimics the job has the advantage of providing an opportunity to observe characteristics that may otherwise be difficult to measure.
A tool that I have found very valuable over the years is what I call the “leaderless group.” This assessment tool brings three to four applicants together at the same time and they are tasked with completing a project. It is while the applicants are working on this project that observers will have an opportunity to observe these less measurable behaviors.
For example, when it comes to assessing leadership, I develop a rating scale for the raters to use that includes performance domains such as:
- Active Listening
- Open to Ideas
- Remains on Task
- Meeting Management
- Takes Leadership Role
Each of these domains will have three behavioral anchors that describe a level of behavior such as Well Qualified, Qualified, Not Qualified. Below are examples of such anchors.
Domain: Active Listening
Well Qualified Response – Makes eye contact while listening to what others say. Takes notes on points others make. Asks clarifying questions in response to what others say before moving to share one’s own ideas. Encourages others to continue in order to fully share their ideas.
Qualified Response – Makes eye contact while listening to what others say. Waits for others to speak without interruption before sharing one’s own ideas.
Not Qualified Response – Interrupts others who are sharing ideas. Body language communicates inattention. May put down the ideas of others. Changes topics without acknowledging the points of others.
The key to effective leaderless groups is to take all steps necessary to make them fair for all applicants. One way to do this is to give applicants an opportunity to inform themselves in the days before the leaderless group activity. A good way to do this is to incorporate a writing exercise for which the topic interleaves with the leaderless group activity. In that way, every applicant has an equal opportunity to conduct research on the topic for their writing exercise.
While applicants should be informed that they will take part in a leaderless group activity, they should remain in the dark about what is being assessed. They key to a leaderless group’s effectiveness is that applicants do not know how to “perform.” Rather, they should engage in the work just like they would in real life.
Another important characteristic of the leaderless group is the amount of time allowed. In general, applicants should be given at least 40 minutes to work on their task. This provides ample time for interactions. It also takes advantage of the fact that while many people can fake good behavior for brief periods, it is hard to sustain when focused on tasks over time.
The task itself should be carefully selected to ensure that it is general enough for outside applicants while being face valid and clearly related to the job. That is why for leadership-related leaderless groups, I nearly always use the format of a task force.
Instructions I provide to applicants may say something like “You have been selected by upper management to serve on a task force. Your assignment is to work together to discuss and make specific recommendations on what the organization should do to address the [Problem Scenario]. You will have 40 minutes to work as a team on this project, after which you will be given 10 minutes to present your findings to management (the panel of observers). You may make use of the materials on the conference table where you will be working as well as the white board and flip chart.
You may begin now.”
It should be noted that that a leaderless group is only one kind of job simulation. While it is very powerful, it has its limitations. It is not appropriate for every situation. There are other forms of job simulation that can be adapted to a wide range of needs.
A parting recommendation is to always make sure that any pre-employment assessment, whether traditional or “creative”, is based on a solid job analysis. To be defensible, any activity should be related to the job. Finally, ensure that any activity is fair in the sense that it does not work to the systematic advantage of any ethnic group or provide advantage to one gender over another.
This blog is based on the BCG Institute for Workforce Development (BCGi) webinar presented by Jim Higgins, Ed.D., on August 23, 2022. If you have any questions about this topic or would like a consultation about a future assessment project, please email firstname.lastname@example.org.