If you have taken a SAS Deployment exam in the past 9-12 months, you may have wondered why there are pre-test test questions. While it might seem like I am providing warmer-upper questions out of the goodness of my heart, these pre-test tests are actually called candidate surveys. Their purpose? To draw a correlation between a candidate’s perceived level of their own preparedness and how well the candidate actually does on the exam.
Pre-test surveys are a fairly common practice and of course here at SAS, we jump at the chance to draw statistical correlations. So what does the data tell us? Let’s look at two examples*.
Survey Question #1: How much experience do you have working with the SAS software covered on this exam?
The Experience Level chart indicates the more experience a candidate has, the greater the likelihood of passing this exam.
Survey Question #2: On a scale from 1 -5**, how would you rate your current level of expertise with this software?
(**Scale details are provided to candidates.)
The Skill Level chart shows some mixed results. Typically the higher the skill level, the greater the likelihood of a candidate passing the exam. However when the survey options are somewhat subjective in nature (for example, “high level” might mean something different to one candidate vs. another), confounding results can occur such as someone reporting an Expert skill level failing the exam.
When I review pre-test survey data, I keep two things in mind. First, the data is subjective in nature and based on a candidate’s perception of their experience (having software installed on their computer for 6 months is not the same as actually USING the software for 6 months). Second, there can be data outliers (a candidate who rates themselves an “Expert” with “No Experience”). However, enough data normalized across a pool of candidates can provide a strong indication of the candidate type who is more likely to pass an exam.
Why are correlating experience and skill level to an exam important? Because part of exam development best practices include properly defining the job role and profile of a minimally qualified candidate. If an exam prerequisite recommends at least 1 year of product experience, then the exam should be designed to meet that standard.
Pre-test surveys can provide insight into a candidate’s level of preparedness and help validate the exam’s target audience is indeed hitting the mark. The surveys can also provide an important feedback loop into training and preparation methodologies. So the next time you are faced with a test before the test, just know your answers are very important. It’s one part of the test you’ll be sure to pass!
*Survey questions are actual. Data provided for illustrative purposes only. Chart formats provided by Robert Allison.
Beyond the Credential, a blog from The SAS Learning Post, goes beyond the impersonal and sometime sterile world of technical exams and shines the light on the more human side of testing.