Validity
CareerFit-Test might be initially viewed as an internet parlor game, but in reality it is a carefully crafted instrument spanning more than two decades of scientific research. Uniqueness should never be equated with shoddiness or amateur presentation.
Most vocational interest tests are developed with a series of self-exploratory written questions. The obvious element necessary for useful results is reading competence. CareerFit-Test breaks that mold by using self-exploration photo preferences to elicit answers, with little reading required. The scoring process for most vocational or career interest surveys inserts a time delay between test- taking and results receiving. The CareerFit-Test authors rely on a body of research suggesting substantial value associated with clear and immediate feedback of relevant test results.
With those introductory general statements, other typical questions and short answers are:
Q. How were the job incumbent work-photo images chosen? (content validity)
A. The authors obtained agreement from hundreds of jobholders, representing a broad spectrum of jobs, and photographed the defining element of their individual work environments. Over a period of time a number of vocational specialists agreed to independently assign a job title to each of these photographs. Only those photos with unanimous job title assignment were considered for placement in the prototype CareerFit-Test. There is a body of published research that shows even young children have the ability to describe the tasks a person is performing, using photo clues. The greater the variety of real-life worker contacts they have, the better their ability to recall those experiences, and the more detail they are capable of describing in those tasks.
These theoretical and empirical recall findings form the rationale for this initial test construction approach.
Q. How was the format of the test determined?
A. After reviewing the literature of potential item response types, the authors determined the forced-choice, with opt-out, item response option was ideal, and would also be amenable to correlational, validity and normative research in the future. The pairs of photos to display in each “item” were determined according to a vocational theory based on worker personalities. This well-studied theory suggests that job titles can be grouped into clusters. Further, the statistical distances between the different clusters can be described.
Each of the two photos included in every “test item” was selected, based on inclusion within clusters furthest apart in theory (and statistical studies). The number of items in the total test was determined by: equal representation from every job cluster, the quantity of photos available in the author’s photo pool, the number of item choices that could reasonably be completed in a school class period and, the number of total responses amenable to statistical power analysis.
Q. Do the photos statistically represent their perceived job cluster? (construct validity)
A. The authors sampled 202 workers representing specific job titles from within all of the theoretical job clusters. Their test responses were analyzed statistically (correlation analysis) to determine whether the photos they preferred were predominately those representing their current job cluster. The results showed a positive statistically significant relationship between actual photo choices and their current, theoretical job cluster involvement.
Post analysis procedures identified several photos that related to multiple clusters rather than a single one. These photos were replaced with substitutes, and additional research showed an anticipated correction of this multiple cluster effect.
That follow-up research utilized the addition of a popular paper-and-pencil vocational interest test for comparison purposes. Those results basically showed that both tests were producing identical outcomes. (comparative validity)
A separate study was next completed among job seekers (sample size of 5412) to determine anticipated response norms. The only extra measured variable characterizing the respondents that yielded significant and useful results was gender differences. A follow-up study also yielded significantly different profiles for male and female respondents. The resulting norms reflect those differences and are used for developing more accurate test reports.
These norms contain two other cautionary scales. For test-takers who repeatedly opt-out from choosing either photo, reports contain a warning about the usefulness of the report for career planning. Additionally, a test-retest element was inserted in the survey to provide an estimate of repeatable reliability that is contained in test reports.
Q. Have any additional findings served to improve the test-taker experience?
A. During all the research sessions it was discovered that background music was appreciated, and this seemed to produce a relaxed atmosphere and attention focus not evident when music was lacking. This is now part of the instruction procedure.
In addition, a full page of suggested resources is included with the test results report for career planning purposes.
Statistical research on the test will continue in the future and adjustments will be made where necessary, to improve the predictive accuracy of the instrument.