Decoding the Labor Market: Can Simple Proxies Replace Complex Data in Mapping Occupational Transitions?

In the modern global economy, workforce mobility is a critical driver of economic resilience. As industries evolve under the pressures of automation, green transitions, and shifting global supply chains, millions of workers face the prospect of changing careers. Identifying viable transition pathways—mapping how easily a worker can move from one occupation to another based on shared skills—has become a central focus for labor economists and policymakers alike.

For years, the gold standard for this type of analysis has been the Occupational Information Network (ONET) database, managed by the U.S. Department of Labor. By quantifying worker characteristics such as abilities, skills, and knowledge, ONET allows researchers to calculate skill gaps between jobs. However, applying these granular data models outside the United States—or in data-scarce environments where local labor markets diverge sharply from American taxonomies—presents significant hurdles.

A recent empirical study explores a pragmatic question: *Can simpler, readily available proxies—specifically, semantic comparisons of job descriptions via small language models and the mathematical distance between standard occupational classification codes—stand in for ONET ratings when granular data is absent?**

Main Facts: The Quest for Accessible Workforce Mobility Metrics

The foundational premise of skills-based transition modeling is straightforward: occupations that demand similar cognitive, physical, and technical attributes are, on average, easier for workers to navigate between.

To test whether alternative metrics can approximate this complex web of O*NET data without requiring millions of dollars in primary data collection, researchers evaluated three distinct ways to measure occupational similarity across nearly 900 detailed occupations:

1. *Skill-Based Ratings (The ONET Benchmark):** Calculating Manhattan distances across 120 standardized worker characteristic variables (encompassing abilities, essential skills, transferable skills, and knowledge) and converting them into a standardized 0–100 similarity score.
2. Semantic Similarity (Job Descriptions): Using a small, open-source transformer model (all-MiniLM-L6-v2) to transform standard-English job descriptions into numerical vectors, subsequently comparing them via cosine similarity.
3. Classification Proximity (SOC Codes): Measuring the structural distance between jobs within the US Standard Occupational Classification (SOC) system, analyzing both list-position rankings and raw code numerical differences.

The central finding? While neither proxy can completely substitute for O*NET’s deep worker characteristic data, both perform remarkably well as complementary tools. Specifically, classification codes excel at separating broad classes of work, while semantic language models prove superior at ranking specific occupations within closely related career clusters.

Chronology: From Truck Driver Pathways to Universal Proxies

The trajectory of this research began with a granular case study focusing on truck drivers, identifying potential transition pathways based on shared knowledge and abilities. That initial investigation exposed a broader methodological dilemma. While advising on labor market interventions in international settings—specifically India—researchers encountered labor ecosystems where the US SOC framework failed to map cleanly onto local employment realities.

This friction forced a strategic pivot: abandoning O*NET entirely in certain contexts while raising fundamental questions about how to evaluate occupational distance in the absence of rich administrative or survey data.

Subsequent data wrangling and exploratory analysis proceeded in distinct phases:

• Data Harmonization: Importing version 31.0 of the O*NET database and standardizing importance (1–5 scale) and level (0–7 scale) metrics into a unified 0–100 scale.
• Vector Embeddings: Processing standardized textual job descriptions through a neural network to capture underlying occupational semantics rather than superficial keyword matches.
• Matrix Comparisons: Generating over 399,000 unique job pairs to test rank correlations, linear regressions, and hierarchical groupings across the SOC structure.

Supporting Data: What the Numbers Tell Us

The empirical analysis reveals a nuanced story about how structural hierarchies and natural language processing intersect with labor market realities.

The Power of the Hierarchy

When comparing similarity measures against the SOC taxonomy, the data shows that average similarity scores rise predictably as occupational groupings narrow. When two jobs share a major occupational group, their similarity scores hover around moderate levels; when they share detailed occupational definitions, the scores increase significantly.

However, a closer look at the data uncovers an important limitation:

• The Overlap Problem: Below the major group level, boxplots of similarity scores overlap heavily. Whether two jobs sit in the same minor group, broad occupation, or differ only by a suffix, the average similarity scores show minimal movement.
• Explanatory Power: Together, semantic descriptions and SOC list positions explain roughly one-third of the total variation in skill-based scores overall, and 10% to 15% of the variation within major occupational groups—which is precisely where real job seekers look for new employment.

Correlation and Predictive Performance

Rank correlation analyses (Spearman’s rho) demonstrate that SOC distance measures correlate strongly with skill-based scores across the entire dataset because both are fundamentally rooted in the logic of occupational sorting.

Yet, when examining pairs within specific major groups, semantic similarity derived from job descriptions consistently outperforms SOC distance metrics in tracking true skill overlap. The lone major exception is Architecture and Engineering, where the SOC structure deliberately separates professional engineers from supporting technicians—a technical distinction that semantic text embeddings occasionally blur.

Official Responses and Methodological Debate

The development of these proxy methodologies has sparked healthy debates among data scientists and labor economists regarding the definition of "cheap" data science versus rigorous empirical policy design.

Critics of proxy-based metrics often point to the risk of circularity: because classification systems like the SOC are explicitly designed to group occupations based on work performed, skills, and training, finding that SOC distance correlates with skill-based scores is hardly surprising. It validates the classification system’s design rather than introducing independent empirical insight.

Conversely, proponents emphasize the radical accessibility of these proxy tools. As noted in methodological discussions, utilizing a transformer model that runs locally on a standard laptop—utilizing a lightweight 100MB model and rudimentary job descriptions—recovers a surprising amount of signal from a vastly richer, more expensive dataset.

Furthermore, researchers candidly acknowledge the implicit assumption of their work: by making the ONET skill-based score the benchmark to predict, the analysis treats it as an infallible standard. In reality, even the richest and most resource-intensive occupational datasets carry blind spots. If ONET metrics fail to capture emerging green-economy skills or localized informal labor realities, blind adherence to them could lead policymakers astray.

Implications: Practical Applications for Workforce Policy

The practical implications of these findings extend far beyond academic data science, offering actionable tools for workforce development agencies, policymakers, and career counselors operating in resource-constrained environments.

1. Robust Sense-Checking for Transition Pathways

When governments or international bodies attempt to build local occupational transition matrices using non-traditional data sources—such as scraped online job postings, vocational training curricula, or administrative tax records—validating the output is notoriously difficult. The dual-proxy approach outlined in this research offers a low-cost, highly accessible validation mechanism:

• Use SOC-style classification distances to screen out implausible macro-transitions (e.g., ensuring policy recommendations do not suggest an absurd leap between radically different economic sectors).
• Use semantic text descriptions to sense-check and rank micro-transitions within a targeted industry cluster.

2. Democratizing Labor Market Analytics

Building bespoke occupational characteristic databases akin to O*NET requires millions of dollars and years of survey collection. For developing economies or regional municipal governments lacking such budgets, leveraging open-source language models alongside national classification standards democratizes advanced labor market analytics. While neither proxy is sophisticated enough to stand alone as an authoritative career counselor, together they serve as an effective defense against implausible policy recommendations.

3. A Nuanced View of the "Blind Spots"

Ultimately, this research serves as both a validation and a cautionary tale. It proves that simple, inexpensive text-matching and structural distance codes can effectively mirror complex skill matrices. Yet, it reminds researchers that classification structures and machine learning embeddings are best deployed as complementary navigational compasses—not absolute maps—when guiding workers toward a secure economic future.