Data Collection and Enrichment Methodology

Platform Overview

Data Collection Philosophy

Canaria treats job postings as observable signals of employer labor demand, not as direct measures of hiring outcomes. A posted job reflects stated intent to hire; it does not confirm a hire occurred, a position was filled, or that a single headcount was added. Researchers should apply this distinction when using posting counts as economic indicators.

The platform focuses on making what is observable reliable, standardized, and analytically usable, while explicitly preserving uncertainty and avoiding inference beyond the data's scope. Raw job postings are ingested through a distributed scraping infrastructure spanning Indeed, LinkedIn Jobs, and 200,000+ employer career portals (Greenhouse, Lever, Workday, iCIMS, and others). Coverage is United States primary, with data beginning in 2022 and updated daily to hourly.

Each posting passes through a multi-stage enrichment pipeline combining deterministic parsing, machine learning classification, and job-level entity resolution. All raw, parsed, and enriched fields are retained for transparency and auditability. When a field is absent or unparseable, the record carries a NULL rather than a synthetic fill-in, and coverage rates are tracked by source and data vintage. The field priority rule throughout the pipeline is: parsed structured data takes precedence over scraped text, which takes precedence over calculated or model-predicted values.

Pipeline Architecture

Every job posting passes through an 8-stage pipeline: ingestion, deduplication, NLP classification, location parsing, salary normalization, skills extraction, title normalization, and company enrichment.

SOC Classification: Accuracy and Coverage

Canaria assigns 6-digit Standard Occupational Classification (SOC) codes using both the job title and the full description text. Title-only classifiers commonly misclassify roles with generic titles such as "Manager" or "Analyst." Using description context brings 6-digit accuracy to 85-92% on postings with sufficient description length (more than 200 words). The taxonomy is aligned with the BLS 2018 SOC standard, enabling direct comparison with government labor statistics.

2-digit major group accuracy exceeds 95%, which is at or above the industry benchmark range of 93-97%. Coverage improves with data vintage as both source quality and model iterations improve.

Accuracy and Coverage Thresholds

Known Classification Challenges

Salary Prediction: Model and Coverage

Only about 2.5% of job postings across all sources include stated compensation. To address this gap, Canaria runs a regression model trained on 50M+ Glassdoor and Indeed salary observations to predict minimum, average, and maximum annual USD compensation for every record where State, Zip code, and SOC code are available. Predictions are annualized to USD; the original stated period (hourly, monthly, annual) is preserved in a separate field.

The dataset includes three complementary salary signals: posted ranges (where disclosed), model-predicted ranges (on all records meeting prerequisites), and employee-reported data (from Glassdoor, 10.5M+ individual reports). Posted ranges reflect what employers say they will pay; predicted ranges reflect market rates for the SOC, state, and zip combination; employee-reported data reflects what workers in those roles actually earned, including years-of-experience and bonus breakdowns not available from postings.

Prediction Accuracy

Accuracy Degradation Factors

Prediction accuracy degrades in the following conditions. Researchers should apply appropriate filters or widen confidence intervals when working with these subsets:

• Very high compensation (above $300K/year)
• Hourly and gig-economy work
• International postings (model trained primarily on US data)
• Older data vintages (salary norms shift over time)

Salary invariants enforced on all records:salary_min ≤ salary_avg ≤ salary_max; all populated values are positive; all values are normalized to annual USD.

Work Mode Detection: Remote, Hybrid, On-site

Work mode classification uses keyword detection combined with location analysis to determine whether a role is Remote, Hybrid, On-site, or Flexible. Researchers must account for a structural break in this field at 2020: before the pandemic, fewer than 1% of postings mentioned remote work, so NULL values in pre-2020 data are correct and expected. The field does not exist as a concept for that era.

As of 2023+, work mode is a standard field across major sources with greater than 85% coverage. A NULL in 2023+ data indicates a parsing gap rather than an implicitly on-site role. Remote postings peaked around 2022 at approximately 15% of all postings and have declined to approximately 10% as of 2024-2025. The term "hybrid" remains poorly standardized: it can mean one day per week or four days per week depending on the employer.

Coverage by Era

Accuracy Metrics

Named Entity Recognition

A fast, dictionary-based keyword processor runs in-pipeline to extract structured entities from job descriptions. Extracted skills are filtered through a title-skill relevance model to remove spurious matches. Multi-valued attributes are represented as arrays; when no signal is present, fields return empty arrays rather than null.

Extraction Accuracy (F1 Scores)

Skills Coverage by Description Length

Skills Extraction QA Thresholds

Skills Taxonomy: Context and Scale

Canaria's 37,000+ skills taxonomy is built from O*NET mappings, public sources, and empirical signals derived from live job postings. It expands continuously as new skills emerge in the data. The table below compares major standards for context.

Semantic Deduplication

A central challenge in online job postings data is severe duplication, driven by reposting behavior, aggregator syndication, and minor text changes over time. Canaria addresses this through a configurable, multi-signal deduplication framework that resolves repeated observations of the same hiring intent into a single canonical job entity. The average deduplication rate across all sources is 40-60% (multi-source).

Deduplication Benchmarks

Location Parsing

Raw location strings from job postings are parsed into structured geographic components and geocoded to latitude/longitude. Country-level parsing exceeds 98% accuracy; city-level accuracy is 85-93% due to ambiguous city names, metro area strings, and multi-location postings. The is_remote_global flag identifies postings where location is not a physical place.

Data Quality by Source

Not all job posting sources are created equal. Data completeness, duplication rates, and field availability vary significantly by source type. ATS direct feeds from employer career portals provide the highest quality data, while aggregator sources like Google Jobs carry substantially higher duplication and null rates.

Coverage by Data Vintage

Null rates vary by time period due to schema evolution, source availability changes, and the rollout of salary transparency laws. Researchers should account for data vintage when interpreting coverage.

Salary Transparency Law Timeline

US salary disclosure laws are the primary driver of improving salary coverage in job posting data. Each new law causes a measurable increase in the share of postings that include compensation information.

Company Enrichment

Canaria maintains a company database of 28.5M companies, enriching each job posting with industry, employee count, headquarters location, revenue, and founding year. Company entity resolution uses a hybrid approach combining fuzzy name matching, domain/URL matching, LinkedIn company ID joins, and ML-based resolution to achieve 90-95% match accuracy while keeping false merge rates below 1%.

Company Matching Methods

Quality Metrics

Data Quality Monitoring

Canaria maintains comprehensive monitoring and quality assurance systems across all ingestion and processing stages. A pipeline component tracks field-level completeness for every scraped batch. Non-empty field counts are aggregated by source and time period, enabling detection of source degradation or scraping failures. Completeness metrics inform prioritization of enrichment efforts. Source URLs are monitored for consecutive failure streaks and flagged for manual review when potentially defunct. Monitoring dashboards visualize trends and trigger alerts for anomalies including queue depth spikes, error rate thresholds, and crawler failures.

Research Applications

The Canaria dataset enables empirical analysis across multiple labor economics domains.