Job boards versus career sites as a data source: duplicates, delay and coverage

What changes when you combine LinkedIn or Indeed with ATS career feeds from Greenhouse, Lever or Ashby. Where the duplicates come from, why boards lag, and which source to trust for each field.

If you need job listings as data, the career site is the better primary source and the job board is the better secondary source. Career sites run on an applicant tracking system (ATS) such as Greenhouse, Lever, Ashby or Workday. The ATS holds the original record of every opening. The board holds a copy of that record, often with delay, often edited, and often mixed with copies from other boards. When you combine the two, the ATS feed gives you the canonical row and the board gives you reach into companies you did not know to look for.

The three problems you will hit when you merge them are predictable. Duplicates appear because one opening gets posted in several places and under several titles. Delay appears because boards receive listings on a schedule and remove them on a different schedule. Coverage gaps appear because many companies never post to a board, and because a visitor who is not logged in only sees part of what a board holds. The rest of this article explains each problem, what you can see without an account, and when a free route is good enough.

Where listings actually come from

Almost every company above a certain size uses an ATS. The recruiter writes the opening once in that system. The ATS then publishes it to the company career page and, if configured, syndicates it to job boards. Greenhouse documents this flow in its public Job Board API, which serves the open jobs for any company that has enabled it, without a login and without an API key. The endpoint is described at developers.greenhouse.io/job-board.html. Lever offers the same thing through its Postings API, documented at github.com/lever/postings-api. Ashby publishes a public job posting API as well.

This matters for data quality. The ATS record has a stable job ID, a creation timestamp, a structured location, department and team fields, and the full original description. Once the listing reaches a board, several of those fields are rewritten. The board may normalize the title, truncate the description, replace the location with its own geocoded version, and assign its own ID. If your pipeline needs the original, the board is already one step removed.

Duplicates: why one opening shows up five times

Duplicates have four distinct causes. You need to handle each one differently.

The first cause is multi-posting. A company posts the same opening to its career site, to LinkedIn, to Indeed and to a niche board. Each copy has a different URL and ID. If you collect from all of them you get four rows for one job. The fix is to match on the ATS job ID when the board includes an apply link that points back to the ATS. Greenhouse apply links contain the job ID in the URL. Lever links do too. When the board strips that link and replaces it with its own apply flow, you lose the join key and have to fall back on title plus company plus location.

The second cause is aggregation. Indeed and similar boards crawl career sites and other boards, then index what they find. A listing can enter Indeed once from the company page and once from a staffing agency that reposted it. The agency copy often has a different company name, a vaguer location and a slightly rewritten description. These are the hardest duplicates to catch, because the text was changed on purpose.

The third cause is location fan-out. An opening that is open in three cities is one record in the ATS with one location field, or three records if the recruiter chose to create one per city. On the board, that same opening frequently appears as one listing per city, and sometimes one per metro area the board thinks is relevant. A remote role can appear in dozens of locations on a board while existing as a single row in the ATS.

The fourth cause is reposting. Boards rank fresh listings higher. Recruiters know this and repost or "refresh" an opening every few weeks. The board shows a new posting date. The ATS still shows the original creation date. If you are measuring time-to-fill or how long a role stays open, use the ATS date. The board date measures recruiter behavior, not the job.

A practical dedup order that works: match on ATS job ID when present, then on apply URL after stripping tracking parameters, then on normalized title plus company domain plus country. Keep the earliest timestamp from any source as the opening date. Treat the ATS row as the master and attach board rows to it as sightings.

Delay: when each source changes

ATS feeds are close to real time. When a recruiter opens or closes a job in Greenhouse, the Job Board API reflects it within minutes. The same is true for Lever and Ashby. Workday career sites are slower to crawl but still reflect the ATS state directly.

Boards lag in both directions. A new listing reaches a board through a feed that the board polls on its own schedule, or through a crawl, or through a manual post by the recruiter. Expect hours to days. Removal is worse. Many boards keep a listing visible for a fixed period after the company stopped accepting applications, and some keep it until the next crawl notices the source page is gone. A listing that is live on a board and closed in the ATS is a common state, and it is the main reason that applying through a board sometimes produces a "this job is no longer available" page.

For a data product this means two things. If you want to know what is open right now, poll the ATS. If you want to know when a company first went public with a role, the ATS creation date is still the better signal, but a board can show you the role earlier if the company does not expose a public ATS feed and you were not already watching its career page.

Coverage: what each source does and does not contain

ATS feeds only cover companies that use a supported ATS and have enabled public access. Greenhouse, Lever and Ashby expose public endpoints by default for most customers. Workday, SuccessFactors, iCIMS and Taleo have no equivalent public JSON feed, so those career sites must be read as HTML. Small companies often have no ATS at all. They post directly to a board, or to a single page on their own site, or nowhere.

Boards fill that gap. If a listing exists only on Indeed because the company has no career page, the board is the only source. That is the strongest argument for combining the two. The ATS gives you depth and accuracy for the companies it covers. The board gives you breadth across companies you cannot enumerate in advance.

Now the honest part about what a visitor who is not logged in can see.

LinkedIn shows a limited public view of job search results to anonymous visitors. Detail pages are often partially visible, then interrupted by a login wall. Salary insights, applicant counts and the full description frequently require an account. Automated collection of LinkedIn content is prohibited by the LinkedIn User Agreement, which is published at linkedin.com/legal/user-agreement. LinkedIn also enforces this technically. Treat LinkedIn as a source you read manually or through products LinkedIn itself licenses, not as a crawl target.

Indeed shows most listing content to anonymous visitors, but it aggressively blocks automated access and its terms restrict it. Indeed's own site terms are published at indeed.com/legal. The listing text on Indeed is often a copy of something you can fetch from the original source with no restriction at all.

Google for Jobs is not a board but an index. It reads structured data that career pages embed in their HTML, documented at developers.google.com/search/docs/appearance/structured-data/job-posting. That same structured data is on the career page for anyone to read, and it is a useful fallback when a company uses an ATS with no public API.

Company career pages are the most open source of the three. They are public by design. The company wants the listing read. Most have no login wall and many expose the ATS JSON directly. The gap is that you have to know the company exists before you can read its page.

Free ways to get the same data

For a handful of companies, you do not need a tool at all. Call the Greenhouse or Lever endpoint for each company slug and you have the complete open job list in JSON. That is free, immediate and the most accurate copy that exists. If you watch fewer than a few dozen companies and you know their names, this is the better choice. Write a short script, run it on a timer, diff the results.

For a single company with no public ATS, the career page itself plus the embedded job posting structured data is the free route. For one-off research, a manual export is fine. Open the career page, copy the listings into a spreadsheet, done.

Boards also offer legitimate free routes in some cases. Saved search alerts on LinkedIn or Indeed send new matches by email at no cost. For a small, fixed query that is a reasonable way to monitor without scraping anything.

A paid scraper only makes sense when the free route stops scaling. That happens when you need to watch hundreds or thousands of companies across several ATS platforms, when you need the HTML-only platforms such as Workday handled, when you need deduplication across sources done consistently, or when you need the result delivered as a dataset instead of a script you maintain.

Which source to trust per field

A short rule of thumb when both sources have a row for the same job. Take the job ID, creation date, department, team, employment type and full description from the ATS. Take the first-seen date from whichever source saw it first. Take the closed status from the ATS and ignore the board until it catches up. Take company discovery from the board, then move that company to your ATS watchlist so future openings come from the primary source. Never take salary from a board unless the ATS has no salary field and the board explicitly shows the figure as employer-provided rather than estimated.

If you want ATS feeds from Greenhouse, Lever, Ashby, Workday and other platforms collected and deduplicated for you, with the ATS row kept as the master record and pricing per result instead of per month, see the ATS job scrapers on this site.


Want the signal instead of the raw filings? Get a free report preview. Prefer the tool to the write-up? Browse all data feeds or connect the free MCP server.