Scraping Indeed in 2026 still works, but only within narrow limits. A visitor who is not logged in can load a search page, read the job cards on it, and open individual postings. Going deeper than the first few result pages, loading hundreds of listings in a row, or hitting the site from a datacenter address runs into bot checks that stop the session. The practical answer is to treat each search as a shallow window of recent results, run many narrow searches instead of one wide one, and repeat them on a schedule so you catch postings as they appear.
That approach gives you a usable feed of new jobs per keyword and location. It does not give you the complete Indeed database, and nothing short of a commercial agreement with Indeed does. This article explains what you can see without an account, why the first page is the only reliably open part, how sponsored listings change what a search returns, and how to set up monitoring that stays within those limits. It also names the free alternatives, because for several use cases they are the better choice.
What a logged out visitor can and cannot see
Indeed serves job search results to anonymous visitors. That is by design, since most job seekers arrive from a search engine and Indeed wants them to see results before asking for an account. A search page contains a list of job cards with title, company name, location, posting age, sometimes a salary line, and a short snippet. Clicking a card loads the full description.
What you do not get without an account:
- Saved searches and email alerts tied to a profile.
- Application history and the apply flow itself for Indeed Apply jobs.
- Some employer details that only show after you start an application.
What you do not get even with an account:
- The full result set for a broad query. Indeed caps how deep any search goes and stops paginating well before the true number of matching jobs.
- Stable job identifiers across time. The job key in the URL is stable for a posting, but the same job reposted by an employer gets a new key.
- Reliable salary data. The salary line is often an Indeed estimate rather than a figure the employer provided, and it is labeled as such on the page.
Indeed's terms of service, published at indeed.com/legal, prohibit automated access without permission. Anyone scraping the site should read that page and decide for themselves whether their use is acceptable. This article describes the technical mechanics and the limits. It is not legal advice.
Why only the first result page is reliably open
People notice this quickly: the first page of a search loads fine, page two or three usually loads, and then something breaks. Either the response is a challenge page instead of results, or the job list comes back empty, or the pagination simply ends. Three separate mechanisms cause this.
First, Indeed limits search depth. The pagination parameter in the URL increases by the page size, but past a certain offset the site returns no more results regardless of how many jobs match. Indeed does not publish the exact cap and it has changed over the years, so do not rely on any specific number. The effect is the same for a human user clicking "next" as for a script.
Second, bot detection is request based. Indeed uses a bot management layer that scores each request on headers, TLS fingerprint, request timing, and IP reputation. A browser loading one page every few seconds from a residential connection scores well. A script loading twenty pages in a burst from a cloud provider scores badly and receives a challenge. The first page often passes because the session is fresh and the request pattern is still indistinguishable from a person. By page three the pattern is not.
Third, the robots file at indeed.com/robots.txt disallows crawling of search result paths for general crawlers. That file is a published statement of what Indeed wants automated clients to avoid, and it is worth reading before you build anything.
Put together, the conclusion is simple. A scraper that tries to exhaust a broad search will fail, and will fail in a way that also blocks the IP for the shallow searches that would have worked. The scrapers that still work in 2026 accept the shallow window and compensate with query design and scheduling.
How sponsored listings distort a search
Indeed makes money when employers pay to promote postings. Sponsored jobs appear at the top of search results and again mixed into the list, and Indeed's employer documentation describes them as jobs that receive better visibility in exchange for a per-click or per-application budget. Indeed explains the product on its own hiring pages under Sponsored Jobs.
For a scraper this has three consequences.
The ranking you see is not relevance alone. Sorting by relevance blends paid placement with match quality. Sorting by date reduces the effect but does not remove it, since sponsored cards can still be pinned above the dated list. If you want a feed that reflects what was actually posted, sort by date and mark sponsored cards separately. The card markup carries a sponsored indicator, and your parser should keep it as a field rather than dropping it.
Sponsored jobs are overrepresented in a shallow window. Because you can only reliably read the first pages, and sponsored jobs cluster there, any count or share you compute from scraped data is skewed toward employers with advertising budgets. A dataset that says a certain company posts the most jobs in a city may really be saying that company spends the most on Indeed. Do not present scraped Indeed counts as market share.
The same job can appear more than once. A sponsored card and its organic twin can both show up in one result set, and the same posting appears under several nearby locations. Deduplicate on the job key, and when the key differs, on a normalized combination of title, company and location.
Query design that stays inside the limits
Since depth is capped, breadth has to come from the number of distinct queries. A search for "data engineer" in "United States" returns a tiny fraction of matching jobs. The same title searched per metro area, per posting age filter, and per remote flag returns far more unique listings in total, because each narrow query has its own shallow window.
A workable pattern:
- Pick a list of titles that matter to you, written the way employers write them.
- Pick a list of locations at city or metro level, not country level.
- Always set the posting age filter to the shortest value that covers your schedule interval. If you run daily, ask for jobs posted in the last day.
- Sort by date.
- Store the job key and skip anything already seen.
The posting age filter is the most important of these. It turns the shallow window from "the top of everything" into "everything new since yesterday", which for most monitoring tasks is exactly what you want and is small enough to fit in the first pages.
Monitoring on a schedule
A one-off scrape is nearly useless for Indeed because the data goes stale within days and the window is shallow. A scheduled scrape that runs every day or every few hours against narrow queries builds a complete picture over time, because each posting only has to be caught once while it is still in the recent window.
Keep these rules in mind when you schedule.
Run at a steady, low rate. Spread requests over the interval instead of firing all queries at the top of the hour. This both reduces challenges and spreads load.
Treat a challenge response as a signal, not an error to retry immediately. Retrying a blocked request from the same address within seconds makes the score worse. Back off, and if challenges persist, reduce the number of queries per run.
Store raw cards, then derive. Keep the parsed card fields and the raw description HTML. Indeed changes its page structure without notice, and keeping raw data lets you re-parse history after a markup change instead of losing it.
Track postings disappearing as well as appearing. A job that was in yesterday's window and is gone today is either filled, expired or removed. That is useful information for hiring signals, and you only get it from a schedule.
Monitor the scraper itself. A parser that silently returns zero jobs because a class name changed is the most common failure. Alert on an empty result for a query that normally returns results.
Free ways to get the same data, and when they are better
Before paying for anything, check whether one of these covers your need.
Indeed job alerts. If you want to know when a job matching a query appears, Indeed will email you for free. For a handful of searches this beats any scraper. It stops being practical past a few dozen queries, and the emails are not structured data.
Employer career sites and ATS feeds. Most companies that post on Indeed also post on their own site, which usually runs on an applicant tracking system. Several of those systems expose public JSON endpoints for the job board, and the data there is first party, complete for that employer, and free. If you care about a known set of companies rather than a keyword across the whole market, go to the source and skip Indeed entirely.
Structured job markup. Many career pages publish job postings with schema.org JobPosting markup, which is what search engines read to build their jobs features. Parsing that is more stable than parsing Indeed's result page.
Government and sector boards. For public sector, academic and some regulated industries, official job boards are the authoritative source and often offer exports.
Indeed's own API. Indeed has not offered a general public job search API for years. The old publisher program that once provided one is closed to new applicants. If your organization is an employer or partner, Indeed has integrations on the hiring side, but those serve posting jobs, not reading the search index.
When is scraping Indeed still the right call? When you need keyword coverage across many employers you do not know in advance, in a specific region, updated on a schedule, as structured rows you can load into a database. That is the one job the alternatives above do not do well.
Honest expectations
Any Indeed dataset built from the open site is a sample. It is biased toward recent postings, toward sponsored employers, and toward whatever queries you chose. It will miss jobs that never ranked in the shallow window. Salaries are often estimates. Company names are free text and need normalizing. Treat the output as a monitoring feed and a lead source, not as a census, and your results will hold up.
If you want this done without maintaining a parser, proxies and a scheduler yourself, our Indeed Jobs Scraper runs narrow, date sorted queries on a schedule, marks sponsored cards, deduplicates on the job key, and charges per result returned rather than per run.
Want the signal instead of the raw filings? Get a free report preview. Prefer the tool to the write-up? Browse all data feeds or connect the free MCP server.