For years, the price of admission to SEC filing analysis was a scraper. You wrote code to walk EDGAR's directory structure. You handled broken HTML. You maintained the whole thing when a page layout changed. That barrier is mostly gone now. An AI agent with an HTTP tool and clear instructions can go from a company name to a scored signal in one session. This article walks through how that pipeline actually works, step by step, and where it still breaks.
One note before we start. Nothing here is investment advice. It is a description of a data pipeline.
Why no scraper is needed anymore
The common belief is that SEC data lives in messy HTML that must be scraped. That was true once. It is not true today. The SEC publishes structured, machine-readable endpoints for almost everything an analyst needs. There is a full-text search system for filings, a JSON API for company submissions, and standardized XBRL data for financial facts. The SEC documents these endpoints on its developer resources page.
An AI agent does not need to parse rendered web pages to use these. It needs three things. It needs to know the endpoints exist. It needs permission to make HTTP requests. It needs a rubric for what to do with the results. The rest is orchestration, and orchestration is exactly what agents are good at.
The SEC does publish fair access guidelines that limit request rates and require a declared User-Agent header identifying your application. Any agent you build should respect those rules. This is not optional politeness. Ignoring the guidelines gets your traffic blocked.
Step one: resolve the name to a CIK
Every EDGAR filer has a Central Index Key, or CIK. Nothing useful happens until you have it. The SEC publishes a complete mapping of tickers to CIK numbers as a single JSON file. An agent's first move is to fetch that file, or a cached copy of it, and match the company name or ticker against it.
This step sounds trivial. It is where naive pipelines fail first. Company names are ambiguous. "Meta" could match several filers. Subsidiaries file under their own CIKs. A good agent handles this by returning candidate matches and either picking the one with the expected ticker or asking for confirmation. A scraper would just take the first result and silently analyze the wrong company. The agent's ability to notice ambiguity and stop is a real advantage, not a soft one.
Step two: pull the filing history
With a CIK in hand, the agent requests the submissions endpoint at data.sec.gov. The response is a JSON document listing every filing the company has made. Each entry includes the form type, the filing date, the accession number, and the primary document name. From those fields the agent can construct a direct URL to any document in the archive.
No HTML parsing has happened yet. No scraper exists. The agent has a complete, structured filing history from one GET request.
Step three: pick the forms that carry signal
A filing history is noise until you filter it. Different questions map to different form types. Insider buying and selling lives in Forms 3, 4, and 5. Institutional holdings live in Form 13F. Private capital raises live in Form D. Material corporate events live in 8-K filings. Annual and quarterly financials live in 10-K and 10-Q filings.
The agent's job here is translation. The user asks a plain-language question. The agent maps it to form types. "Has anyone inside this company been buying stock" becomes "fetch recent Form 4 filings and read the transaction codes." "Who are the big holders" becomes "find 13F filings that report a position in this issuer." This mapping is knowledge work, not engineering work, which is why an agent can do it and a scraper cannot.
Each form type comes with its own reporting lag, and the lag changes what a signal means. We have covered the 13F deadline and its 45-day reporting window in detail in our piece on 13F deadlines, so the short version here is that institutional holdings data is always a look backward, never a live feed. Form D has its own quirks around timing and amendments, covered in our guide to reading a Form D filing.
Step four: read the actual document
Now the agent fetches a specific filing. This is where large language models change the economics. A Form 4 is structured XML and easy to parse either way. But a 10-K risk factors section, an 8-K describing an executive departure, or the "use of proceeds" language around a Form D raise are prose. Before language models, extracting meaning from prose meant either a human reader or a brittle keyword system. An agent reads the section, summarizes what changed against the prior filing, and flags language that a keyword match would miss.
The honest caveat is that this is also where agents are least reliable. A model can misread a table, confuse the filer with the issuer, or state a figure that appears nowhere in the document. The fix is structural. The agent should quote the exact passage it relied on, with a link to the source document, so a human can verify the claim in seconds. A signal that cannot be traced back to a specific line in a specific filing should not survive the pipeline.
Step five: turn a reading into a score
A summary is not a signal. A signal needs a score, and a score needs consistent rules applied to every filing the same way. This is the step where you take the agent's structured output and run it through fixed logic. The logic itself should not be improvised by the model on each run. It should be written down once and applied mechanically.
The scoring dimensions that matter are mostly boring and mostly stable. How recent is the filing relative to its deadline. How large is the transaction relative to the filer's history. Is this one event or part of a cluster. Does the direction of the transaction agree with what other filers are doing in the same name. None of these require secret data. They require discipline in applying the same rubric every day, which is exactly what an automated pipeline provides and a casual EDGAR browsing session does not.
The same architecture works beyond SEC data. Congressional trading disclosures follow a parallel pipeline, from the House Clerk's disclosure database to a scored trade record. We have written up how those disclosures work end to end, and the reporting deadline that governs them is covered in our piece on the 45-day rule, so we will not rebuild that ground here.
What you can do for free
You do not need a paid product to do any of this. EDGAR full-text search is free at sec.gov and covers filings back to 2001. The submissions and XBRL APIs at data.sec.gov are free and need no API key. The SEC also publishes RSS feeds for new filings by form type. Open-source libraries like edgartools and sec-edgar-downloader wrap these endpoints if you prefer code over agents. If your question is about one company and one filing, the free path is genuinely the right answer, and building anything on top of it would be wasted effort.
The gap the free path leaves is breadth and consistency. Reading one 10-K with an agent takes minutes. Reading every relevant filing across thousands of filers, every day, with the same scoring rubric applied identically each time, is a different kind of problem. That is a pipeline problem, not a reading problem, and it is the part worth automating properly.
Where this pipeline still breaks
Three failure modes deserve naming. First, amendments. Filers correct earlier filings, and a pipeline that scores the original without checking for an amendment scores stale facts. Second, entity confusion. Similar names, parent and subsidiary CIKs, and fund families with dozens of related filers all create matching errors that look correct until they are not. Third, model error. An agent that summarizes without quoting its source will eventually assert something the filing does not say. Every one of these is manageable. None of them is manageable by accident. The pipeline has to check for amendments, resolve entities deliberately, and require citations by design.
The end state is worth restating plainly. Company name in, scored and source-linked signal out, with no scraper written and no HTML parsed. The building blocks are official SEC endpoints, an agent that knows how to use them, and a fixed scoring rubric that runs the same way every time.
If you want to see what this pipeline produces when it runs across the full universe of 13F filers instead of one company at a time, the output is our Smart-Money 13F Consensus report, which scores where institutional filers agree, updated as new filings arrive.
Want the signal instead of the raw filings? Get a free report preview. Prefer the tool to the write-up? Browse all data feeds or connect the free MCP server.