Knowledge sources reference

Supported source types

• Website / URL — Single page — Only the URL you give us.
• Website / URL — Crawl site — Follows same-site links from the start page, breadth-first, up to Max pages (1–200, default 50), minus anything matched by Exclude paths.
• Website / URL — Sitemap — Fetches every <loc> entry of a sitemap.xml, including nested sitemap indexes, up to Max pages. Exclude paths apply here too.
• File upload — PDF, DOCX, Markdown, or plain text — up to 20 MB per file, several files per source.
• Paste text — A doc or FAQ pasted directly as Markdown or plain text.

Not available: Notion or Google Drive connectors, and a crawl-depth setting. Scope a crawl with Max pages and Exclude paths instead.

Step-by-step

1. Open Knowledge. Dashboard → Knowledge → Add source. Sources are workspace-level, not tied to one assistant.
2. Pick a source type. Website / URL (Single page, Crawl site, or Sitemap), File upload (PDF, DOCX, MD, TXT — max 20 MB each), or Paste text (Markdown or plain text).
3. Scope a crawl. Set Max pages (1–200, default 50) and optional Exclude paths — one path pattern per line, * matches the rest of the path (e.g. /blog/*). Excluded pages don't count toward Max pages. There is no crawl-depth setting: crawls follow same-site links breadth-first until the page budget is spent.
4. Attach it to assistants. Tick the assistants that should answer from this source. A source attached to no assistant is indexed but never used.
5. Wait for sync. Indexing runs in the background and resumes if interrupted; the source row shows pages done, chunk count and the last page fetched, then turns ready.
6. Set the refresh cadence. URL sources re-index automatically — Daily by default; pick every 6 hours, 12 hours, 2 days or weekly per source. Re-sync refreshes right now.

What happens at ingest

1. Pages are fetched and reduced to their main text; files are parsed (PDF and DOCX text extraction, Markdown/TXT as-is).
2. Text is chunked heading-aware at roughly 1,000 characters with a small overlap, so a chunk rarely straddles two topics.
3. Each chunk is embedded once and stored in your workspace; a re-sync replaces the previous documents for that source.
4. On every question the assistant embeds the query and retrieves the nearest chunks from the sources attached to it.

Limits

• Per file — 20 MB (PDF, DOCX, MD, TXT).
• Per URL source — Max pages 1–200 (default 50); up to 50 exclude patterns.
• Auto-refresh — 6 hours to weekly, per URL source; default daily. Uploads and pasted text don't auto-refresh.
• Per workspace — 50 MB knowledge storage and 1 URL source on the trial; 50 MB / 2 URL sources on AI Adopter; 250 MB / 10 URL sources on AI Pilot.