Sources

By the end of this page you'll have a mental model of what a source is, and you'll know the four API calls to list, add, update, and remove them.

Before you start

• A bot you've already created (see Create your first bot).
• The root URL of an additional corpus you want this bot to learn from.

What a source is

A source is a URL root the crawler walks. Every bot starts with one default source seeded from the clientUrl you passed at bot creation — that's the website the bot was born around.

You can attach more sources to the same bot when your content lives in more than one place: your marketing site and your docs subdomain, your product help center and your changelog.

The source fields

Field	Type	Default	Description
type	'website' | 'sitemap' | 'single_page'	website	How the crawler walks this root.
rootUrl	string (URL)	—	The starting URL for the crawl.
includePatterns	string[]	[]	Glob patterns the crawler must match to keep a URL.
excludePatterns	string[]	[]	Glob patterns the crawler drops on sight.
maxDepth	number (1–20)	5	How many link-hops from rootUrl the crawler is allowed to follow.
maxPages	number (1–5000)	500	Per-source ceiling on pages fetched.

Listing the sources on a bot

curl https://api.mimicbot.app/api/bots/{botId}/sources \
  -H "Authorization: Bearer $MIMICBOT_TOKEN"

Returns { "sources": [...] } — every source belonging to the bot, including the default one seeded at creation.

Adding a source

curl -X POST https://api.mimicbot.app/api/bots/{botId}/sources \
  -H "Authorization: Bearer $MIMICBOT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "website",
    "rootUrl": "https://docs.acme.example",
    "maxDepth": 5,
    "maxPages": 1000
  }'

Adding a source enqueues an immediate crawl for that source only — the bot's existing sources aren't re-crawled. The bot's status flips to indexing during the job.

Updating a source

curl -X PATCH https://api.mimicbot.app/api/sources/{sourceId} \
  -H "Authorization: Bearer $MIMICBOT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "excludePatterns": ["**/legal/**", "**/archive/**"],
    "maxPages": 800
  }'

Only includePatterns, excludePatterns, maxDepth, and maxPages are editable. To change rootUrl or type, delete the source and recreate it.

Deleting a source

curl -X DELETE https://api.mimicbot.app/api/sources/{sourceId} \
  -H "Authorization: Bearer $MIMICBOT_TOKEN"

Deleting a source hard-deletes its pages from the bot's corpus via ON DELETE CASCADE — the next chat turn will no longer cite them.

Next

Configuration complete. Actions section lands in the next release.