Skip to main content
Knowledge Bases

Add Website Pages and Crawl a Site

Add one web page, or a page and every page underneath it, to a VAW Native knowledge base. Pages are re-read every day so answers stay current.

Written By voiceaiwrapper

Last updated 26 days ago

Your client's pricing page, FAQ and service pages already answer most of what callers ask. Add them to a knowledge base by web address, and your agents can answer from them. You can add a single page, or a page plus every page underneath it, up to 500 pages.

Web pages are read again every day, so when the page changes, your agent's answers change with it.

Applies to: VAW Native Pods. Vapi, Retell AI and ElevenLabs Pods show their own knowledge bases read-only. See Provider Knowledge Bases and Tools in VoiceAIWrapper.

Before you start

  • A knowledge base to add the pages to. See Create a Knowledge Base and Upload Files.
  • A public address. Pages behind a login, or on a private network, can't be read.
  • For a whole section, the right starting page. The crawl only goes down from the address you give it (see below), so start at the section you want, such as https://example.com/help.

Add a single page

  1. Open the knowledge base and click Add to knowledge base.
  2. Select Web address.
  3. Paste the full address into Web address, starting with https://.
  4. Leave Also add every page underneath this one off.
  5. Click Add page.

You'll see Page queued and the sheet closes. The page appears on the Documents tab and goes from Waiting to Reading to Ready.

Only the main content of the page is kept. Menus, headers, footers and cookie banners are dropped. The address can also point straight at a PDF or CSV file.

Add a page and everything underneath it

  1. Open the knowledge base and click Add to knowledge base.
  2. Select Web address and paste the starting address.
  3. Turn on Also add every page underneath this one.
  4. Optionally, list parts of the site to leave out in Skip addresses containing, one per line (for example /blog and /careers).
  5. Click Add site.
The Add to knowledge base sheet on the Web address option: an address such as https://example.com/help typed in the Web address field, the Also add every page underneath this one switch turned on, "/blog" and "/careers" on separate lines in Skip addresses containing, the blue background-fetch note, and the Add site button.

You'll see Crawl queued and the sheet closes. The site is read in the background. Pages appear as they're read, which usually takes a few minutes and can take longer for a large site. You can carry on working in the meantime.

The crawl appears as one row on the Documents tab (a Site section). Click the arrow next to its name to see every page it found.

Which pages a crawl adds

The crawl starts at your address and follows the links it finds on each page.

RuleWhat it means
Only pages underneath the startStarting at example.com/help adds example.com/help/billing, but not example.com/pricing. Start at the home page to cover the whole site.
Same site onlyLinks to other domains are never followed.
Up to 5 links deepPages more than 5 clicks from the starting page aren't reached.
Up to 500 pagesOnce 500 pages have been read, the rest are left out with Page limit reached.
Skip listAny address that contains one of your entries is left out. Entries are plain text, not patterns, and you can add up to 50.
robots.txt is respectedPages the site's robots.txt file blocks are left out. The crawler identifies itself as VoiceAIWrapperBot.
Readable pages onlyImages, scripts, stylesheets, data feeds and similar files are left out.

The crawler reads each page as the server sends it. It doesn't run JavaScript, so pages that build their content in the browser may come back with little or no text. If that happens, upload the content as a file or paste it as text instead.

See what was added and what was left out

Expand the site section row to see a summary like "42 pages found underneath Β· 38 added, 4 left out". Each page shows Added or the reason it was left out.

Pages that are still being read count as left out and show Left out until they're ready, so check again once the crawl has finished.

The Documents tab with a site section row expanded (chevron clicked): the "n pages found underneath Β· n added, n left out" summary and a list of pages, most with an Added badge and a few with reasons such as Matched an exclusion rule or Disallowed by robots.txt. Hover tooltip on the Last synced date reading "Syncs again every day."
ReasonWhat happened
Matched an exclusion ruleThe address contains an entry from your skip list.
Disallowed by robots.txtThe site asks crawlers not to read this page.
Page limit reachedThe crawl had already read 500 pages.
Not human-readable contentIt's a file such as an image, script or data feed, not a page.
No readable text foundThe page had no text we could use, or it couldn't be loaded (for example, a broken link).
Address is not publicly reachableThe address points somewhere that isn't public.
Left out (no reason)The page is still being read, or it was found but couldn't be read afterwards.

Left-out pages aren't used for answers. In most cases there's nothing to fix. For more on statuses, see Document Statuses, Syncing and Failed Documents.

Every page a crawl finds counts toward the 200-document limit, including pages it leaves out. A crawl of a large site can fill the knowledge base, after which you can't add anything else to it. To avoid this, start from a narrower section, use the skip list, or give the crawl a knowledge base of its own.

How web pages stay up to date

Web pages and site sections sync once a day. The Last synced column shows when each one was last checked. If a page hasn't changed since the last check, it's left as it is.

To sync a page now instead of waiting, hover over its row and click the sync icon next to the Last synced date. You'll see Refresh queued.

If a sync fails (for example, the site is briefly down), the page keeps answering from its last good copy. Its status changes to Out of date and Last synced shows Failed with the date. It tries again the next day.

A few things to know:

  • The daily sync re-reads the pages already in the list. It doesn't look for new pages. To pick up pages added to the site since your crawl, remove the site section and add it again.
  • Syncing a site section row re-reads its starting page only. The pages underneath it are kept up to date by the daily sync.
  • Adding an address that's already in the knowledge base doesn't create a duplicate. You'll see "That address is already here β€” refreshing it" and the existing page is read again. This happens even with Also add every page underneath this one turned on, so it doesn't start a new crawl.
  • Removing a site section removes every page underneath it too.

Troubleshooting

Paste the full address, including https:// (or http://). Copying it from your browser's address bar is the easiest way.

The address points somewhere we can't connect to, such as a private network or a local address. Use the public address of the page, and open it in a private browser window to check it loads without signing in.

The page couldn't be loaded or had no usable text. Common causes are a page that needs a login, a page that no longer exists, or a page that builds its content with JavaScript. Hover over the status to see the reason. Fix the cause, then click Try again in the question-mark popover on the row.

Check that:

  • You started at the right level. Pages outside the starting address aren't followed.
  • The pages are linked from other pages in that section. The crawl only finds pages through links, up to 5 clicks deep.
  • Your skip list isn't too broad. /help would also skip /helpdesk.
  • The site's robots.txt isn't blocking crawlers. Expand the row to see the reasons.

The crawl filled the 200-document limit. Remove the site section and crawl a narrower part of the site, or move the crawl into its own knowledge base and attach both to your agent.