Skip to main content
Website import is similar to Web Search but creates an indexed copy of crawled pages instead of searching in real-time. This significantly improves search quality. The crawler automatically follows links within the same domain and structure.
  • Best for: Static websites, FAQs or knowledge bases.
  • Limitations: This method has a capacity limit 100 pages, afterwards it stops looking for new pages. It is not suitable for indexing entire large webshops or sites with thousands of pages.

​
Configuration

FieldDescription
TitleThe name of the source configuration.
When to use this source?Instructions for the AI on when to use this specific source.
Analyze exact URLIf checked, only the specific Start URL will be analyzed, and links will not be followed.
Start URLThe URL where the crawler should begin indexing (e.g., https://example.com/blog).
Include URL(s)Specific URLs to include in the import. If set, only the paths that match the specified patterns will be included in the final result. For e.g. https://example.com/faq will only add pages under the faq subpath
Exclude URL(s)specific URLs to exclude from the import. If set, the paths that match the specified patterns will be excluded in the final result. For e.g. https://example.com/news/* will not scrape https://example.com/news/latest-company-announcement.asp
Import frequencyHow often the website should be re-indexed (e.g., Once, Daily, Weekly).
Access levelControls which agents or users can access this source. Access levels