Import website - Typewise Documentation

Import website creates an indexed copy of crawled pages instead of searching in real-time. This significantly improves search quality. The crawler automatically follows links within the same domain and structure.

Best for: Static websites, FAQs or knowledge bases.
Limitations: This method has a capacity limit of 100 pages, after which it stops looking for new pages. It is not suitable for indexing entire large sites with thousands of pages.

Configuration

Field	Description
Title	The name of the source configuration.
When to use this source?	Instructions for the AI on when to use this specific source.
Analyze exact URL	If checked, only the specific Start URL will be analyzed, and links will not be followed.
Start URL	The URL where the crawler should begin indexing (e.g., `https://example.com/blog`).
Include URL(s)	Specific URLs to include in the import. If set, only the paths that match the specified patterns will be included in the final result. For e.g. `https://example.com/faq` will only add pages under the `faq` subpath
Exclude URL(s)	specific URLs to exclude from the import. If set, the paths that match the specified patterns will be excluded in the final result. For e.g. `https://example.com/news/*` will not scrape `https://example.com/news/latest-company-announcement.asp`
Import frequency	How often the website should be re-indexed (e.g., Once, Daily, Weekly).
Access level	Controls which agents or users can access this source. Access levels

See also

Choosing a method: Compare website lookup methods
Live URL fetch: Fetch the latest content from any URL in real time
Live web search: Search the web in real time