There are three primary ways to integrate website data. Choosing the right method depends on your specific needs regarding data freshness, reliability, and the target website’s structure.
This method works similarly to a standard search engine (like Google). You can set it up to include domains, which narrows the results to a specific website, or you can exclude domains to remove certain websites from the search.
Best for:
Big websites with a lot of pages, like e-commerce sites that can’t be imported.
Search for general information across the internet.
Limitations: It relies on the search engine’s indexing and search algorithms.
This method crawls specific web pages and stores their contents, optionally including those of the subpages, directly in the Typewise database. It features an advanced search algorithm that can find information more reliably and faster.
Best for: Static websites, FAQs or knowledge bases.
Limitations: This method has a capacity limit 100 pages with a maximum depth of 10 subpages, afterwards it stops looking for new pages. It is not suitable for indexing entire large webshops or sites with thousands of pages.
This method accesses a specific website in real-time to get the latest information. It supports dynamically generated URLs, e.g. to get an order status. The AI agent decides how to fill the dynamic parameters automatically based on the context.
How it works: You provide a URL template with placeholders.
Example 1:https://www.hotel-search.com/search?minPrice={minimum_price}&maxPrice={maximum_price} (Agent fills in price range)
Example 2:https://www.online-shop.com/en/search?q={product} (Agent searches for a specific product)
Example 3:https://www.search-engine.com/en/search/?q={search_query} (Agent inputs general search terms)
Best for: Sites with predictable search URL structures (e.g., e-commerce sites, booking platforms). It is highly effective when you need to pass dynamic values like product names, dates, or price ranges.
Limitations: This method will only analyze a single page of the website. It won’t follow links to other pages.
Recommendation: Only use this method if you need dynamic parameters or the latest information. If the URL is static, Website Import is usually a better choice.
