There are two ways you can customize what a bot will learn. Let's assume you want a bot to just learn about the products on your website but nothing else.
Approach 1: Add URL
When you add a URL, the bot will add all URLs below that base path. So if your site had a product section under the base url named /products/, you would pass "www.petzparadice.com/products/", and the bot would crawl all items under this, e.g. "www.petzparadice.com/products/item_1", "www.petzparadice.com/products/item_2" and so on. This would mean the bot will not find anything above or adjacent to the /products/ section. For example "www.petzparadice.com/meet-the-team/" would not be included.
Approach 2: Exclude URL
You can pass a list of all URLs you do not want to be included. So you could simply add the base URL "www.petzparadice.com/" and manually exclude all pages which you don't want the bot to know about, e.g. "www.petzparadice.com/meet-the-team/".
Depending on how your site is structured, approach 1 or 2 or a combination might be the best approach.
