How to use IBM App Connect with Website Crawler

Default filter properties:

Allow self-signed certificate

• To crawl a website that has self-signed certificates, set this property to true.
• Supported values are true (the default) and false.

Crawl interval in milliseconds

• This property specifies how frequently Website Crawler crawls each page URL that is retrieved by the action. This value is used to reduce the load effect on the website that is being crawled.
• Supported values are slow (2500 ms), medium (1500 ms), and fast (700 ms). The default value is Slow.

Exclude header and footer tags

• To exclude header and footer tags, set this property to true. Links in header and footer tags are not crawled.
• Supported values are true and false (the default).

Filter by domain

• If this property is set to true, Website Crawler does not crawl links that contain a host domain that is different from the domain that is configured in Fully qualified web URL. (Any such links are still listed in the links collection in the action response.)
• To crawl a website on a private network through the IBM Secure Gateway, you must set Filter by domain to true because public website pages cannot be crawled through the IBM Secure Gateway.
• Supported values are true (the default) and false.

Fully qualified web URL

• Specify the URL of the initial page to be crawled.
• Specify a URL that starts with the same protocol and domain as is specified on the Website Crawler connection parameter Fully qualified web domain name to be crawled

Maximum depth of pages to be crawled

• Set the maximum depth of pages to be crawled to an integer value, such as 1, 5 (the default), or 20. This property directs Website Crawler to crawl pages only until the specified depth level, as demonstrated in the following examples.
  • Maximum depth = 0: Crawl only the initial page (which is specified by the Fully qualified web URLs property) to return a collection of the links that the page contains.
  • Maximum depth = 1: Crawl the initial page, then crawl any HTML page links that were discovered in the initial page. This setting returns a collection of the links that are contained in the initial page and in any discovered HTML page links (at 'depth 1').
  • Maximum depth = 2: Crawl the initial page, then crawl any discovered HTML page links (at 'depth 2'), then crawl any HTML page links that were discovered in web pages at 'depth 2'.

Timeout in milliseconds

• This property specifies the maximum time that Website Crawler waits for a web page to respond before it stops trying to crawl that page and moves on to the next page in the list.
• The default value is 30000 milliseconds.

Respect robots TXT rules set by Web master

• This property defines that the connector adheres to the robots.txt rule, which is defined by the websites, for all the publicly accessible websites and websites hosted behind a firewall.
• If a website is behind a firewall and the Filter by Domain option is set to true, the property can be overridden. If it’s a public site, changing this property takes no effect.
• Supported values are true (the default) and false.

Extra filter properties:

Blocklist strings

• This filter specifies one or more substrings for URLs to be excluded from the website crawl. Separate multiple strings with a semicolon ;.
• If a string matches part of a URL, that URL is not crawled.

For example, a website page contains the links: https://my.domain/afolder/open.html, https://my.domain/bfolder/private_ano.html, and https://my.domain/afolder/private.html. If you set Blocklist strings to /private, the response for the website page contains the links list with only https://my.domain/afolder/open.html. Website Crawler ignores the pages https://my.domain/bfolder/private_ano.html and https://my.domain/afolder/private.html.

Filter by subtree

• If this property is set to true, only pages under the same initial page URL are crawled. For example, if the page URL in Fully qualified web URL is specified as https://www.example.com/partners, and Filter by subtree is set to true, only URLs that begin with https://www.example.com/partners/ are crawled.
• If the value is set to false, Website Crawler crawls all pages from the initial page URL. For example, the page URL in Fully qualified web URL is specified as https://www.example.com/partners, and Filter by subtree is set to false. In this case, all links with URLs that begin with https://www.example.com/ are crawled down to the specified maximum depth. (Links that begin with different URLs can also be crawled if Filter by domain is set to false and other website conditions allow crawling.)
• Supported values are true and false. Default value is false.

• http://my.co.host.com/helloworld/web_page.html (depth 1) with links to site.css and page_2.html.
• http://my.co.host.com/helloworld/page_2.html (depth 2) with links to site.css, page_3.html, and web_page.html.
• http://my.co.host.com/helloworld/page_3.html (depth 3) with links to site.css and page_2.html.

• Fully qualified web URL is set to http://my.co.host.com/helloworld/web_page.html.
• Maximum depth of pages to be crawled is set to 3.

{"response":{"payload":[
{"Id":"fc0af263-cbea-5e58-b3e0-52f3020e6d5e","Url":"http://my.co.host.com/helloworld/web_page.html","ContentType":"text/html","LastModified":"2018-11-17T17:50:52.000Z","Links":[{"id":"a02d2a25-77a3-5352-b612-59b2a965d55d","url":"http://my.co.host.com/helloworld/page_2.html"}]},
{"Id":"a02d2a25-77a3-5352-b612-59b2a965d55d","Url":"http://my.co.host.com/helloworld/page_2.html","ContentType":"text/html","LastModified":"2018-11-17T17:50:52.000Z","Links":[{"id":"05884593-6b22-59bb-8696-fbbcd86a0898","url":"http://my.co.host.com/helloworld/page_3.html"},{"id":"fc0af263-cbea-5e58-b3e0-52f3020e6d5e","url":"http://my.co.host.com/helloworld/web_page.html"}]},
{"Id":"05884593-6b22-59bb-8696-fbbcd86a0898","Url":"http://my.co.host.com/helloworld/page_3.html","ContentType":"text/html","LastModified":"2018-11-17T17:50:52.000Z","Links":[{"id":"a02d2a25-77a3-5352-b612-59b2a965d55d","url":"http://my.co.host.com/helloworld/page_2.html"}]}
}

URL

This property is mandatory, and specifies the fully qualified URL of the web page to be downloaded.

Ignore invalid SSL

• To download the content of a web page from a website that uses self-signed certificates, set this property to true.
• Supported values are true (the default) and false.

Content Type

• This property is mandatory, and specifies whether the file contains binary or text content.
• Supported values are Binary (the default) and Text. (You can use the Custom option to map a value from a preceding event or action.)