WebFetchManager

Need help? Ask Kademi

Retrieves remote web pages and converts HTML to Markdown for AI/machine consumption, so callers get readable content and metadata instead of raw markup. Fetching always behaves as a good citizen: requests identify as KademiBot (with a non-configurable identifying header) and respect robots.txt, and outbound connections are restricted from internal hosts by the Kademi name resolver. https://github.com/Kademi/kademi-dev/issues/23796

Group: Managers

Methods

htmlToMarkdown(String html) · htmlToMarkdown(String html, Map<String,Object> options) · urlToMarkdown(String url) · urlToMarkdown(String url, Map<String,Object> fetchOptions) · urlToMarkdown(String url, Map<String,Object> fetchOptions, Map<String,Object> markdownOptions) · urlToMarkdownAsync(String url, Map<String,Object> fetchOptions, Map<String,Object> markdownOptions) · urlToMarkdownAsync(String url, Map<String,Object> fetchOptions) · urlToMarkdownAsync(String url) · htmlToMarkdownAsync(String html, Map<String,Object> options) · htmlToMarkdownAsync(String html)

`htmlToMarkdown(String html)`

Returns: Map<String,Object>

Converts the given HTML to Markdown using sensible defaults: navigation chrome and forms are stripped and document metadata is extracted.

Parameter	Description
`html`	the HTML document to convert, may be null or blank

`htmlToMarkdown(String html, Map<String,Object> options)`

Returns: Map<String,Object>

Converts the given HTML to Markdown.

Parameter	Description
`html`	the HTML document to convert, may be null or blank
`options`	conversion options, may be null for defaults. Keys mirror the camelCased xberg ConversionOptions setters (e.g. "skipImages", "extractMetadata", "headingStyle", "bullets", nested "preprocessing" map) plus the local "sanitize" and "stripFrontmatter". Keys whose default differs from the library: sanitize=true, stripFrontmatter=true, skipImages=true, extractMetadata=true, preprocessing {enabled=true, preset="minimal"}

`urlToMarkdown(String url)`

Returns: Map<String,Object>

Fetches the given URL and converts the response to Markdown using default fetch and conversion options.

Parameter	Description
`url`	the absolute http(s) URL to fetch

`urlToMarkdown(String url, Map<String,Object> fetchOptions)`

Returns: Map<String,Object>

Fetches the given URL and converts the response to Markdown using default conversion options.

Parameter	Description
`url`	the absolute http(s) URL to fetch
`fetchOptions`	fetch options, may be null for defaults. Supported keys: "allowedDomains" (List of host names, enforced on the initial URL and every redirect hop), "maxRedirects" (Integer, default 5, 0 disables redirects), "userAgent" (String override), "timeoutMs" (Integer), "maxBytes" (Integer), "respectRobots" (Boolean, default true), "followMarkdownLink" (Boolean, default false - when the HTML advertises a {@code <link rel="alternate" type="text/markdown">} alternate, fetch that instead of converting the HTML)

`urlToMarkdown(String url, Map<String,Object> fetchOptions, Map<String,Object> markdownOptions)`

Returns: Map<String,Object>

Fetches the given URL and converts the response to Markdown.

Parameter	Description
`url`	the absolute http(s) URL to fetch
`fetchOptions`	fetch options, may be null for defaults. Supported keys: "allowedDomains" (List of host names, enforced on the initial URL and every redirect hop), "maxRedirects" (Integer, default 5, 0 disables redirects), "userAgent" (String override), "timeoutMs" (Integer), "maxBytes" (Integer), "respectRobots" (Boolean, default true), "followMarkdownLink" (Boolean, default false - when the HTML advertises a {@code <link rel="alternate" type="text/markdown">} alternate, fetch that instead of converting the HTML)
`markdownOptions`	conversion options, may be null for defaults. Supported keys are described on htmlToMarkdown; the "sanitize" key (default true) also controls scrubbing of unsafe link schemes from markdown obtained directly (server-negotiated or via a markdown alternate)

`urlToMarkdownAsync(String url, Map<String,Object> fetchOptions, Map<String,Object> markdownOptions)`

Returns: Object

Fetches the given URL asynchronously, returning a JavaScript Promise that resolves with the same result map as urlToMarkdown. The blocking fetch and conversion run on a worker thread so the GraalJS event loop is never blocked; the per-tenant throttle is applied up front on the calling (loop) thread. GraalJS only.

Parameter	Description
`url`	the absolute http(s) URL to fetch
`fetchOptions`	fetch options, may be null for defaults (see {@link #urlToMarkdown(String, Map, Map)})
`markdownOptions`	conversion options, may be null for defaults (see {@link #urlToMarkdown(String, Map, Map)})

`urlToMarkdownAsync(String url, Map<String,Object> fetchOptions)`

Returns: Object

Fetches the given URL asynchronously using default conversion options. See the three-argument urlToMarkdownAsync method for details.

Parameter	Description
`url`	the absolute http(s) URL to fetch
`fetchOptions`	fetch options, may be null for defaults

`urlToMarkdownAsync(String url)`

Returns: Object

Fetches the given URL asynchronously using default fetch and conversion options. See the three-argument urlToMarkdownAsync method for details.

Parameter	Description
`url`	the absolute http(s) URL to fetch

`htmlToMarkdownAsync(String html, Map<String,Object> options)`

Returns: Object

Converts the given HTML to Markdown asynchronously, returning a JavaScript Promise that resolves with the same result map as htmlToMarkdown. The CPU-bound native conversion runs on a worker thread so the GraalJS event loop is never blocked. GraalJS only; not throttled per-tenant (it does no network I/O, like the synchronous method).

Parameter	Description
`html`	the HTML document to convert, may be null or blank
`options`	conversion options, may be null for defaults (see {@link #htmlToMarkdown(String, Map)})

`htmlToMarkdownAsync(String html)`

Returns: Object

Converts the given HTML to Markdown asynchronously using default options. See the two-argument htmlToMarkdownAsync method for details.

Parameter	Description
`html`	the HTML document to convert, may be null or blank

To get full access to the Kademi Hub existing customers can login here, or new customers can register here.