The most important aspect of Shaman is its automatic connector generator. Starting from a simple URL, Awdee can infer the relationships between the different kinds of data in a web site, and the CSS selectors for all their properties.

Connectors can be then used to programmatically access the data on the website, or to simply dump all or part of its content.

1. Specify a starting URL

Click the New analysis button and paste the URL of the website you want to analyze. Next, press the Start button and wait for Shaman to analyze the HTML of the page.

2. Ensure the analysis was performed correctly

The first analysis is more prone to mistakes, because Shaman has to rely on a single example. If a sequence of elements was misinterpreted (for example, a series of divs where the first one is not conceptually a sibling of the others, like a header), you can let Shaman know which is the correct interpretation.

3. Assign names to tables and columns

Assign names to tables (types) and columns (fields). You can also remove useless fields by clicking on the X button of each column. If you want to see which column was assigned to each element, click the Select element button (or press F12) and select the element on the rendered page. The corresponding HTML pattern will be expanded on the workspace tree

4. Review CSS selector problems

Shaman tries as best as it can to create elegant yet precise CSS selectors for the fields of a table. In some cases manual intervention might be however required. Fields where Shaman couldn't manage to write a satisfying CSS selectors are highlighted with a warning icon. Expand the field and navigate to Extraction rule > CSS selector You can now manually type a CSS selector for the field or the element list you need. Elements that don't match Shaman's expectations will be hightlighted accordingly. You don't have however to follow Awdee's expectations.

5. Analyze detail pages

Because some of the fields you need might be only available from the detail pages (eg. /items/{Id}), you have to analyze them as well. Right click on a table and select Analyze values like [URL] as details.

Alternatively, you can right click an URL set and click Analyze as details for [table].

6. Save the connector

By pressing the Build connector button, the declarative classes will be generated for you. You can now fine-tune the code, compose a query against it, or publish it. Grayed out tables won't be exported. You can toggle the export flag by right clicking a table and selecting Enable/disable type export.