Tokeniser is a program used for converting charts from S-57 to S-101 and vice versa, for assessing data quality, comparing them, editing, and maintaining them up to date.Detailed answers about what tokens are and why we use them to solve the aforementioned tasks can be found in articles and presentations on the website dkart.fi. Here we will only briefly note the main points.

Due to fact that semantic difference between the formats is very insignificant, less than 10% the formats duplicate each other by 90% or more, which from a technical standpoint is entirely unnecessary. Therefore, the idea of a single universal data description deems logical and pragmatic. A token is a set of all descriptors of hydrographic reality necessary for representing real-world entities in accordance with the requirements of both S-57/S-101 standards. In other words, tokens are a unification of S-57 and S-101.

A universal token data model and Feature Catalogue are used to describe tokens, which we called S-157. Tokens are stored in an open HML format. The general data workflow looks as shown in the diagram. We convert original official S-57 ENC into tokens. Then the tokens are converted into S-57/S-101 charts.

Tokenizer SW: S-57 to S-101 /

English

Open the dKart Tokenizer program.

The image shows the dKart Tokenizer program menu. The left part of the window contains the following elements:

• Source chart - section for selecting files for conversion.

There are two types of conversion:
- S-57 chart — conversion of S-57 format charts (in our case, this option)
- Tokens — Conversion of charts with Tokens
- Package loading function — batch mode for loading charts.

• Options - section for program settings (profiles).
• Database — section for displaying the result. After conversion, two columns with chart names will appear on the Datasets panel on the left: the left column for the source chart, the right column for the tokenset chart.
• Application — information section.

The right part of the program window will display the path to the loaded chart (in batch mode, the list of all charts).

1) Check the box next to the Package Loading item to activate the batch processing mode.

4) Go to the Options section. Here, select the S-57 to TokenSet profile.

5) Check the interface settings:

• Number of threads (Threads) is set to 12 (recommended value for most configurations).
• Ensure the correct conversion rules are specified:

⁃ Output Folder: specify the destination folder for the resulting files.
⁃ Target FC: must be set to S-100_FC_Tokens.xml.
⁃ Forward Rules rule: S-100_Token to S-57.xml.
⁃ Backward Rules rule: S-57 to S-100_Token.xml.

6)Save the changes with the Apply button.
7)Return to the Source Chart section and click the Start button to begin conversion.

2) Select the conversion type S-57 Chart. Click the button to select the folder with chart files.
3)After selecting the folder, verify the path is correct in the right pane of the program. The list of charts will appear on the right.

recommended value for most configurations

The list of charts

Conversion type

Batch mode

Specify the path to the chart directory

Profile for converting
S-57 to S-101 cards

specify the destination folder for the resulting files

Save the changes with the Apply button.

Upon completion of the conversion, two columns will appear on the left in the Datasets panel:

• Left column — source charts.
• Right column — converted charts (tokensets).

Check the conversion status by the disappearance of icons next to each file (all icons on the right should disappear). This means the process has completed successfully.

Check the contents of the output folder (Output Folder) you specified earlier for the correct structure:

• Inside each data folder, there will be token sets with the extension .TKN...000.
• The TKNSET folder contains tokenset files and a chart in S-101 format.

After converting charts from S-57 format to S-101 format, we have a Token Set dataset, including token charts and their corresponding S-101 format charts, as well as the source S-57 data. We can edit the token charts and subsequently convert them back to S-101 and S-57.

1) The Token chart is taken from the folder obtained after the S-57 to Tokensets conversion (step 5)

2) Conversion settings

• The checkbox in the Package Loading window must be unchecked
• Select the conversion type S-57 Chart. Click the button to select the folder with chart files.
• The loaded chart will appear in the right part of the Tokeniser.

• Number of threads (Threads) is set to 12 (recommended value for most configurations).

⁃ Ensure the correct conversion rules are specified:
◦ Source FC: must be set to S-100_FC_Tokens.xml
◦ Target FC: must be set to 101_Feature_Catalogue_2.0.0.xml
◦ Forward Rules: S-100_Token to S-101.xml
◦ Uncheck the Make MSD checkbox
◦ Verify that the Load source chart and Load converted chart checkboxes are checked

• Save the changes with the Apply button.
• Return to the Source Chart section and click the Start button to begin conversion.

3) Result Verification
Upon completion of the conversion, two columns will appear on the left in the Datasets panel:

• Left column — source tokenset chart
• Right column — converted S-101 chart

4) Final Check

• Check the contents of the output folder you specified earlier for the correct structure:

The loaded chart

specify the folder for the result here

don't forget to click the apply button

1) The Token chart is taken from the folder obtained after the S-57 to Tokensets conversion (step 5)

2) Conversion settings

• The checkbox in the Package Loading window must be unchecked
• Select the conversion type S-57 Chart. Click the button to select the folder with chart files.
• The loaded chart will appear in the right part of the Tokenizer.

• Number of threads is set to 12 (recommended value for most configurations).

⁃ Ensure the correct conversion rules are specified:
◦ Source FC: must be set to S-100_FC_Tokens.xml
◦ Forward Rules: S-100_Token to S-57.xml
◦ Backward Rules: S-57 to S-100_Token.xml
◦ Check the Make MSD checkbox
◦ Verify that the Load source chart and Load converted chart checkboxes are checked

• Save the changes with the Apply button.
• Return to the Source Chart section and click the Start button to begin conversion.

3) Result Verification
Upon completion of the conversion, two columns will appear on the left in the Datasets panel:

• Left column — reverse S-57 chart
• Right column — source tokenset chart

4) Final Check

• Check the contents of the output folder you specified earlier for the correct structure:

Result: the specified output folder will contain the reverse S-57 chart and TXT files (if any were generated during conversion to tokens). The name of the reverse chart will be identical to the primary one.

The loaded chart

specify the folder for the result here

don't forget to click the apply button