To get full access to the Kademi Hub existing customers can login here, or new customers can register here.
Encodes text into tokens, decodes tokens back into text, and counts tokens, delegating to the first registered TokeniserService. It is registered in the root context and its methods are exported for use from GraalJS. The encoder names currently supported by the registered TikTokenService implementation are gpt2, r50k_base, p50k_base, p50k_edit and cl100k_base.
Group: Managers
Properties
| Property | Returns | Description |
|---|---|---|
| tokeniserServices | List<TokeniserService> |
Methods
encode(String encoder, String content)
Returns: List<Integer>
Encodes the given content using the specified encoder using the first available TokeniserService
| Parameter | Description |
|---|---|
encoder | the encoder to use |
content | the content to encode |
decode(String encoder, Collection<Integer> tokens)
Returns: String
Decodes the given tokens back into text using the specified encoder and the first available TokeniserService. Returns null if no TokeniserService is registered.
| Parameter | Description |
|---|---|
encoder | the encoder that was used to produce the tokens |
tokens | the tokens to decode |
countTokens(String encoder, String content)
Returns: Integer
Counts the number of tokens the given content would encode to, using the specified encoder and the first available TokeniserService.
| Parameter | Description |
|---|---|
encoder | the encoder to use |
content | the content to count tokens for |
© 2025 Kademi Ltd