Enter the word for search

tokenize

/ˈtəʊkənaɪz/

/ˈtoʊkənaɪz/

Translations

tokenizeverb

tokenized

Past Simple

tokenized

Past Participle

tokenizing

Gerund

/ˈtəʊkənaɪz/

/ˈtoʊkənaɪz/

токенизироватьКомпьютерный и финтех-термин; используется как для разбиения текста на токены, так и для замены чувствительных данных суррогатными токенами.

The software can tokenize the text for further analysis.

Программа может токенизировать текст для дальнейшего анализа.

Synonyms

break down

Antonyms

Definitions

tokenizeverb

To break down text into smaller components, typically words or phrases, for processing in computational linguistics or natural language processing.

The software was designed to tokenize the input text before analyzing its syntax.

To substitute sensitive data, such as payment card numbers, with non-sensitive surrogate tokens for secure storage or transmission.

The gateway tokenizes credit card details before forwarding the transaction to the merchant.