Top Qs
Timeline
Chat
Perspective

tokenizer

From Wiktionary, the free dictionary

Remove ads

English

Etymology

From tokenize +‎ -er.

Noun

tokenizer (plural tokenizers)

  1. (computing) A system that parses an input stream into its component tokens.

Wikiwand - on

Seamless Wikipedia browsing. On steroids.

Remove ads
Remove ads