Tokens are a big reason today's generative AI falls short

Generative AI models, like GPT-4o, use a token-based approach to process text due to technical and pragmatic reasons, facing challenges like handling spaces and punctuation, case sensitivity, language inequities, and difficulties with numbers and patterns.

• Transformers use tokenization to process text.
• Tokenization introduces biases and errors.
• English tokenization assumptions don't apply globally.
• Token-efficient languages perform better.
• Models struggle with number and pattern tasks.