Package com.axelor.ai.chat.plugin

Class TokenEstimator

java.lang.Object
com.axelor.ai.chat.plugin.TokenEstimator
public class TokenEstimator extends Object
Estimates token counts for UIMessage lists using provider-specific tokenizers when available, falling back to a character-based heuristic for providers without public tokenizers.

  • Method Summary

    Modifier and Type
    Method
    Description
    static int
    countMessageTokens(UIMessage msg, dev.langchain4j.model.TokenCountEstimator estimator)
    Counts tokens for a single message by extracting all text content from its parts.
    static int
    countTokens(List<UIMessage> messages, dev.langchain4j.model.TokenCountEstimator estimator)
    Counts the total tokens across all messages.
    static dev.langchain4j.model.TokenCountEstimator
    forModel(AiModel model)
    Returns a TokenCountEstimator appropriate for the given model's provider.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • forModel

      public static dev.langchain4j.model.TokenCountEstimator forModel(AiModel model)
      Returns a TokenCountEstimator appropriate for the given model's provider.

    • countTokens

      public static int countTokens(List<UIMessage> messages, dev.langchain4j.model.TokenCountEstimator estimator)
      Counts the total tokens across all messages.

    • countMessageTokens

      public static int countMessageTokens(UIMessage msg, dev.langchain4j.model.TokenCountEstimator estimator)
      Counts tokens for a single message by extracting all text content from its parts.