Searched defs:SentencePieceTokenizer (Results 1 – 4 of 4) sorted by relevance
| /aosp_15_r20/external/executorch/examples/models/llama/ |
| H A D | eval_llama_lib.py | 20 Tokenizer as SentencePieceTokenizer, unknown
42 tokenizer: Union[SentencePieceTokenizer, Tiktoken],
92 tokenizer: Union[SentencePieceTokenizer, Tiktoken],
140 tokenizer: Union[SentencePieceTokenizer, Tiktoken],
|
| /aosp_15_r20/external/tflite-support/tensorflow_lite_support/cc/text/tokenizers/ |
| H A D | sentencepiece_tokenizer.h | 35 explicit SentencePieceTokenizer(const std::string& path_to_model) { in SentencePieceTokenizer() function
39 explicit SentencePieceTokenizer(const char* spmodel_buffer_data, in SentencePieceTokenizer() function
|
| /aosp_15_r20/external/executorch/examples/models/llama/evaluate/ |
| H A D | eager_eval.py | 13 Tokenizer as SentencePieceTokenizer, unknown
29 tokenizer: Union[SentencePieceTokenizer, Tiktoken],
|
| /aosp_15_r20/external/executorch/extension/llm/tokenizer/ |
| H A D | utils.py | 9 Tokenizer as SentencePieceTokenizer, unknown
|