Searched refs:convert_tokens_to_string (Results 1 – 2 of 2) sorted by relevance
| /aosp_15_r20/external/executorch/examples/mediatek/aot_utils/llm_utils/tokenizers_/ | ||
| H A D | tokenization_llama.py | 327 def convert_tokens_to_string(self, tokens): member in LlamaTokenizer |
| H A D | tokenization_utils_fast.py | 656 def convert_tokens_to_string(self, tokens: List[str]) -> str: member in PreTrainedTokenizerFast |