Searched defs:suffix_indicator (Results 1 – 4 of 4) sorted by relevance
| /aosp_15_r20/external/libtextclassifier/native/utils/ |
| H A D | wordpiece_tokenizer.cc | 29 const std::string& suffix_indicator, in Lookup()
50 const std::string& suffix_indicator, const int max_chars_per_subtoken, in LongestMatchStartingAt()
113 const std::string& suffix_indicator, in AddWord()
132 int byte_end, const std::string& suffix_indicator, in AddUnknownCharacter()
161 const int max_chars_per_subtoken, const std::string& suffix_indicator, in TokenizeL2RGreedy()
211 const int max_chars_per_subtoken, const std::string& suffix_indicator, in WordpieceTokenize()
236 const std::string& suffix_indicator, bool use_unknown_token, in WordpieceTokenize()
|
| H A D | bert_tokenizer.h | 61 std::string suffix_indicator = kDefaultSuffixIndicator; member
|
| /aosp_15_r20/external/tflite-support/tensorflow_lite_support/cc/text/tokenizers/ |
| H A D | bert_tokenizer_jni.cc | 44 std::string suffix_indicator(raw_suffix_indicator); in Java_org_tensorflow_lite_support_text_tokenizers_BertTokenizer_nativeLoadResource() local
|
| H A D | bert_tokenizer.h | 62 std::string suffix_indicator = kDefaultSuffixIndicator; member
|