Searched defs:use_unknown_token (Results 1 – 4 of 4) sorted by relevance
| /aosp_15_r20/external/libtextclassifier/native/utils/ |
| H A D | wordpiece_tokenizer.cc | 92 LookupStatus NoTokenFound(const absl::string_view token, bool use_unknown_token, in NoTokenFound()
133 bool use_unknown_token, in AddUnknownCharacter()
162 bool use_unknown_token, const std::string& unknown_token, in TokenizeL2RGreedy()
212 bool use_unknown_token, const std::string& unknown_token, in WordpieceTokenize()
236 const std::string& suffix_indicator, bool use_unknown_token, in WordpieceTokenize()
|
| H A D | bert_tokenizer.h | 62 bool use_unknown_token = kDefaultUseUnknownToken; member
|
| /aosp_15_r20/external/tflite-support/tensorflow_lite_support/cc/text/tokenizers/ |
| H A D | bert_tokenizer_jni.cc | 36 jboolean use_unknown_token, jstring junknown_token, in Java_org_tensorflow_lite_support_text_tokenizers_BertTokenizer_nativeLoadResource()
|
| H A D | bert_tokenizer.h | 63 bool use_unknown_token = kDefaultUseUnknownToken; member
|