Home
last modified time | relevance | path

Searched defs:unknown_token (Results 1 – 6 of 6) sorted by relevance

/aosp_15_r20/external/libtextclassifier/native/utils/
H A Dwordpiece_tokenizer.cc93 const std::string& unknown_token, in NoTokenFound()
134 const std::string& unknown_token, in AddUnknownCharacter()
162 bool use_unknown_token, const std::string& unknown_token, in TokenizeL2RGreedy()
212 bool use_unknown_token, const std::string& unknown_token, in WordpieceTokenize()
237 const std::string& unknown_token, const WordpieceVocab* vocab_map, in WordpieceTokenize()
H A Dbert_tokenizer.h63 std::string unknown_token = kDefaultUnknownToken; member
/aosp_15_r20/external/tflite-support/tensorflow_lite_support/cc/test/text/
H A Dregex_tokenizer_test.cc112 int unknown_token; in TEST() local
129 int unknown_token; in TEST() local
/aosp_15_r20/external/tflite-support/tensorflow_lite_support/cc/text/tokenizers/
H A Dbert_tokenizer_jni.cc48 std::string unknown_token(raw_unknown_token); in Java_org_tensorflow_lite_support_text_tokenizers_BertTokenizer_nativeLoadResource() local
H A Dregex_tokenizer.cc108 bool RegexTokenizer::GetUnknownToken(int* unknown_token) { in GetUnknownToken()
H A Dbert_tokenizer.h64 std::string unknown_token = kDefaultUnknownToken; member