Home
last modified time | relevance | path

Searched defs:token_unicode (Results 1 – 3 of 3) sorted by relevance

/aosp_15_r20/external/libtextclassifier/native/utils/
H A Dbert_tokenizer.cc89 const UnicodeText token_unicode = UTF8ToUnicodeText(token, /*do_copy=*/false); in TokenizeSingleToken() local
111 const UnicodeText token_unicode = in TokenizeIntoWordpieces() local
H A Dtoken-feature-extractor.cc110 UnicodeText token_unicode = in ExtractDenseFeatures() local
140 UnicodeText token_unicode = in ExtractDenseFeatures() local
/aosp_15_r20/external/libtextclassifier/native/utils/grammar/parsing/
H A Dlexer.cc35 UnicodeText token_unicode = UTF8ToUnicodeText(value.data(), value.size(), in AppendTokenSymbols() local