|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.cjk.CJKTokenizer
public final class CJKTokenizer
CJKTokenizer was modified from StopTokenizer which does a decent job for
most European languages. It performs other token methods for double-byte
Characters: the token will return at each two charactors with overlap match.
Example: "java C1C2C3C4" will be segment to: "java" "C1C2" "C2C3" "C3C4" it
also need filter filter zero length token ""
for Digit: digit, '+', '#' will token as letter
for more info on Asia language(Chinese Japanese Korean) text segmentation:
please search google
Field Summary |
---|
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
CJKTokenizer(java.io.Reader in)
Construct a token stream processing the given input. |
Method Summary | |
---|---|
Token |
next()
Returns the next token in the stream, or null at EOS. |
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
close |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
---|
reset |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public CJKTokenizer(java.io.Reader in)
in
- I/O readerMethod Detail |
---|
public final Token next() throws java.io.IOException
next
in class TokenStream
java.io.IOException
- - throw IOException when read error
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |