|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.CharTokenizer
org.apache.lucene.analysis.ru.RussianLetterTokenizer
public class RussianLetterTokenizer
A RussianLetterTokenizer is a tokenizer that extends LetterTokenizer by additionally looking up letters in a given "russian charset". The problem with LeterTokenizer is that it uses Character.isLetter() method, which doesn't know how to detect letters in encodings like CP1252 and KOI8 (well-known problems with 0xD7 and 0xF7 chars)
Field Summary |
---|
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
---|
input |
Constructor Summary | |
---|---|
RussianLetterTokenizer(java.io.Reader in,
char[] charset)
|
Method Summary | |
---|---|
protected boolean |
isTokenChar(char c)
Collects only characters which satisfy Character.isLetter(char) . |
Methods inherited from class org.apache.lucene.analysis.CharTokenizer |
---|
next, normalize |
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
---|
close |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
---|
reset |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public RussianLetterTokenizer(java.io.Reader in, char[] charset)
Method Detail |
---|
protected boolean isTokenChar(char c)
Character.isLetter(char)
.
isTokenChar
in class CharTokenizer
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |