fields module

Contains functions and classes related to fields.

Schema class

class whoosh.fields.Schema(**fields)

Represents the collection of fields in an index. Maps field names to FieldType objects which define the behavior of each field.

Low-level parts of the index use field numbers instead of field names for compactness. This class has several methods for converting between the field name, field number, and field object itself.

All keyword arguments to the constructor are treated as fieldname = fieldtype pairs. The fieldtype can be an instantiated FieldType object, or a FieldType sub-class (in which case the Schema will instantiate it with the default constructor before adding it).

For example:

s = Schema(content = TEXT,
           title = TEXT(stored = True),
           tags = KEYWORD(stored = True))
add(name, fieldtype, glob=False)

Adds a field to this schema.

Parameters:
  • name – The name of the field.
  • fieldtype – An instantiated fields.FieldType object, or a FieldType subclass. If you pass an instantiated object, the schema will use that as the field configuration for this field. If you pass a FieldType subclass, the schema will automatically instantiate it with the default constructor.
analyzer(fieldname)

Returns the content analyzer for the given fieldname, or None if the field has no analyzer

copy()

Returns a shallow copy of the schema. The field instances are not deep copied, so they are shared between schema copies.

has_vectored_fields()

Returns True if any of the fields in this schema store term vectors.

items()

Returns a list of (“fieldname”, field_object) pairs for the fields in this schema.

names()

Returns a list of the names of the fields in this schema.

scorable_names()

Returns a list of the names of fields that store field lengths.

stored_names()

Returns a list of the names of fields that are stored.

vector_names()

Returns a list of the names of fields that store vectors.

class whoosh.fields.SchemaClass(**fields)

All keyword arguments to the constructor are treated as fieldname = fieldtype pairs. The fieldtype can be an instantiated FieldType object, or a FieldType sub-class (in which case the Schema will instantiate it with the default constructor before adding it).

For example:

s = Schema(content = TEXT,
           title = TEXT(stored = True),
           tags = KEYWORD(stored = True))

FieldType base class

class whoosh.fields.FieldType(format, vector=None, scorable=False, stored=False, unique=False, multitoken_query='first')

Represents a field configuration.

The FieldType object supports the following attributes:

  • format (fields.Format): the storage format for the field’s contents.
  • vector (fields.Format): the storage format for the field’s vectors (forward index), or None if the field should not store vectors.
  • scorable (boolean): whether searches against this field may be scored. This controls whether the index stores per-document field lengths for this field.
  • stored (boolean): whether the content of this field is stored for each document. For example, in addition to indexing the title of a document, you usually want to store the title so it can be presented as part of the search results.
  • unique (boolean): whether this field’s value is unique to each document. For example, ‘path’ or ‘ID’. IndexWriter.update_document() will use fields marked as ‘unique’ to find the previous version of a document being updated.
  • multitoken_query is a string indicating what kind of query to use when a “word” in a user query parses into multiple tokens. The string is interpreted by the query parser. The strings understood by the default query parser are “first” (use first token only), “and” (join the tokens with an AND query), “or” (join the tokens with OR), and “phrase” (join the tokens with a phrase query).

The constructor for the base field type simply lets you supply your own configured field format, vector format, and scorable and stored values. Subclasses may configure some or all of this for you.

clean()

Clears any cached information in the field and any child objects.

index(value, **kwargs)

Returns an iterator of (termtext, frequency, weight, encoded_value) tuples.

parse_query(fieldname, qstring, boost=1.0)

When self_parsing() returns True, the query parser will call this method to parse basic query text.

parse_range(fieldname, start, end, startexcl, endexcl, boost=1.0)

When self_parsing() returns True, the query parser will call this method to parse range query text. If this method returns None instead of a query object, the parser will fall back to parsing the start and end terms using process_text().

process_text(qstring, mode='', **kwargs)

Returns an iterator of token strings corresponding to the given string.

self_parsing()

Subclasses should override this method to return True if they want the query parser to call the field’s parse_query() method instead of running the analyzer on text in this field. This is useful where the field needs full control over how queries are interpreted, such as in the numeric field type.

sortable_type

alias of unicode

sortable_values(ixreader, fieldname)

Returns an iterator of (term_text, sortable_value) pairs for the terms in the given reader and field. The sortable values can be used for sorting. The default implementation simply returns the texts of all terms in the field.

The value of the field’s sortable_type attribute should contain the type of the second item (the sortable value) in the pairs, e.g. unicode or int.

This can be overridden by field types such as NUMERIC where some values in a field are not useful for sorting, and where the sortable values can be expressed more compactly as numbers.

to_text(value)

Returns a textual representation of the value. Non-textual fields (such as NUMERIC and DATETIME) will override this to encode objects as text.

Pre-made field types

class whoosh.fields.ID(stored=False, unique=False, field_boost=1.0)

Configured field type that indexes the entire value of the field as one token. This is useful for data you don’t want to tokenize, such as the path of a file.

Parameters:
  • stored – Whether the value of this field is stored with the document.
class whoosh.fields.IDLIST(stored=False, unique=False, expression=None, field_boost=1.0)

Configured field type for fields containing IDs separated by whitespace and/or puntuation.

Parameters:
  • stored – Whether the value of this field is stored with the document.
  • unique – Whether the value of this field is unique per-document.
  • expression – The regular expression object to use to extract tokens. The default expression breaks tokens on CRs, LFs, tabs, spaces, commas, and semicolons.
class whoosh.fields.STORED

Configured field type for fields you want to store but not index.

class whoosh.fields.KEYWORD(stored=False, lowercase=False, commas=False, scorable=False, unique=False, field_boost=1.0)

Configured field type for fields containing space-separated or comma-separated keyword-like data (such as tags). The default is to not store positional information (so phrase searching is not allowed in this field) and to not make the field scorable.

Parameters:
  • stored – Whether to store the value of the field with the document.
  • comma – Whether this is a comma-separated field. If this is False (the default), it is treated as a space-separated field.
  • scorable – Whether this field is scorable.
class whoosh.fields.TEXT(analyzer=None, phrase=True, vector=None, stored=False, field_boost=1.0, multitoken_query='first')

Configured field type for text fields (for example, the body text of an article). The default is to store positional information to allow phrase searching. This field type is always scorable.

Parameters:
  • analyzer – The analysis.Analyzer to use to index the field contents. See the analysis module for more information. If you omit this argument, the field uses analysis.StandardAnalyzer.
  • phrase – Whether the store positional information to allow phrase searching.
  • vector – A whoosh.formats.Format object to use to store term vectors. By default, fields do not store term vectors.
  • stored – Whether to store the value of this field with the document. Since this field type generally contains a lot of text, you should avoid storing it with the document unless you need to, for example to allow fast excerpts in the search results.
class whoosh.fields.NUMERIC(type=<type 'int'>, stored=False, unique=False, field_boost=1.0, decimal_places=0, shift_step=4, signed=True)

Special field type that lets you index int, long, or floating point numbers in relatively short fixed-width terms. The field converts numbers to sortable text for you before indexing.

You specify the numeric type of the field when you create the NUMERIC object. The default is int.

>>> schema = Schema(path=STORED, position=NUMERIC(long))
>>> ix = storage.create_index(schema)
>>> w = ix.writer()
>>> w.add_document(path="/a", position=5820402204)
>>> w.commit()

You can also use the NUMERIC field to store Decimal instances by specifying a type of int or long and the decimal_places keyword argument. This simply multiplies each number by (10 ** decimal_places) before storing it as an integer. Of course this may throw away decimal prcesision (by truncating, not rounding) and imposes the same maximum value limits as int/long, but these may be acceptable for certain applications.

>>> from decimal import Decimal
>>> schema = Schema(path=STORED, position=NUMERIC(int, decimal_places=4))
>>> ix = storage.create_index(schema)
>>> w = ix.writer()
>>> w.add_document(path="/a", position=Decimal("123.45")
>>> w.commit()
Parameters:
  • type – the type of numbers that can be stored in this field: one of int, long, float, or Decimal.
  • stored – Whether the value of this field is stored with the document.
  • unique – Whether the value of this field is unique per-document.
  • decimal_places – specifies the number of decimal places to save when storing Decimal instances as int or float.
  • shift_steps – The number of bits of precision to shift away at each tiered indexing level. Values should generally be 1-8. Lower values yield faster searches but take up more space. A value of 0 means no tiered indexing.
  • signed – Whether the numbers stored in this field may be negative.
class whoosh.fields.DATETIME(stored=False, unique=False)

Special field type that lets you index datetime objects. The field converts the datetime objects to sortable text for you before indexing.

Since this field is based on Python’s datetime module it shares all the limitations of that module, such as the inability to represent dates before year 1 in the proleptic Gregorian calendar. However, since this field stores datetimes as an integer number of microseconds, it could easily represent a much wider range of dates if the Python datetime implementation ever supports them.

>>> schema = Schema(path=STORED, date=DATETIME)
>>> ix = storage.create_index(schema)
>>> w = ix.writer()
>>> w.add_document(path="/a", date=datetime.now())
>>> w.commit()
Parameters:
  • stored – Whether the value of this field is stored with the document.
  • unique – Whether the value of this field is unique per-document.
class whoosh.fields.BOOLEAN(stored=False)

Special field type that lets you index boolean values (True and False). The field converts the boolean values to text for you before indexing.

>>> schema = Schema(path=STORED, done=BOOLEAN)
>>> ix = storage.create_index(schema)
>>> w = ix.writer()
>>> w.add_document(path="/a", done=False)
>>> w.commit()
Parameters:
  • stored – Whether the value of this field is stored with the document.
class whoosh.fields.NGRAM(minsize=2, maxsize=4, stored=False, field_boost=1.0, queryor=False, phrase=False)

Configured field that indexes text as N-grams. For example, with a field type NGRAM(3,4), the value “hello” will be indexed as tokens “hel”, “hell”, “ell”, “ello”, “llo”. This field chops the entire

Parameters:
  • minsize – The minimum length of the N-grams.
  • maxsize – The maximum length of the N-grams.
  • stored – Whether to store the value of this field with the document. Since this field type generally contains a lot of text, you should avoid storing it with the document unless you need to, for example to allow fast excerpts in the search results.
  • queryor – if True, combine the N-grams with an Or query. The default is to combine N-grams with an And query.
  • phrase – store positions on the N-grams to allow exact phrase searching. The default is off.
class whoosh.fields.NGRAMWORDS(minsize=2, maxsize=4, stored=False, field_boost=1.0, tokenizer=None, at=None, queryor=False)

Configured field that breaks text into words, lowercases, and then chops the words into N-grams.

Parameters:
  • minsize – The minimum length of the N-grams.
  • maxsize – The maximum length of the N-grams.
  • stored – Whether to store the value of this field with the document. Since this field type generally contains a lot of text, you should avoid storing it with the document unless you need to, for example to allow fast excerpts in the search results.
  • tokenizer – an instance of whoosh.analysis.Tokenizer used to break the text into words.
  • at – if ‘start’, only takes N-grams from the start of the word. If ‘end’, only takes N-grams from the end. Otherwise the default is to take all N-grams from each word.
  • queryor – if True, combine the N-grams with an Or query. The default is to combine N-grams with an And query.

Exceptions

exception whoosh.fields.FieldConfigurationError
exception whoosh.fields.UnknownFieldError

Table Of Contents

Previous topic

analysis module

Next topic

formats module

This Page