searching module

This module contains classes and functions related to searching the index.

Searching classes

class whoosh.searching.Searcher(reader, weighting=<class 'whoosh.scoring.BM25F'>, closereader=True, fromindex=None, parent=None)

Wraps an IndexReader object and provides methods for searching the index.

Parameters:
  • reader – An IndexReader object for the index to search.
  • weighting – A whoosh.scoring.Weighting object to use to score found documents.
  • closereader – Whether the underlying reader will be closed when the searcher is closed.
  • fromindex – An optional reference to the index of the underlying reader. This is required for Searcher.up_to_date() and Searcher.refresh() to work.
doc_count()

Returns the number of UNDELETED documents in the index.

doc_count_all()

Returns the total number of documents, DELETED OR UNDELETED, in the index.

document(**kw)

Convenience method returns the stored fields of a document matching the given keyword arguments, where the keyword keys are field names and the values are terms that must appear in the field.

This method is equivalent to:

searcher.stored_fields(searcher.document_number(<keyword args>))

Where Searcher.documents() returns a generator, this function returns either a dictionary or None. Use it when you assume the given keyword arguments either match zero or one documents (i.e. at least one of the fields is a unique key).

>>> stored_fields = searcher.document(path=u"/a/b")
>>> if stored_fields:
...   print stored_fields['title']
... else:
...   print "There is no document with the path /a/b"
document_number(**kw)

Returns the document number of the document matching the given keyword arguments, where the keyword keys are field names and the values are terms that must appear in the field.

>>> docnum = searcher.document_number(path=u"/a/b")

Where Searcher.document_numbers() returns a generator, this function returns either an int or None. Use it when you assume the given keyword arguments either match zero or one documents (i.e. at least one of the fields is a unique key).

Return type:int
document_numbers(**kw)

Returns a generator of the document numbers for documents matching the given keyword arguments, where the keyword keys are field names and the values are terms that must appear in the field.

>>> docnums = list(searcher.document_numbers(emailto=u"matt@whoosh.ca"))
documents(**kw)

Convenience method returns the stored fields of a document matching the given keyword arguments, where the keyword keys are field names and the values are terms that must appear in the field.

Returns a generator of dictionaries containing the stored fields of any documents matching the keyword arguments.

>>> for stored_fields in searcher.documents(emailto=u"matt@whoosh.ca"):
...   print "Email subject:", stored_fields['subject']
idf(fieldname, text)

Calculates the Inverse Document Frequency of the current term (calls idf() on the searcher’s Weighting object).

key_terms(docnums, fieldname, numterms=5, model=<class 'whoosh.classify.Bo1Model'>, normalize=True)

Returns the ‘numterms’ most important terms from the documents listed (by number) in ‘docnums’. You can get document numbers for the documents your interested in with the document_number() and document_numbers() methods.

>>> docnum = searcher.document_number(path=u"/a/b")
>>> keywords = list(searcher.key_terms([docnum], "content"))

“Most important” is generally defined as terms that occur frequently in the top hits but relatively infrequently in the collection as a whole.

Parameters:
  • fieldname – Look at the terms in this field. This field must store vectors.
  • docnums – A sequence of document numbers specifying which documents to extract key terms from.
  • numterms – Return this number of important terms.
  • model – The classify.ExpansionModel to use. See the classify module.
key_terms_from_text(fieldname, text, numterms=5, model=<class 'whoosh.classify.Bo1Model'>, normalize=True)

Return the ‘numterms’ most important terms from the given text.

Parameters:
  • numterms – Return this number of important terms.
  • model – The classify.ExpansionModel to use. See the classify module.
postings(fieldname, text, qf=1)

Returns a whoosh.matching.Matcher for the postings of the given term. Unlike the whoosh.reading.IndexReader.postings() method, this method automatically sets the scoring functions on the matcher from the searcher’s weighting object.

reader()

Returns the underlying IndexReader.

refresh()

Returns a fresh searcher for the latest version of the index:

my_searcher = my_searcher.refresh()

If the index has not changed since this searcher was created, this searcher is simply returned.

This method may CLOSE underlying resources that are no longer needed by the refreshed searcher, so you CANNOT continue to use the original searcher after calling refresh() on it.

search(q, limit=10, sortedby=None, reverse=False, groupedby=None, optimize=True, scored=True, filter=None, collector=None)

Runs the query represented by the query object and returns a Results object.

Parameters:
  • query – a whoosh.query.Query object.
  • limit – the maximum number of documents to score. If you’re only interested in the top N documents, you can set limit=N to limit the scoring for a faster search.
  • sortedby – the name of a field to sort by, or a tuple of field names to sort by multiple fields. This is a shortcut for using a whoosh.sorting.Sorter object to do a simple sort. To do complex sorts (where different fields are sorted in different directions), use Searcher.sorter() to get a sorter and use it to perform the sorted search.
  • reverse – if sortedby is not None, this reverses the direction of the sort.
  • groupedby – a list of field names or facet names. If this argument is not None, you can use the Results.groups() method on the results object to retrieve a dictionary mapping field/facet values to document numbers.
  • optimize – use optimizations to get faster results when possible.
  • scored – if False, the results are not scored and are returned in “natural” order (the order in which they were added).
  • collector – (expert) an instance of Collector to use to collect the found documents.
  • filter – a query, Results object, or set of docnums. The results will only contain documents that are also in the filter object.
Return type:

Results

sorter(*args, **kwargs)

Returns a whoosh.sorting.Sorter object for this searcher. See the documentation for Sorter for how to use the sorter object to get sorted search results.

up_to_date()

Returns True if this Searcher represents the latest version of the index, for backends that support versioning.

class whoosh.searching.Collector(limit=10, usequality=True, replace=10, groupedby=None, scored=True, timelimit=None, greedy=False)

A Collector finds the matching documents, scores them, collects them into a list, and produces a Results object from them.

Normally you do not need to instantiate an instance of the base Collector class, the Searcher.search() method does that for you.

If you create a custom Collector instance or subclass you can pass it to the Searcher.search() method using the collector keyword argument:

mycollector = MyCollector()
results = searcher.search(myquery, collector=mycollector)

Note that when you call Searcher.search() with a custom collector the method will overwrite several attributes on the collector instance with the values of keyword arguments to Searcher.search(). To avoid this, start the search from the collector instead:

mycollector = MyCollector()
results = mycollector.search(searcher, myquery)

Do not re-use or share Collector instances between searches. You should create a new Collector instance for each search.

To limit the amount of time a search can take, pass the number of seconds to the timelimit keyword argument:

# Limit the search to 4.5 seconds
col = Collector(timelimit=4.5, greedy=False)
# If this call takes more than 4.5 seconds, it will raise a
# whoosh.searching.TimeLimit exception
try:
    r = searcher.search(myquery, collector=col)
except TimeLimit:
    # You can still retrieve partial results from the collector
    # after a time limit exception
    r = col.results()

If the greedy keyword is True, the collector will finish adding the most recent hit before raising the TimeLimit exception.

add_all_matches(searcher, matcher)

Adds the matched documents from the given matcher to the collector’s list of matched documents.

add_matches(searcher, matcher)

Calls either :meth:Collector.add_top_matches` or Collector.add_all_matches() depending on whether this collector needs to examine all documents.

This method should record the current document as a hit for later retrieval with Collector.items().

add_searcher(searcher, q)

Adds the documents from the given searcher with the given query to the collector. This is called by the Collector.search() method.

add_top_matches(searcher, matcher)

Adds the matched documents from the given matcher to the collector’s priority queue.

collect(score, id)

This method is called for each found document. This method is only called by Collector.add_all_matches().

Parameters:
  • score – the score for this document. This may be None if the collector is not set up to compute scores.
  • id – the document number of the document.
items()

Returns the collected hits as a list of (score, docid) pairs.

pull_matches(matcher, usequality)

Low-level method yields (docid, quality) pairs from the given matcher. Called by Collector.add_top_matches() and Collector.add_all_matches(). If usequality is False or the matcher doesn’t support quality, the second item in each pair will be None.

results(runtime=None)

Returns the collected hits as a Results object.

score(searcher, matcher)

Called to compute the score for the current document in the given whoosh.matching.Matcher.

search(searcher, q, filter=None)

Top-level method call which uses the given Searcher and whoosh.query.Query objects to return a Results object.

This method takes care of calling Collector.add_searcher() for each sub-searcher in a collective searcher. You should only call this method on a top-level searcher.

should_add_all()

Returns True if this collector needs to add all found documents (for example, if limit=None), or alse if this collector should only add the top N found documents.

class whoosh.searching.TermTrackingCollector(*args, **kwargs)

This collector records which parts of the query matched which documents in the final results. The results for each part of the query are available as a dictionary in the catalog attribute of the collector after the search, where the keys are representations of the parts of the query and the values are sets of document numbers that matched that part of the query.

How to choose a key to represent query objects in the catalog dictionary was not entirely clear. The current implementation uses the unicode representation of the query object, which usually returns something at least recognizable (for example, unicode(Term("f", u"a")) == u"f:a" and unicode(Prefix("f", "b")) == u"f:b*").

>>> myparser = qparser.QueryParser("content", myindex.schema)
>>> myquery = myparser.parse(u"apple OR bear NOT camel")
>>> col = TermTrackingCollector()
>>> results = searcher.search(myquery, collector=col)
>>> # The docnums in the results that contained "apple"
>>> col.catalog["content:apple"]
set([1, 2, 3])
>>> for hit in results:
...     print hit.rank, ":", hit["title"]
...     for key, docset in col.catalog.keys():
...         if hit.docnum in docset:
...             print "   - Contains", key

Results classes

class whoosh.searching.Results(searcher, q, top_n, docset, groups=None, runtime=-1)

This object is returned by a Searcher. This object represents the results of a search query. You can mostly use it as if it was a list of dictionaries, where each dictionary is the stored fields of the document at that position in the results.

Parameters:
  • searcher – the Searcher object that produced these results.
  • query – the original query that created these results.
  • top_n – a list of (docnum, score) tuples representing the top N search results.
  • scores – a list of scores corresponding to the document numbers in top_n, or None if the results do not have scores.
  • runtime – the time it took to run this search.
copy()

Returns a copy of this results object.

docnum(n)

Returns the document number of the result at position n in the list of ranked documents.

docs()

Returns a set-like object containing the document numbers that matched the query.

estimated_length()

The estimated maximum number of matching documents, or the exact number of matching documents if it’s known.

estimated_min_length()

The estimated minimum number of matching documents, or the exact number of matching documents if it’s known.

extend(results)

Appends hits from ‘results’ (that are not already in this results object) to the end of these results.

Parameters:
  • results – another results object.
fields(n)

Returns the stored fields for the document at the n th position in the results. Use Results.docnum() if you want the raw document number instead of the stored fields.

filter(results)

Removes any hits that are not also in the other results object.

groups(name)

If you generating groupings for the results by using the groups keyword to the search() method, you can use this method to retrieve the groups.

>>> results = searcher.search(my_query, groups=["tag"])
>>> results.groups("tag")

Returns a dictionary mapping category names to lists of document IDs.

has_exact_length()

True if this results object already knows the exact number of matching documents.

highlights(n, fieldname, text=None, top=3, fragmenter=None, formatter=None, order=<function FIRST at 0x93a4fb4>)

Returns highlighted snippets for the document in the Nth position in the results. It is usually more convenient to call this method on a Hit object instead of the Results.

See the docs for the Hit.highlights() method.

key_terms(fieldname, docs=10, numterms=5, model=<class 'whoosh.classify.Bo1Model'>, normalize=True)

Returns the ‘numterms’ most important terms from the top ‘numdocs’ documents in these results. “Most important” is generally defined as terms that occur frequently in the top hits but relatively infrequently in the collection as a whole.

Parameters:
  • fieldname – Look at the terms in this field. This field must store vectors.
  • docs – Look at this many of the top documents of the results.
  • terms – Return this number of important terms.
  • model – The classify.ExpansionModel to use. See the classify module.
Returns:

list of unicode strings.

score(n)

Returns the score for the document at the Nth position in the list of ranked documents. If the search was not scored, this may return None.

scored_length()

Returns the number of scored documents in the results, equal to or less than the limit keyword argument to the search.

>>> r = mysearcher.search(myquery, limit=20)
>>> len(r)
1246
>>> r.scored_length()
20

This may be fewer than the total number of documents that match the query, which is what len(Results) returns.

upgrade(results, reverse=False)

Re-sorts the results so any hits that are also in ‘results’ appear before hits not in ‘results’, otherwise keeping their current relative positions. This does not add the documents in the other results object to this one.

Parameters:
  • results – another results object.
  • reverse – if True, lower the position of hits in the other results object instead of raising them.
upgrade_and_extend(results)

Combines the effects of extend() and increase(): hits that are also in ‘results’ are raised. Then any hits from the other results object that are not in this results object are appended to the end.

Parameters:
  • results – another results object.
class whoosh.searching.Hit(results, docnum, pos=None, score=None)

Represents a single search result (“hit”) in a Results object.

This object acts like a dictionary of the matching document’s stored fields. If for some reason you need an actual dict object, use Hit.fields() to get one.

>>> r = searcher.search(query.Term("content", "render"))
>>> r[0]
<Hit {title=u"Rendering the scene"}>
>>> r[0].rank
0
>>> r[0].docnum
4592L
>>> r[0].score
2.52045682
>>> r[0]["title"]
"Rendering the scene"
>>> r[0].keys()
["title"]
Parameters:
  • results – the Results object this hit belongs to.
  • pos – the position in the results list of this hit, for example pos=0 means this is the first (highest scoring) hit.
  • docnum – the document number of this hit.
  • score – the score of this hit.
fields()

Returns a dictionary of the stored fields of the document this object represents.

highlights(fieldname, text=None, top=3, fragmenter=None, formatter=None, order=<function FIRST at 0x93a4fb4>)

Returns highlighted snippets from the given field:

r = searcher.search(myquery)
for hit in r:
    print hit["title"]
    print hit.highlights("content")

See how to highlight terms in search results for more information.

You can set the fragmenter and formatter attributes on the Results object instead of specifying the fragmenter and formatter arguments to this method. For example, to return larger fragments and highlight them by converting to upper-case instead of with HTML tags:

from whoosh import highlight

r = searcher.search(myquery)
r.fragmenter = highlight.ContextFragmenter(surround=40)
r.formatter = highlight.UppercaseFormatter()
for hit in r:
    print hit["title"]
    print hit.highlights("content")
Parameters:
  • fieldname – the name of the field you want to highlight.
  • text – by default, the method will attempt to load the contents of the field from the stored fields for the document. If the field you want to highlight isn’t stored in the index, but you have access to the text another way (for example, loading from a file or a database), you can supply it using the text parameter.
  • top – the maximum number of fragments to return.
  • fragmenter – A whoosh.highlight.Fragmenter object. This controls how the text is broken in fragments. The default is whoosh.highlight.ContextFragmenter. For some applications you may find that a different fragmenting algorithm, such as whoosh.highlight.SentenceFragmenter gives better results. For short fields you could use whoosh.highlight.WholeFragmenter which returns the entire field as a single fragment.
  • formatter

    A whoosh.highlight.Formatter object. This controls how the search terms are highlighted in the snippets. The default is whoosh.highlight.HtmlFormatter with tagname='b'.

    Note that different formatters may return different objects, e.g. plain text, HTML, a Genshi event stream, a SAX event generator, etc.

  • order – the order of the fragments. This should be one of whoosh.highlight.SCORE(), whoosh.highlight.FIRST(), whoosh.highlight.LONGER(), whoosh.highlight.SHORTER(), or a custom sorting function. The default is highlight.FIRST.
class whoosh.searching.ResultsPage(results, pagenum, pagelen=10)

Represents a single page out of a longer list of results, as returned by whoosh.searching.Searcher.search_page(). Supports a subset of the interface of the Results object, namely getting stored fields with __getitem__ (square brackets), iterating, and the score() and docnum() methods.

The offset attribute contains the results number this page starts at (numbered from 0). For example, if the page length is 10, the offset attribute on the second page will be 10.

The pagecount attribute contains the number of pages available.

The pagenum attribute contains the page number. This may be less than the page you requested if the results had too few pages. For example, if you do:

ResultsPage(results, 5)

but the results object only contains 3 pages worth of hits, pagenum will be 3.

The pagelen attribute contains the number of results on this page (which may be less than the page length you requested if this is the last page of the results).

The total attribute contains the total number of hits in the results.

>>> mysearcher = myindex.searcher()
>>> pagenum = 2
>>> page = mysearcher.find_page(pagenum, myquery)
>>> print("Page %s of %s, results %s to %s of %s" %
...       (pagenum, page.pagecount, page.offset+1,
...        page.offset+page.pagelen, page.total))
>>> for i, fields in enumerate(page):
...   print("%s. %r" % (page.offset + i + 1, fields))
>>> mysearcher.close()
Parameters:
  • results – a Results object.
  • pagenum – which page of the results to use, numbered from 1.
  • pagelen – the number of hits per page.
docnum(n)

Returns the document number of the hit at the nth position on this page.

is_last_page()

Returns True if this object represents the last page of results.

score(n)

Returns the score of the hit at the nth position on this page.

Table Of Contents

Previous topic

scoring module

Next topic

spans module

This Page