Load Balancing Distributed Inverted Files: Query Ranking

Abstract

Search engines use inverted files as index data structures to speed up the solution of user queries. The index is distributed on a set of processors forming a cluster of computers and queries are received by a broker machine and scheduled for solution in the cluster. The broker must use a scheduling algorithm to assign queries to processors since the computations associated with the ranking of documents that form part of the solutions to queries can take a significant fraction of the total running time. The cost of this task can be highly variable and depends on the particular user preferences for words when formulating queries in a given period of time. Thus the scheduling algorithm must be able to cope efficiently with a highly dynamic and very large amount of jobs being assigned in an on-line manner to the processors. In this paper we evaluate a number of scheduling algorithms proposed in the literature in the context of scheduling queries on a search engine.