Analysis of load balance in hybrid partitioning

Abstract

In information retrieval systems, there are three types of index partitioning schemes - term partitioning, document partitioning, and hybrid partitioning. The hybrid-partitioning scheme combines both term and document partitioning schemes. Term partitioning provides high concurrency, which means that queries can be effectively directed to their relevant nodes while the document partitioning and the hybrid partitioning schemes provide high parallelism which means that a single query is sent over the whole node in the system to get the query-answer. In this paper, directing-queries to their relevant nodes by carrying-out a set of real experiments in a parallel information retrieval (IR) system is investigated and a new approach for improving the load balance for hybrid partitioning scheme and increasing the throughput of the parallel IR system by directing hybrid queries to their relevant nodes is proposed.