Profiset Evaluation Platform

The Profiset platform was created to provide a testbed for large-scale image search systems. It consists of three parts – the Profiset collection, a set of query topics, and a partial ground truth for each query topic. In addition, we provide services for results evaluation and collaborative exploitation of the query topics and the respective ground truth data.

Profiset data

The collection contains 20M high-quality images with rich and systematic annotations, which were obtained from Profimedia, a web-site selling stock images produced by photographers from all over the world. For each image, we have extracted five MPEG7 global visual descriptors. Each entry in the dataset consists of the following information:

  • a thumbnail image;
  • a link to the corresponding page on the Profimedia web-site;
  • two types of image annotation: a title (typically 3 to 10 words) and keywords (about 20 keywords per image in average) mostly in English (about 95 %);
  • five MPEG-7 visual descriptors extracted from the original image content: Scalable Color, Color Structure, Color Layout, Edge Histogram and Region Shape.

To take a look at the collection, you can use the public demo and try our content-based searching in the images. The dataset can be downloaded and freely used for research purposes. Before the download, a registration and agreement to the usage terms is needed.

Query topics and ground truth

To enable efficient testing of search methods, we provide a set of 100 query topics and for each of them semi-automatically collected ground truth data verified by users. Each query topic is formed by single query image and a few keywords (typically one or two). The topics were selected using the Profimedia search logs and several examples of queries that we know from experience to be either easy or difficult to process in content-based searching. We also checked that there are enough relevant results for each query on the dataset. The following categories are represented by the topics: activity (5 queries), animal (8), art (6), body part (5), building (3), event (3), food (8), man-made objects (16), nature (16), people (12), place (9), plant (2), speciļ¬c building (4), vehicle (3).

A ground truth for a given query topic should in ideal case contain an indicator of relevance for each object in the dataset. However, creating such ground truth requires enormous amount of human labour. Therefore, we only provide a partial ground truth for each query. We employed a set of 140 different search methods exploiting different approaches (text-based search, content-based search, combination of both) to obtain a set of candidate objects. These were then manually evaluated by our judges (lab members). The judges were asked to mark each object as very good, acceptable, or irrelevant, which we transformed into relevance levels of 100 %, 50 % and 0 %, respectively. Each query-object pair was evaluated at least twice, the final relevance of a result object is computed as an average of the collected evaluations.

Related publications

More details about the creating of the ground truth as well as web services that allow to evaluate user-submitted results and expand our ground truth can be found in the following publication:

  • P. Budikova, M. Batko, P. Zezula. Evaluation Platform for Content-based Image Retrieval Systems. In International Conference on Theory and Practice of Digital Libraries 2011, LNCS 6966. Berlin : Springer, 2011. ISBN 978-3-642-24468-1, pp. 130-142. 26.9.2011, Berlin, Germany.

Profiset download

To download the Profiset data, you first need to confirm the Profiset usage agreement. Subsequently, we shall provide you with the access details.