NFDI4Culture Linked Data Resources

About: <>

Further developments in prometheus

Further developments in prometheus

In the prometheus image archive further developments were completed by the Research Tools and Data Services task area.

Image similarity search

An image similarity search was integrated using image vectors created based on the self-supervised learning algorithm SwAV (Swapping Assignments between Views). The algorithm is described in Caron et al. and an implementation is provided by Facebook. A SwAV model pre-trained with the ImageNet dataset was used as the model. Resulting image vectors, constrained to 80 dimensions sufficient for the result, were precomputed for all images in the image archive and stored in the index.

The queries of the search engine are thus reduced to the calculation of the distance between the vectors stored in the index. The least distance can be calculated by either the Euclidean distance, the cosine similarity or the Manhattan distance. The Euclidean distance is used to calculate the distance, an evaluation of the different methods will be done in the next step. Furthermore, a background calculation of the vectors is implemented as a concurrent process to integrate it into the indexing process. The image similarity search is now also available on the production server and can be used by the public.

Integration of additional formats

The 3D viewer from kompakkt (code here) and the corresponding metadata for the 3D objects have now been integrated into prometheus. The new kompakkt index contains datasets with an iframe_url, which is then rendered by the application. The iframe contains the kompakkt widget to navigate the 3D object. A preview image can also be retrieved to display the results list.