Yandex has updated its image search, adding simultaneous analysis of the image itself and the text accompanying it on the page. The company claims this will allow for more accurate determination of image content, especially in cases where the caption or description does not match what is shown in the photo.

Yandex image search algorithms have learned to simultaneously analyze an image and the text on the page. The new technology improves results in cases where the text in the image description contradicts the image itself.

The update is based on multimodal technology. First, specialists trained a large model, and then, based on it, developed several compact models for various data processing stages.

The most resource-intensive operations are performed in advance, allowing the new algorithm to work in real time without increasing the time it takes to generate search results.

Previously, the system analyzed the image and text separately, which could lead to errors when visual information and description diverged. Now, this data is considered jointly. According to Yandex, after the technology was implemented, the share of relevant images at the top of the search results significantly increased.