Scott Services: Yandex upgrades search algorithm to better understand user searches

Wednesday, 23 August 2017

Yandex upgrades search algorithm to better understand user searches

Yandex, Russia’s largest search engine, announced they have transitioned to an upgraded search platform named “Korolyov.”

Named after a Russian satellite town northeast of Moscow that has long served as the center of Russia’s space exploration, Korolyov adds two major upgrades above the Palekh system which was launched last November.

Korolyov matches the meaning of a search query to all of the content of a web page. Palekh only looked at headlines.
Yandex applies Korolyov to a far greater number of pages than Palekh (200,000 vs. 150 per search query).

Palekh was Yandex’s attempt at Google’s RankBrain, and now Korolyov makes it a lot better. The new version of the algorithm is meant to better understand user intent and handle long-tail queries.

The press release explains:

First, “Korolyov” is better at understanding user intent than its predecessor because it examines the entirety of web pages rather than just their headlines. Second, “Korolyov” can scale to analyze a thousand times more documents in real time than “Palekh.”

Like all modern AI-based systems, “Korolyov” improves itself with each incremental data point. Yandex’s position as the largest search engine in Russia creates a positive feedback loop for our deep neural network algorithm, which leads to superior search results for our users.

To me, the interesting point was that the previous version, Palekh, only looked at the headlines of the pages and not the full content of the pages. That has now changed with this upgrade.

Yandex also announced that data from Yandex.Toloka, a mass-scale crowd-sourced platform, will now be fed into Yandex MatrixNet, the company’s proprietary algorithm, along with anonymized feedback data. Yandex uses Yandex.Toloka to have humans analyze and evaluate web content, feeding that data back to train the machine-learning algorithm. It is available at https://toloka.yandex.com/.