Data collection is underway for the creation of a large Kazakh language model

In order to create and launch a large Kazakh language model, JSC “National Information Technologies” is beginning to use the service from a leader in the field of machine learning — Hugging Face. Based on the platform, an open data collection will be organized, to which the professional IT community of Kazakhstan and holders of open data can join.

Hugging Face is a leading platform designed for exchanging research in the field of machine learning, on the basis of which users can develop tools and create AI models.

The creation of a modern Kazakh language model is an important step towards strengthening the digital independence of Kazakhstan and promoting national culture in the global digital space. The first stage in the creation of any language model is data collection.

As a result of the general data collection, a high-quality model for processing the Kazakh language will be created. In the future, this will help not only to improve automatic translation, but also to increase the quality and accuracy of processing texts in the Kazakh language as a whole.

Representatives of the professional IT community and holders of open data can join the collection. Data collection will be carried out on a specially created account of JSC “NIT”. Users, after logging into the platform, can upload files to the account. Text files of different styles and genres in the Kazakh language in txt, .csv, .json formats are subject to upload.