IrbisGPT. How the first neural network in the Kazakh language works

Armen Atayan, CEO, Founder of Gen2B and Co-founder of Irbis.ai, @armenlp

About Me

I am the founder of Gen2B. Previously, my team and I were engaged in web development and blockchain, creating applications for market makers and a crypto trading platform. Additionally, I have invested in seven startups as a business angel, five of which are still operational today. I have participated in IPOs and other exchange-related events. Currently, I am fully focused on Gen2B.

About the Irbis AI Neural Network

This is essentially an interesting story. I remember discussing with Bakhyt Niyazov and Pavel Koktyshev at MOST Hub the need for solutions in a closed-loop system. I argued that using language models poses significant risks and advocated for transitioning to a closed-loop system, as this could pose a threat and endanger national security.

Someone asked: "Can we create our own national model in the Kazakh language?" Ideas and solutions immediately came to mind, and I replied: "Yes, we can, but it will take time." I discussed this with the head of our AI department, who confirmed that it was possible but would require time.

Thus, the development of Irbis GPT began. We had no resources or data, only a concept and an untested hypothesis that we believed would work. The hypothesis was based on our experience in creating commercial language models, including the Kaspi model. Teaching a completely different language, especially Kazakh with limited data, was a real challenge for us.

Irbis is an open-source model and not a commercial project. Our neural network has 7 billion parameters and is trained to recognize and respond with high accuracy in the Kazakh language. We used various methods to improve the quality of responses and language recognition, making this model unique and effective.

Irbis GPT was developed according to our roadmap. Our goal was to create a model that correctly understands Kazakh grammar and can construct proper sentences. This was the primary task, after which we trained the model in various fields.

We went through several stages by testing models such as Gemi and LAM, as well as fine-tuned LAM-70. We made many iterations and tested hypotheses, which allowed us to create an effective path to achieve our goal. The head of our AI department deeply understood the process thanks to his experience working with GPT. An interesting fact: in 2020, he gained direct access to GPT-3 from OpenAI president Greg Brockman.

We collaborated with the creators of Chinese Llama, choosing methods for training the tokenizer and retraining the model layers. We also received support in the form of credits from NVIDIA.

Our goal was to achieve grammatically correct understanding. Short-term results were satisfactory.

About the Use of Irbis GPT

The application area of Irbis GPT is not different from other language models, but its understanding of the Kazakh language makes it particularly useful. The model also understands Russian and English, but it provides grammatically correct responses in Kazakh, making it suitable for sensitive sectors and areas with limited internet access.

The model is also interesting for researchers. Within a few weeks after its release, our models were downloaded more than 35,000 times, including forked and quantized versions. Irbis GPT is also used in educational platforms.

The main function of the model is to automate the processing of input information. In this area, especially in teaching the model to work with context, there is still much work to be done. In the next improved version, we will focus on enhancing these capabilities.

Plans

We plan to develop Irbis GPT and improve its cognitive abilities, particularly focusing on working with context, which is important for automating business processes. Currently, we are creating a roadmap, are open to collaboration, and are conducting negotiations with major global companies. We also have plans to involve students in the project and continue testing hypotheses about the model architecture. Our outlook for the future is optimistic.