IrbisGPT. How the first neural network in the Kazakh language is structured

Armen Atayan, CEO, founder of Gen2B and co-founder of Irbis.ai, @armenlp

About Me

I am the founder of Gen2B. Previously, my team and I were engaged in web development and blockchain, creating applications for market makers and a platform for crypto trading. I am also a business angel and have invested in seven startups, five of which are still active. I have participated in IPOs and other stock market activities. Currently, I am fully focused on Gen2B.

About Gen2B

When GPT-3 was released, I was in Indonesia and gained access to a closed early beta test because I had an American company at the time. I experimented with the technology and was thrilled by its capabilities. At that time, I had a small team of developers, and we decided to create a Telegram bot based on GPT-3. We built it but did not release it, as it was merely a bridge to the technology without significant product contribution.

Later, we developed a "Swiss Army knife" of neural networks combined into a Telegram bot. It included child modes, avatar generation via the DALL-E API, and our own APIs. One of our services was related to educational simulations, where we trained through situational immersion.

With the release of GPT-4, we decided to change direction. We noticed that many startups investing in wrappers around publicly available technology did not survive. This became especially evident with the emergence of GPT-based chatbots, when many realized they had invested in interfaces to a common technology and suffered losses.

With the advent of GPT-4, I decided that we needed to develop in a direction where OpenAI could not compete—in closed loops. We assumed that with growing awareness of the threats associated with the use of language models in social hacking, regulations would tighten, and solutions for closed loops would become in demand.

Since then, we have been developing AI solutions for closed business loops. Our product Gen2Chat includes speech analytics, as well as analytics related to Gen2Call. Currently, we are finalizing the prototyping and testing of Gen2Agent—our speech bot for automatically answering calls, which we plan to release at the end of August.

About Irbis AI Neural Network

This is an interesting story. I remember sitting with Bakht Niyazov and Pavel Koktyshov at MOST Hub, discussing the need for solutions in closed loops. I advocated the idea that the use of language models poses serious threats, and therefore we need to move to closed loops. This is unsafe and can pose a threat to national security.

Someone asked: "Can we create our own national model in Kazakh?" Ideas and solutions immediately came to mind, and I replied: "Yes, we can, but it will take time." I discussed this with our Head of AI, who confirmed that it was possible but would take time.

Thus began the development of Irbis GPT. We had no resources or data, only a concept and an untested hypothesis that we believed would work. The hypothesis was based on our experience in creating commercial language models, including a model for Kaspi. Training a model for a completely different language, especially Kazakh, with limited data was a real challenge for us.

Irbis is an open-source model, and our project is not commercial. Our neural network has 7 billion parameters and is trained to recognize and respond in Kazakh with high accuracy. We used various techniques to improve the quality of responses and language recognition, making the model unique and effective.

Irbis GPT was created according to our roadmap. Our goal was to create a model that correctly understands Kazakh grammar and forms correct sentences. This was the basic task, after which we trained the model on various domains.

We went through several stages, testing models such as Gemi and LAM, as well as fine-tuning LAM-70. We conducted numerous iterations and tested hypotheses, which allowed us to build an effective path to the goal. Our Head of AI, deeply immersed in the process thanks to his experience with GPT. An interesting fact: he personally received access to GPT-3 from OpenAI president Greg Brockman in 2020.

We collaborated with the creators of Chinese Llama and chose the method of training the tokenizer and retraining the model layers, which was not applied very often. We also received support from NVIDIA in the form of credits.

Our goal was to achieve grammatically correct expression. We went through several stages: training a new tokenizer, fine-tuning, and creating an instruct model. The results after a short training time were satisfactory.

About the Application of Irbis GPT

The application area of Irbis GPT is no different from other language models, but it is particularly useful due to its understanding of the Kazakh language. The model also understands Russian and English but responds grammatically correctly in Kazakh, making it suitable for sensitive sectors and work without internet access.

The model is also interesting to researchers. We were pleasantly surprised that literally within a few weeks after the release, our models were downloaded more than 35,000 times, including forked and quantized versions. Irbis GPT is also used in educational platforms.

The main function of the model is the automation of incoming information processing. In this direction, there is still much work to be done, especially in training the model to work with context. In the next more advanced version, we will focus on improving these capabilities.

Risks in the Development of Artificial Intelligence in the Future

There is a concern that AI may threaten humanity, but in my opinion, everything depends on whose hands this tool will fall into and under the control of which group of people it will be. Artificial intelligence is a tool for processing information, and its effectiveness depends on who and for what purpose they use it.

If AI reaches the level of superintelligence, it may become a collective consciousness surpassing individual minds.

The key factor will be how exactly AI will be used and what goals those who control it will have. AI can be a powerful tool for improving life and solving global problems, but it is necessary to carefully manage its development and application to minimize risks and maximize benefits for humanity.

As for the development of artificial intelligence in Central Asia and Kazakhstan, there are several key trends and challenges. One of the global trends is the optimization of the contact center service sector. This is a trend also observed in Central Asia and helps increase business margins.

In the future, I see the creation of fully autonomous agents, that is, digital copies of people, who will be connected to contact centers. These agents will be trained on the best cases from all calls and chats, allowing them to form a digital personality with their own voice. This will ensure high efficiency of the digital employee, who will be practically indistinguishable from a real person.

We are already actively working in this direction and creating prototypes in our laboratory. In the near future, it can be expected that such technologies will develop and be implemented, providing new opportunities for business and improving service quality.

Plans

We plan to develop Irbis GPT, focusing on improving its cognitive abilities, especially in working with context, which is important for automating business processes. Currently, we are forming a roadmap, are open to cooperation, and are negotiating with major global companies. We also plan to attract students to participate in the project and continue to test hypotheses about the model architecture. Our forecasts remain optimistic.

Our solutions at Gen2B were ahead of the market, and now we see great potential for our products both in Kazakhstan and abroad. We have verified that many of our features have no analogs in the world and plan to hold a press conference after completing the platform formation.

Currently, we are actively cooperating with several companies in Kazakhstan. We have been closely working with some of them for over a year, refining the platform to meet their specific needs. Our platform is distinguished by a high level of customization, and we are confident that in the future our solutions will have significant global importance.