Shuka-1 — это инновационная мультимодальная модель, созданная специально для понимания индийских языков напрямую из аудио. Объединяя мощный энкодер и декодер Llama 3, этот ИИ обеспечивает бесшовную трансформацию речи в текст, учитывая все лингвистические нюансы.
Shuka v1 is a language model which natively understands audio in Indic languages. It is an encoder-decoder model built by combining two models: Our state-of-the-art, in-house, audio encoder: Saaras v1 Meta’s Llama3-8B-Instruct as the decoder The encoder and decoder are connected by a small projector with ~60M parameters. During training, only the projector weights are finetuned while the rest of the network is frozen. Following our tradition of training models frugally, we train Shuka v1 on less than 100 hours of audio. Though we only finetune the projector on English and Hindi data, the multilingual nature of our encoder makes Shuka v1 perform well on zero-shot QA in other Indic languages as well. We have tested on the model on Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu.