GPT-NeoX-Japanese — это специализированная языковая модель, мастерски адаптированная под сложную письменность Японии. Благодаря архитектуре GPT-NeoX, этот ИИ эффективно работает с иероглифами кандзи и азбуками хирагана и катакана, обеспечивая естественную генерацию текста.
https://huggingface.co/docs/transformers/model_doc/gpt_neox_japanese We introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of https://github.com/EleutherAI/gpt-neox. Japanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts. To address this distinct structure of the Japanese language, we use a special sub-word tokenizer. We are very grateful to tanreinama for open-sourcing this incredibly helpful tokenizer. Following the recommendations from Google’s research on PaLM, we have removed bias parameters from transformer blocks, achieving better model performance. Please refer this article in detail. Development of the model was led by Shinya Otani, Takayoshi Makabe, Anuj Arora, and Kyo Hattori from ABEJA, Inc.. For more information on this model-building activity, please refer here (ja).