Japanese-GPT-1B — это мощный текстовый генератор от rinna, обученный на огромном массиве японских данных, включая Wikipedia и C4. Модель с 1 миллиардом параметров идеально справляется с созданием естественного контента и пониманием нюансов японского языка.
Model architecture A 24-layer, 2048-hidden-size transformer-based language model. Training The model was trained on Japanese C4, Japanese CC-100 and Japanese Wikipedia to optimize a traditional language modelling objective. It reaches around 14 perplexity on a chosen validation set from the same data. Tokenization The model uses a sentencepiece-based tokenizer. The vocabulary was first trained on a selected subset from the training data using the official sentencepiece training script, and then augmented with emojis and symbols.