360Zhinao-7B — это продвинутая языковая модель от 360 Security Technology, обученная на тщательно очищенных данных. Главная фишка этого ИИ — поддержка контекстного окна до 360K токенов, что позволяет анализировать огромные массивы текста без потери смысла.
We present 360Zhinao models with 7B parameter size and context lengths spanning 4K, 32K and 360K, all available at this https URL. For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. Under such guidance, we perfect our data cleaning and composition strategies to pretrain 360Zhinao-7B-Base on 3.4T tokens. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size.