续下载了一点点的数据集,准备搓一个小的模型玩玩(约 270 GB) - 科技与创作 / AI - NodeLoc的4天后,也就是今天,终于清洗完成了!
AAI_IndustryCorpus_literature: 176 GB (2832 万条)
sedthh_gutenberg_english: 28 GB (~4.6 万条)
HiTZ_Multilingual-Medical-Corpus: 3.5 GB
stjiris_portuguese-legal-sentences-v0: 7.9 GB (部分)
BEE-spoke-data_gutenberg-en-v1-clean: 1.6 GB
sedthh_gutenberg_multilang: 1.7 GB
joelniklaus_legal-mc4: 545 MB
AI-team-UoA_greek_legal_code: 254 MB
BEE-spoke-data_fineweb-literature-100k: 424 MB (部分)
rzgar_kurdish_sorani_literatures: ~13 MB, 22727 条
rzgar_kurdish_sorani_literatures_it: 894 MB
izumi-lab_open-text-books: 87 MB
CalvinU_project-gutenberg: 25569 条
Medilora_us_medical_license_exam_textbooks_en: 23 MB
QingHong258_Ancient-Chinese-Language-CUZ: 24 MB
Yotam_economics-textbook: 255 条
总产出约 220+ GB,约 3500 万条记录。