How to get train-splits.txt and valid-splits.txt before training tr11-176B-ml

[robinfang7]

• Big Science version: latest
• Python version: 3.8.8
• Operating System: Ubuntu 20.04.5 LTS

Description

How to get train-splits.txt and valid-splits.txt at Line39 in train/tr11-176B-ml/tr11-176B-ml.slurm. Thx.
TRAIN_DATA_PATH=$MEGATRON_DEEPSPEED_REPO/data/train-splits.txt
VALID_DATA_PATH=$MEGATRON_DEEPSPEED_REPO/data/valid-splits.txt

[robertLiuLinFeng]

Hello, has this problem been solved? I also had this problem, and I don't know where to download the data in train-splits.txt/valid-splits.txt.