OOM issue with large dataset streaming

[John6666]

Is the data set file itself too large…?
If it seems like a bug, let’s call lhonestq.

How to load a large hf dataset efficiently?

I am trying to load a dataset axiong/pmc_oa · Datasets at Hugging Face The dataset size is around 22 gb and I have ram ~10 GB, the dataset object is stuck at extracting file point [Screenshot 2024-01-16 at 8.41.53 AM] I also tried streaming mode but that’s giving another error. from datasets import load_dataset dataset = load_dataset("axiong/pmc_oa", 'pmc_oa', split='train', streaming=True) print(next(iter(dataset))) [Screenshot 2024-01-16 at 9.22.15 AM] Any suggestion on how to deal with…