Add C4 Streaming dataset

[abhi-mosaic]

This PR adds an adapter for HF streaming datasets, so that we can train with DDP and sharding. There is also a small QoL fix for DeepSpeed.

I've also added a new GPT3-125m YAML which can be the start of our new GPT3 family of benchmarks. I think we should deprecate the GPT2 collection of YAMLs (-52m, -83m -125m) in the near future, say Composer v0.5.

Some TODOs, which can be follow-up PRs:

• Add the rest of the GPT3 family YAMLs
• Update the GPT2 model with QoL fixes
• Update the BERT-base benchmark to use C4
• Deprecate / combine the old LMDataset object

[abhi-mosaic]

This PR has been refactored, so instead of a general HF streaming adapter (which is a bit unfeasible, given that HF datasets are user-generated and do not have a consistent implementation of shards or sample dicts), this PR now adds a specific C4Dataset that happens to be backed by HF Datasets.

The intent is for C4Dataset to be an example for users who want to train with their own HF datasets, but they will have to construct their CustomDataset class independently. This is similar to how our ImageNet dataset is an example of a vision dataset that happens to be backed by torchvision.datasets.ImageFolder.

In the future, we can try to abstract some things away, and/or coordinate with HF upstream to build better sharding support, but right now C4Dataset is just a standalone example.

[hanlint]

Did mostly a code style/ readability review. Will have to rely on more knowledgable (e.g. @moinnadeem ) to check the functional correctness.

[moinnadeem]

Added a few comments, correctness looks good to me from what I can tell! Good work Abhi!

[abhi-mosaic]

Hey @hanlint @moinnadeem , sorry for the delay on this PR. I think I should have addressed all the comments. Hope we can get this merged today! And then I can work with @moinnadeem to update the BERT yaml to use C4, as well as convert some of the large GPT2 yamls (> 125M) to GPT3.

[hanlint]

LGTM from code quality stand point, defer to @moinnadeem for NLP correctness

[abhi-mosaic]

Also, I'm seeing a lot of new pyright errors.. but I don't think there are any related to c4.py. Please let me know if this is expected / what I can do.

[abhi-mosaic]

Turns out the self documenting f-string syntax f"{var=}" did not appear until python3.8, so our python3.7 tests were failing 🙃 . Reverted...