[Train][Doc] Update PyTorch Data Ingestion User Guide

[woshiyyya]

Why are these changes needed?

Restructured the "Starting from PyTorch Data" section for better readability

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

Nice! Love the added benefits & cleaner steps!

[matthewdeng]

IMO the previous documentation here was more clear.

[woshiyyya]

Ah I actually changed the order and put the last 2 sentences into a later section. Previously, we mixed Ray Data and framework utilities, and discussed them together. This PR tries to disaggregate them and try to tell the story step-by-step:

• Existing methods to ingest PyTorch data
• The relationship between the existing methods and Ray Data
• Why migrate to Ray Data (The benefits of Ray Data)
• How to migrate to Ray Data

But of course this is just my preference, we can switch it back if it doesn't sounds better.

[matthewdeng]

I see! We can definitely still do that, but in that case I think we should change the wording a bit in this line, and move the comparison table to the sub-section below. Right now it's just not super clear what it means that "You can still use these framework data utilities" and why there is a comparison table with Ray Data concepts.

[matthewdeng]

The original names were more explicit to make it clear that this is referring to the dataset framework, rather than the training framework.

[matthewdeng]

Suggested change

	Why using Ray Data?
	Comparison with Ray Data

[matthewdeng]

Let's add another sub-header here? Since the table is no longer explaining why to use Ray Data, but instructions on how to integrate an existing dataset with Ray Train. Maybe we can name it something like "Using PyTorch data with Ray Train"

[woshiyyya]

Yeah that's a good idea. I also feel that we are missing something here. Will add a sub-header

[matthewdeng]

nit:

Suggested change

	1. Convert your PyTorch Dataset to a Ray Dataset and
	1. Convert your PyTorch Dataset to a Ray Dataset.

There are some other small typos/formatting errors that I'll review more thoroughly in a follow-up review.

[matthewdeng]

nit: I understand why you chose to do this but I'm also a little worried this might be confusing since Option 1 technically does also use Hugging Face Datasets.

[woshiyyya]

Oh I realized the difference now. Previously, this section aims at teaching users how to convert their HF dataset to Ray Dataset, then do training. But this PR tries to directly categorize on what we eventually use in the training function.

# prev
HF Dataset -> Ray Data -> HF Transformers
            HF Dataset -> HF Transformers

# now
Ray Data -> HF Transformers
HF Dataset -> HF Transformers

My consideration here is that we'd better not force everyone to take the "HF Dataset -> Ray Data" conversion step.

For example, their original datasets format could be parquet, and before onboarding Ray, they already build a HF Dataset from parquet file, then feed it to HF Trainer.

In this case, they can either build ray dataset from parquet or from HF dataset.

# Before onboarding Ray
raw data -> HF dataset -> HF transformer

# After onboarding Ray
option 1: raw data -> HF dataset -> Ray Data -> HF transformer

v.s. 

option 2:  raw data -> Ray Data -> HF transformer

We can discuss more in person next week.

[matthewdeng]

This is good content that I think everyone should read, regardless of whether or not they are starting with PyTorch data. Do you think we could bring this higher up in the guide (e.g. even in the introduction), and then reference it from here?

[woshiyyya]

OK. Sounds good to me.

[justinvyu]

High level comments:

1. Can we copy some content from this blog post? This user guide should be the place to compare Ray Data against other data ingest solutions. Particularly, I'm thinking of copying over the diagrams as well as the table comparing against torch dataloader, HF dataset, tf data, etc.
2. Proposed restructure of this guide:

(Ray Data + Ray Train) Quickstart
    -> Code examples, with the "Option 1: Ray Data" moved over here under each framework.
Why use Ray Data?
    -> Comparison with Other Data Ingest Solutions
        -> Comparison table
Alternative to Ray Data Ingest (Framework-native Dataloaders)
    -> "Ray Data is the recommended data loading solution for scalable blah blah blah, but Ray Train still integrates well with existing dataloading solutions you may be using, such as X, Y, Z.
    -> Link to the framework user guides, since we already go over how to set up the framework native dataloaders.
Ray Data Configurations
    -> all the remaining sections become subsections.

I think this structure fixes the problem where I was getting lost in the middle of the user guide because it suddenly starts talking about pytorch dataloader -- it wasn't clear that there are 2 separate paths: Ray Data vs. Alternatives. Now, we first put Ray Data front and center and make the case for it. Then, we talk about alternatives that are still integrated nicely.

3. For a follow-up PR, it would be nice to have some more realistic examples. For example, show read_parquet("s3://...") instead of the from_items dummy dataset that we have right now in the torch ray data quickstart. Can borrow this from the blog post again.