Support Infinite PyDataset

[LarsKue]

Toy datasets are often generated on the fly, making their effective size infinite. Model.fit() nicely supports this idea by allowing to pass a batch_size and steps_per_epoch parameter. However, we already need both of these in a fully generated PyDataset since we need to return a positive length and return batches from __getitem__. This leads to code like this:

dataset = MyPyDataset(batch_size=128, steps_per_epoch=100)
model.fit(dataset, batch_size=128, steps_per_epoch=100)

which creates redundancy, because the parameters to fit() are ignored, or both must have the same value.

Conceptually, it makes the most sense to pass this information in model.fit(), so I propose two changes to PyDataset:

1. Allow signalling a dataset of infinite length, e.g. by returning -1 or raising a TypeError like a tf.data.Dataset
2. Allow fetching dynamically sized batches in __getitem__, e.g. by returning only single items instead of a batch, like a torch.utils.data.Dataset

I would be willing to help out on a pull request for this, but would appreciate some initial pointers and whether these changes could see merging. Thank you.

[fchollet]

There are two separate issues here: support for infinite cardinality, and support for dynamically sized batches.

For infinite cardinality support, we can use None to indicate it. We can definitely support it, similar to what we do for infinite tf.data.Dataset instances. If you're able, feel free to open a PR!

[LarsKue]

Yes, I felt the two issues are similar enough to group them in one issue. Apologies if this is not how you see it.

As for returning None as the length, I feel this is a little risky, since forgetting a return statement also looks like returning None from the outside. This could be an issue for a dataset that has a more complex length computation, such as one that increases in length as training goes on (e.g. experience replay buffers).

[hertschuh]

Hi @LarsKue ,

Thank you for reporting this.

For item 2., this is already how it works. __getitem__ returns a full batch, and the batch size can be different from batch to batch.

For item 1. I just pushed a change that enables this. You implement the num_batches property and return None to signal an infinite dataset. I understand the concern that if you forget a return clause, you could return None by accident. But weighing this against other options, it seemed relatively less error prone (you could easily do incorrect math with -1).

Note that you can also easily feed a infinite dataset to model.fit() by using a generator.

def generate():
    while True:
        x = ...
        y = ...
        yield x, y

model.fit(generate(), ...)

It's less code than implementing a PyDataset and doesn't require being able to access your data by index. However, it doesn't support multiprocessing / multithreading.