Non-linear history and max_fragment_id and next_row_id

[wjones127]

We need fragment_id and row_id to be globally unique across the table history. Re-using fragment ids can cause issue if we cache objects based on their fragment id. For example:

1. Create a dataset with stable Row IDs enabled; current version is 1.
2. Run compaction to produce version 2.
3. Checkout and restore back to version 1, adjust compaction params, run compaction again to produce version 3. The new compacted fragments will have the same ids as the compacted fragments from step 2.
4. Query version 3, then checkout to version 2 and query again. There will be cache collisions for keys based on (dataset_uri, version).

This is an issue with restore, but you can likely cause similar issues with branches. There might be similar bugs with duplicate row ids.

We need a way to keep these globally unique, if possible.

[majin1102]

This is an issue with restore, but you can likely cause similar issues with branches

I’m not sure I fully understand your point. From my understanding, branches use different dataset_uris, so their cached keys won’t conflict.

For example, given a dataset at s3://bucket/test.lance, its dev branch would have the URI s3://bucket/test.lance/tree/dev. This ensures that branches don’t share any dataset cache

[zhangyue19921010]

globally unique across the table history

Maybe related to globally unique not cache only

[wjones127]

I’m not sure I fully understand your point. From my understanding, branches use different dataset_uris, so their cached keys won’t conflict.

Okay, if they use unique URIs, then that is fine and branches aren't a worry.

[majin1102]

To clarify my point, I’ve opened a PR #5554. As I understand it, restore is currently the only operation that can cause fragment_id duplication across versions. cc @zhangyue19921010