Deprecate list_indices

[westonpace]

In #5221 we introduce describe_indices which should replace list_indices. I'm creating this issue to deprecate list_indices which is actually a bit of a task as it is used in many tests.

[fenfeng9]

I'd like to work on this issue and will start this weekend.

[fenfeng9]

I have a few migration questions to confirm:

1. Should the old version of the datagen script retain list_indices?

This script is pinned to lance==0.21.0, which doesn’t have describe_indices / stats.index_stats.

lance/test_data/v0.21.0/datagen.py

Lines 10 to 39 in a902892

	# To generate the test file, we should be running this version of lance
	assert lance.__version__ == "0.21.0"
	data = pa.table(
	{
	"vector": pa.FixedSizeListArray.from_arrays(
	pc.random(16 * 256).cast(pa.float32()), 16
	)
	}
	)
	ds = lance.write_dataset(data, "bad_index_fragment_bitmap")
	ds.create_index("vector", index_type="IVF_PQ", num_partitions=1, num_sub_vectors=1)
	data2 = pa.table(
	{
	"vector": pa.FixedSizeListArray.from_arrays(
	pc.random(16 * 32).cast(pa.float32()), 16
	)
	}
	)
	ds.insert(data2)
	ds.optimize.optimize_indices(num_indices_to_merge=0)
	ds.cleanup_old_versions(older_than=timedelta(0))
	indices = ds.list_indices()
	assert len(indices) == 2
	# There is overlap in fragment_ids, which is not allowed
	assert indices[0]["fragment_ids"] == {0}
	assert indices[1]["fragment_ids"] == {0, 1}

2. Should the two internal calls in dataset.py be migrated to describe_indices?

For these two locations, is it recommended to continue using list_indices (internally suppressing warnings) or to switch to describe_indices / stats.index_stats?

• has_index:
  

  lance/python/python/lance/dataset.py

  Lines 666 to 670 in a902892

  	@property
  	def has_index(self):
  	return len(self.list_indices()) > 0
  	def _apply_default_scan_options(self, builder: ScannerBuilder):

• _default_vector_index_for_column:
  

  lance/python/python/lance/dataset.py

  Lines 3861 to 3871 in a902892

  	Raises KeyError if no such index exists.
  	"""
  	for meta in self.list_indices():
  	if column in meta["fields"] and meta["type"].startswith("IVF"):
  	return meta["name"]
  	raise KeyError(f"No IVF index for column '{column}'")
  	def centroids(
  	self,
  	*,

3. Should these scenarios be changed to stats.index_stats, or kept as is?

Some cases don’t seem to work with describe_indices (missing index_details or vector index type is Unknown):

• Manual index commit:
  

  lance/python/python/tests/test_commit_index.py

  Lines 93 to 99 in a902892

  	# Verify that both datasets have the index
  	assert len(dataset_with_index.list_indices()) == 1
  	assert len(dataset_without_index.list_indices()) == 1
  	assert (
  	dataset_without_index.list_indices()[0] == dataset_with_index.list_indices()[0]
  	)

• Distributed FTS index:
  

  lance/python/python/tests/test_scalar_index.py

  Lines 2792 to 2810 in a902892

  	def test_build_distributed_fts_index_basic(tmp_path):
  	"""
  	Test basic functionality of build_distributed_fts_index helper function.
  	"""
  	# Generate test dataset with multiple fragments
  	ds = generate_multi_fragment_dataset(
  	tmp_path, num_fragments=3, rows_per_fragment=100
  	)
  	# Build distributed FTS index
  	distributed_ds = build_distributed_fts_index(
  	ds, "text", with_position=False, remove_stop_words=False
  	)
  	# Verify the index was created
  	indices = distributed_ds.list_indices()
  	assert len(indices) > 0, "No indices found after distributed index creation"

• Specific vector index types (IVF_PQ / IVF_HNSW_*):
  

  lance/python/python/tests/test_vector_index.py

  Lines 566 to 597 in a902892

  	def test_index_type(dataset, tmp_path):
  	ann_ds = lance.write_dataset(dataset.to_table(), tmp_path / "indexed.lance")
  	ann_ds = ann_ds.create_index(
  	"vector",
  	index_type="IVF_PQ",
  	num_partitions=4,
  	num_sub_vectors=16,
  	replace=True,
  	)
  	assert ann_ds.list_indices()[0]["type"] == "IVF_PQ"
  	ann_ds = ann_ds.create_index(
  	"vector",
  	index_type="IVF_HNSW_SQ",
  	num_partitions=4,
  	num_sub_vectors=16,
  	replace=True,
  	)
  	assert ann_ds.list_indices()[0]["type"] == "IVF_HNSW_SQ"
  	ann_ds = ann_ds.create_index(
  	"vector",
  	index_type="IVF_HNSW_PQ",
  	num_partitions=4,
  	num_sub_vectors=16,
  	replace=True,
  	)
  	assert ann_ds.list_indices()[0]["type"] == "IVF_HNSW_PQ"
  	def test_create_dot_index(dataset, tmp_path):

[fenfeng9]

@westonpace, could you take a look at these deprecate list_indices migration questions? Thanks!

[westonpace]

1. Should the old version of the datagen script retain list_indices?

Yes, it will need to keep list_indices. That should be fine.

2. Should the two internal calls in dataset.py be migrated to describe_indices?

Yes, ideally both of those methods can change to use describe_indices.

3. Should these scenarios be changed to stats.index_stats, or kept as is?

Ideally they should change. I think the specific vector index type is the one thing we can't report as part of a call to describe_indices today. In those cases we might need to keep list_indices for today.