deltalake and delta-rs breaking changes in v1.0.0

[MABERG13-github]

When importing from catalog like glue or unity catalog dask-deltatable uses the deltalake package. This package just had a big overhaul with the v1.0.0 release. where functions like DeltaTable.from_data_catalog is removed see their documentation here. Also The DataCatalog class is removed.

They suggest moving to create path to the catalog which works quite well with existing setup by creating a

  full_url = f"s3://{database_name}/{table_name}"
  dt = DeltaTable(full_url)
  file_paths = dt.file_uris()
  ddf = dd.read_parquet(file_paths, **kwargs)

in the _read_from_catalog function.

[HiromuHota]

@jacobtomlinson are you planning to address this any time soon? If you haven't started, I can take a look at this this weekend. Lmk how I can help.

[jacobtomlinson]

@HiromuHota please go ahead! Just getting tests passing and CI green with the latest versions would be a great start.

[HiromuHota]

@jacobtomlinson I was able to adapt to all changes but the removal of DeltaTable.from_data_catalog.
Seems like this interface was removed in delta-io/delta-rs#1941 a long time ago. Does dask-deltatable still want to support this?
Other than that, I can clean up my code and put up a PR.

[HiromuHota]

I just saw the description of this issue about supporting catalog like glue or unity catalog, but I don't have access to any of these catalog.
Alright, I won't remove the support to read from catalog in my PR and would just leave it to you folks.

[HiromuHota]

@jacobtomlinson I'd appreciate your review #96!