Implement Lance columnar storage for inference results

[Copilot]

Change Description

Replaces batched .npy file storage with Lance columnar format for inference results (infer, test, umap, engine verbs). New writes default to Lance for optimal performance; existing .npy files remain readable via auto-detection.

Solution Description

New classes (src/hyrax/data_sets/result_dataset.py):

• ResultDatasetWriter: Includes tensor metadata in PyArrow schema and validation for tensor shape/dtype consistency across batches.
• ResultDataset: PyTorch Dataset + HyraxQL getters (get_data, get_object_id)

Factory functions (src/hyrax/data_sets/result_factories.py):

• create_results_writer(): Returns Lance writer
• load_results_dataset(): Auto-detects Lance vs .npy on disk, returns appropriate reader

Shared utilities (src/hyrax/config_utils.py):

• resolve_results_dir(): Centralized results directory resolution logic, now used by both InferenceDataSet and result_factories.py (eliminates code duplication)

Updated verbs:

• infer, test, umap, engine: Use factories, call writer.commit() instead of write_index()
• lookup, save_to_database: Updated to use load_results_dataset() for format-agnostic reading
• pytorch_ignite.create_save_batch_callback(): Uses factory
• InferenceDataSetWriter: Added commit() method for API compatibility

Dependencies: Added lancedb, pyarrow, pylance to pyproject.toml

Example usage:

# Writing (automatic in verbs, uses async internally)
writer = create_results_writer(dataset, results_dir)
writer.write_batch(object_ids, tensors)
writer.commit()  # optimization disabled by default for performance

# Reading (auto-detects format)
results = load_results_dataset(config, results_dir)
data = results[0]  # Returns numpy array
obj_id = results.get_object_id(0)  # HyraxQL getter

Testing: 16 unit tests covering batch writes, multi-dim tensors, dtypes, edge cases (NaN/Inf), indexing, iteration, cross-platform compatibility, async/sync modes.

Documentation: Updated lance_design_hyrax.md to reflect actual implementation (Lance-only writes, auto-detection on read, correct getter names).

Future work: Migration script for .npy → Lance conversion, visualize verb support, deprecation of InferenceDataSet.

Code Quality

• I have read the Contribution Guide and agree to the Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

Original prompt

Start implementation

The user has attached the following file paths as relevant context:

• .github/copilot-instructions.md

[Chronological Review: The conversation began with the user asking about the differences between `lance.dataset` and the combination of `lance.connect` and `.open_table`. This was followed by inquiries regarding the support of index types in `table.take_offsets` and the return type of `HyraxDataset`. The user then requested a plan for implementing a design document in a tractable and testable manner, and finally initiated the implementation phase.]

[Intent Mapping:

1. "Is there a difference between using lance.dataset and the combination of lance.connect and .open_table? Is one preferred?" - User seeks clarification on the usage of Lance library functions.
2. "Does table.take_offsets support all of these index types?" - User wants to confirm compatibility of index types with HyraxDataset.
3. "What about returning Torch.tensor? Is that promised by HyraxDataset?" - User is questioning the return type consistency of HyraxDataset.
4. "Take a look at this design document and develop a plan for implementing it..." - User requests a structured implementation plan for a design document.
5. "Start implementation" - User indicates readiness to begin the implementation process.]

[Technical Inventory:

• lance.dataset: A function for accessing datasets in the Lance library.
• lance.connect: A function to establish a connection to a Lance database.
• table.take_offsets: A method for retrieving offsets from a table, with concerns about index type support.
• HyraxDataset: A dataset class that the user is evaluating for compatibility with Torch tensors.]

[Code Archaeology:

• No specific code files or functions were modified in this conversation, but the user is working within the context of /Users/derekjones/code/work/hyrax/specs/lance_db_spec.md. The focus is on understanding and implementing features related to the Lance library and HyraxDataset.]

[Progress Assessment:

• Completed Tasks: User has clarified several technical aspects regarding the Lance library and HyraxDataset.
• Partially Complete Work: The user has requested a plan for implementation and has initiated the implementation phase, but specific tasks within that phase are not detailed yet.]

[Context Validation: All necessary context for continuing the work on the Lance library and HyraxDataset is captured, including user inquiries and the current file being worked on.]

[Recent Commands Analysis:

• Last Agent Commands: The user executed commands related to understanding the Lance library and initiating implementation.
• Tool Results Summary: No specific tool results were returned as the conversation focused on inquiries and planning rather than executing code.
• Pre-Summary State: The agent was actively working on the implementation of a design document in the context of the Lance library and HyraxDataset when the token budget was exceeded.
• Operation Context: The user’s commands were aimed at clarifying technical details and preparing for implementation, aligning with their goal of developing a structured approach to the design document.]

1. Conversation Overview: - Primary Objectives: 1. "Is there a difference between using `lance.dataset` and the combination of `lance.connect` and `.open_table`? Is one preferred?" 2. "Does `table.take_offsets` support all of these index types?" 3. "What about returning Torch.tensor? Is that promised by HyraxDataset?" 4. "Take a look at this design document and develop a plan for implementing it..." 5. "Start implementation" - Session Context: The conversation has revolved around understanding the Lance library's functionalities and preparing for the implementation of a design document related to HyraxDataset. - User Intent Evolution: The user transitioned from seeking clarifications on library functions to requesting a structured implementation plan and finally initiating the implementation phase.

2. Technical Foundation:

• lance.dataset: Function for accessing datasets in the Lance library.
• lance.connect: Function to establish a connection to a Lance database.
• table.take_offsets: Method for retrieving offsets from a table, with concerns about index type support.
• HyraxDataset: Dataset class under evaluation for compatibility with Torch tensors.

3. Codebase Status:

• File Name: /Users/derekjones/code/work/hyrax/specs/lance_db_spec.md
• Purpose: This file is important for documenting tests and specifications related to the Lance library and HyraxDataset.
• Current State: The user is in the process of implementing a design document.
• Key Code Segments: Not specified in the conversation.
• Dependencies: Related to the Lance library and HyraxDataset functionalities.

4. Problem Resolution:

• Issues Encountered: User is uncertain about the compatibility of index types and return types in the context of Hyr...

Created from VS Code.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[github-actions]

Before [0aebbf0]	After [c36d109]	Ratio	Benchmark (Parameter)
38.5±0.09s	54.7±0.2s	~1.42	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(16384, 'qdrant')
457±6ms	3.41±0.01s	7.46	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(64, 'chromadb')
560±1ms	3.69±0s	6.59	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(256, 'chromadb')
1.53±0s	6.11±0.01s	4.00	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(2048, 'chromadb')
9.60±0.06s	25.7±0.1s	2.67	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(16384, 'chromadb')
2.18±0.02s	5.01±0s	2.30	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(64, 'qdrant')
2.87±0.03s	5.94±0s	2.07	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(256, 'qdrant')
6.86±0.01s	11.4±0s	1.66	vector_db_benchmarks.VectorDBInsertBenchmarks.time_load_vector_db(2048, 'qdrant')
1.27G	1.47G	1.16	vector_db_benchmarks.VectorDBInsertBenchmarks.peakmem_load_vector_db(2048, 'qdrant')
1.03G	1.19G	1.15	vector_db_benchmarks.VectorDBInsertBenchmarks.peakmem_load_vector_db(2048, 'chromadb')

Click here to view all benchmarks.

[codecov]

Codecov Report

❌ Patch coverage is 88.54167% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.17%. Comparing base (0aebbf0) to head (f4fe719).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/hyrax/config_utils.py	63.15%	7 Missing ⚠️
src/hyrax/verbs/lookup.py	0.00%	5 Missing ⚠️
src/hyrax/data_sets/result_dataset.py	96.29%	4 Missing ⚠️
src/hyrax/verbs/engine.py	0.00%	3 Missing ⚠️
src/hyrax/data_sets/result_factories.py	90.00%	2 Missing ⚠️
src/hyrax/data_sets/inference_dataset.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #699      +/-   ##
==========================================
+ Coverage   63.25%   64.17%   +0.91%     
==========================================
  Files          59       61       +2     
  Lines        5770     5892     +122     
==========================================
+ Hits         3650     3781     +131     
+ Misses       2120     2111       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[drewoldag]

Overall this is looking good to me, I'm ok clicking the approve button. I'll plan on working with it a bit tomorrow morning.

[gitosaurus]

@copilot The benchmark performance regressions appear to arise from .infer, which suggest that Lance is not fast enough in the current implementation. Use the recommended async/await approach for best speed.

[Copilot]

@copilot The benchmark performance regressions appear to arise from .infer, which suggest that Lance is not fast enough in the current implementation. Use the recommended async/await approach for b...

Implemented async/await support in 5842f93.

The ResultDatasetWriter now uses LanceDB's async operations (connect_async and AsyncTable.add) internally while maintaining a synchronous API for backward compatibility via asyncio.run().

Key changes:

• Added use_async parameter (default True) to enable async Lance operations
• Removed automatic table optimization from commit() - now opt-in via optimize parameter
• Per Lance documentation, optimization is expensive and typically only needed after 100k+ records or 20+ write operations

All 16 unit tests pass with the new async implementation.