Skip to content

refactor: use HfApi instead of Repository for hub sync#732

Open
Ashutosh0x wants to merge 1 commit intohuggingface:mainfrom
Ashutosh0x:refactor/use-hf-api
Open

refactor: use HfApi instead of Repository for hub sync#732
Ashutosh0x wants to merge 1 commit intohuggingface:mainfrom
Ashutosh0x:refactor/use-hf-api

Conversation

@Ashutosh0x
Copy link

Description: Refactor hub synchronization to use HfApi

This PR addresses #503 by refactoring the .github/hub/push_evaluations_to_hub.py script to use the modern huggingface_hub.HfApi instead of the legacy huggingface_hub.Repository class.

Key Changes

  • Removed Git Dependency: By switching to HfApi.upload_folder and HfApi.create_tag, we eliminate the need for a local git clone and the overhead of managing a local Git repository.
  • Improved Performance: Using HfApi is generally faster and more reliable than shell-based Git operations, especially in CI environments.
  • Cleaner Implementation: Replaced ~40 lines of subprocess calls and manual Git handling with a concise tempfile.TemporaryDirectory and HfApi call.
  • Simplified Dependency Management: The script now focuses purely on uploading the evaluation modules.

Benefits

  • No more "Could not clone" errors due to environment/shell quirks.
  • Reduced memory and disk footprint during CI execution.
  • Aligning with the current best practices of the huggingface_hub ecosystem.

Fixes #503

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move away from Repository to HfApi upload files

1 participant