Create Debian/Ubuntu package dataset

[nightlark]

Create a package dataset for Debian/Ubuntu that maps file names to the package(s) that could have installed the files. Relates to #5 and #8 for determining how file names should be normalized to use as a "key" for lookups. We should also consider how we may want to split up the dataset into smaller chunks based on how it will be used (e.g. only includes, only binary files, etc).

Some potential sources of data for this are:

• SQLite database from linux-package-analyzer
• Contents-amd64 gzipped file from http://security.ubuntu.com/ubuntu/dists/ for noble or oracular (sources also may have interesting info to add); Debian equivalent is https://ftp.debian.org/debian/dists/stable-updates/main/
  • The package name given isn't the best one (usually contains version information in its name) -- in general the source package name is a better option that is more recognizable; though there are exceptions like libstdc++-12-dev comes from the gcc-12 source package (a few special cases for some well-known libraries like that may be needed.

[nightlark]

Update/write script to output results as a sqlite database with an index generated on the file name for fast lookups.

[nightlark]

1/16 - regenerate dataset using normalized file name function for quick lookup file name key (ran into issue with a haskell file name normalization corner case).

[nightlark]

1/23 - Steven updated dataset using the updated/fixed normalization function (fixed in #52)

[nightlark]

2/13 - Steven generated sqlite database from Debian bookworm Contents file. About 25% of files in Debian don't appear Ubuntu -- not sure if it's Ubuntu lagging behind or choosing not to pull certain packages. Which of the packages those files belong to don't appear in Ubuntu?

• 3/6 - most "missing" packages seem to be dev related; maybe just lagging behind in synchronizing

Ubuntu has 4x the number of files in Debian.

[nightlark]

Once #6 is done, upload current Ubuntu dataset, consider also uploading Debian dataset.

[nightlark]

Name ubuntu/debian dataset releases with major version of distro -- ubuntu-24.04.db or ubuntu-noble.db

[nightlark]

@nightlark: create the repos for this on HF and Zenodo.

[nightlark]

Steven uploaded datasets to 3 debian (bookworm, bullseye, buster) and 3 ubuntu (jammy, focal, noble) versions to HuggingFace (https://huggingface.co/dapper-datasets).