Contents-amd64.gz

[LucidOne]

I'm not sure if this issue belongs somewhere else, but I'm not sure where it goes.

I use apt-file regularly to figure out which package is responsible for a file, but it doesn't work with the ROS repository.

$ apt-file update
Downloading Index http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/Contents-amd64.diff/Index:
No Index available.
Downloading complete file http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 77.7M  100 77.7M    0     0  8572k      0  0:00:09  0:00:09 --:--:-- 8602k

.....

Ignoring source without Contents File:
  http://packages.ros.org/ros/ubuntu/dists/xenial/Contents-amd64.gz

Is it possible to generate this data?

[nuclearsandwich]

I moved this issue here from Bloom as the Contents index is generated by the repository tool not bloom. According to the reprepro man page "is quite slow, you have been warned". I'm not sure how slow is slow but I expect that it would be prohibitive for us to apply the config option to generate them globally as we'd have to rebuild it on every package import.

It might be something that we could add to just the main repository depending on how it affects the running time of a sync-to-main job which would be enough to get it on the main ROS repositories.

[LucidOne]

Even if the data was fairly stale I think it would be a huge help trying to figure out which package owns which file in /opt/ros. Right now I have to guess the package name and do a dpkg -L, which doesn't always work.

[gavanderhoorn]

I was about to post a new issue asking for this.

It would indeed be very nice if packages.ros.org could offer that file.

dpkg -S only works for installed packages, which essentially means it can't really be used to figure out which package you should install to make a certain file available on a specific host (as I'd have to have it installed already for dpkg -S to work, but I don't know what to install).

Additionally, it would make things like this possible/easier.

@nuclearsandwich: you mention you're hesitant to enable generation of Contents-*.gz files as it's apparently "quite slow". You would seem to be in the perfect position to figure out how slow exactly. I would do it myself, but I don't maintain packages.ros.org's reprepro configuration nor deployment.

Would you see a chance to test exactly what sort of overhead it would add?

[nuclearsandwich]

You would seem to be in the perfect position to figure out how slow exactly. I would do it myself, but I don't maintain packages.ros.org's reprepro configuration nor deployment.
Would you see a chance to test exactly what sort of overhead it would add?

Yeah, this is definitely a situation where pushing this onto the community support is a harder technical challenge. In all honesty, my maintenance backlog is pretty full up at present with other build farm efforts. I always try to use the time surrounding ROSCon to deep dive on community issues, maybe this year I'll spend that time getting Contents file generation working for the main repos at least.

We sync those repositories infrequently enough that as long as the runtime impact is in the minutes to seconds range for each platform there should be no serious blockers to rolling it out there. Whether it can be enabled on the testing repo will depend on how it works on main and honestly even if the impact is extremely minor we've got a lot of work going into reducing the time it takes to import single packages into the building repo and re-generating a Contents file on each import is probably out of the question. But with very few exceptions the building repository is not meant to be used directly except by buildfarm processes.

[gavanderhoorn]

I would personally not be interested in testing. So if that's not included, that would be fine by me.

[gavanderhoorn]

I've been looking, but I can't really find whether reprepro supports some sort of batched update for Contents-*.gz files, instead of updating per pkg import.

[nuclearsandwich]

I've been looking, but I can't really find whether reprepro supports some sort of batched update for Contents-*.gz files, instead of updating per pkg import.

I don't think reprepro itself distinguishes between individual package imports and our larger sync processes or has any batching capability at all. The ROS build farm operations are just wrappers around reprepro commands. I'm watching the architecture of createrepo-agent with a lot of excitement hoping that we can adapt the general framework for apt repositories either with reprepro or another low level repository tool which can really accelerate performance tailored to our workload.

removefilter, deleteunreferenced, and update for syncs.

removefilter and include for package imports.

The contents file generation is apparently configurable per distribution so it's something that we could readily control for just main, or main and testing but not building since each of those configs is generated by either the buildfarm deployment or ros_buildfarm itself via reprepro_updater.