Align repo, crawlers, and schemas for packages and libraries

[braver]

I don't know enough details to fully describe the discussion. The meat of it though, as I understand it right now, is ...

• packages are still on the v3 schema because the old website doesn't know any better (and cannot be updated)
• libraries are however already on v4.
• thecrawl crawls packages, https://github.com/packagecontrol/channel crawls libraries (?)
• there is crawler-like tech in the PC client package to resolve package

Perhaps more. But there is some consolidation that can happen here, especially once the old site is no longer needed.

[deathaxe]

The primary change between v3 and v4 is new wording for "dependencies", which are named "libraries". This wording applies to all aspects of those JSON files.

Another change is switching from SemVer to PEP440 versions, which was required to support all official python packages. To the extend versioning is used, PEP440 actually defines better and more conistent semantics for pre-releases. Ignore its oddities with local releases etc.

For packages v3 vs. v4 schema is probably less important as intented new keys, like python_version to override local .python-version files for unmaintained packages becomes more or less obsolete with new ST releases and dependencies vs. libraries keys within releases to specify dependencies upstream has never been practically used, yet being supported by PC3 and 4.

The important part for libraries is python_version being used to specify dedicated release branches for each supported python version. load_order key is no longer needed (and therefore disallowed by PC client).

Even though the long term goal may be to teach PC to resolve libraries on client-side, libraries have to be resolved from GH, BB, GL, CB and pypi for compatibility reasons with existing PC4 clients AND because there are a couple of ST-only python packages, such as mdpopups, sublime_aio, sublime_lib, lsp_utils, which are unlikely to be deployed via pypi.

To get rid of external libraries channel, thecrawl must learn to crawl them on its own.

Yes, PC has its own crawler logic as it is basically used for both, the client and server-side (packagecontrol.io) functionality. The code however is primarily python 3.3 and if it is to be continued to support all ST4 builds it can be lifted to py3.8 only. thecrawl uses py3.10+ code and thus can't be used for the client in its current form.

The primary question however is how much "crawling" logic the client needs. Most likely a couple of users will complain about any feature drop. Being able to specify a git repo as repository without a dedicated repository.json is a neat niche feature, despite suffering from its own edge cases.

It may be needed for version pinning, if PC needs to pull available versions of specified installed packages.

I never wanted to maintain and manage 2 code bases to handle all the logic. Despite a bit hard to keep track of, I always found the way package_control is used on both ends a very affective and efficiant solution, ensuring consistency between server and client part. Appears those days are over. At least it is not useful to bundle all required dependencies for aiohttp or httpx to PC client. That's just too much overhead and would require platform-specific package releases.