Track file extensions for syntax definitions

[michaelblyons]

The rewrite opens up the possibility of a feature I'd been kicking around in my head for a couple years, though it might be too ambitious: I think it would be interesting and useful to index the file extensions that a package claims to provide syntax for.

Benefits

• PC can suggest packages from inside ST for un-highlighted (or wrongly, badly, etc. highlighted) files.
• It's another view into the package database, potentially exposing some additional utility for packages.
• The first time we do this, we may discover a bunch of packages that should have the language syntax tag, but don't.

Drawbacks

• Requires downloading, parsing, and saving results for packages. This can use a lot of API rate-limit.
  • Mitigation: Maybe running only on language syntax packages with new updates will drop the API / CPU requirements sufficiently. A one-time run of every package to fill in missing tags is probably still needed.

Prior Art

• This feature does not exist in the current ST PC.
• VS Code's marketplace does track file extensions that could be highlighted. They don't need to do all the parsing, though, because the manifest file for a package tracks the extensions that it has syntax for. (Then Microsoft somewhat aggressively pushes those packages on people, which we would not want to do.)

Discussion

Taken from Discord

@kaste: I never had problems just searching for the actual "language" and then each of them should have the language syntax label. I often rather know that name (e.g. Clojure) before its file extension.

@michaelblyons:

• I don't actually trust the language syntax tag. Some packages have syntax definitions but not the tag.
  • For some, we could one-time scan and add the tag to the appropriate Channel repo.
  • But for others, arguably the language is kind of incidental to something else that the package is doing. I don't have strong opinions that we should add a tag if that is the case.
• There's also the case where it's not immediately apparent what the file extension is for. Granted, you could $favorite_search_engine it and probably comes up with one of those ad-stuffed file-extensions.info type sites that gives you a good guess.
• And lastly, there's the case where the name of the package might not signal well enough how much it does.
  1. PackageDev: If you find a *.tmLanguage file, you don't know to search for a Sublime configuration editing package because the internet tells you it's a TextMate file.
  2. SSH Config has minimal syntaxes thrown in for known hosts, authorized keys, and a couple other things that don't suggest "Go look for SSH packages" to a user.

Disclaimer

I'm not going to be upset if the amount of work outweighs the benefit.

[kaste]

In fact Package Control - the client moves a very slow pace right now. And within Sublime Text this would be most useful.

What is possible: Prepare a Sublime Text package, that exposes one command "Show suggested packages for this view.". You make a simple script/crawler that running on GitHub fetches the syntax files, reads them and writes a map "ext -> list[package_name]". You publish that on a release (like we do here in this repo). The seed could be the workspace file from here, which has the details key and the last_modified key per package. (details being the Hub URL.)

"Your" package / the add-on fetches "your" map, and then call PCs window.run_command("install_packages", {"packages": []})

We could also collate this additional data and feed it to our search index, so that ".py" matches some Python syntax package.

[michaelblyons]

Prepare a Sublime Text package, that exposes one command "Show suggested packages for this view.". You make a simple script/crawler that running on GitHub fetches the syntax files, reads them and writes a map "ext -> list[package_name]".

This is a good idea.

Current status

For anyone who hasn't been tracking this:

• @TerminalFi made an offline crawler to scan the first 3 folder levels for *.sublime-syntax or *.tmLanguage files. Out of the 5k+ packages, 1334 have at least one syntax file.
• I filtered that output to find 613 packages with a syntax which don't have the language syntax label. See this spreadsheet which is doing some filtering to show items that are not yet sorted or resolved.
• And by "resolved," I mean that I have Language syntax tag (A-Z) sublimehq/package_control_channel#9150 opened to tag packages whose syntax is not merely for its own build output or other transient data.

Future work

• Sorting the last 50-60 packages whose language-syntax is somewhat arguable.
• Once Language syntax tag (A-Z) sublimehq/package_control_channel#9150 is merged, Terminal's GraphQL or any future crawler can skip packages without the language syntax label.
• Setting up a crawler to actually download the syntax files and collate their file extensions.
• Write a package to ask for PC install options for the current file, as @kaste suggests.

Observations

When I was doing the sorting of packages missing tags, I noticed these things.

• There are a significant number of "Lang X support was abandoned, so I made a new one" packages.

• There are a significant number of "Here is Lang X, but with extra platform keywords" that could be an inherited syntax. Especially Ruby, JavaScript, maybe PHP.

• … the case where the name of the package might not signal well enough how much it does.

  holds true for several other packages, too. Syntax for files not mentioned in the package name or blurb. Sometimes not even in the ReadMe.

[michaelblyons]

make a simple script/crawler that running on GitHub fetches the syntax files, reads them and writes a map "ext -> list[package_name]".

I have a local one of these, now. It's somewhat naive. Outputs 10k lines or so.

• It turns out that a lot of packages' syntax files are not strictly valid PList or YAML. I am currently skipping them.
  • Edit: With some pre-processing, I've gotten it down to 29 15 3 failing syntax files.
• I discovered https://github.com/hronro/sublime-linguist-syntax with the biggest pile of tmLanguages I've ever seen.
• Themes and icon packs have a bunch of dummy syntaxes to make the sidebar look nice.