Overhaul of the plugin maintenance/system

[JoeKar]

Description

This is a topic which spans over multiple repositories:

• plugin-channel
• updated-plugins
• micro
• all dedicated plugin repositories (present inside micro-editor)
• 3rd-party channels etc.

Current situation:
We have a lot of plugins which are currently "maintained" inside one single "large" updated-plugins repository, which are somehow old forks or the last "actual" state. On top of that we have plugin-channel which point to plugins inside updated-plugins or the upstream locations.
Both repositories have in common that they store the binary artifacts of every tracked version in one release...

• https://github.com/micro-editor/plugin-channel/releases/tag/plugins
• https://github.com/micro-editor/updated-plugins/releases/tag/v1.0.0
  ...so they are uploaded manually...for each plugin.
  The traceability is actually an absolute pain.
  Right now there is no authentication check available for any plugin.

channel.json points to...

• https://github.com/micro-editor/plugin-channel/tree/master/plugins/*.json
• https://github.com/micro-editor/updated-plugins/tree/master/**/*.json
  ...which point to "internal" as well as external resources.

Target situation:
Each plugin shall be tracked within one single repository, whether it's one of micro-editor or a third party.
Each version of each plugin must be tracked with its version number AND tag fitting together. Based on the tags GitHub will already create "snapshots" of the actual repository content, archive them to a zip and tar.gz, which then can be linked as artifact (actually no need to track further manually uploaded binaries/artifacts).
The *.json's pointing to the concrete files/versions should include (for future) revisions an entry for the hash of the file, to provide at least some sort of authenticity.
Unfortunately GitHub's tag snapshots don't generate a SHA256 on its own.

We need a plan where the master *.json is tracked and which location it shall point to.
I think we can agree, that the micro team can't maintain every possible plugin, but the question is, if it shall curate the plugins and point to fixed releases?

[JoeKar]

@Andriamanitra, @Neko-Box-Coder, @niten94:
I look into your direction, because you're heavily using and maintaining plugins and channels on your own. 😉

Community feedback is highly welcome here, because the actual plugin situation is a mess.

[niten94]

I'll state my perspective, however I believe I'm not significantly involved with plugins, and I don't have a definite idea how plugins should be maintained.

Both repositories have in common that they store the binary artifacts of every tracked version in one release...

The traceability is actually an absolute pain.
Right now there is no authentication check available for any plugin.

If reducing the amount of things that could be manually checked isn't enough, I think relying on GitHub workflows to upload archives would reduce the effort of verification to a good degree.

Each plugin shall be tracked within one single repository, whether it's one of micro-editor or a third party.
Each version of each plugin must be tracked with its version number AND tag fitting together.

If the plugins are small and irregularly updated, I find it more appropiate to maintain them together under one repository. A tag named like plugin/version (based on Go) could be made when one plugin updates, together with a GitHub release providing only the updated plugin archive.

The *.json's pointing to the concrete files/versions should include (for future) revisions an entry for the hash of the file, to provide at least some sort of authenticity.

I'm not sure what you mean by "concrete" but yes, it helps (mostly) in ensuring authenticity and additionally download integrity.

I think we can agree, that the micro team can't maintain every possible plugin, but the question is, if it shall curate the plugins and point to fixed releases?

I assume you're asking about the official plugin channel here, but I think we should continue to do so. It's likely done because the content pointed by URLs specified in JSON within plugin repositories could be replaced with a malicious version. I believe channel maintainers are more trusted to not conduct such.

Unfortunately, I can't come up with a better way and location to upload the archives provided through channels. @Neko-Box-Coder's stable channel also provides copies of plugin content and has similar problems.

[JoeKar]

The *.json's pointing to the concrete files/versions should include (for future) revisions an entry for the hash of the file, to provide at least some sort of authenticity.

I'm not sure what you mean by "concrete" [...]

I was referring to something like this: palettero.json#L8-L14

    {
      "Version": "0.0.5",
      "Url": "https://github.com/terokarvinen/palettero/archive/0.0.5.zip",
      "Require": {
        "micro": ">=2.0.0"
      }
    },

Inside we should place the hash for this particular release.

It's likely done because the content pointed by URLs specified in JSON within plugin repositories could be replaced with a malicious version.

Exactly, that was also my train of thought. We should at least maintain the revision lists (+ hashes) pointing to a specific archive, like above for the palettero plugin. What we most probably shouldn't do ist to link from our channel json directly to the repo.json provided by the plugin, because then we lose any control and were open for malicious plugins by whatever reason.

Based on the tags GitHub will already create "snapshots" of the actual repository content, archive them to a zip and tar.gz, which then can be linked as artifact (actually no need to track further manually uploaded binaries/artifacts).

[...] together with a GitHub release [...]

Seems like there is no way around the explicit GitHub release asset¹, as we already use it in micro, when we rely on stable hashes.

Footnotes

1. https://github.blog/open-source/git/update-on-the-future-stability-of-source-code-archives-and-hashes/ ↩

[Neko-Box-Coder]

Inside we should place the hash for this particular release.

Yes. I always wanted some sort of hash (which is tracked by git) that micro can verify.

We should at least maintain the revision lists (+ hashes) pointing to a specific archive, like above for the palettero plugin.

Unfortunately, I can't come up with a better way and location to upload the archives provided through channels.

I did think about putting the zip in the git tree itself (A different branch maybe?) for my unofficial channel, so that it can be tracked "officially" instead of just uploading to a release. Haven't done it yet because migrating all the plugins to git-tree is a bit of effort.

But at the moment, I am just doing what the offical channel doing, putting the version in the zip name itself.

What we most probably shouldn't do ist to link from our channel json directly to the repo.json provided by the plugin, because then we lose any control and were open for malicious plugins by whatever reason.

Absolutely. That's why I have 2 channels (link to upstream or link to channel release zips) for my unofficial channel where people can choose.

I think we can agree, that the micro team can't maintain every possible plugin, but the question is, if it shall curate the plugins and point to fixed releases?

I think we should be maintaining the ones that are in the updated-plugin repo since most of the upstreams in there are stale and not maintained.

As for ones that are not in there, i.e. 3rd party & future plugins, they should be maintained by the maintainer of the plugins, and updates should be submitted by them.

For my unofficial channel, I check the upstreams and update my stable releases from time to time but I think that is too much effort for you guys, not to mention not scalable as well.

[Edit]:
Might have misread what @JoeKar meant by "curate". If you are talking about storing all the plugins releases/zips in the repo, then yes I think we should be doing that.

Since the upstream could be removed or changed. Introducing a hash might help to detect change but then we might have trouble doing that with older/existing plugins, and also might end-up having a mishmash of half of the plugins are stored here and half of them are stored externally.

---

I think the "easy" first step(s) for now would be

1. Merging the updated-plugins and plugin-channel into one, instead of having to manage 2.
2. Remove any external link from the official plugin to point zips from internal release.

Then maybe we can continue from there and think about where to put the zips?

---

[Edit]

_{I was also planning to add the official plugins to my channel, gonna link it to this issue as well since the origin of the plugins may change depending on the outcome of this convo}

_{Neko-Box-Coder/unofficial-plugin-channel#19}

[Andriamanitra]

I don't use the built-in plugin handling much because managing plugins directly with git is much more convenient. I usually want to make some changes (and track them), and if I install a plugin with micro -plugin install I won't get the git history.

I think we can agree, that the micro team can't maintain every possible plugin, but the question is, if it shall curate the plugins and point to fixed releases?

Yes, I think that's important to make plugins discoverable for the casual user. It's nice that they're listed on the github.io page. The plugin-channel repo will need a major update to bring the list up to date and introduce the checksum field (ideally in a way that still allows old versions of micro to use the same repo.json). Some automation to keep track of upstream releases to make sure it also stays up to date would also be worthwhile.

Each plugin shall be tracked within one single repository, whether it's one of micro-editor or a third party.
Each version of each plugin must be tracked with its version number AND tag fitting together.

If the plugins are small and irregularly updated, I find it more appropiate to maintain them together under one repository. A tag named like plugin/version (based on Go) could be made when one plugin updates, together with a GitHub release providing only the updated plugin archive.

I don't like the idea of having multiple plugins in the same repo. It makes it harder to find the plugin you're looking for, and to see if/how they're different from the upstream.

I would rather start by archiving the micro-editor/updated-plugins repo and moving the plugins that are still relevant to their own repos (ones that Github sees as forks of their upstreams). In some cases (eg. manipulator-plugin) we don't even need a fork any more because the upstream has been updated to work with 2.0. The updated-plugins repo has not been updated in a while and the way it's currently set up none of the plugins track their upstreams so it's difficult to even tell when it is out of date.

[JoeKar]

I don't like the idea of having multiple plugins in the same repo. It makes it harder to find the plugin you're looking for, and to see if/how they're different from the upstream.

[...] moving the plugins that are still relevant to their own repos (ones that Github sees as forks of their upstreams).

I agree here. 👍

I did think about putting the zip in the git tree itself (A different branch maybe?) for my unofficial channel, [...]

I'm no friend of tracking binaries within git (git-lfs is a different story).

Then maybe we can continue from there and think about where to put the zips?

Right now I tend to use GH release action...for plugins we maintain, as we do for micro, to create and upload the artifacts to a tag/release. Sure, just for the new/latest ones. The old locations remain active and will not receive any hash. So micro needs to validate as opt-in.