question about sbo-bot

[rizitis]

Hello,
I prefer to open here a topic that I encountered from the emaιl list because my question mainly concerns sbo-bot.

In the last update one of the installed packages on my system was cinelerra which needed updating. Both the 2 package managers I use (sbopkg, slpkg) failed to download the source code and the update failed. The reason for the failure was that in the .info file the cinelerra url contains semicolon characters (;)
As we all know In Unix shells (bash, sh,
zsh), a semicolon means “end of command”.
So when the URL is pasted into a terminal without quotes, the shell
splits it into multiple commands and wget only receives the first part
of the URL: https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git
As a result, the server returns an HTML page instead of the tarball.
Browsers don’t have this issue because they don’t interpret ;
specially — they send the full URL as-is.

I thought it would be a good idea to inform the package maintainer and as a solution I proposed to escape the URL in the info file, so the shell passes
it correctly to wget:
https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git%5C;a=snapshot%5C;h=1914b32609fcead7518186b3069a3df2e07a9bc0%5C;sf=tgz%5C;fn=cinelerra-1914b32.tar.gz

Once escaped, wget sends the same request as the browser and the
download works normally.

The answer I received was that:

1. If package managers are not supporting valid .info files, it's
   their issue, I suggest filing a bug report with them.
2. Pasting an URL into a terminal without quotes is the most ridiculous
   idea I have ever heard. Why would you ever do that?
3. I am not adding work-arounds for sloppy garbage code that cannot escape
   its shell arguments. This is an insult to human conscience.

The second argument (2), in essence, finds me agreeing, because when someone does their work manually, then they will either click through a browser to download the source code and that works, or they will copy and paste the url through a terminal and through curl, wget they will download the file, so they have the option to use quotes, which is also unique.

The third (3) is his opinion and I respect it, if that's what he thinks.

For the first one (1), before I open a bug report in sbopkg, slpkg etc.. I would like an answer on how the specific package passed the sbo-bot check and possibly other similar ones?
When sbo-bot reads the .info file during the build, does it put the urls in quotes?
I say this because there is no relevant information in the README file, so the info file is indeed correct as is.

If sbo-bot actually puts urls in quotes that would be ideal and there should be a relevant announcement in mail list to those who develop package managers for SBo so that they can make their patches.

thanks.

[willysr]

For me, easier way is to put the tarball somewhere public and URL-friendly instead of having to deal with this naming scheme.
sbopkg doesn't put them in quotes as far as I know

[rizitis]

Basically it's not a sbopkg or slpkg etc issue, since even with quotes: wget "https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git;a=snapshot;h=1914b32609fcead7518186b3069a3df2e07a9bc0;sf=tgz;fn=cinelerra-1914b32.tar.gz"
the file is not downloaded with the real name but as index.html?p=goodguy%2Fcinelerra.git;a=snapshot;h=1914b32609fcead7518186b3069a3df2e07a9bc0;sf=tgz;fn=cinelerra-1914b32.tar.gz

sbo-bot works like browser agent so it can download it correctly,
imo cinelerra should have a notice in the README "download source from the browser", but maybe i m wrong.

For me, the case is closed. Feel free to close the issue when you decide.

---

BTW: I change sbopkg like this:

if [[ -z $NO_DL_LOOP ]]; then
               - wget $TWGETFLAGS ${DOWNLOAD[$i]} >> $SBOPKGOUTPUT &
               + wget $TWGETFLAGS -- "${DOWNLOAD[$i]}" >> $SBOPKGOUTPUT &
                echo "$!" >> $SBOPKG_PIDLIST 2>> $SBOPKGOUTPUT
                wait $!
                WGETRC=$?

And it looks it work without issues, this way it quotes urls
tested with coppwr which has more than 400 source packages to download in the .info file and with ocenaudio-bin which is also a problematic file to download, both passed.
So willysr I think you can add this improvement to sbopkg

[aclemons]

For the first one (1), before I open a bug report in sbopkg, slpkg etc.. I would like an answer on how the specific package passed the sbo-bot check and possibly other similar ones? When sbo-bot reads the .info file during the build, does it put the urls in quotes? I say this because there is no relevant information in the README file, so the info file is indeed correct as is.

sbo-bot is actually just a github webhook that calls Jenkins to build packages with slackrepo with some caching etc.

The download code for slackrepo is here:

https://github.com/aclemons/slackrepo/blob/98760827384d50228a4d14c4364e1e34652eda54/libexec/functions.d/srcfunctions.sh#L111-L234

To answer the question, yes it quotes the url - splits by whitespace to get each download link for the ones which have multiple, but each is quoted. It generally uses wget but some packages are forced to use curl with a so-called hint.

[pghvlaans]

Basically it's not a sbopkg or slpkg etc issue, since even with quotes ... the file is not downloaded with the real name

Well, not exactly. Using --content-disposition with wget as in the helpful link from @aclemons fixes that. The next version of sbotools will follow suit.

Granted, out of over 9000 scripts, only three of them appear to have download links that act this way.

[rizitis]

--content-disposition

First of all congratulations for the work you are doing on sbotools .
But my dilemma remains. So since there is no official package manager for SBo , what is the supposedly supported way for someone to build a package?

• Either clone the git repository locally and download the source code from the .info url
• or from the web interface of https://slackbuilds.org/

In the second case, user use the browser and after downloading the package, move it into the .SlackBuild folder.
In the first case, how can you know that the source code needs --content-disposition so that it can be downloaded correctly?
I want to say that any improvement made to package managers is good and welcome, but what does the typical SBo official require?

As you said, 3-4 packages are all that are needed --content-disposition, this is not a "serious" problem.
The point for me is: is an .info file with such a url that needs --content-disposition valid without a note in the README, yes or no?
Personally, if I could answer, I would say that for me, anything that passes the sbo-bot check should be considered valid automatically and it is in the maintainer's good will to mention it in the README.

[pghvlaans]

Thank you kindly.

In the first case, how can you know that the source code needs --content-disposition so that it can be downloaded correctly?

My best guess (implemented in another commit) is that if there's a question mark or semicolon in the link basename, it might be necessary. Using --content-disposition in other cases can change the file name unexpectedly. GraphicsMagick is one example. Maybe a sourceforge issue?

what is the supposedly supported way for someone to build a package?

The website (or at least a browser) is pretty much required for downloads with license agreements. It can't be ignored.

I'm not admin, but my personal opinion is that URLs should behave the same for browsers and CLI (there's a guide about GitHub source links and everything). Still, I also think it falls to the helper script to accommodate the repository as it is, so I don't mind making changes in that direction.