DMX and CA_PID improvements - no kernel header change required anymore

[masjerang]

@Strunzdesign
I've been playing a bit more with descrambler and did some changes to it (ie. remove DMX and ca_pid things depending on change to kernel headers).
If you have some time, can you check and try: https://github.com/masjerang/descrambler
Thanks!

[Strunzdesign]

Challenge accepted ;-)

Thank you very much, I just read the mail with the notification. I'm going to perform a quick review (hoping that I understand what you did, but I'm curious) then give it a try on my box. I may have a problem as my "kernel headers" of the system are "vanilla" as they are separate from the kernel source tree that I modified for the dvblookback driver. Is this an issue?

Regards,
Florian

[masjerang]

Thanks! Good that we have a volunteer. :-) The idea is that you should be able to compile without modifying the vanilla kernel sources. Not sure if you easily ‘remove’ the patched kernel sources. This removes the need to patch sources outside descrambler.
Please let me know your results.

[Strunzdesign]

Hi,

okay, now I'm confused. You write that I do not need to modify the vanilla kernel sources. What I currently do is this: I do need to patch them to add the dvbloopback driver, a process that involves the copying a driver directory, two "sed" commands, and the application of five patches. Omitting one of these steps breaks the compilation of the kernel and the contained dvbloopback driver.

Ok. Given that I managed to compile a kernel together with the dvbloopback module and successfully rebooted into the system. I end up with:

• A patched / sed'ed folder containing kernel sources and compiled objects at /usr/src/linux/
• The never modified kernel headers provided by the OS residing at /usr/include/linux. They are not relevant for the dvblookback driver or the kernel itself, only for userspace applications (such as descambler?!). No patches were applied here.

Reading your last comment, do you suggest to revert the patches of the kernel sources after the compilation (restoring the vanilla kernel sources + make clean + make distclean)? What do you mean that there is no need to patch sources "outside descrambler"? Outside descrambler, do you refer to the kernel containing the dvbloopback-related additions? Please clarify...

Regards,
Florian

[masjerang]

Sorry, let me explain.
Let split into dvbloopback and descrambler.

dvbloopback
Normal build procedure, patching dvb-core and build dvbloopback, like you did before. No changes.

descrambler
There was a patch called dmx.patch which patches the kernel sources (dmx.h and ca.h). Those patches were required to get descrambler compiling with a recent kernel. Instead of doing this dmx.patch, I've adapted descrambler to skip using the DMX stuff and added the required ca_pid structs etc to a separate header file calles ca_pid.h and included in descrambler header files.
In order to compile the userspace binary descrambler, there is no need to patch any system source/header files anymore (for example, no root privileges required to build descrambler binary, but only need root to install).

Hope that makes more sense.
Thanks!

[Strunzdesign]

Thank you very much for that clarification. Now I also see my mistake, that in fact I do patch the linux-headers package on my system, with the dmx-patch you mentioned. I forgot about it because some time ago I placed it into a "magic" folder on my Gentoo box to have it applied automatically by the package manager during each install / update of the linux-headers package. To summarize, I have to disable this dmx patch, reinstall linux-headers to revert the patch effectively, then give your repo a try. Straightforward, no problem.

I'm going to test that tomorrow as I my box is busy right now.

Regards,
Florian

[bas-t]

@masjerang please do not do your development on the master branch.
You should use a temporary branch to do that. Of course such temp branch should be initialy be a clone of the master branch.
Should you need more info, let me know.
Regards, Tycho.

[masjerang]

@bas-t and @Strunzdesign
I've updated my fork and did how @bas-t recommened.
My changes are now in testing-branch.
Looking forward to testing.

[warpme]

Guys,
I'm upgrading my server and decided to give a test for dvblopback+descrambler.
OS is archlinux with 5.9.10 kernel.
I compiled 5.9.10 kernel with dvblopback.
it loads ok.
Issue is with descrambler :-(
Starting descrambler log:

Nov 26 12:34:33.952 INIT: Loading v1.0-10-gf30995a-Master
Nov 26 12:34:33.952 CAM: initializing FFdecsawrapper, A software emulated CAM
Nov 26 12:34:33.952 CAM(core.load): ** Key updates (AU) are enabled (active CAIDs) (no prestart)
Nov 26 12:34:33.952 CAM(core.load): ** Local systems DON'T take priority over cached remote
Nov 26 12:34:33.952 CAM(core.load): ** Force transfermode with digital audio
Nov 26 12:34:33.952 CAM(core.load): ** ECM cache is set to enabled
Nov 26 12:34:33.952 CAM(core.load): ** TsBufferSize is 32 MB
Nov 26 12:34:33.952 CAM(core.load): ** ScCaps are 1 2 0 0 0 0 0 0 0 0
Nov 26 12:34:33.952 CAM(general.info): loading overrides from /etc/sasc-ng/override.conf
Nov 26 12:34:33.952 CAM(core.load): loaded 0 overrides from /etc/sasc-ng/override.conf
Nov 26 12:34:33.952 CAM(general.info): loading smartcard data from /etc/sasc-ng/smartcard.conf
Nov 26 12:34:33.952 CAM(core.load): loaded 0 smartcard data from /etc/sasc-ng/smartcard.conf
Nov 26 12:34:33.952 CAM(general.info): loading cardslot config from /etc/sasc-ng/cardslot.conf
Nov 26 12:34:33.952 CAM(general.info): loading ecm cache from /etc/sasc-ng/ecm.cache
Nov 26 12:34:33.953 CAM(general.info): loading keys from /etc/sasc-ng/SoftCam.Key
Nov 26 12:34:33.953 CAM(core.load): loaded 0 keys from /etc/sasc-ng/SoftCam.Key
Nov 26 12:34:33.953 CAM(general.info): loading Viaccess cards from /etc/sasc-ng/Viaccess.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Viaccess cards from /etc/sasc-ng/Viaccess.KID
Nov 26 12:34:33.953 CAM(general.info): loading Seca cards from /etc/sasc-ng/Seca.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Seca cards from /etc/sasc-ng/Seca.KID
Nov 26 12:34:33.953 CAM(general.info): loading Irdeto cards from /etc/sasc-ng/Ird-Beta.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Irdeto cards from /etc/sasc-ng/Ird-Beta.KID
Nov 26 12:34:33.953 CAM(general.info): loading cardclient config from /etc/sasc-ng/cardclient.conf
Nov 26 12:34:33.953 CAM(cardclient.newcamd): now using protocol version 525 (cdLen=8)
Nov 26 12:34:33.953 CAM(cardclient.core): hostname=192.168.1.254 port=24000 emm=1 emmCaids 0100/ffff
Nov 26 12:34:33.953 CAM(cardclient.core): client 'Newcamd' ready
Nov 26 12:34:33.954 CAM(core.net): connecting to 192.168.1.254:24000/tcp (192.168.1.254)
Nov 26 12:34:33.954 THREAD: Netwatcher thread started (pid=574, tid=140013054121536)
Nov 26 12:34:33.957 CAM(cardclient.login): Newcamd: CaID=xxxx admin=1 srvUA=xxxxxxxxx provider 000000/0xxxxxxxxx
Nov 26 12:34:33.957 CAM(core.load): ** registered systems:
Nov 26 12:34:33.957 CAM(core.load): ** Viaccess          (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Seca              (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** SC-VideoGuard2    (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Viaccess       (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Seca           (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Nagra          (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Irdeto         (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Cryptoworks    (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Conax          (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** Fake-NDS          (pri -12)
Nov 26 12:34:33.957 CAM(core.load): ** Nagra2            (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Nagra             (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Irdeto2           (pri  -8)
Nov 26 12:34:33.957 CAM(core.load): ** Irdeto            (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Cryptoworks       (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** ConstCW           (pri -20)
Nov 26 12:34:33.957 CAM(core.load): ** Conax             (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Cardclient        (pri -15)
Nov 26 12:34:33.958 THREAD: SC housekeeper thread started (pid=574, tid=140013045728832)
Illegal instruction (core dumped)

Oops in kernel:

Nov 26 12:34:35 test systemd-coredump[578]: [LNK] Process 574 (ffdecsawrapper) of user 0 dumped core.

                                            Stack trace of thread 574:
                                            #0  0x000055e115bfe4bb _ZL18key_schedule_blockPhS_ (ffdecsawrapper + 0xca4bb)
                                            #1  0x000055e115bfed71 _ZL12schedule_keyP9csa_key_tPKh (ffdecsawrapper + 0xcad71)
                                            #2  0x000055e115c07411 _Z17set_control_wordsPvPKhS1_ (ffdecsawrapper + 0xd3411)
                                            #3  0x000055e115c0747a _Z14get_key_structv (ffdecsawrapper + 0xd347a)
                                            #4  0x000055e115beae90 connect_ffd (ffdecsawrapper + 0xb6e90)
                                            #5  0x000055e115b58e1e main (ffdecsawrapper + 0x24e1e)
                                            #6  0x00007f5755716152 __libc_start_main (libc.so.6 + 0x28152)
                                            #7  0x000055e115b5940e _start (ffdecsawrapper + 0x2540e)

                                            Stack trace of thread 575:
                                            #0  0x00007f57557b5c51 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xc7c51)
                                            #1  0x00007f57557bb137 __nanosleep (libc.so.6 + 0xcd137)
                                            #2  0x00007f57557bb06e sleep (libc.so.6 + 0xcd06e)
                                            #3  0x000055e115bd6bd8 _ZN11cNetWatcher6ActionEv (ffdecsawrapper + 0xa2bd8)
                                            #4  0x000055e115beee75 _ZN7cThread11StartThreadEPS_ (ffdecsawrapper + 0xbae75)
                                            #5  0x00007f5755bff3e9 start_thread (libpthread.so.0 + 0x93e9)
                                            #6  0x00007f57557ee293 __clone (libc.so.6 + 0x100293)

                                            Stack trace of thread 576:
                                            #0  0x00007f5755c059c8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf9c8)
                                            #1  0x000055e115bef00f _ZN9cCondWait4WaitEi (ffdecsawrapper + 0xbb00f)
                                            #2  0x000055e115bef080 _ZN9cCondWait7SleepMsEi (ffdecsawrapper + 0xbb080)
                                            #3  0x000055e115bbbf1a _ZN14cScHousekeeper6ActionEv (ffdecsawrapper + 0x87f1a)
                                            #4  0x000055e115beee75 _ZN7cThread11StartThreadEPS_ (ffdecsawrapper + 0xbae75)
                                            #5  0x00007f5755bff3e9 start_thread (libpthread.so.0 + 0x93e9)
                                            #6  0x00007f57557ee293 __clone (libc.so.6 + 0x100293)
Nov 26 12:34:35 test systemd[1]: systemd-coredump@9-577-0.service: Succeeded.

this is from code https://github.com/masjerang/descrambler/tree/testing (but I have also the same trap on bas-t descrambler code and also on old vdr code.)
After some investigations i discovered issue seems to be FFDECSA optimisation related cpu march selection.
FFdecsa_optimizer script checks is compiler supporting march=native - and if it is - then script suggest to go with march=native.
Issue is that this is compile time check - not run-time check and this will make run of descrambler failed when runtime cpu arch is different than compile-time cpu arch.
Ideal solution will be to implement few (most frequently meet) precompiled optimised FFDCSA binaries, benchmark them at descrambler start and select fastest one to use. This will require probably not small amount of work...
Simpler solution might be: ask at script start: "will descrambler also run on this compile machine?". If yes then we can go with march=native. If not we should go with more safe i.e. march=x86-64
Let me know what you think about above...

[masjerang]

Hi Piotr,

I'm using 5.8.0 kernel without any issues on descrambler and dvbloopback. I haven't tested on 5.9.x kernel.
I can try to look into it. Are you able to provide some additional logging?
I'm not an expert on CPU arch selection....