acces ROM waveforms when installed on E2 synth ?

[chaocrator]

would it be possible at some point to acces ROM waveforms on E2 synth ?

at least drums/percussions in E2 synth are perfectly usable.

also, E2 synth owners have no option to export factory samples, so after Hacktribe installation we get only VA/FM oscillators, and that's all. we have to ask for factory waveforms (from E2S) elsewhere if we need them.

[bangcorrupt]

Yes, this will be possible in the future. When the PCM data is loaded from the flash, it's decompressed and sent to DSP RAM one word at a time. I can hijack this to copy PCM to CPU RAM, then dump it over sysex as a single blob. If I can get a debugger attached to the DSP then I may be able to fix the issue with sample saving and have the factory samples loading like normal.

If you're short of samples until then, check out the carved beats set by @BKLronin.

[maks]

What a great collection of samples! Thanks for making them available @BKLronin !

[BKLronin]

Thank you very much. They were exclusively recorded and synthesised for hacktribe.

[maks]

@bangcorrupt I've just reread this and thought about the implications: would this mean that its in theory possible that you would be able to in the future modify the firmware to send/receive sample data via Midi sysex?

[alexreisinger]

Hey I would like to bump this as I am stuck on this exact step.
I patched the firmware and got everything running on my E2 synth.
Connected the usb cable and checked mido.get_input_names() and mido.get_output_names()
and they return 'electribe2 sampler 0' for the input and 'electribe2 sampler 1' for the output so I adapted the script to use those resepctively:

e2sysex.py:

def __init__(self, input_port='electribe2 sampler 0', output_port='electribe2 sampler 1', midi=None):
        logging.debug('Initialise SysEx')
        
        if midi is not None:
            self.midi = midi
            self.inport = self.midi.inport
            self.outport = self.midi.outport
        else:
            self.inport = mido.open_input(input_port)
            self.outport = mido.open_output(output_port)

when running the script I am stuck at the same spot as @F-Thom (maybe you got more progress by now?).
If I leave the ports standard I also get the error of the Ports not being found.

output of the script:
PS C:\Users\alexr\hacktribe> python .\scripts\dump_extracted_pcm.py
DEBUG:root:Initialise SysEx
INFO:root:Found electribe
INFO:root:Global channel 0
INFO:root:Firmware version 2.2
DEBUG:root:Getting decompressed PCM from CPU RAM
DEBUG:root:Called read_cpu_ram
DEBUG:root:Sending message

That is where everything blocks. I cant seem to get a response from the other SysEx functions you mentioned, the e.get_current_pattern() and so on - it just runs indefinitley.

Using Windows 10 and VSCode for that matter. Hopefully you can help and I am happy to provide any further information if necessary! Thanks for the effort in the whole project, looking forward to testing it with the PCM Samples.

[bangcorrupt]

I don't know why this isn't working for people. I have just tested and it worked for me, taking a little over a minute and giving me a 16MB wav file of the PCM data. I did get a 'Resource temporarily unavailable' message, possibly due to flooding the input with a large sysex message. Maybe this is handled differently on Windows and causes an error. I will try to test with Windows, but it might not be soon.

[user@host hacktribe] $ time python scripts/dump_extracted_pcm.py
DEBUG:root:Initialise SysEx
INFO:root:Found electribe
INFO:root:Global channel 0
INFO:root:Firmware version 2.2
DEBUG:root:Getting decompressed PCM from CPU RAM
DEBUG:root:Called read_cpu_ram
DEBUG:root:Sending message

MidiInAlsa::alsaMidiHandler: unknown MIDI input error!
System reports: Resource temporarily unavailable
DEBUG:root:Received response
DEBUG:root:Decoding binary
DEBUG:root:Writing pcm-dump.wav

real    1m9.706s
user    0m10.973s
sys     0m0.904s

[bangcorrupt]

Can you check you have 'python-rtmidi' installed, not 'rtmidi-python' (these are different packages)?

Could you also check which backend mido defaults to on your system?

In a python console:

>>> import mido
>>> mido.backend
<backend mido.backends.rtmidi (not loaded)>

[F-Thom]

I don't know why this isn't working for people. I have just tested and it worked for me, taking a little over a minute and giving me a 16MB wav file of the PCM data. I did get a 'Resource temporarily unavailable' message, possibly due to flooding the input with a large sysex message. Maybe this is handled differently on Windows and causes an error. I will try to test with Windows, but it might not be soon.

[user@host hacktribe] $ time python scripts/dump_extracted_pcm.py
DEBUG:root:Initialise SysEx
INFO:root:Found electribe
INFO:root:Global channel 0
INFO:root:Firmware version 2.2
DEBUG:root:Getting decompressed PCM from CPU RAM
DEBUG:root:Called read_cpu_ram
DEBUG:root:Sending message

MidiInAlsa::alsaMidiHandler: unknown MIDI input error!
System reports: Resource temporarily unavailable
DEBUG:root:Received response
DEBUG:root:Decoding binary
DEBUG:root:Writing pcm-dump.wav

real    1m9.706s
user    0m10.973s
sys     0m0.904s

First of all, thank you very much for coming back to us and taking time to check this topic again.
I didn't check further after my last message and I can try when I'll have time again. I'm not in a hurry. I can report whenever I've some news.

[alexreisinger]

Ok I finally got time to check on that again.
I got it to work, I now have a 16MB wav file.
On Windows and on my Linux VM inside Windows I always got stuck at: DEBUG:root:Sending message, but now on an M3 MacBook Air I got it to work flawlessly.
I just had to adapt the Input and Output ports in the e2sysex.py to
input_port='electribe2 sampler PAD/KNOB'
output_port='electribe2 sampler SOUND'

The script however does not display the time stats that your log stated, it just stops at Writing pcm-dump.wav.

The result of the mido.backend is:

import mido
mido.backend
<backend mido.backends.rtmidi (loaded)>

Thanks again for your work, and I will now continue the workflow. If anyone has any questions maybe I can help a bit on that part!

[drmindflip]

I joined up here to ask this! I'm an E2 Synth user and I mostly use it for making ambient sequences and textures - I literally never use the 90% of the stock sounds (OSC types 001- 325 never get a look in) but I do get a lot of use out of 326-428 (the Synth and Instrument categories)

My ideal would be to keep those, trash the rest, get those lovely new synth oscs that you've added, and maybe eventually add a few samples. Is this ever likely to be possible?

Thanks for all your inspiring work

[AndrewSosna]

Thanks, sent you an email

[bangcorrupt]

The firmware you sent me matches the one I tested and when I run it I get the samples dumped correctly. The audio files you sent are the correct length but look like unitialised RAM, matching the data in the raw binary files. This is what I would expect to see if the script was run against the main Hacktribe branch, or if the read address is somehow getting mangled during transmission.

Can you confirm that you only have one Hacktribe connected via USB when you run the script?
Are you able to test using a Linux system? Maybe mido interacts with the MIDI drivers differently on a Mac.

[AndrewSosna]

As it appeared, my SD card already contained a folder for Sampler version with e2sample.all so E2 never decompressed PCMs to memory and I was getting empty memory dump. I removed the SD card before dumping PCM samples and everything went fine, I got the proper WAV. Thank you very much.

Two following questions:

1. Is there some document about the structure of this WAV, to make it easier for me to split it automatically, or loading it to Audacity or something similar and just splitting it by silence is the way to go?
2. Now I'm curious, is there a way do the reverse? To update this WAV by replacing samples and pushing them back to E2 flash, basically getting a regular E2 synth version with updated samples?

[bangcorrupt]

split it automatically

I'm working on format definitions similar to those in e2_formats.py, along with a script to read the full PCM section from flash. These will allow parsing the binary to a dictionary, including a list of offsets for each sample. I think there is at least few bytes of zeroes between each sample, so splitting by silence may work, but if you can wait a while I should be able to script it.

replacing samples and pushing them back to E2 flash

I don't know the compression algorithm used, but reversing it could be possible. If I can't get the remaining features from the synth working in Hacktribe I might look at loading custom samples to the synth. In the long run I want to move all the content to the SD card in a single project file, maybe using the flash for alternate firmware apps.

Thank you again for your continued support 😃 !

[alexreisinger]

Hello i would like to ask if there is more information on the structure of the wav or a script that splits the audio into the individual samples already? I have loaded it into Audacity and tried to label the samples by silence but that did not work well, there seem to be a minimum of a few audio points (samples) between those samples that i see.
Thanks for the dedication again!

[Ghillermo]

Hi, iirc the old "turn synth into sampler" header change hack allowed us to get the samples too. What happens if I run the python patch script but just use it to change header on src file (without applying the hack patch) would commenting a few lines of code (and calculating the new hash) do the trick or will it brick?

If we can do this then we'd just export samples to SD, move them to 501 an on and use them.

[bangcorrupt]

It will prompt to install the synth firmware. Also, sampler firmware won't read the multi-sampled synth PCM correctly.

[AndrewSosna]

@bangcorrupt , Could you please tell more about multi-sampling support on E2/E2s?
As I understand the Sampler doesn't support multi-sampling but the Synth does?
Looks like not all but only some PCM samples are multi-sampled on Synth. So, theoretically, what would be the best way to get multi-sampling, I guess the only way is to replace PCM samples on Synth, right? And are there, like, "hardcoded multisample-dedicated PCM slots" on Synth, so that we would be only able to replace those, or theoretically all the "slots" (or does it even work this way, are there "slots" or just a memory area?) could contain multi-sampled samples?

[bangcorrupt]

They both support mullti-sampling but the sampler doesn't implement it fully. The sampler flash PCM section is the same format as the synth, but every sample is in a group of one. On playback the sampler reads and uses the empty key and velocity group data, I was able to map samples by editing values in CPU RAM. I didn't look into it much further, but I think when the sampler loads PCM from flash it assumes each sample is its own group which leads to bugs with instrument indexing and slice data saving. I think some day we should at least be able to load a map file into RAM after booting, hopefully it can be integrated into the existing sample loading.

[AndrewSosna]

Yes, I only have one E2… I don’t have Linux, but I could install it on a virtual machine… or on WindowsAlso I could try with SysEx Librarian, it’s an app that sends SysEx from files and records the response. I was able to send “dump pattern” with it, but i would need a SysEx file. I can also try with Apple native API CoreAudio, I don’t have experience with it but it would be interesting to try. Still I’ll need to read your code and replicate it with Swift and form SysEx messages same way as you do (first time I’m dealing with this stuff)…Before I try the options above, are there any specific requirements when dumping PCM? I mean, maybe i should have removed the SE card before switching E2 on?Sent from a mobile deviceOn 2 Jan 2023, at 17:52, bangcorrupt ***@***.***> wrote: The firmware you sent me matches the one I tested and when I run it I get the samples dumped correctly. The audio files you sent are the correct length but look like unitialised RAM, matching the data in the raw binary files. This is what I would expect to see if the script was run against the main Hacktribe branch, or if the read address is somehow getting mangled during transmission. Can you confirm that you only have one Hacktribe connected via USB when you run the script? Are you able to test using a Linux system? Maybe mido interacts with the MIDI drivers differently on a Mac. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[bangcorrupt]

remove the SD card before switching E2 on

That could be it, didn't think of it. If it loads an e2sample.all from the SD card it won't decompress the PCM from flash, so it will never copy it to CPU RAM. When it shows the sampler boot screen, do you see the loading bar move across the bottom? The first half moves in steps, this is reading the compressed PCM from flash. The second half moves smoothly, this is decompressing chunks and sending them to the DSP RAM. If it shows a percentage loading, then it is reading from the SD Card.

[AndrewSosna]

I’m not at home, will check when I go back, but as far as I remember indeed there was a progress indicator loading, with percentages Sent from a mobile deviceOn 2 Jan 2023, at 18:16, bangcorrupt ***@***.***> wrote: remove the SD card before switching E2 on That could be it, didn't think of it. If it loads an e2sample.all from the SD card it won't decompress the PCM from flash, so it will never copy it to CPU RAM. When it shows the sampler boot screen, do you see the loading bar move across the bottom? The first half moves in steps, this is reading the compressed PCM from flash. The second half moves smoothly, this is decompressing chunks and sending them to the DSP RAM. If it shows a percentage loafing, the it is reading from the SD Card. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[f-galland]

Andrew, is it possible you can share your wav file with the rest of us?

Thanks a lot, glad you got it working!

[AndrewSosna]

I'm sorry, I don't really want to get into copyright troubles, that's why I didn't ask @bangcorrupt to just send me his extracted file.
I could try to help you with extracting your samples by writing a more detailed step-by-step guide, if that would help

[f-galland]

Thanks! It was more of a lazy request trying to avoid risk bricking my electribe xD

I didn't consider the copyright issue, though.

Happy new year!

[richardarnezjames]

DId you ever make a step by step guide for this? I'm having issues extracting the synth PCM file

[richardarnezjames]

I've tried for two days to get this to work and no matter what I get the soft-bricked screen that's says SOFTWARE UPDATE and nothing else. someone email me the the PCM file please.
my E2 Synth's mother board is no. 13, I can't take anymore chances.

[email protected]

[ptowns]

Not sure if this us relevant or of any use...
https://www.chickensys.com/products2/korgcreator/index.html

This is a software for making multisample samples for other earlier korg devices, wonder if its the same?

[positionhigh]

i got the e2-synth-pcm.wav extracted with linux (attempts in windows failed).

Now, is there a script to get them back into an .all file or does that currently not exist, (yet) ? Maybe i am missing something obvious ?

[inkapmb]

@vananasun The waveforms are included in the those factory sample, when you extract them with sysex the waveforms are included in the .wav

@f-tom

but more as additional instruments loaded into RAM only if in use in a given project

Yes sur, its exactly whta i want to do

The maximum buffer size shall be 16x the longest ROM sample (16 tracks) but you can in the practice also allow only 4 or 6 tracks to use ROM samples to save RAM.

You thinks it will be too big ? Do you know how the synth of the e2 currently work ?

[vananasun]

So to clarify, are the waveforms stored in the CPU ROM, or on the Macronix flash chip? Because if it's stored in flash, that's good news

[inkapmb]

So to clarify, are the waveforms stored in the CPU ROM, or on the Macronix flash chip? Because if it's stored in flash, that's good news

According to @bangcorrupt in this comment :

When the PCM data is loaded from the flash, it's decompressed and sent to DSP RAM one word at a time. I can hijack this to copy PCM to CPU RAM, then dump it over sysex as a single blob.

Therefore I guess the PCM data are natively decompressed and sent to the DSP RAM for the ROM (?), but this process have been modified in a special branch to load PCM data to the CPU RAM to be able to dump them via sysex.

[F-Thom]

Do you know how the synth of the e2 currently work ?
@inkapmb : unfortunately not but I can only guess: Korg will never integrate more RAM that it would need. Now there is DSP RAM and CPU RAM.
The DSP RAM is filled with waveforms and this cover the 240 s of samples we use. If we reserve a bit of the DSP RAM to load ROM samples "on demand" while choosing a ROM sample as a sample.
You thinks it will be too big ?
I think the more we reserve, the less room remains for own samples to load or to sample/resample. Will any user need more than 6 ROM sample is a good question. If you decide to load 16 ROM sample, then you don't need Hacktribe at all, you can simply use the E2 stock firmware... But it may depend on the pattern you create.

A question remains though: if the ROM samples are compressed and loaded all together, it's difficult to access them, you need to decompress everything, get the sample you want, knowing the address and the length and dismiss the rest. Meanwhile, for this operation, you'll never have enough RAM, neither DSP or CUP... So you'll need to dismiss on the fly what you don't need and I think it's complicated to get.
Getting this working would be a dream but I don't consider this really feasible.

[inkapmb]

@F-Thom

If you decide to load 16 ROM sample, then you don't need Hacktribe at all, you can simply use the E2 stock firmware...

Sur but when switch to another pattern or switching synth/sample will get latency of loading ?

A question remains though: if the ROM samples are compressed and loaded all together, it's difficult to access them, you need to decompress everything, get the sample you want, knowing the address and the length and dismiss the rest. Meanwhile, for this operation, you'll never have enough RAM, neither DSP or CUP... So you'll need to dismiss on the fly what you don't need and I think it's complicated to get.
Getting this working would be a dream but I don't consider this really feasible.

I fully agree with that from what I know. However, to avoid taking up too much memory space with certain synth wave form, it could be (potentially) possible to add another parameter to synth such as [shift] + (osc) to be able to morph these wave forms instead of having them in memory. Ex : There is a lot of vpm-sine, vpm-saw, etc... (from hack tribe) so it could be possible to one vpm-sine, one vpm-saw etc, consequently it will be lot of free memory. In addition, according to the e2 doc (p11), there is only 134 waveforms (from 246 to 380) rest being sample. I guess, hack tribe synth (modified) + e2 synth + sample (with decent size) could fit, but I did not have dig enough in the firmware to confirm that.