Consider using esp_new_jpeg

[NaxGH]

Hi
Have you ever consider using esp_new_jpeg for motion detect?

[s60sc]

I'll take a look

[NaxGH]

it ~30ms faster at same resolution. I test it at SXGA.

D (11:38:33.743) MAIN: [jpg2rgb] Decoded 160x128 -> 61440 bytes in 171 ms
D (11:38:33.746) MAIN: [checkMotion] JPEG to rescaled grayscale bitmap conversion 20480 bytes in 175ms
D (11:38:33.757) MAIN: [checkMotion] Bitmap rescale to 9216 bytes in 7ms
D (11:38:33.760) MAIN: [checkMotion] Detected 18 changes, threshold 164, light level 60, in 2ms
D (11:38:33.766) MAIN: [checkMotion] Overall time took 194 ms
D (11:38:33.773) MAIN: [checkMotion] ============================

using esp_new_jpeg library

D (11:42:04.327) MAIN: [jpg2rgb] Decoded 160x128 -> 61440 bytes in 138 ms
D (11:42:04.331) MAIN: [checkMotion] JPEG to rescaled grayscale bitmap conversion 20480 bytes in 142ms
D (11:42:04.342) MAIN: [checkMotion] Bitmap rescale to 9216 bytes in 7ms
D (11:42:04.344) MAIN: [checkMotion] Detected 7 changes, threshold 164, light level 59, in 3ms
D (11:42:04.350) MAIN: [checkMotion] Overall time took 161 ms
D (11:42:04.357) MAIN: [checkMotion] ============================

[NaxGH]

my test code:

static void jpeg_get_scaled_dims(int width, int height, esp_jpeg_image_scale_t scale, int *out_width, int *out_height) {
    uint8_t div = 1;
    switch (scale) {
        case JPEG_IMAGE_SCALE_1_2: div = 2; break;
        case JPEG_IMAGE_SCALE_1_4: div = 4; break;
        case JPEG_IMAGE_SCALE_1_8: div = 8; break;
        case JPEG_IMAGE_SCALE_0:
        default: div = 1; break;
    }

    // minimum allowed scaled size = ceil(orig / divisor)
    int min_w = (width  + div - 1) / div;
    int min_h = (height + div - 1) / div;

    // round UP to multiple of 8
    int w = ((min_w + 7) / 8) * 8;
    int h = ((min_h + 7) / 8) * 8;

    // clamp so it doesn’t exceed orig size (rounded DOWN to multiple of 8)
    int max_w = (width  / 8) * 8;
    int max_h = (height / 8) * 8;
    if (w > max_w) w = max_w;
    if (h > max_h) h = max_h;

    // make sure ≥ min
    if (w < min_w) w = min_w;
    if (h < min_h) h = min_h;

    // enforce non-zero 
    if (w <= 0) w = 8; 
    if (h <= 0) h = 8;

    *out_width = w;
    *out_height = h;
}

// base on esp_jpeg_decode_one_picture() from esp_new_jpeg/test_app/test_decoder.c
static jpeg_error_t jpg2rgb(uint8_t *input_buf, int len, uint16_t width, uint16_t height, uint8_t *out_buf) {
    uint32_t sTime = millis();

    jpeg_error_t ret;

    jpeg_dec_config_t config = DEFAULT_JPEG_DEC_CONFIG();
    config.output_type  = JPEG_PIXEL_FORMAT_RGB888; // @todo maybe use YUV442 (CbYCrY) ?
    config.scale.width = width;
    config.scale.height = height;

    jpeg_dec_handle_t jpeg_dec = NULL;
    ret = jpeg_dec_open(&config, &jpeg_dec);
    if (ret != JPEG_ERR_OK) {
        LOG_ERR("Failed to open jpeg decoder");
        return ret;
    }

    jpeg_dec_io_t jpeg_io = {
        .inbuf = input_buf,
        .inbuf_len = len,
        .outbuf = out_buf,
    };

    jpeg_dec_header_info_t out_info = {};
    ret = jpeg_dec_parse_header(jpeg_dec, &jpeg_io, &out_info);
    if (ret != JPEG_ERR_OK) {
        LOG_ERR("Failed to parse jpeg header");
        jpeg_dec_close(jpeg_dec);
        return ret;
    }

    ret = jpeg_dec_process(jpeg_dec, &jpeg_io);
    if (ret != JPEG_ERR_OK) {
        LOG_ERR("JPEG decode failed");
        jpeg_dec_close(jpeg_dec);
        return ret;
    }

    jpeg_dec_close(jpeg_dec);

    int out_len = out_info.width * out_info.height * RGB888_BYTES;
    LOG_DBG("Decoded %dx%d -> %d bytes in %d ms", out_info.width, out_info.height, out_len, millis() - sTime);
    return JPEG_ERR_OK;
}

static void rgb2gray(uint8_t *buffer, int width, int height) {
    int num_pixels = width * height;
    uint8_t *src = buffer;
    uint8_t *dst = buffer;

    for (int i = 0; i < num_pixels; i++) {
        uint8_t r = src[0];
        uint8_t g = src[1];
        uint8_t b = src[2];
        // Calculate grayscale value using luminance formula
        uint8_t gray = (uint8_t)((r * 77 + g * 150 + b * 29) >> 8);
        *dst++ = gray;
        src += 3;
    }
}

usage in checkMotion():

#ifdef USE_NEW_JPG_DECODER
    int sampleWidth, sampleHeight;
    jpeg_get_scaled_dims(fb->width, fb->height, (esp_jpeg_image_scale_t)scaling, &sampleWidth, &sampleHeight);
    static uint8_t *img_buf = (uint8_t *)heap_caps_aligned_calloc(16, 1, sampleWidth * sampleHeight * RGB888_BYTES, MALLOC_CAP_SPIRAM);  // no need to free
    if (jpg2rgb(fb->buf, fb->len, sampleWidth, sampleHeight, img_buf) != JPEG_ERR_OK) return motionStatus;
    if (colorDepth == GRAYSCALE_BYTES) rgb2gray(img_buf, sampleWidth, sampleHeight);
#else

[s60sc]

Thanks
I'll use your code to get started. There are some other optimisations I could do in checkMotion()

[s60sc]

esp_new_jpeg doesnt seem to be included in Arduino

[NaxGH]

ah right I forgot, you need to manually add the library.
its part of esp-adf-libs -> esp_new_jpeg or idf.py add-dependency "espressif/esp_new_jpeg^0.6.1"

also need to guide user if they want to use esp_new_jpeg

[NaxGH]

I am not familiar with Arduino IDE. maybe I am wasting your time.
well let me take a look if I can make it work with Arduino IDE. maybe someone can help with this too.

[NaxGH]

@s60sc Hi, I manage to make it work and compile the project.
please include the library Sketch -> Include Library -> Add .ZIP Library
espressif__esp_new_jpeg.zip
or you can make it yourself

espressif__esp_new_jpeg/
 ├── library.properties
 └── src/
      ├── esp_jpeg_common.h
      ├── esp_jpeg_dec.h
      ├── esp_jpeg_enc.h
      ├── esp_jpeg_version.h
      └── esp32/
           └── libesp_new_jpeg.a
      └── other/
           └── other.a

library.properties

name=espressif__esp_new_jpeg
version=0.6.1
author=Espressif
maintainer=Espressif <https://github.com/espressif>
sentence=Espressif JPEG encoder and decoder
paragraph=ESP_NEW_JPEG is Espressif's lightweight JPEG encoder and decoder library. The memory and CPU loading are optimized to make better use of Espressif chips.
category=Data Processing
url=https://github.com/espressif/esp-adf-libs/tree/master/esp_new_jpeg
architectures=esp32
precompiled=full

build log:

....
....\sketch\motionDetect.cpp -o nul
Alternatives for esp_jpeg_dec.h: [[email protected]]
ResolveLibrary(esp_jpeg_dec.h)
  -> candidates: [[email protected]]
Skipping dependencies detection for precompiled library espressif__esp_new_jpeg
....
esptool v5.1.0
Creating ESP32 image...
Merged 2 ELF sections.
Successfully created ESP32 image.
....
Using library espressif__esp_new_jpeg at version 0.6.1 in folder: D:\Documents\Arduino\Projects\libraries\espressif__esp_new_jpeg 
"....\\AppData\\Local\\arduino\\sketches\\....../ESP32-CAM_MJPEG2SD.ino.elf"
Sketch uses 1780955 bytes (90%) of program storage space. Maximum is 1966080 bytes.
Global variables use 98088 bytes (29%) of dynamic memory, leaving 229592 bytes for local variables. Maximum is 327680 bytes.

[s60sc]

Thanks
I'll look at implementing the stream version of the example as it should only be necessary to open & close handler when resolution changes
I've raised discussion 11873 on arduino-esp32