Design blob processing pipeline (e.g. EXIF removal, malware scan, optimization)

[SwimmingRieux]

We must remove the metadata like location from blobs. We can use external services, micro services and more. or maybe some general way to process blobs for:

1. malware check.
2. exif(?) data removing.
3. harmful content detection.
4. generating compressed versions to save space maybe?

We don't need all of them on v1. just an overview of how we may develop it in future so we can make our next step more suitable for these milestones.

[SwimmingRieux]

1- I'm gonna use ClamAV for malware check.
2- For metadata extraction and exif removing I'm probably going with this, this or this, but I'm not sure, so tell me if there are better options.
4- Regarding the compression, seems like MinIO has it built-in.

@ZigBalthazar @kehiy

[kehiy]

@SwimmingRieux it seems ok. but malware detection and exif removers seems to be old and their updates are for more than 2 years ago at least. what do you think? would they accept prs from us in case that they need updates?

all tools you find are ok. you may want to take a look at tools written in c and rust as well. 🤔

how about harmful object detection?

[SwimmingRieux]

would they accept prs from us in case that they need updates?

What about forking their repos?

about harmful object detection?

Still researching, let me know if you have any good sources.

[kehiy]

@SwimmingRieux

if we can fork them, why not. it can be a good option as well.

ok. my idea is cloudflare and Microsoft services. what do you think?

[SwimmingRieux]

@kehiy

Cool! Would you share some links or useful resources?

[kehiy]

@SwimmingRieux

here are some resources that may help:

https://developers.cloudflare.com/cache/reference/csam-scanning
https://www.microsoft.com/en-us/photodna
https://www.missingkids.org/theissues/csam#whatncmecisdoingaboutit

also, #17 would help us to prevent this stuff manually!

[SwimmingRieux]

just dropping links here, they might be useful too

https://huggingface.co/mbehbooei/vit-base-patch16-224-in21k-finetuned-moderation
https://blog.min.io/object-detection-minio-yolo/

[kehiy]

@SwimmingRieux Thanks. The first link from hugging face seems not contains that much information. What it does?

[SwimmingRieux]

@kehiy The first link is the model used by nostr.build people.
nevent1qy88wumn8ghj7mn0wvhxcmmv9uq32amnwvaz7tmjv4kxz7fwv3sk6atn9e5k7tcpzemhxue69uhhyetvv9ujumt0wd68ytnsw43z7qgcwaehxw309ahxverz9ehx7umhdpjhyefwvdhk6tcpr3mhxue69uhkummnw3ezuur0wf6x2mt0dejhymewvdhk6tcqyzefj57spc5qe6gjsptpf4m36tezdu9nudhd64s0v25rkvlqkst6cxp3akc

[SwimmingRieux]

I think for now we can just set up the cloudflare, and then maybe in future improve it...

[kehiy]

@SwimmingRieux Thank you for event ID. It seems actually nice. We are also able to finetune as well or integrate them into modstr. But yes, for you lets keep everything as lightweight as possible.