from_immunarch() fails with DuckDB conversion error when numeric columns contain NA values

[giuliabriss]

❓ Questions and Help

When using from_immunarch() to convert an immdata object (created with repLoad()) to the new ImmunData format, the function fails with a DuckDB conversion error when numeric columns like V.end, D.start, D.end, and J.start contain NA values.

Error Message

Error in `compute_parquet()`:
! {"exception_type":"Conversion","exception_message":"Failed to write '..../output/annotations.parquet': 
CSV Error on Line: 23064
Error when converting column \"J.start\". Could not convert string \"NA\" to 'BIGINT'

Column J.start is being converted as type BIGINT
This type was auto-detected from the CSV file.
Possible solutions:
* Override the type for this column manually by setting the type explicitly, e.g., types={'J.start': 'VARCHAR'}
* Set the sample size to a larger value to enable the auto-detection to scan more values, e.g., sample_size=-1
* Use a COPY statement to automatically derive types from an existing table.
* Check whether the null string value is set correctly (e.g., nullstr = 'N/A')

Can you help me please?

Thank you!

[vadimnazarov]

Hi @giuliabriss thank you. Could you share the code you used to do it, please? Was it just idata <- from_immundata(your_data, "folder-name")? I couldn't reproduce the bug on my side.

Do you really need the J.start, etc., columns? You can remove them, but this is not a great solution if you need those, obviously.

immdata$data <- lapply(immdata$data, function (df) df |> select(-V.end, -J.start, -D.end, -D.start, -Proportion) )
idata <- from_immunarch(immdata, ...)

[giuliabriss]

Hi @vadimnazarov,

Thanks for your prompt response and suggestions.

For context, I’m uploading immunoSEQ .tsv files exported from Adaptive (v1). This is the code I used:

`library(immunarch)

path <- "data/v1"
output <- "output"

immdata <- immunarch::repLoad(path, .coding = TRUE)

idata <- from_immunarch(
imm = immdata,
schema = c("CDR3.nt"),
output_folder = output
)
`

I tried your suggestions and they worked—thank you!

In the meantime, I found another workaround that also seems to work.

immdata <- immunarch::repLoad(path, .coding = TRUE)

immdata$data <- lapply(immdata$data, function(df) {
numeric_cols <- c("V.end", "D.start", "D.end", "J.start")
for (col in numeric_cols) {
if (col %in% names(df)) {
# Replace NA with empty string
df[[col]][is.na(df[[col]])] <- ""
}
}
df
})

idata <- from_immunarch(
imm = immdata,
schema = c("CDR3.nt"),
output_folder = output
)

Thank you again!

[vadimnazarov]

Ah, I see. Maybe you could directly read the Adaptive files into R using read_repertoires? https://immunomind.github.io/docs/guides/io/ingesting/ It would be much convenient. What columns do those files have? Could you send me the first ~5 lines from one file, please?

[giuliabriss]

Thanks for the suggestion! I'll try that approach.

Regarding the file format, please find attached a .tsv file with the first few lines from an Adaptive Biotechnologies immunoSEQ file from one sample.

adaptive_preview_ok.tsv