Implement the python implementation of PreprocessData (Step 2 and Step 3)

[laurensWe]

• Translate step 2 and step 3 of the matlab function PreprocessData.m
• This step will rotate the images to be vertical and then extract the features
• Added tests (both unit as integration tests)

[SimoneAriens]

I think this code is functional, I only saw a few possible deviances from the matlab code. But there are a few places where you could reuse (and maybe slightly adapt) some of the already existing functions/classes, and make the flow a bit more concise and logical

[github-actions]

Diff Coverage

Diff: origin/main..HEAD, staged and unstaged changes

• packages/scratch-core/src/conversion/filter.py (91.9%): Missing lines 176,416-417
• packages/scratch-core/src/conversion/mask.py (100%)
• packages/scratch-core/src/conversion/preprocess_striation/parameters.py (100%)
• packages/scratch-core/src/conversion/preprocess_striation/preprocess_data.py (91.6%): Missing lines 148,213,223-225,283,286,289-291,342,349,484
• packages/scratch-core/src/conversion/preprocess_striation/preprocess_data_filter.py (78.9%): Missing lines 33,109-115

Summary

• Total: 241 lines
• Missing: 24 lines
• Coverage: 90%

packages/scratch-core/src/conversion/filter.py

Lines 172-180

  172         return result[
  173             pad_y : -pad_y if pad_y else None, pad_x : -pad_x if pad_x else None
  174         ]
  175     else:
! 176         raise ValueError("This padding mode is not implemented")
  177 
  178 
  179 def _apply_order0_filter(
  180     data: NDArray[np.floating],

---

Lines 412-421

  412     if axis == 0:
  413         kernel_y = kernel_1d
  414         kernel_x = np.array([1.0])
  415     else:
! 416         kernel_y = np.array([1.0])
! 417         kernel_x = kernel_1d
  418 
  419     # Match MATLAB SmoothMod.m: different boundary modes based on NaN presence
  420     # - NaNs present: zero padding + edge correction via normalization (NanConv path)
  421     # - No NaNs: symmetric padding (DIPimage smooth() path)

---

packages/scratch-core/src/conversion/preprocess_striation/preprocess_data.py

Lines 144-152

  144     :returns: Rotated depth data.
  145     """
  146     # Skip rotation for angles smaller than ~0.1° (0.00175 rad)
  147     if abs(angle_rad) <= 0.00175:
! 148         return depth_data.copy()
  149 
  150     height, width = depth_data.shape
  151 
  152     # Calculate total vertical shift: tan(angle) * width

---

Lines 209-217

  209     """
  210     # Determine subsampling factor
  211     sub_samp = extra_sub_samp
  212     if scale_x < 1e-6:
! 213         sub_samp = round(1e-6 / scale_x) * extra_sub_samp
  214 
  215     # Determine sigma for smoothing (minimum of 3)
  216     sigma = max(3, round(1.75e-5 / scale_x / sub_samp))

---

Lines 219-229

  219     data_subsampled = depth_data
  220     mask_subsampled = mask
  221 
  222     if sub_samp > 1 and depth_data.shape[1] // sub_samp >= 2:
! 223         data_subsampled = resample_image_array(depth_data, factors=(sub_samp, 1))
! 224         if mask is not None:
! 225             mask_subsampled = resample_image_array(mask, factors=(sub_samp, 1)) > 0.5
  226 
  227     # Smooth data
  228     smoothed = _smooth_2d(data_subsampled, (sigma, sigma))

---

Lines 279-295

  279     :param cut_y_after_shift: If True, crop NaN borders after shifting.
  280     :returns: Tuple of (rotated_data, rotated_mask).
  281     """
  282     if mask is not None:
! 283         data_rotated = _rotate_data_by_shifting_profiles(
  284             depth_data, angle_rad, cut_y_after_shift
  285         )
! 286         mask_rotated = _rotate_data_by_shifting_profiles(
  287             mask.astype(float), angle_rad, cut_y_after_shift
  288         )
! 289         mask_binary = mask_rotated == 1
! 290         y_slice, x_slice = _determine_bounding_box(mask_binary)
! 291         return data_rotated[y_slice, x_slice], mask_binary[y_slice, x_slice]
  292 
  293     return _rotate_data_by_shifting_profiles(
  294         depth_data, angle_rad, cut_y_after_shift
  295     ), None

---

Lines 338-346

  338                 depth_data, mask, a_tot_rad, cut_y_after_shift
  339             )
  340 
  341             if a == a_last:
! 342                 iteration = max_iter - 1
  343             else:
  344                 a_last = a
  345 
  346         iteration += 1

---

Lines 345-353

  345 
  346         iteration += 1
  347 
  348     if iteration >= max_iter:
! 349         return 0.0
  350     return a_tot
  351 
  352 
  353 def fine_align_bullet_marks(

---

Lines 480-488

  480 
  481     :returns: Tuple of (aligned_data, profile, mask, total_angle).
  482     """
  483     if params is None:
! 484         params = PreprocessingStriationParams()
  485 
  486     data_filtered = scan_image.data.copy()
  487     mask_filtered = mask

---

packages/scratch-core/src/conversion/preprocess_striation/preprocess_data_filter.py

Lines 29-37

  29     valid_cols = np.any(valid_data, axis=0)
  30 
  31     if not np.any(valid_rows) or not np.any(valid_cols):
  32         # No valid data at all - return empty arrays
! 33         return (
  34             np.array([]).reshape(0, data.shape[1]),
  35             np.array([], dtype=bool).reshape(0, data.shape[1]),
  36             np.array([], dtype=np.intp),
  37         )

---

Lines 105-118

  105     cropped_data = output
  106     cropped_mask = mask
  107 
  108     if cut_borders_after_smoothing:
! 109         if has_masked_regions:
! 110             output_with_nan = output.copy()
! 111             output_with_nan[~mask] = np.nan
! 112             cropped_data, cropped_mask, _ = _remove_zero_border(output_with_nan, mask)
! 113         elif sigma_int > 0 and scan_image.height > 2 * sigma_int:
! 114             cropped_data = output[sigma_int:-sigma_int, :]
! 115             cropped_mask = mask[sigma_int:-sigma_int, :]
  116 
  117     return cropped_data, cropped_mask

---

[github-actions]

Package	Line Rate	Branch Rate	Health
.	95%	88%	✔
comparators	100%	100%	✔
container_models	99%	100%	✔
conversion	96%	89%	✔
conversion.leveling	100%	100%	✔
conversion.leveling.solver	100%	75%	✔
conversion.preprocess_impression	99%	92%	✔
conversion.preprocess_striation	90%	69%	✔
extractors	95%	75%	✔
parsers	98%	67%	✔
parsers.patches	89%	60%	✔
preprocessors	100%	100%	✔
processors	100%	100%	✔
renders	98%	50%	✔
utils	91%	75%	✔
Summary	96% (1360 / 1417)	81% (174 / 216)	✔

Minimum allowed line rate is 50%