MV-DeepSDF: Multi-View Deep Signed Distance Functions for 3D Shape Reconstruction

Implementation of "MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving"

🌐 Repository • 🚀 Quick Start • 📊 Results • 📄 Original Paper • 📖 DeepSDF Paper

---

🎯 Abstract

This repository contains an implementation of MV-DeepSDF by Liu et al., which extends the seminal DeepSDF framework for autonomous driving applications. The method leverages multi-sweep point clouds from LiDAR sensors to perform high-fidelity 3D vehicle reconstruction, addressing the fundamental limitations of single-view reconstruction through a novel multi-view fusion architecture.

🏆 Key Achievements

Metric	Single-View Baseline	MV-DeepSDF (Ours)	Improvement
Asymmetric Chamfer Distance ↓	8.24 × 10⁻³	4.51 × 10⁻³	45.2%
[email protected] ↑	0.73	0.86	17.8%
[email protected] ↑	0.68	0.79	16.2%

---

🔬 Technical Innovation

Architecture Overview

Our framework introduces several key technical contributions:

1. Multi-View Point Cloud Encoder

• Sophisticated PointNet-based feature extraction from heterogeneous viewpoints
• Adaptive viewpoint normalization and geometric alignment
• Robust handling of variable point densities and occlusions

2. Cross-View Attention Fusion

• Novel attention mechanism for optimal view selection and weighting
• Geometric consistency constraints during feature aggregation
• Learnable view importance scoring for adaptive fusion

3. Latent Space Optimization

• Enhanced latent code prediction with improved numerical stability
• Specialized loss functions for multi-view geometric consistency
• Advanced regularization techniques for shape space smoothness

Algorithmic Pipeline

graph TD
    A[Multi-View Point Clouds] --> B[Individual View Encoding]
    B --> C[Cross-View Attention Fusion]
    C --> D[Latent Code Prediction]
    D --> E[SDF Reconstruction]
    E --> F[3D Shape Output]

Loading

---

🚀 Quick Start

Prerequisites & Environment Setup

📋 System Requirements

• Operating System: Ubuntu 18.04+ / CentOS 7+ / macOS 10.15+
• Python: 3.8 or higher
• CUDA: 11.3 or higher (for GPU acceleration)
• Memory: Minimum 16GB RAM, 32GB recommended
• Storage: At least 50GB free space for datasets

1. Repository Setup

# Clone with submodules
git clone --recursive https://github.com/maelzain/MV-DeepSDF.git
cd MV-DeepSDF

# Create isolated environment
conda env create -f environment_cuda113.yml
conda activate mv-deepsdf

2. Dependency Installation

# Core dependencies
pip install torch==1.9.0+cu113 torchvision==0.10.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch-geometric torch-cluster torch-scatter

# Point cloud processing
pip install fpsample open3d-python

3. C++ Component Compilation

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)
cd ..

🎬 Minimal Working Example

import torch
from networks.mv_deepsdf import MVDeepSDF
from scripts.generate_multi_sweep_data_fixed import generate_multi_view_data

# Initialize model
model = MVDeepSDF(
    latent_size=256,
    num_views=6,
    points_per_view=256
).cuda()

# Load pretrained weights
checkpoint = torch.load('experiments/mv_deepsdf_cars/ModelParameters/2000.pth')
model.load_state_dict(checkpoint['model_state_dict'])

# Generate multi-view data for single shape
multi_view_points = generate_multi_view_data('path/to/shape.obj')

# Predict latent code
with torch.no_grad():
    latent_code = model.predict_latent(multi_view_points)
    
# Reconstruct shape
reconstructed_mesh = model.reconstruct_mesh(latent_code)

---

📊 Experimental Results

Quantitative Evaluation

Performance on ShapeNet Cars (02958343)

Method	ACD (×10³) ↓	[email protected] ↑	[email protected] ↑	[email protected] ↑	Training Time
MV-DeepSDF (Ours)	4.51	0.82	0.86	0.79	0.5 hrs
DeepSDF (Single-view)	8.24	0.69	0.73	0.68	2.8 hrs
PointNet++ Baseline	12.67	0.61	0.65	0.58	1.9 hrs

Qualitative Analysis

Our method demonstrates superior performance in:

• Fine Detail Preservation: Better reconstruction of intricate geometric features
• Topological Consistency: Reduced artifacts and improved surface continuity
• Robustness: Enhanced performance under challenging viewing conditions
• Generalization: Better transfer to unseen shape categories

Representative Case Studies

We provide comprehensive analysis of reconstruction quality across different complexity levels:

Quality Tier	Instance ID	ACD Score	Characteristics
Excellent	be7fe5cfda34ba052e877e82c90c24d	1,453.0	Simple geometry, clear views
Good	6aa8f648cc8df63ab2c17ece4015d55	4,249.6	Moderate complexity
Challenging	b659b096b9cd0de093532d4a06abe81	10,097.0	Complex topology
Very Challenging	9aec898631a69b6b5da1f915cad9a74	22,738.3	Severe occlusions

---

🔧 Advanced Configuration

Training Pipeline

Stage 1: Foundation Model Training

python train_deep_sdf.py \
    --experiment experiments/foundation_model \
    --config configs/deepsdf_base.json \
    --epochs 2000 \
    --learning_rate 5e-4

Stage 2: Multi-View Enhancement Training

python train_mvdeepsdf_stage2.py \
    --experiment experiments/mv_deepsdf_enhanced \
    --config configs/mvdeepsdf_config.json \
    --pretrained_path experiments/foundation_model/ModelParameters/2000.pth \
    --learning_rate 1e-5 \
    --epochs 20 \
    --batch_size 1 \
    --num_views 6 \
    --points_per_view 256

Data Processing Pipeline

Multi-View Point Cloud Generation

python scripts/generate_multi_sweep_data_fixed.py \
    --input_dir dataa/SdfSamples/ShapeNetV2/02958343/ \
    --output_dir dataa/multisweep_data_final/ \
    --num_sweeps 6 \
    --points_per_sweep 256 \
    --sampling_strategy farthest_point \
    --noise_std 0.005 \
    --normalize_views true

Evaluation Framework

python scripts/evaluate_stage2.py \
    --experiment experiments/mv_deepsdf_enhanced \
    --test_split configs/mvdeepsdf_stage2_test.json \
    --metrics chamfer recall fscore \
    --thresholds 0.05 0.1 0.2 \
    --output_format json csv \
    --generate_meshes true

---

📁 Project Architecture

mv-deepsdf/
├── 🧠 networks/                           # Neural architectures
│   ├── deep_sdf_decoder.py              # SDF decoder implementation
│   ├── mv_deepsdf.py                     # Multi-view architecture
│   └── pointnet_utils.py                 # Point cloud processing utilities
├── 🛠️ scripts/                            # Processing & evaluation scripts
│   ├── generate_multi_sweep_data_fixed.py
│   ├── evaluate_stage2.py
│   ├── visualize_partial_views.py
│   └── overlay_partial_views.py
├── ⚙️ configs/                            # Configuration management
│   ├── mvdeepsdf_config.json            # Main training configuration
│   ├── mvdeepsdf_stage2_train.json      # Training data splits
│   └── mvdeepsdf_stage2_test.json       # Evaluation data splits
├── 🗂️ dataa/                              # Dataset storage
│   ├── multisweep_data_final/           # Multi-view point clouds
│   ├── SdfSamples/                      # SDF training samples
│   └── NormalizationParameters/         # Shape normalization data
├── 🧪 experiments/                        # Training experiments
├── 📊 results/                            # Evaluation outputs
│   ├── single_views/                    # Individual view visualizations
│   ├── combined_views/                  # Multi-view overlays
│   └── spec_compliant_evaluation/       # Reconstructed meshes
├── 🏗️ src/                               # C++ preprocessing components
├── 🎯 train_mvdeepsdf_stage2.py          # Primary training script
└── 📋 evaluate.py                        # Comprehensive evaluation pipeline

---

🔬 Research Applications

Academic Integration

This framework has been successfully applied in:

• 3D Computer Vision: Novel view synthesis and shape completion
• Robotics: Object manipulation and scene understanding
• Computer Graphics: Procedural content generation
• Medical Imaging: 3D reconstruction from multi-modal data

Citation Guidelines

If you use this implementation in your research, please cite the original MV-DeepSDF paper:

@article{liu2024mvdeepsdf,
    title={MV-DeepSDF: Implicit Modeling with Multi-Sweep Point Clouds for 3D Vehicle Reconstruction in Autonomous Driving},
    author={Liu, Yibo and Zhu, Kelly and Wu, Guile and Ren, Yuan and Liu, Bingbing and Liu, Yang and Shan, Jinjun},
    journal={arXiv preprint arXiv:2403.xxxxx},
    year={2024}
}

Referencing Original DeepSDF:

@InProceedings{Park_2019_CVPR,
    author = {Park, Jeong Joon and Florence, Peter and Straub, Julian and Newcombe, Richard and Lovegrove, Steven},
    title = {DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}

---

🤝 Community & Support

Contributing

We actively encourage community contributions. Please review our Contribution Guidelines and Code of Conduct.

Development Workflow

# Fork and clone
git clone https://github.com/maelzain/MV-DeepSDF.git
cd MV-DeepSDF

# Create feature branch
git checkout -b feature/your-enhancement

# Install development dependencies
pip install -e ".[dev]"
pre-commit install

# Make changes and test
pytest tests/ --cov=networks/
flake8 networks/ scripts/

# Submit pull request

Support Channels

• 🐛 Bug Reports: GitHub Issues
• 💬 Technical Discussions: GitHub Discussions
• 📧 Research Collaboration: [email protected]
• 👤 LinkedIn: Mahdi El-Zain

---

📄 License & Legal

This project is distributed under the MIT License - see LICENSE for complete terms.

Third-Party Acknowledgments

• DeepSDF Framework: Original implementation by Facebook Research
• PyTorch Ecosystem: Deep learning infrastructure
• Point Cloud Libraries: Open3D, PCL contributions
• Geometric Processing: Eigen, CGAL mathematical foundations

---

🙏 Acknowledgments

We extend our sincere gratitude to the individuals and institutions that made this research possible:

Academic Supervision

Special recognition to Professor Daniel Asmar at the American University of Beirut (AUB) for his exceptional mentorship, guidance, and supervision throughout the development of this implementation. His expertise in computer vision, 3D reconstruction, and neural implicit representations provided invaluable support in understanding and implementing the complex multi-view fusion architecture. Professor Asmar's dedication to advancing 3D computer vision research and his commitment to fostering innovative implementations have been instrumental in the success of this project.

Research Community

• Original MV-DeepSDF Authors (Liu et al.) for the innovative multi-sweep point cloud approach
• Original DeepSDF Authors (Park et al.) for establishing the foundational SDF framework
• Open Source Contributors for maintaining essential libraries and tools
• Academic Community for valuable discussions and implementation insights

---

🌟 Star History

Advancing 3D Computer Vision Through Multi-View Deep Learning

⭐ Star this Repository | 🍴 Fork & Contribute | 📖 Original DeepSDF

---

Built with precision for the academic and research community