MoCA: Multi-modal Cross-masked Autoencoder for Time-Series in Digital Health

• Demonstration
• Setting up Environment
• Data and Checkpoints
• Pre-training
• Downstream Classification Tasks

Demonstration

For reconstruction and classification demonstration on the UCI-HAR dataset, go to UCIHAR_mae_visualize.ipynb.

Setting up Environment

This repository uses Git Large File Storage (LFS) for checkpoints. Before cloning the repository, run

git lfs install

to install LFS.

To get started, clone the repository to your local or cluster machine by running the following commands in your terminal

git clone https://github.com/HowonRyu/MoCA.git
cd MoCA

You can set up the environment by installing the required dependencies:

pip install -r requirements.txt

The list of dependencies can be found in the requirements.txt file.

Data and Checkpoints

• MoCA defaults to using the pre-processed UCI-HAR[1] dataset located in MoCA/data/{data_length}. Default input length is 200 in MoCA.
• The pre-trained checkpoints can be found in MoCA/checkpoints.

Pre-training

You can pre-train from scratch by using submitit_pretrain.py, which calls the main_pretrain.py script, using the following:

python {path_to_submitit.pretrain.py} \
	--wk_dir {path_to_working_directory}  --job_name {job_nickname} \
	--data_path {path_to_data} \
	--nodes 1  --use_volta32 x \
	--batch_size 50 --num_workers 4 \
	--model mae_vit_base_patch16 --mask_ratio 0.25 --patch_size1 1 --patch_num 10 --alt \
	--epochs {epochs} --warmup_epochs {warm_up_epochs} --dump_freq {checkpoint_save_frequency(epochs)} \
	--lr 0.0005 --weight_decay 0.05 --loss_type 'all' --device 'cuda'

Downstream Classification Tasks

Given a pre-training checkpoint (weights), you can finetune/linear probe using the following scripts:

• Finetuning

python {path_to_submitit_finetune.py} \
    --wk_dir  {path_to_working_directory} --job_name {job_nickname} \
    --data_path {path_to_working_directory} \
    --nodes 1 --batch_size 5 --epochs {epochs}  \
    --cls_token --nb_classes 7 --patch_num 10 --patch_size1 1 --alt \
    --model vit_base_patch16  \
    --lr 0.001 --weight_decay 0 --device 'cuda' --dump_freq 50 \
    --finetune {path_to_pretrained_checkpoints} 

• Linear Probing

python {path_to_submitit_linprobe.py} \
    --wk_dir  {path_to_working_directory} --job_name {job_nickname} \
    --data_path {path_to_working_directory} \
    --nodes 1 --batch_size 5 --epochs {epochs}  \
    --cls_token --nb_classes 7 --patch_num 10 --patch_size1 1 --alt \
    --model vit_base_patch16  \
    --lr 0.001 --weight_decay 0 --device 'cuda' --dump_freq 50 \
    --finetune {path_to_pretrained_checkpoints} 

Configuration	UCI-HAR	WISDM	IMWSHA	RealWorld	OPPORTUNITY	PAMAP2
Optimizer	AdamW	AdamW	AdamW	AdamW	AdamW	AdamW
Learning Rate	1e-3	1e-3 (1e-2)	2.5e-4 (1e-3)	2.5e-4 (2.5e-3)	2.5e-4 (2.5e-3)	2.5e-4 (2.5e-3)
Weight Decay	5e-2 (0)	5e-2 (1e-4)	5e-2 (1e-4)	5e-2 (1e-4)	1e-1 (5e-2)	5e-2 (1e-4)
Optimizer Momentum	β₁=0.9, β₂=0.999	β₁=0.9, β₂=0.999	β₁=0.9, β₂=0.999	β₁=0.9, β₂=0.999	β₁=0.9, β₂=0.999	β₁=0.9, β₂=0.999
Batch Size	50	256	64 (28)	64	64	64
LR Schedule	Cosine Decay	Cosine Decay	Cosine Decay	Cosine Decay	Cosine Decay	Cosine Decay
Warm-up Epochs	5 (10)	10	10	10	10	10
Training Epochs	50	50	50	50	50	50

• Finetuning configuration (linear probing configuration if different from finetuning)

Reference

[1] Reyes-Ortiz, J., Anguita, D., Ghio, A., Oneto, L., & Parra, X. (2013). Human Activity Recognition Using Smartphones [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C54S4K.