WACV 2027 / 4D point cloud representation learning
Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
1Nagoya University 2Northeastern University 3KC Machine Learning Lab 4University of Science, VNU-HCM 5University of California, Los Angeles
Each of the 2,048 input points takes the feature of its nearest encoder token, projected to RGB with one PCA basis per encoder that is shared across actions and frames, so similar colors mean similar features. The dense encoder's colors change along the body, from head to torso to legs. The global encoder never sees a per-point target, and on most actions its colors vary less within the body. Drag to rotate, scroll or pinch to zoom, double-click to reset. Both encoders are the released Table 1 seed-0 checkpoints.
In short
Self-supervised pretraining for 4D point clouds has mostly used intra-modal pretexts, and the few methods that borrow from 2D foundation models distill one global embedding per clip, throwing away the per-patch features those models compute. Cross4D-JEPA renders each frame from geometry alone, maps every visible 3D point to the teacher patch it projects to, and trains the student to predict those features densely, in latent space, with no masking, negatives, or decoder.
Controlled studies attribute the gain to the granularity of the correspondence rather than to the teacher: image and video teachers tie, and an untrained teacher recovers most of the gain.
Method
The teacher runs once per clip before training; its targets are cached and the teacher is freed, so pretraining costs one GPU for about 35 minutes on MSR-Action3D.
Hover a patch (or focus the render and use the arrow keys) to light up the points that inherit its feature. Gray points are hidden from the camera in this frame and get no target.
Results
Frozen-encoder linear-probe top-1 unless noted; mean ± standard deviation over seeds. Hover any mark for its value, or open the table under each chart.
With the same teacher, student and budget, predicting per-point targets beats predicting one global target per clip by 0.18 to 0.29, several times either error bar.
Table 1. MSR-Action3D uses DINOv2 (5 dense / 8 global seeds); DeformingThings4D and NTU RGB+D 60 use V-JEPA 2 (5 seeds each).
Swapping DINOv2 for DINOv3, V-JEPA 2 or VideoMAE V2 moves dense results by at most 0.023, while the dense–global gap stays between 0.21 and 0.29 for every teacher.
Table 2, MSR-Action3D, 5 seeds per teacher (8 for DINOv2 global).
Same render, correspondence, 9 × 9 grid and objective; only the teacher's training changes. Raw rendered pixels already reach 0.685, and an untrained ViT 0.776. The same ordering holds for zero-shot HOI4D segmentation: 13.95 mIoU from raw pixels, 15.81 from the untrained ViT, 16.61 from DINOv2, against a 4.6 floor.
Table 3, MSR-Action3D, 3 seeds.
Holding Point4D's parameters at 3.28M, dense supervision stays flat down to 4 tokens per frame and degrades below that; global never overtakes it. On the official P4Transformer, dense also beats global (0.617 vs. 0.376), and shrinking the ball-query radius from 0.7 to 0.1 leaves dense between 0.58 and 0.68.
Table 4, MSR-Action3D, 3 seeds per point (more for n2 = 64).
Re-implemented on Point4D with the same data and probe, at their best budget from a 12× sweep, masked and latent intra-modal pretexts reach 44.7 to 65.8%; dense distillation reaches 82.9%. Fine-tuned, it reaches 92.0%, close to published methods that use 3–13× larger backbones.
Table 6, MSR-Action3D. Published rows use their own backbones and are not directly comparable.
Fine-tuning from the dense encoder beats training the same architecture from scratch at every label fraction, by 0.17 with 10% of labels and still by 0.10 with all of them.
MSR-Action3D end-to-end fine-tuning, 3 seeds.
Dense downstream task
A linear layer on frozen per-point features labels every point of every frame. Below, encoders pretrained on HOI4D itself (in-domain), each with the same probe.
Each color is one semantic class; a point whose color matches the ground truth is correct. The number on each view is the point accuracy on this clip, over all 24 frames. The table reports the full test split.
Code and checkpoints
The repository has the method, the baselines, one script per paper table, and all 128 pretrained encoders behind the six main tables, each listed with its training command and logged result.
git clone https://github.com/tuananh1007/Cross4D-JEPA.git
cd Cross4D-JEPA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e ".[preprocess,dev]"
pytest # CPU smoke testspython -m cross4d_jepa.engine.linear_probe \
configs/msr_point4d.yaml \
--encoder checkpoints/main/msr_dense_dinov2_s0.pt
# linear probe: acc=0.8509Citation
@inproceedings{nguyen2027cross4djepa,
title = {Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning},
author = {Nguyen, Trung Thanh and Nguyen-Truong, Hai and Vo, Tu and Truong, Hoang M. and Vu, Tuan-Anh},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year = {2027}
}