Multiple Pose Virtual Try-On Based on 3D Clothing Reconstruction

Recently, image-based virtual try-on (VTON) systems using deep generative models have drawn significant research attention. However, the 2D clothing shape transform methods in the earlier works show serious limitations in 3D clothing deformation required in multiple-pose VTON scenarios. In this paper, we develop a 3D-MPVTON system of two pipelines and show that a 3D clothing model reconstruction approach provides much better results for the multi-pose VTON scenario. First, the 3D clothing model reconstruction pipeline is based on CloTH-VTON+. The try-on clothing is matched to the target clothing regions in the simply-shaped reference human model and its 3D model is reconstructed using the associated 3D human body model. For natural clothing rendering from an arbitrary view, accurate texture mapping is crucial. We have developed a highly accurate texture matching method. The try-on pipeline first generates target segmentation from the target pose for conditional information for the following stage's network models. Correct target segmentation was one of the main performance bottlenecks in previous VTON studies. Our proposed equalized entropy loss for the target segmentation generation network greatly reduces the segmentation label imbalance, and results in high-quality segmentation and reduced training time. The rigged reconstructed 3D clothing model can be easily deformed into the target pose and the human body shape while retaining the textures of the clothes. The remaining parts, i.e., the non-target clothing regions of the human in the target pose are generated through a deep generative human pose transfer model. Finally, the generated clothing and the remaining parts are combined using conditional generative networks to in-paint the dis-occluded areas and blend them together. Our proposed hybrid pipeline outperforms the previous 2D-based approaches by substantial margins in both objective evaluation and subjective user study, especially in the cases of the large pose and viewpoint changes.