Random Swin Transformer

After deep learning appeared, the convolutional neural network (CNN) dominated various applications of image classification, object detection, and semantic segmentation. Recently, a transformer based on various attention mechanisms performed better than the CNN. But, the transformer requires a large amount of memory for full attention among tokens. Recently, a Swin transformer has been proposed to solve that memory issue. It applies the attention per sub-regions on an image. Also, it solves a problem caused by not using full attention on an image by shifting window that guarantees more tokens are involved in attention. In this paper, we investigate a method of randomly selecting tokens in Swin transformer. We randomly choose tokens within a certain range rather than using a fixed shift value in the Swin transformer. Experimental results show the feasibility of the proposed method.