Shot Boundary Detection with 3D Depthwise Convolutions and Visual Attention
Author:
Esteve Brotons Miguel Jose1ORCID, Lucendo Francisco Javier1, Javier Rodriguez-Juan2ORCID, Garcia-Rodriguez Jose2ORCID
Affiliation:
1. Telefónica I+D, 28050 Madrid, Spain 2. Computers Technology Department, University of Alicante, 03080 Alicante, Spain
Abstract
Shot boundary detection is the process of identifying and locating the boundaries between individual shots in a video sequence. A shot is a continuous sequence of frames that are captured by a single camera, without any cuts or edits. Recent investigations have shown the effectiveness of the use of 3D convolutional networks to solve this task due to its high capacity to extract spatiotemporal features of the video and determine in which frame a transition or shot change occurs. When this task is used as part of a scene segmentation use case with the aim of improving the experience of viewing content from streaming platforms, the speed of segmentation is very important for live and near-live use cases such as start-over. The problem with models based on 3D convolutions is the large number of parameters that they entail. Standard 3D convolutions impose much higher CPU and memory requirements than do the same 2D operations. In this paper, we rely on depthwise separable convolutions to address the problem but with a scheme that significantly reduces the number of parameters. To compensate for the slight loss of performance, we analyze and propose the use of visual self-attention as a mechanism of improvement.
Funder
European Union NextGenerationEU/ PRTR HORIZON-MSCA-2021-SE-0
Subject
Electrical and Electronic Engineering,Biochemistry,Instrumentation,Atomic and Molecular Physics, and Optics,Analytical Chemistry
Reference33 articles.
1. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv. 2. Chollet, F. (2017). Xception: Deep Learning with Depthwise Separable Convolutions. arXiv. 3. Hu, J., Shen, L., Albanie, S., Sun, G., and Wu, E. (2019). Squeeze-and-Excitation Networks. arXiv. 4. Li, X., Wang, W., Hu, X., and Yang, J. (2019). Selective Kernel Networks. arXiv. 5. Souček, T., Moravec, J., and Lokoč, J. (2019). TransNet: A deep network for fast detection of common shot transitions. arXiv.
|
|