Shiming Chen1,
Peng Zhang1,
Qinmu Peng1,
Guo-Sen Xie2,
Zehong Cao3,
Wei Yuan1,
and Xinge You1,
1Huazhong University of Science and Technology (HUST), China 2 Nanjing University of Science and Technology, China 3University of South Australia, Australia
{shimingchen, zp_zhg, youxg, pengqinmu}@hust.edu.cn ling.shao@ieee.org
Dynamic texture (DT) exhibits statistical stationarity in the spatial domain and stochastic repetitiveness in the temporal dimension, indicating that different frames of DT possess a high similarity correlation that is critical prior knowledge. However, existing methods cannot effectively learn a synthesis model for high-dimensional DT from a small number of training samples. In this paper, we propose a novel DT synthesis method, which makes full use of similarity as prior knowledge to address this issue. Our method is based on the proposed kernel similarity embedding, which can not only mitigate the high-dimensionality and small sample issues, but also has the advantage of modeling nonlinear feature relationships. Specifically, we first put forward two hypotheses that are essential for the DT model to generate new frames using similarity correlations. Then, we integrate kernel learning and the extreme learning machine into a unified synthesis model to learn kernel similarity embeddings for representing DTs. Extensive experiments on DT videos collected from the internet and two benchmark datasets, i.e., Gatech Graphcut Textures and Dyntex, demonstrate that the learned kernel similarity embeddings can provide discriminative representations for DTs. Further, our method can preserve the long-term temporal continuity of the synthesized DT sequences with excellent sustainability and generalization. Meanwhile, it effectively generates realistic DT videos with higher speed and lower computation than the current state-of-the-art methods. The code and more synthesis videos are available at our project page https://shiming-chen.github.io/Similarity-page/Similarit.html.
There are two different benchmark datasets used for evaluating our method: Gatech Graphcut Textures dataset and Dyntex dataset. Some of these generated dynamic texture videos are used in paper, in which supports better vision quality. Indeed, we also evaluate our method on others DTs in-the-wild. If you are interested in the quantitative evaluation results (PSNR, SSIM), please take it from our paper.
In each example, the first one is the observed video, the others are the generated videos synthesized by Similarity-DT using different kernel function (left-to-right: Linear kernel, Rational Quadratic kernel, Polynomial kernel, Multiquadric kernel, Sigmoid kernel, Gaussian kernel)
In each row, the first three are the observed videos with 200 frames, the other three are the synthesized videos generated by Similarity-DT with 1000 frames. Here, we display 18 dynamic texture sequences of 6 classes (top-to-bottom: bulb, elevator, flowers swaying in the current, rotating wind ornament, water wave, windmill).
In each group, the first row displays the observed videos (used for testing). As for other rows, the first video is the observed video (used for training), the others are the synthesized videos corresponding to the first row.
The first group displays observed DT videos, the others are the generated videos synthesized by different methods (top-to-bottom: Two-Stream [4], STGCN [3], DG [2], ours(Similarity-DT))
This work was supported in part by the National Natural Science Foundation of China~(61571205 and 61772220), the Key Program for International S\&T Cooperation Projects of China~(2016YFE0121200), the Special Projects for Technology Innovation of Hubei Province~(2018ACA135), the Key Science and Technology Innovation Program of Hubei Province~(2017AAA017), the Natural Science Foundation of Hubei Province~(2018CFB691), fund from Science, Technology and Innovation Commission of Shenzhen Municipality~(JCYJ20180305180637611, JCYJ20180305180804836 and JSGG20180507182030600). We thank Dr. Jianwen Xie for his suggestions.
[1] Xinge You et al. "Kernel Learning for Dynamic Texture Synthesis." IEEE Transactions on Image Processing (TIP), 2016.
[2] Jianwen Xie et al. "Learning Dynamic Generator Model by Alternating Back-Propagation through Time." In AAAI, 2019.
[3] Jianwen Xie et al. "Energy-based spatial-temporal generative convNet." IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
[4] Matthew Tesfaldet et al. "Two-stream convolutional networks for dynamic texture synthesis." In CVPR, 2018.
[5] Gatys Leon A. et al. "Texture synthesis using convolutional neural networks." In NeurIPS, 2015.
[6] Gianfranco Doretto et al. "Dynamic textures." International Journal of Computer Vision (IJCV), 2003.