, NIQSV+ improves the original NIQSV by considering more distortion types: "black hole" and "stretching". The experimental resuls show that it achieves a gain of 7.68% in correlation with subjective measurement (PLCC) compared to the original NIQSV, and even outperforms the widely used FR and NR 2D IQA metrics, NR quality metric (NIQSV+) of the NIQSV, vol.101
, This model can be used on any pixel based quality assessment metric. The performance improvements of PSNR and SSIM are 36.85% and 13.33% respectively in terms of PLCC. Besides, the improved PSNR outperforms all the tested DIBR quality metrics including NR and FR significantly, A FR quality model for DIBR-synthesized views (SC-DM) (detailed in chapter
, NIQSV+: A No Reference Synthesized View Quality Assessment Metric, IEEE Transactions on Image Processing, vol.27, issue.4, pp.1652-1664, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01668626
, A benchmark of DIBR Synthesized View Quality Assessment Metrics on a new database for Immersive Media Applications". IEEE Transactions on Multimedia, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899625
, NIQSV: A No Reference Image Quality Assessment Metric for 3D Synthesized Views, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
URL : https://hal.archives-ouvertes.fr/hal-01622686
, A full-reference Image Quality Assessment metric for 3D Synthesized Views". Image Quality and System Performance Conference, at IS&T Electronic Imaging, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01637084
, Performance comparison of objective metrics on free-viewpoint videos with different depth coding algorithms, SPIE Optical Engineering + Applications, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899633
, SC-IQA: Shift compensation based image quality assessment for DIBR-synthesized views, IEEE international conference on Visual Communications and Image Processing (VCIP), 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899637
, A novel depth-based virtual view synthesis method for free viewpoint video, IEEE Transactions on Broadcasting, vol.59, pp.614-626, 2013.
, A survey of augmented reality, Presence: Teleoperators and virtual environments, vol.6, pp.355-385, 1997.
, Hypothesis testing, type I and type II errors, Industrial psychiatry journal, vol.18, p.127, 2009.
, Objective image quality assessment of 3D synthesized views, Signal Processing, vol.30, pp.78-88, 2015.
URL : https://hal.archives-ouvertes.fr/hal-01150648
, Surf: Speeded up robust features, European conference on computer vision, pp.404-417, 2006.
, Quality assessment of stereoscopic images, EURASIP journal on image and video processing, vol.1, p.659024, 2008.
URL : https://hal.archives-ouvertes.fr/hal-00362891
, Navier-stokes, fluid dynamics, and image and video inpainting, Computer Vision and Pattern Recognition, vol.1, pp.I-I, 2001.
, New quality evaluation method suited to multimedia context: Samviq, Proceedings of the Second International Workshop on Video Processing and Quality Metrics, VPQM, vol.6, 2006.
, An edge-based structural distortion indicator for the quality assessment of 3D synthesized views, 2012 Picture Coding Symposium, pp.249-252, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00724399
, Compression of Multi-View-plus-Depth (MVD) data: from perceived quality analysis to MVD coding tools designing, 2012.
URL : https://hal.archives-ouvertes.fr/tel-00777710
, A quality assessment protocol for Free-viewpoint video sequences synthesized from decompressed depth data, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), pp.100-105, 2013.
URL : https://hal.archives-ouvertes.fr/hal-01011243
, Towards a new quality metric for 3-D synthesized view assessment, IEEE Journal of Selected Topics in Signal Processing, vol.5, pp.1332-1343, 2011.
URL : https://hal.archives-ouvertes.fr/hal-00628070
, Visual quality assessment of synthesized views in the context of 3D-TV, 3D-TV system with depth-image-based rendering, pp.439-473, 2013.
URL : https://hal.archives-ouvertes.fr/hal-00748518
, The emerging MVC standard for 3D video services, EURASIP Journal on Applied Signal Processing, p.8, 2009.
, Spatio-Temporal Disocclusion Filling Using Novel Sprite Cells, IEEE Transactions on Multimedia, vol.20, pp.1376-1391, 2018.
, Objective view synthesis quality assessment, IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, pp.82881-82881, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00681304
, Region filling and object removal by exemplar-based image inpainting, IEEE Transactions on image processing, vol.13, pp.1200-1212, 2004.
, Learning a deep convolutional network for image superresolution, European conference on computer vision, pp.184-199, 2014.
, New full-reference quality metrics based on HVS, Proceedings of the Second International Workshop on Video Processing and Quality Metrics, vol.4, 2006.
, 3DSwIM Source Code, 2017.
, Perceptual quality assessment of 3D synthesized images, 2017 IEEE International Conference on, pp.505-510, 2017.
, A 3D-TV approach using Depth-image-based rendering (DIBR), Proc. of VIIP, vol.3, p.3, 2003.
, Depth-Image-Based Rendering (DIBR), compression, and transmission for a new approach on 3D-TV, Electronic Imaging, pp.93-104, 2004.
, Measurement of ringing artifacts in JPEG images, Proceedings of SPIE, vol.6076, pp.74-83, 2006.
, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Readings in computer vision, pp.726-740, 1987.
, Generative adversarial nets, Advances in neural information processing systems, pp.2672-2680, 2014.
, FINAL REPORT FROM THE VIDEO QUALITY EXPERTS GROUP ON THE VALIDATION OF OBJECTIVE MODELS OF MUL-TIMEDIA QUALITY ASSESSMENT, 2008.
, Deep learning network for blind image quality assessment, IEEE International Conference on, pp.511-515, 2014.
, Model-based referenceless quality metric of 3D synthesized images using local image description, IEEE Transactions on Image Processing, 2017.
, Quality assessment of a stereo pair formed from decoded and synthesized views using objective metrics, 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), pp.1-4, 2012.
, Binocular vision and stereopsis, 1995.
, Globally and locally consistent image completion, ACM Transactions on Graphics (TOG), vol.36, p.107, 2017.
, Human Factors of 3D Displays, p.9780470022733, 2006.
, Image-to-image translation with conditional adversarial networks, 2017.
, Methodology for the subjective assessment of video quality in multimedia applications, Rapport technique, International Telecommunication Union, 2007.
, IVC-IRCCyN lab, IRCCyN/IVC DIBR image database, 2017.
, Object-based Layered Depth Images for improved virtual view synthesis in rate-constrained context, 18th IEEE International Conference on, pp.125-128, 2011.
URL : https://hal.archives-ouvertes.fr/hal-00628013
, Salient object detection: A discriminative regional feature integration approach, Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pp.2083-2090, 2013.
, Light field compression using depth image based view synthesis, 2017 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pp.19-24, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01591329
, Critical binocular asymmetry measure for the perceptual quality assessment of synthesized stereo 3D images in view synthesis, IEEE Transactions on Circuits and Systems for Video Technology, vol.26, pp.1201-1214, 2016.
, Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering, 2010 IEEE International Conference on Image Processing, pp.1809-1812, 2010.
, On the accuracy of objective image and video quality models: New methodology for performance evaluation, Quality of Multimedia
URL : https://hal.archives-ouvertes.fr/hal-01395440
, A convolutional neural network approach for objective video quality assessment, IEEE Transactions on Neural Networks, vol.17, issue.5, pp.1316-1327, 2006.
URL : https://hal.archives-ouvertes.fr/hal-00287426
, Examplar-based inpainting based on local geometry, 18th IEEE International Conference on, pp.3401-3404, 2011.
, Quality Assessment of DIBR-Synthesized Images by Measuring Local Geometric Distortions and Global Sharpness, IEEE Transactions on Multimedia, vol.20, pp.914-926, 2018.
, Hole Filling with Multiple Reference Views in DIBR View Synthesis, IEEE Transactions on Multimedia, 2018.
, Key-Frame-Based Background Sprite Generation for Hole Filling in Depth Image-Based Rendering, IEEE Transactions on Multimedia, vol.20, pp.1075-1087, 2018.
, How to Learn the Effect of Non-Uniform Distortion on Perceived Visual Quality? Case Study Using Convolutional Sparse Coding for Quality Assessment of Synthesized Views, 25th IEEE International Conference on Image Processing (ICIP), pp.286-290, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01970888
, Image Quality Assessment for DIBR Synthesized Views using Elastic Metric, Proceedings of the 2017 ACM on Multimedia Conference, pp.1157-1163, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01971127
, Quality assessment for synthesized view based on variable-length context tree, Multimedia Signal Processing (MMSP), 2017 IEEE 19th International Workshop on, pp.1-6, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01675918
, Subjective and objective video quality assessment of 3D synthesized views with texture/depth compression distortion, IEEE Transactions on Image Processing, vol.24, pp.4847-4861, 2015.
, LIVE lab, LIVE Software Releases, 2017.
, A hole filling approach based on background reconstruction for view synthesis in 3D video, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1781-1789, 2016.
, End-to-end blind image quality assessment using deep neural networks, IEEE Transactions on Image Processing, vol.27, pp.1202-1213, 2018.
, On Constructing z-Dimensional DIBRSynthesized Images, IEEE Transactions on Multimedia, vol.18, pp.1453-1468, 2016.
, Multi-View Video Plus Depth Representation and Coding, 2007 IEEE International Conference on Image Processing, vol.1, pp.201-204, 2007.
, Making a completely blind image quality analyzer, IEEE Signal Processing Letters, vol.20, pp.209-212, 2013.
, A two-step framework for constructing blind image quality indices, IEEE Signal processing letters, vol.17, pp.513-516, 2010.
, Blind image quality assessment: From natural scene statistics to perceptual quality, IEEE transactions on Image Processing, vol.20, pp.3350-3364, 2011.
, Visual importance pooling for image quality assessment, IEEE journal of selected topics in signal processing, vol.3, pp.193-201, 2009.
, View generation with 3D warping using depth information for FTV, Signal Processing, vol.24, pp.65-72, 2009.
, View synthesis for advanced 3D video systems, EURASIP Journal on Image and Video Processing, vol.1, pp.1-11, 2008.
, Depth image based rendering with advanced texture synthesis, 2010 IEEE International Conference on Multimedia and Expo (ICME), pp.424-429, 2010.
, Depth image-based rendering with advanced texture synthesis for 3-D video, IEEE Transactions on Multimedia, vol.13, pp.453-465, 2011.
, On between-coefficient contrast masking of DCT basis functions, Proceedings of the third international workshop on video processing and quality metrics, vol.4, 2007.
, Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
, 500-11, Methodology for the Subjective Assessment of the Quality of Television Pictures, Recommendation ITU-R BT. 500-11, ITURBT Recommendation, vol.7, 2002.
, Anchor view allocation for collaborative free viewpoint video streaming, IEEE Transactions on Multimedia, vol.17, pp.307-322, 2015.
, Virtual reality: exploring the brave new technologies, 1991.
, Blind image quality assessment: A natural scene statistics approach in the DCT domain, IEEE Transactions on Image Processing, vol.21, pp.3339-3352, 2012.
, DCT statistics modelbased blind image quality assessment, 18th IEEE International Conference on Image Processing (ICIP), pp.3093-3096, 2011.
, Dataset and Pipeline for Multi-view Light-Field Video, Computer Vision and Pattern Recognition Workshops, pp.1743-1753, 2017.
, Free viewpoint video quality assessment based on morphological multiscale metrics, 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), pp.1-6, 2016.
, , 2017.
, DIBR-synthesized image quality assessment based on morphological multi-scale approach, EURASIP Journal on Image and Video Processing, vol.1, p.4, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01675151
, DIBR synthesized image quality assessment based on morphological pyramids, 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), pp.1-4, 2015.
, DIBR synthesized image quality assessment based on morphological wavelets, p.2015
, Seventh International Workshop onQuality of Multimedia Experience (QoMEX), pp.1-6, 2015.
, Multi-Scale Synthesized View Assessment Based on Morphological Pyramids, Journal of Electrical Engineering, vol.67, pp.3-11, 2016.
, No reference image quality assessment for JPEG2000 based on spatial features, Signal Processing, vol.23, pp.257-268, 2008.
, Middlebury stereo datasets, 2012.
, Faithful disocclusion filling in depth image based rendering using superpixel-based inpainting, IEEE Transactions on Multimedia, vol.17, pp.2160-2173, 2015.
, Feature based volumetric video compression for interactive playback, Proceedings of the 2002 IEEE symposium on Volume visualization and graphics, pp.89-96, 2002.
, 3VQM: A vision-based quality measure for DIBR-based 3D videos, 2011 IEEE International Conference on Multimedia and Expo, pp.1-6, 2011.
, A no-reference quality measure for DIBR-based 3D videos, 2011 IEEE International Conference on Multimedia and Expo (ICME), pp.1-6, 2011.
, Hierarchical hole-filling for depth-based view synthesis in FTV and 3D video, IEEE Journal of Selected Topics in Signal Processing, vol.6, pp.495-504, 2012.
, MCL-3D: A database for stereoscopic image quality assessment using 2D-image-plus-depth source, 2014.
, Image processing, analysis, and machine vision, 2014.
, Standardized extensions of high efficiency video coding (HEVC), IEEE Journal of selected topics in Signal Processing, vol.7, issue.6, pp.1001-1016, 2013.
, Learning a blind measure of perceptual image quality, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.305-312, 2011.
, Free-viewpoint TV, IEEE Signal Processing Magazine, vol.28, pp.67-76, 2011.
, Reference softwares for depth estimation and view synthesis, p.15377, 2008.
, Overview of the multiview and 3D extensions of high efficiency video coding, IEEE Transactions on Circuits and Systems for Video Technology, vol.26, pp.35-49, 2016.
, An image inpainting technique based on the fast marching method, Journal of graphics tools 9, pp.23-34, 2004.
, The Future of Video: Enabling Immersion
, View synthesis techniques for 3D video, SPIE Optical Engineering+ Applications, International Society for Optics and Photonics, pp.74430-74430, 2009.
, View synthesis techniques for 3D video, SPIE Optical Engineering+ Applications, International Society for Optics and Photonics, pp.74430-74430, 2009.
, A benchmark of DIBR Synthesized View Quality Assessment Metrics on a new database for Immersive Media Applications, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899625
, A full-reference Image Quality Assessment metric for 3D Synthesized Views, Image Quality and System Performance Conference, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01637084
, NIQSV: A NO REFERENCE IMAGE QUALITY ASSESS-MENT METRIC FOR 3D SYNTHESIZED VIEWS, IEEE International Conference on Acoustics, Speech and Signal Processing, 2017.
, NIQSV+: A No-Reference Synthesized View Quality Assessment Metric, IEEE Transactions on Image Processing, vol.27, pp.1652-1664, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01668626
, Performance comparison of objective metrics on freeviewpoint videos with different depth coding algorithms, Applications of Digital Image Processing XLI, vol.10752, p.107520, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899633
, SC-IQA: Shift compensation based image quality assessment for DIBR-synthesized views, IEEE International Conference on Visual Communications and Image Processing, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01899637
, CR-GAN: learning complete representations for multi-view generation, 2018.
, Overview of the stereo and multiview video coding extensions of the H. 264/MPEG-4 AVC standard, Proceedings of the IEEE 99, vol.4, pp.626-642, 2011.
, Quality prediction of asymmetrically distorted stereoscopic 3D images, IEEE Transactions on Image Processing, vol.24, pp.3400-3414, 2015.
, A universal image quality index, IEEE Signal Processing Letters, vol.9, pp.1070-9908, 2002.
, Modern image quality assessment, Synthesis Lectures on Image, Video, and Multimedia Processing, vol.2, pp.1-156, 2006.
, Blind measurement of blocking artifacts in images, International Conference on Image Processing, vol.3, pp.981-984, 2000.
, Information content weighting for perceptual image quality assessment, IEEE Transactions on Image Processing, vol.20, pp.1185-1198, 2011.
, Multiscale structural similarity for image quality assessment, Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers, vol.2, pp.1398-1402, 2003.
, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, vol.13, pp.1057-7149, 2004.
, Inter-view consistent hole filling in view extrapolation for multi-view image generation, Image Processing (ICIP), 2014 IEEE International Conference on, pp.2883-2887, 2014.
, A survey of hybrid MC/DPCM/DCT video coding distortions, Signal processing, vol.70, pp.247-278, 1998.
, , 2017.
, Multi-view image generation from a single-view, 2018 ACM Multimedia Conference on Multimedia Conference, pp.383-391, 2018.
, A perceptual metric for evaluating quality of synthesized sequences in 3DV system, Visual Communications and Image Processing 2010, International Society for Optics and Photonics, pp.77440-77440, 2010.
, Stereo Magnification: Learning view synthesis using multiplane images, 2018.
, Scale-aware edge-preserving image filtering via iterative global optimization, IEEE Transactions on Multimedia, 2017.
, Depth image based view synthesis: New insights and perspectives on hole generation and filling, IEEE Transactions on Broadcasting, vol.62, pp.82-93, 2016.
, A no-reference sharpness metric sensitive to blur and noise, 2009 International Workshop on Quality of Multimedia Experience, 2009.
, IEEE, pp.64-69, 2009.
,
, 13 3 Example of Canon's Free Viewpoint Video System, p.15, 2001.
, Examples of depth information from monocular cues, p.22
, Panum's Area of Fusion
, Example of view synthesis
, 26 1.5 Object shifting caused by depth low-passing filter, the right borders of the character's faces are slightly modified
, Example of ghosting effect distortions caused by Gaussian noised depth map, p.28
, Example of object warping distortion caused by imperfect image inpainting method, the "newspaper" and the "girl's nose" are extremely stretched, and their shapes are greatly changed
, Synthesis distortions caused by imperfect image inpainting method. (a) and (b) stretching; (c) and (d) blurry regions. The "girl's " hair and clothes in image (b); and the right border of the sculpture in (d), p.29
, The Crumbling in the "chair arms" in image (b) and black hole around the "man's arm", 10 Categories of quality assessment metrics for DIBR-synthesized views, p.32
, Scheme of MP-PSNR, m r and m e represent the reduce and expand filter respectively. 9
, 47 1.14 Example relationship between DMOS and objective quality scores, p.51
, Example of feature points matching and transform, p.54
, Example of dis-occluded mask
, 58 2.4 Block matching: (a), (b) are the patches in the synthesized and the reference image; (c) block in the synthesized image; (d) matched block in the reference image: for direct 8x8 block-matching (red block), or multiresolution block-matching (green)
, Scatter plot of DMOS versus the predicted quality score, p.62
, Performance dependency of the proposed metric with the changing ratios (?%)
, Scatter plot of DMOS versus the predicted quality score on MCL-3D database
, Scatter plot of SC-IQA (? = 1) quality score versus DMOS on MCL-3D database
, 68 3.2 Examples of morphological operations on binary images, p.69
, Examples of morphological operations on gray level images, p.70
,
, Examples of intermediate results in the NIQSV measurement for one synthesized view in the "Newspaper"sequence. The distortions marked in (a) are well detected in (d), while in the non-distortion regions, such as the girl's hair, the distortion values are very low
,
, Synthesized Image and its corresponding average horizontal gradient, p.77
, Scatter plots of DMOS versus DMOSp of each IQA method, p.80
, Proposed quality metric scores of reference image and images synthesized by A2
,
,
,
,
, The subscript L means this virtual view is synthesized from the neighboring left view, while the subscript R means from the right. V SRS2 denotes the view interpolation inter-view mode of VSRS. The error bars indicates the corresponding confidence intervals of the tested images. The bars referring to interview synthesize views are marked by red, the leftextrapolated views marked by green, and the right-extrapolated views marked by blue
, 108 4.11 Scatter plots of DMOS versus DMOSp of each IQA method, p.112
, 13 Results of different/similar on IETR database (Metric 1-20 represent PSNR, vol.114, p.115
, Results of better/worse on IETR database (Metric 1-20 represent
, Results of different/similar on IVC database (Metric 1-20 represent
, 118 of the existing metrics. The features in the first column indicate frequency domain feature (FF), contour/gradient, JND, Multi-scale decomposition (MSD), local image description (LID), p.33
, 63 2.2 Performance comparison of the proposed method with the state-of-theart metrics on MCL-3D database
, , p.83
, The best three results are marked in bold), NIQSV+_s means NIQSV with stretching detection, NIQSV+_b means NIQSV with black hole detection 84
, The green colored algorithms indicate that the changing of ranking position is acceptable; the blue ones are medium; and the red ones are non acceptable
, Execution time of each IQA metric normalized base on PSNR. The metrics A-Z indicate NIQSV+
, Res. Var. denotes Residual Variance), the metrics A-Z indicate NIQSV+, p.86
, The symbol "1" indicates that the statistical performance of the IQA metric in the row is significantly superior to the one in the column, the symbol "-1" means the opposite, while "0" indicates that there is no significant difference between the metrics in the row and in the column. The metrics A-Z indicate NIQSV+, Statistical significance table based on residuals between model predictions DM OS p and DM OS, vol.87
, 93 4.2 Introduction of the tested MVD sequences
,
,
, where the symbol 1 indicates that the DIBR synthesis method in the row is significantly superior to the one in the column, the symbol -1 means the opposite, while 0 indicates that there is no significant difference between the DIBR synthesis methods in the row and in the column, Student T-test with obtained DM OS scores
, SV FR metric" indicates the side view based FR metric, p.111
, SE" indicates Structural Element size