]. O. Bibliographie1, D. G. Agazzi, D. A. Messerschmitt, and . Hodges, Non linear echo cancellation of data signals, IEEE Trans. Commun, vol.30, issue.11, pp.2421-2433, 1982.

B. G. Agee, S. V. Schell, and W. A. Gardner, Spectral self-coherence restoral: a new approach to blind adaptive signal extraction using antenna arrays, Proceedings of the IEEE, vol.78, issue.4, pp.753-767, 1990.
DOI : 10.1109/5.54812

A. Ahmed and A. M. , All-Digital Self-Interference Cancellation Technique for Full-Duplex Systems, IEEE Transactions on Wireless Communications, vol.14, issue.7, pp.3519-3532, 2015.
DOI : 10.1109/TWC.2015.2407876

P. O. Amblard, M. Gaeta, and J. L. Lacume, Statistics for complex variables and signals ??? Part I: Variables, Signal Processing, vol.53, issue.1, pp.1-25, 1996.
DOI : 10.1016/0165-1684(96)00071-0

L. Antilla and M. Valkama, Blind Signal Estimation in Widely-Linear Signal Models With Fourth-Order Circularity: Algorithms and Application to Receiver I/Q Calibration, IEEE Signal Processing Letters, vol.20, issue.3, pp.221-224, 2013.
DOI : 10.1109/LSP.2012.2230625

M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE Transactions on Signal Processing, vol.50, issue.2, pp.174-188, 2002.
DOI : 10.1109/78.978374

S. Benedetto, E. Bigieri, and R. Daffara, Modeling and performance evaluation of non-linear satellite links -A Volterra series approach, IEEE Trans. Aerosp. Elect. Syst, vol.15, pp.494-507, 1979.

J. Benesty, J. Chen, and Y. Huang, Binaural Noise Reduction in the Time Domain With a Stereo Setup, IEEE Transactions on Audio, Speech, and Language Processing, vol.19, issue.8, pp.2260-2272, 2011.
DOI : 10.1109/TASL.2011.2119313

E. Biglieri, A. Gersho, R. D. Gitlin, and T. L. Lim, Adaptive Cancellation of Nonlinear Intersymbol Interference for Voiceband Data Transmission, IEEE Journal on Selected Areas in Communications, vol.2, issue.5, pp.765-777, 1984.
DOI : 10.1109/JSAC.1984.1146107

G. Bottomley and S. Chennakeshu, Unification of MLSE receivers and extension to time-varying channels, IEEE Transactions on Communications, vol.46, issue.4, pp.464-472, 1998.
DOI : 10.1109/26.664302

J. Capon, R. J. Greenfield, and R. J. Kolker, Multidimensional maximum-likelihood processing of a large aperture seismic array, Proc. IEEE, pp.191-211, 1967.
DOI : 10.1109/PROC.1967.5439

J. Capon, High resolution frequency wave number spectrum analysis, Proc. IEEE, pp.1408-1418, 1969.

J. Chen and J. Benesty, On the Time-Domain Widely Linear LCMV Filter for Noise Reduction With a Stereo System, IEEE Transactions on Audio, Speech, and Language Processing, vol.21, issue.7, pp.1343-1354, 2013.
DOI : 10.1109/TASL.2013.2248719

V. G. Chavali, Signal detection and modulation classification in non-gaussian noise environments, Thèse de doctorat non publiée, 2012.

P. Chevalier, P. Duvaut, and B. Picinbono, Le filtrage de Volterra transverse réel et complexe en traitement du signal Traitement du Signal, Numéro spécial " Non linéaire et non gaussien, pp.451-476, 1990.

P. Chevalier, P. Duvaut, and B. Picinbono, Complex transversal Volterra filters optimal for detection and estimation, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing, pp.3537-3540, 1991.
DOI : 10.1109/ICASSP.1991.150234

P. Chevalier and B. Picinbono, Complex linear-quadratic systems for detection and array processing, IEEE Transactions on Signal Processing, vol.44, issue.10, pp.2631-2634, 1996.
DOI : 10.1109/78.539049

P. Chevalier and A. Maurice, Constrained beamforming for cyclostationary signals, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp.3789-3792, 1997.
DOI : 10.1109/ICASSP.1997.604702

P. Chevalier, L. Albera, A. Ferreol, and P. Comon, On the virtual array concept for higher order array processing, IEEE Transactions on Signal Processing, vol.53, issue.4, pp.1254-1271, 2005.
DOI : 10.1109/TSP.2005.843703

P. Chevalier and F. Pipon, New insights into optimal widely linear array receivers for the demodulation of BPSK, MSK, and GMSK signals corrupted by noncircular interferences-application to SAIC, IEEE Transactions on Signal Processing, vol.54, issue.3, pp.870-883, 2006.
DOI : 10.1109/TSP.2005.862946

P. Chevalier and A. Blin, Widely Linear MVDR Beamformers for the Reception of an Unknown Signal Corrupted by Noncircular Interferences, IEEE Transactions on Signal Processing, vol.55, issue.11, pp.5323-5336, 2007.
DOI : 10.1109/TSP.2007.899394

P. Chevalier, J. P. Delmas, and A. Oukaci, Optimal widely linear MVDR beamforming for noncircular signals, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 2009.
DOI : 10.1109/ICASSP.2009.4960398
URL : https://hal.archives-ouvertes.fr/hal-01361058

P. Chevalier, A. Oukaci, and J. P. Delmas, Third order widely nonlinear Volterra MVDR beamforming, Proc. ICASSP, 2011.
DOI : 10.1109/icassp.2011.5947029
URL : https://hal.archives-ouvertes.fr/hal-01126072

P. Chevalier, J. P. Delmas, and A. Oukaci, Properties, performance and practical interest of the widely linear MMSE beamformer for nonrectilinear signals, Signal Processing, vol.97, pp.269-281, 2014.
DOI : 10.1016/j.sigpro.2013.09.017
URL : https://hal.archives-ouvertes.fr/hal-01285813

P. Comon, R. Liu, and D. Slock, Path-wise wide-sense polynomial receiver for UMTS communications, Proc. 39th Allerton Conf. Com. Contr. Comput., Allerton, Illinois (USA), 2001.

J. P. Delmas, A. Oukaci, and P. Chevalier, On the asymptotic distribution of GLR for impropriety of complex signals, Signal Processing, vol.91, issue.10, pp.2259-2267, 2011.
DOI : 10.1016/j.sigpro.2011.04.007
URL : https://hal.archives-ouvertes.fr/hal-00664248

J. P. Delmas, P. Chevalier, S. Sallem, and M. Sadok, R??cepteurs SIMO MLSE Widely Linear. Structures et performances en pr??sence d???interf??rences non circulaires, Traitement du signal, vol.32, issue.1, pp.7-37, 2015.
DOI : 10.3166/ts.32.7-37

P. M. Djuric, J. H. Kotecha, J. Zhang, Y. Huang, T. Ghirmai et al., Particle Filtering, IEEE Signal Processing Magazine, vol.20, issue.5, pp.19-38, 2003.
DOI : 10.1109/MSP.2003.1236770
URL : https://hal.archives-ouvertes.fr/hal-01437041

B. A. Dumitrescu, Positive Trigonometric Polynomials and Signal Processing Applications, 2007.
DOI : 10.1007/978-3-319-53688-0

J. Eriksson and V. Koivunen, Complex random vectors and ICA models: identifiability, uniqueness, and separability, IEEE Transactions on Information Theory, vol.52, issue.3, pp.1017-1029, 2006.
DOI : 10.1109/TIT.2005.864440
URL : http://arxiv.org/pdf/cs/0512063

J. Eriksson, E. Ollila, and V. Koivunen, Essential Statistics and Tools for Complex Random Variables, IEEE Transactions on Signal Processing, vol.58, issue.10, pp.5400-5408, 2010.
DOI : 10.1109/TSP.2010.2054085

J. Feinstein and Y. Shamash, Spectral factorization of a rational matrix (Corresp.), IEEE Transactions on Information Theory, vol.23, issue.4, pp.534-538, 1977.
DOI : 10.1109/TIT.1977.1055743

G. Forney, Maximum-likelihood sequence estimation of digital sequences in the presence of intersymbol interference, IEEE Transactions on Information Theory, vol.18, issue.3, pp.363-378, 1972.
DOI : 10.1109/TIT.1972.1054829

O. L. Frost and . Iii, An algorithm for linearly constrained adaptive array processing, Proc. IEEE, pp.926-935, 1972.
DOI : 10.1109/PROC.1972.8817

P. Forster, T. Asté, and L. Fety, Multisensors receivers using a filtered reference: application to GSM, ICUPC '98. IEEE 1998 International Conference on Universal Personal Communications. Conference Proceedings (Cat. No.98TH8384)
DOI : 10.1109/ICUPC.1998.733029

W. A. Gardner, A. Napolitano, and L. Paura, Cyclostationarity: Half a century of research, Signal Processing, vol.86, issue.4, pp.641-697, 2006.
DOI : 10.1016/j.sigpro.2005.06.016

L. M. Garth and H. V. Poor, Narrowband interference suppression in impulsive channels, IEEE Transactions on Aerospace and Electronic Systems, vol.28, issue.1, pp.15-34, 1992.
DOI : 10.1109/7.135430

L. J. Griffiths and C. W. Jim, An alternative approach to linearly constrained adaptive beamforming, IEEE Transactions on Antennas and Propagation, vol.30, issue.1, pp.27-34, 1982.
DOI : 10.1109/TAP.1982.1142739

A. Hakkarainen, J. Werner, K. R. Dandekar, and M. Valkama, Widely-linear beamforming and RF impairment suppression in massive antenna arrays, Journal of Communications and Networks, vol.15, issue.4, pp.383-397, 2013.
DOI : 10.1109/JCN.2013.000069

S. Han, C. , Z. Xu, and C. Rowell, Large-scale antenna systems with hybrid analog and digital beamforming for millimeter wave 5G, IEEE Communications Magazine, vol.53, issue.1, pp.186-194, 2015.
DOI : 10.1109/MCOM.2015.7010533

D. A. Harville, Matrix algebra from a statistician's perspective, 1997.

R. Hu and H. M. Ahmed, Echo cancellation in high speed data transmission systems using adaptive layered bilinear filters, IEEE Trans. Commun, vol.42, issue.234, 1994.

L. Huang, Z. Zhang, L. Zhang, and Z. Ye, Widely linear minimum dispersion beamforming for sub-Gaussian noncircular signals, Signal Processing, vol.122, pp.123-128, 2016.
DOI : 10.1016/j.sigpro.2015.10.012

X. Jiang, W. J. Zeng, A. Yasotharan, H. C. So, and T. Kirubarajan, Minimum Dispersion Beamforming for Non-Gaussian Signals, IEEE Transactions on Signal Processing, vol.62, issue.7, pp.1879-1893, 2014.
DOI : 10.1109/TSP.2014.2305639

C. Jutten and P. Comon, Handbook of blind source separation independent component analysis and applications, 2010.
URL : https://hal.archives-ouvertes.fr/hal-00460653

S. Kanna, M. Xiang, and D. P. Mandic, Real-time detection of rectilinear sources for wireless communication signals, 2015 International Symposium on Wireless Communication Systems (ISWCS), 2015.
DOI : 10.1109/ISWCS.2015.7454403

S. A. Kassam and J. B. Thomas, Signal detection in non-gaussian noise series, 1988.

T. Koh and E. J. Powers, Second order Volterra filtering and its application to non linear system identification, IEEE Trans. Acou. Speech Signal Proc, vol.33, issue.6, pp.1445-1455, 1985.

P. A. Laurent, Exact and Approximate Construction of Digital Phase Modulations by Superposition of Amplitude Modulated Pulses (AMP), IEEE Transactions on Communications, vol.34, issue.2
DOI : 10.1109/TCOM.1986.1096504

G. Lazzarin, S. Pupolin, and A. Sarti, Nonlinearity compensation in digital radio systems, IEEE Transactions on Communications, vol.42, issue.2/3/4, pp.988-998, 1994.
DOI : 10.1109/TCOMM.1994.580207

E. L. Lehmann, Elements of large sample theory, 1998.
DOI : 10.1007/b98855

E. Lindskog, Multi-channel maximum likelihood sequence estimation, 1997 IEEE 47th Vehicular Technology Conference. Technology in Motion, pp.715-719, 1997.
DOI : 10.1109/VETEC.1997.600422

J. R. Magnus and H. Neudecker, Matrix differential calculus with applications in statistics and econometrics, Wiley series in probability and statistics, 1999.

R. Meyer, W. H. Gerstacker, R. Schober, and J. B. Huber, A single antenna interference cancellation algorithm for increased gsm capacity, IEEE Transactions on Wireless Communications, vol.5, issue.7, pp.1616-1621, 2006.
DOI : 10.1109/TWC.2006.1673070

G. Mileounis and N. Kalouptsidis, Blind Identification of Second Order Volterra Systems With Complex Random Inputs Using Higher Order Cumulants, IEEE Transactions on Signal Processing, vol.57, issue.10, pp.4129-4135, 2009.
DOI : 10.1109/TSP.2009.2023372

J. W. Modestino and M. V. Eyuboglu, Integrated multielement receiver structures for spatially distributed interference channels, IEEE Transactions on Information Theory, vol.32, issue.2, pp.195-219, 1986.
DOI : 10.1109/TIT.1986.1057158

A. Mohammadi and K. N. Plataniotis, Complex-Valued Gaussian Sum Filter for Nonlinear Filtering of Non-Gaussian/Non-Circular Noise, IEEE Signal Processing Letters, vol.22, issue.4, pp.440-444, 2015.
DOI : 10.1109/LSP.2014.2361459

D. Molteni, M. Nicoli, and G. Edge, Joint OSC Receiver for Evolved GSM/EDGE Systems, IEEE Transactions on Wireless Communications, vol.12, issue.6, pp.2608-2619, 2013.
DOI : 10.1109/TWC.2013.050613.112296

A. Mostafa, R. Kobylinski, I. Kostanic, and M. Austin, Single antenna interference cancellation (SAIC) for GSM networks, 2003 IEEE 58th Vehicular Technology Conference. VTC 2003-Fall (IEEE Cat. No.03CH37484), pp.1089-1093, 2004.
DOI : 10.1109/VETECF.2003.1285189

. Murosa, 3GPP TSG Geran Tdoc GP-090114 MUROS uplink receiver performance, 2009.

. Murosb, 3GPP TSG Geran Tdoc GP-090115 MUROS downlink receiver performance for interference and sensitivity, 2009.

R. D. Nowak and B. D. Van-veen, Volterra filter equalization: a fixed point approach, IEEE Transactions on Signal Processing, vol.45, issue.2, pp.377-388, 1997.
DOI : 10.1109/78.554302

J. C. Olivier and W. Kleynhans, Single antenna interference cancellation for synchronised GSM networks using a widely linear receiver, IET Communications, vol.1, issue.1, pp.131-136, 2007.
DOI : 10.1049/iet-com:20050435

E. Ollila and V. Koivunen, Generalized complex elliptical distributions, Processing Workshop Proceedings, 2004 Sensor Array and Multichannel Signal, 2004.
DOI : 10.1109/SAM.2004.1502990

R. C. Paiva, R. D. Vieira, R. Lida, F. M. Tavares, M. Saily et al., GSM voice evolution using orthogonal subchannels, IEEE Commun. Magazine, pp.80-86, 2012.

B. Pincibono and P. Duvaut, Geometrical properties of optimal Volterra filters for signal detection, IEEE Transactions on Information Theory, vol.36, issue.5, pp.1061-1068, 1990.
DOI : 10.1109/18.57205

B. Picinbono, On circularity, IEEE Transactions on Signal Processing, vol.42, issue.12, pp.3473-3482, 1994.
DOI : 10.1109/78.340781
URL : https://hal.archives-ouvertes.fr/hal-01660497

B. Picinbono and P. Chevalier, Widely linear estimation with complex data, IEEE Transactions on Signal Processing, vol.43, issue.8, pp.2030-2033, 1995.
DOI : 10.1109/78.403373

B. Pincibono and P. Duvaut, Optimal linear-quadratic systems for detection and estimation, IEEE Transactions on Information Theory, vol.34, issue.2, pp.304-311, 1988.
DOI : 10.1109/18.2638

S. M. Razavizadeh, M. Ahn, and I. Lee, Three-Dimensional Beamforming: A new enabling technology for 5G wireless networks, IEEE Signal Processing Magazine, vol.31, issue.6, pp.94-101, 2014.
DOI : 10.1109/MSP.2014.2335236

I. S. Reed, J. D. Mallet, and L. E. Brennan, Rapid Convergence Rate in Adaptive Arrays, IEEE Transactions on Aerospace and Electronic Systems, vol.10, issue.6, pp.853-863, 1974.
DOI : 10.1109/TAES.1974.307893

W. Roh, J. Y. Seol, J. Park, B. Lee, J. Lee et al., Millimeter-wave beamforming as an enabling technology for 5G cellular communications: theoretical feasibility and prototype results, IEEE Communications Magazine, vol.52, issue.2, pp.106-113, 2014.
DOI : 10.1109/MCOM.2014.6736750

M. Ruder, R. Meyer, F. Obernosterer, H. Kalveram, R. Schober et al., Receiver Concepts and Resource Allocation for OSC Downlink Transmission, IEEE Transactions on Wireless Communications, vol.13, issue.3, pp.1568-1581, 2014.
DOI : 10.1109/TWC.2014.012814.131015

M. Sadok, J. Delmas, and P. Chevalier, Enhanced single antenna interference cancellation from MMSE third-order complex Volterra filters, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
DOI : 10.1109/ICASSP.2017.7952947
URL : https://hal.archives-ouvertes.fr/hal-01629454

M. Sadok, J. Delmas, and P. Chevalier, SAIC pour une interférence circulaire au secondordre par filtrage de Volterra complexe d'ordre 3, XXVIe Colloque sur le Traitement du Signal et des Images du GRETSI, pp.5-8, 2017.

P. J. Schreier and L. L. Scharf, Statistical Signal Processing of Complex-Valued Data : The Theory of Improper and Noncircular Signals, 2012.
DOI : 10.1017/CBO9780511815911

M. Schetzen, The Volterra and Wiener theory of non linear systems, 1980.

P. J. Schreier, L. L. Scharf, and A. Hanssen, A generalized likelihood ratio test for impropriety of complex signals, IEEE Signal Processing Letters, vol.13, issue.7, pp.433-436, 2006.
DOI : 10.1109/LSP.2006.871858

P. J. Schreir and L. L. Scharf, Statistical signal processing of complex-valued data, the theory of improper and noncircular signals, 2010.

M. Schwartz, Abstract vector spaces applied to problems in detection and estimation theory, IEEE Transactions on Information Theory, vol.12, issue.3, pp.327-336, 1966.
DOI : 10.1109/TIT.1966.1053900

R. J. Serfling, Approximation Theorems of Mathematical Statistics, 1980.

T. L. Singal, Wireless Communications, 2010.

N. Song, W. U. Alokozai, R. C. De-lamare, and M. Haardt, Adaptive Widely Linear Reduced-Rank Beamforming Based on Joint Iterative Optimization, IEEE Signal Processing Letters, vol.21, issue.3, pp.265-269, 2014.
DOI : 10.1109/LSP.2013.2295943

A. Souloumiac, P. Chevalier, and C. Demeure, Improvement in non-Gaussian jammers rejection with a non-linear spatial filter, IEEE International Conference on Acoustics Speech and Signal Processing, pp.670-673, 1993.
DOI : 10.1109/ICASSP.1993.319901

P. Stavroulakis, Interference Analysis and Reduction for Wireless Systems Artech House mobile communications series, 2003.

A. Swami and B. M. Sadler, Hierarchical digital modulation classification using cumulants, IEEE Transactions on Communications, vol.48, issue.3, pp.416-429, 2000.
DOI : 10.1109/26.837045

H. Trigui and D. Slock, Cochannel interference cancellation within the current GSM standard, ICUPC '98. IEEE 1998 International Conference on Universal Personal Communications. Conference Proceedings (Cat. No.98TH8384), pp.511-515, 1998.
DOI : 10.1109/ICUPC.1998.733028

H. Trigui and D. Slock, Performance bounds for cochannel interference cancellation within the current GSM standard, Signal Processing, vol.80, issue.7, pp.1335-1346, 2000.
DOI : 10.1016/S0165-1684(00)00040-2

G. and T. Wilson, The Factorization of Matricial Spectral Densities, SIAM Journal on Applied Mathematics, vol.23, issue.4, pp.420-426, 1972.
DOI : 10.1137/0123044

G. Ungerboeck, Adaptive Maximum-Likelihood Receiver for Carrier-Modulated Data-Transmission Systems, IEEE Transactions on Communications, vol.22, issue.5, pp.624-636, 1974.
DOI : 10.1109/TCOM.1974.1092267

B. D. Van-veen and K. M. Buckley, Beamforming: a versatile approach to spatial filtering, IEEE ASSP Magazine, vol.5, issue.2, pp.4-24, 1988.
DOI : 10.1109/53.665

P. Vila, F. Pipon, D. Pirez, and L. Fety, MLSE antenna diversity equalization of a jammed frequency selective fading channel, European signal processing conference (EUSIPCO)

M. Vutukuri, R. Malladi, K. Kuchi, and R. Koilpillai, SAIC receiver algorithms for VAMOS downlink transmission, 2011 8th International Symposium on Wireless Communication Systems, pp.31-35, 2011.
DOI : 10.1109/ISWCS.2011.6125304

G. Wang, J. P. Lie, and C. S. See, A robust approach to optimum widely linear MVDR beamformer, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.2593-2596, 2012.
DOI : 10.1109/ICASSP.2012.6288447

F. Wen, Q. Wan, H. Wei, R. Fan, and Y. Luo, Robust Capon beamforming exploiting the second-order noncircularity of signals, Signal Processing, vol.102, pp.100-111, 2014.
DOI : 10.1016/j.sigpro.2014.03.002

T. , M. Whorter, and P. Schreier, Widely linear beamforming, Proc. 37th Asilomar Conference, Pacific Groeve (USA), pp.753-759, 2003.

J. M. Wozencraft and I. M. Jacobs, Principles of Communication Engineering, 1965.

G. Wunder, 5GNOW: non-orthogonal, asynchronous waveforms for future mobile applications, IEEE Communications Magazine, vol.52, issue.2, pp.97-105, 2014.
DOI : 10.1109/MCOM.2014.6736749

C. Q. Xu, C. L. Law, and S. Yosshida, Interference rejection in non-Gaussian noise for satellite communications using non-linear beamforming, Int. Journal on Satellite Communications and Networking, vol.21, pp.13-22, 2003.

D. Xu, L. Huang, X. Xu, and Z. Ye, Widely linear MVDR beamformers for noncircular signals based on time-averaged Second-Order Noncircularity Coefficient Estimation, IEEE Trans. on Vehicular Technology, vol.62, issue.7, pp.3219-3227, 2013.

D. Xu, C. Gong, S. Cao, X. Xu, and Z. Ye, Robust widely linear beamforming based on spatial spectrum of noncircularity coefficient, Signal Processing, vol.104, pp.167-173, 2014.
DOI : 10.1016/j.sigpro.2014.03.043

J. Zhang, L. Huang, L. Zhang, B. Zhang, and Z. Ye, Robust widely linear beamformer based on a projection constraint, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.2509-2513, 2015.
DOI : 10.1109/ICASSP.2015.7178423