Positive solutions to nonlinear systems involving fully nonlinear fractional operators

Abstract

Abstract In this paper we consider the following fractional system $$\begin{array}{} \displaystyle \left\{ \begin{gathered} F(x,u(x),v(x),{\mathcal{F}_\alpha }(u(x))) = 0,\\ G(x,v(x),u(x),{\mathcal{G}_\beta }(v(x))) = 0, \\ \end{gathered} \right. \end{array}$$ where 0 < α, β < 2, 𝓕α and 𝓖β are the fully nonlinear fractional operators: $$\begin{array}{} \displaystyle {\mathcal{F}_\alpha }(u(x)) = {C_{n,\alpha }}PV\int_{{\mathbb{R}^n}} {\frac{{f(u(x) - u(y))}} {{{{\left| {x - y} \right|}^{n + \alpha }}}}dy} ,\\ \displaystyle{\mathcal{G}_\beta }(v(x)) = {C_{n,\beta }}PV\int_{{\mathbb{R}^n}} {\frac{{g(v(x) - v(y))}} {{{{\left| {x - y} \right|}^{n + \beta }}}}dy} . \end{array}$$ A decay at infinity principle and a narrow region principle for solutions to the system are established. Based on these principles, we prove the radial symmetry and monotonicity of positive solutions to the system in the whole space and a unit ball respectively, and the nonexistence in a half space by generalizing the direct method of moving planes to the nonlinear system.