Fractional‐order operators on nonsmooth domains

Abstract

AbstractThe fractional Laplacian , , and its generalizations to variable‐coefficient ‐order pseudodifferential operators , are studied in ‐Sobolev spaces of Bessel‐potential type . For a bounded open set , consider the homogeneous Dirichlet problem: in , in . We find the regularity of solutions and determine the exact Dirichlet domain (the space of solutions with ) in cases where has limited smoothness , for , . Earlier, the regularity and Dirichlet domains were determined for smooth by the second author, and the regularity was found in low‐order Hölder spaces for by Ros‐Oton and Serra. The ‐results obtained now when are new, even for . In detail, the spaces are identified as ‐transmission spaces , exhibiting estimates in terms of near the boundary.The result has required a new development of methods to handle nonsmooth coordinate changes for pseudodifferential operators, which have not been available before; this constitutes another main contribution of the paper.