Fractional‐order operators on nonsmooth domains

The fractional Laplacian $(-\Delta )^a$, $a\in(0,1)$, and its generalizations to variable-coefficient $2a$-order pseudodifferential operators $P$, are studied in $L_q$-Sobolev spaces of Bessel-potential type $H^s_q$. For a bounded open set $\Omega \subset \mathbb R^n$, consider the homogeneous Dirichlet problem: $Pu =f$ in $\Omega $, $u=0$ in $ \mathbb R^n\setminus\Omega $. We find the regularity of solutions and determine the exact Dirichlet domain $D_{a,s,q}$ (the space of solutions $u$ with $f\in H_q^s(\overline\Omega )$) in cases where $\Omega $ has limited smoothness $C^{1+\tau }$, for $2a
Comment: 52 pages. In this version some clarifications and minor corrections were done. The version is accepted for publication in "J. London Math. Soc."