We study time-frequency localization operators of the form
A
a
φ
1
,
φ
2
A_a^{\varphi _1\!,\varphi _2}
, where
a
a
is the symbol of the operator and
φ
1
,
φ
2
\varphi _1 , \varphi _2
are the analysis and synthesis windows, respectively. It is shown in an earlier paper by the authors that a sufficient condition for
A
a
φ
1
,
φ
2
∈
S
p
(
L
2
(
R
d
)
)
A_a^{\varphi _1,\varphi _2}\in S_p(L^2(\mathbb {R}^d))
, the Schatten class of order
p
p
, is that
a
a
belongs to the modulation space
M
p
,
∞
(
R
2
d
)
M^{p,\infty }(\mathbb {R}^{2d})
and the window functions to the modulation space
M
1
M^1
. Here we prove a partial converse: if
A
a
φ
1
,
φ
2
∈
S
p
(
L
2
(
R
d
)
)
A_a^{\varphi _1,\varphi _2}\in S_p(L^2(\mathbb {R}^d))
for every pair of window functions
φ
1
,
φ
2
∈
S
(
R
2
d
)
\varphi _1,\varphi _2\in \mathcal {S}(\mathbb {R}^{2d})
with a uniform norm estimate, then the corresponding symbol
a
a
must belong to the modulation space
M
p
,
∞
(
R
2
d
)
M^{p,\infty }(\mathbb {R}^{2d})
. In this sense, modulation spaces are optimal for the study of localization operators. The main ingredients in our proofs are frame theory and Gabor frames. For
p
=
∞
p=\infty
and
p
=
2
p=2
, we recapture earlier results, which were obtained by different methods.