To accurately explain social group inequalities, people must consider structural explanations, which are causal explanations that appeal to societal factors such as discriminatory institutions and policies. Structural explanations are a distinct type of extrinsic explanation—they identify stable societal forces that are experienced by specific social groups. We argue that a novel framework is needed to specify how people infer structural causes of inequality. The proposed framework is rooted in counterfactual theories of causal judgment, positing that people infer structural causes by discerning whether structural factors were “difference-making” for the inequality they observe. Building on this foundation, our framework makes the following novel contributions: First, we propose specific types of evidence that support this inference, and second, we consider the unique contextual, cognitive, and motivational barriers to the availability and acceptance of this evidence. We conclude by exploring how the framework might be applied in future research examining people’s explanations for inequality.