The semantic congruity effect refers to the facilitation of judgements (i) when the direction of the comparison of two items coincides with the relative position of the items along the dimension comparison or (ii) when the relative size of a standard and a target stimulus coincides. For example, people are faster in judging 'which is bigger?' for two large items, than judging 'which is smaller?' for two large items (selection paradigm). Also, people are faster in judging a target stimulus as smaller when compared to a small standard, than when compared to a large standard, and vice versa (classification paradigm). We use the Drift Diffusion Model (DDM) to explain the time course of a semantic congruity effect in a classification paradigm. Formal modelling of semantic congruity allows the time course of the decision process to be described, using an established model of decision making. Moreover, although there have been attempts to explain the semantic congruity effect within evidence accumulation models, two possible accounts for the congruity effect have been proposed but their specific predictions have not been compared directly, using a model that could quantitatively account for both; a shift in the starting point of evidence accumulation or a change in the rate at which evidence is accumulated. With our computational investigation we provide evidence for the latter, while controlling for other possible explanations such as a variation in non-decision time or boundary separation, that have not been taken into account in the explanation of this phenomenon.