The prevalence of diabetes is increasing worldwide, and it is very important to study new hypoglycemic active substances. In this study, we investigated the hypoglycemic effect of <i>Chroogomphus rutilus</i> crude polysaccharide (CRCP) in HepG2 cells and streptozotocin-induced diabetic mice. A glucose consumption experiment conducted in HepG2 cells demonstrated the <i>in vitro</i> hypoglycemic activity of CRCP. Furthermore, CRCP exhibited significant hypoglycemic effects and effectively ameliorated insulin resistance in insulin resistant HepG2 cells. In high-fat diet and streptozotocin-induced diabetic mice, after 4 weeks of CRCP administration, fasting blood glucose, fasting serum insulin, triglyceride, total cholesterol, low-density lipoprotein cholesterol, glutamate transaminase, alanine transaminase, and insulin resistance index significantly decreased, while high-density lipoprotein cholesterol and insulin sensitivity index (ISI) were markedly increased. Moreover, hematoxylin-eosin (HE) staining and immunofluorescence labeling of tissue sections indicated that CRCP attenuated the pathological damage of liver and pancreas in diabetic mice. These results indicate that CRCP is a potential hypoglycemic agent.