Large class sizes significantly limit EFL (English as a Foreign Language) learners' speaking opportunities in class, and it is also quite difficult for them to have a regular language partner if they want to practice English after class. Microsoft Xiaoying, a chatbot available on smart phones, offers EFL learners a virtual partner for them to conduct conversational practice with, and engage in language learning anytime and anywhere. This study evaluates 50 EFL learners’ accuracy, errors, and improvements in oral tasks during 28 days of training employing this chatbot, which located their grammar and pronunciation errors. The subjects’ grammar and pronunciation accuracy increased, and their oral performance was significantly improved. In addition, the chatbot was positively perceived by most participants. This study suggests that this chatbot could be one useful learning tool for after-class oral practice and that students might significantly improve their grammar and pronunciation through conversational practice with this chatbot.