Table tennis is a popular sport around the world. A key technology in table tennis education and analysis system is reconstructing the trajectory of the fast-moving ball from videos. Typically the table tennis ball is too small and barely visible in the video, making it difficult to be recognized directly by detection models like YOLO. However, table tennis balls usually has obvious motion features, which are usually not found in similar false targets. It inspired the authors to first find all candidate targets and then use the motion features of table tennis ball to select them out. In this article, the authors propose a tree-based algorithm named T-FORT to track the ball and reconstruct its trajectory. Specifically, they consider all the possible objects in a tree-framework, and identify the real target by integrating visual features and moving patterns. The authors conduct a set of experiments on three datasets to evaluate the effectiveness and performance of the proposed algorithm. The experimental results show that the proposed method is more precise than existing algorithms, and is robust in various scenarios.