The segmentation performance of fresh tea sprouts is inadequate due to the uncontrollable posture. A novel method for Fresh Tea Sprouts Segmentation based on Capsule Network (FTS-SegCaps) is proposed in this paper. The spatial relationship between local parts and whole tea sprout is retained and effectively utilized by a deep encoder-decoder capsule network, which can reduce the effect of tea sprouts with uncontrollable posture. Meanwhile, a patch-based local dynamic routing algorithm is also proposed to solve the parameter explosion problem. The experimental results indicate that the segmented tea sprouts via FTS-SegCaps are almost coincident with the ground truth, and also show that the proposed method has a better performance than the state-of-the-art methods.