A relay assignment scheme is proposed in this paper that minimizes the mean delay of transmission for energy harvesting (EH) cooperative communication systems, whose source node and relay nodes are all equipped with energy harvesters. We jointly consider the long-term channel side information (CSI) and energy side information (ESI) of all nodes, and formulate the delay minimization problem as an integer programming problem. To solve this problem, a refined cyclic coordinate method (RCCM) is proposed that considers the cases of fixed-packet-length (FPL) and variable-packet-length (VPL) transmission. Simulation results show that the proposed scheme achieves performance close to that of the real-time relay selection (RRS) scheme with instantaneous CSI and ESI, which gives upper bound of the performance. Moreover, compared with the simple relay rotation (SRR) scheme where each relay has equal service time, the performance of the proposed scheme is significantly improved.