In this letter, we investigate the physical layer security in multi-user multi-relay networks, where each relay is not merely a traditional helper, but at the same time, can become a potential eavesdropper. We first propose an efficient low-complexity user and relay selection scheme to significantly reduce the amount of channel estimation as well as the amount of potential links for comparison. For the proposed scheme, we derive the closed-form expression for the lower bound of ergodic secrecy rate (ESR) to evaluate the system secrecy performance. Simulation results are provided to verify the validity of our expressions and demonstrate how the ESR scales with the number of users and relays.