In this paper, we investigate a novel joint multi-relay and jammer selection (JMRJS) scheme in order to improve the physical layer security of wireless networks. In the JMRJS scheme, all the relays succeeding in source decoding are selected to assist in the source signal transmission and meanwhile, all the remaining relay nodes are employed to act as friendly jammers to disturb the eavesdroppers by broadcasting artificial noise. Based on the more general Nakagami-m fading channel, we analyze the security performance of the JMRJS scheme for protecting the source signal against eavesdropping. The exact closed-form expressions of outage probability (OP) and intercept probability (IP) for the JMRJS scheme over Nakagami-m fading channel are derived. Moreover, we analyze the security-reliability tradeoff (SRT) of this scheme. Simulation results show that as the number of decode-and-forward (DF)relay nodes increases, the SRT of the JMRJS scheme improves notably. And when the transmit power is below a certain value, the SRT of the JMRJS scheme consistently outperforms the joint single-relay and jammer selection (JSRJS) scheme and joint equal-relay and jammer selection (JERJS) scheme respectively. In addition, the SRT of this scheme is always better than that of the multi-relay selection (MRS) scheme.

In this paper, we investigate a cooperative communication system comprised of a source, a destination, and multiple decode-and-forward (DF) relays in the presence of a potential malicious eavesdropper is within or without the coverage area of the source. Based on the more general Nakagami-m fading channels, we analyze the security performance of the single-relay selection and multi-relay selection schemes for protecting the source against eavesdropping. In the single-relay selection scheme, only the best relay is chosen to assist in the source transmission. Differing from the single-relay selection, multi-relay selection scheme allows multiple relays to forward the source to the destination. We also consider the classic direct transmission as a benchmark scheme to compare with the two relay selection schemes. We derive the exact closed-form expressions of outage probability (OP) and intercept probability (IP) for the direct transmission, the single-relay selection as well as the multi-relay selection scheme over Nakagami-m fading channel when the eavesdropper is within and without the coverage area of the source. Moreover, the security-reliability tradeoff (SRT) of these three schemes are also analyzed. It is verified that the SRT of the multi-relay selection consistently outperforms the single-relay selection, which of both the single-relay and multi-relay selection schemes outperform the direct transmission when the number of relays is large, no matter the eavesdropper is within or without the coverage of the source. In addition, as the number of DF relays increases, the SRT of relay selection schemes improve notably. However, the SRT of both two relay selection approaches become worse when the eavesdropper is within the coverage area of the source.