This paper presents a novel threshold-based selection scheme to combine adaptive transmit antenna selection with an adaptive quadrature amplitude modulation (AQAM) for a spatial multiplexing (SM) multiple-input multiple-output (MIMO) system with linear receivers in practical uncorrelated and correlated channel conditions. The proposed scheme aims to maximize the average spectral efficiency (ASE) for a given bit error rate (BER) constraint and also to lower the hardware complexity. Our simulations are run on a general MIMO channel model, under the assumption that the channel state information (CSI) is known at the receiver and the adaptive control signaling can be perfectly fed back to the transmitter. We deploy the low rank-revealing QR (LRRQR) algorithm in transmit antenna subset selection. LRRQR is computationally less expensive than a singular value decomposition (SVD) based algorithm while the two algorithms achieve similar error rate performances. We show that both the conventional AQAM scheme (i.e., without adaptive transmit antenna selection) and the SM scheme perform poorly in a highly correlated channel environment. We demonstrate that our proposed scheme provides a well-behaved trade-off between the ASE and BER under various channel environments. The ASE (i.e., throughput) can be maximized with a proper choice of the channel quality threshold and AQAM mode switching threshold levels for a target BER.