This paper proposes a new selective encryption scheme and a key management scheme for layered access control of H.264/SVC. This scheme encrypts three domains in hierarchical layers using different keys: intra prediction modes, motion vector difference values and sign bits of texture data. The proposed scheme offers low computational complexity, low bit-overhead, and format compliance by utilizing the H.264/SVC structure. It provides a high encryption efficiency by encrypting domains selectively, according to each layer type in the enhancement-layer. It also provides confidentiality and implicit authentication using keys derived in the proposed key management scheme for encryption. Simulation results show the effectiveness of the proposed scheme.