To mitigate the interference caused by frequency reuse between inter-layer and intra-layer users for Non-Orthogonal Multiple Access (NOMA) based device-to-device (D2D) communication underlaying cellular systems, this paper proposes a joint optimization strategy that combines user grouping and resource allocation. Specifically, the optimization problem is formulated to maximize the sum rate while ensuring the minimum rate of cellular users, considering three optimization parameters: user grouping, sub channel allocation and power allocation. However, this problem is a mixed integer nonlinear programming (MINLP) problem and is hard to solve directly. To address this issue, we divide the problem into two sub-problems: user grouping and resource allocation. First, we classify D2D users into D2D pairs or D2D NOMA groups based on the greedy algorithm. Then, in terms of resource allocation, we allocate the sub-channel to D2D users by swap matching algorithm to reduce the co-channel interference, and optimize the transmission power of D2D by the local search algorithm. Simulation results show that, compared to other schemes, the proposed algorithm significantly improves the system sum rate and spectral utilization.