The application of network function virtualization (NFV) and software-defined networking (SDN) to passive optical networks (PONs) is attracting attention for the deployment of cost-effective access network systems. This paper presents a novel architecture of a programmable finite state machine (P-FSM) as a hardware accelerator for protocol processing in an optical line terminal (OLT). The P-FSM is programmable hardware that manages various types of FSMs to enhance flexibility in OLTs and achieve wired-rate performance with a negligible increase in total chip area. The P-FSM is implemented using three key technologies: a specific architecture for state management of communications protocols to minimize the logic area, a memory distributed implementation to minimize the program memory, and a new branch operation to minimize the memory area and reduce processing time. Evaluation results show that the P-FSM can handle 10G-EPON/NG-PON2 communications protocols in the same architecture while achieving wired-rate performance. The increase in the total designed area is only 1.5% to 4.9% depending on the number of protocols supported compared to the area of a conventional communications SoC without flexibility. We also clarify that our architecture has the scalability needed to modify the number of FSMs and the maximum number of ONU connections according to the system scale.