Field Programmable Gate Array (FPGA) implementation of Elliptic Curve Cryptography (ECC) over GF(p) is commonly not fast enough to meet the request of high-performance applications. There are three critical factors to determine the performance of ECC processor over GF(p): multiplication structure, modular multiplication algorithm, and scalar point multiplication scheduling. This work proposes a novel multiplication structure which is a two-stage pipeline on the basis of Karatsuba-Ofman algorithm. With the proposed multiplication structure, we design a 256-bit modular multiplier based on Improved Barret Modular Multiplication algorithm. Upon the modular multiplier, we finish the scalar point multiplication scheduling and implement a high-performance ECC processor on FPGA. Compared with the previous modular multipliers, our modular multiplier reduces the 256-bit modular multiplication time by 28% at least. Synthesis result on Altera Stratix II shows that our ECC processor can complete a 256-bit ECC scalar point multiplication in 0.51ms, which is at least 1.3 times faster than the currently reported FPGA ECC processors over GF(p).