In this paper, we propose a real-time vibration extraction system, which extracts vibration component within a given frequency range from videos in real time, for realizing tremor suppression used in microsurgery assistance systems. To overcome the problems in our previous system based on the mean Lucas-Kanade (LK) optical flow of the whole frame, we have introduced a new architecture combining dense optical flow calculated with simple feature matching and block-based band-pass filtering using band-limited multiple Fourier linear combiner (BMFLC). As a feature of optical flow calculation, we use the simplified rotation-invariant histogram of oriented gradients (RIHOG) based on a gradient angle quantized to 1, 2, or 3 bits, which greatly reduces the usage of memory resources for a frame buffer. An obtained optical flow map is then divided into multiple blocks, and BMFLC is applied to the mean optical flow of each block independently. By using the L1-norm of adaptive weight vectors in BMFLC as a criterion, blocks belonging to vibrating objects can be isolated from background at low cost, leading to better extraction accuracy compared to the previous system. The whole system for 480p and 720p resolutions can be implemented on a single Xilinx Zynq-7000 XC7Z020 FPGA without any external memory, and can process a video stream supplied directly from a camera at 60fps.

This paper shows design and implementation of an image-based vibration detection system on a field-programmable gate array (FPGA), aiming at application to tremor suppression for microsurgery assistance systems. The system can extract a vibration component within a user-specified frequency band from moving images in real-time. For fast and robust detection, we employ a statistical approach using dense optical flow to derive vibration component, and design a custom hardware based on the Lucas-Kanade (LK) method to compute optical flow. And for band-pass filtering without phase delay, we implement the band-limited multiple Fourier linear combiner (BMFLC), a sort of adaptive band-pass filter which can recompose an input signal as a mixture of sinusoidal signals with multiple frequencies within the specified band, with no phase delay. The whole system is implemented as a deep pipeline on a Xilinx Kintex-7 XC7K325T FPGA without using any external memory. We employ fixed-point arithmetic to reduce resource utilization while maintaining accuracy close to double-precision floating-point arithmetic. Empirical experiments reveal that the proposed system extracts a high-frequency tremor component from hand motions, with intentional low-frequency motions successfully filtered out. The system can process VGA moving images at 60fps, with a delay of less than 1 µs for the BMFLC, suggesting effectiveness of the deep pipelined architecture. In addition, we are planning to integrate a CNN-based segmentation system for improving detection accuracy, and show preliminary software evaluation results.