Detecting key frames in videos has garnered substantial attention in recent years, it is a point-level task and has deep research value and application prospect in daily life. For instances, video surveillance system, video cover generation and highlight moment flashback all demands the technique of key frame detection. However, the task is beset by challenges such as the sparsity of key frame instances, imbalances between target frames and background frames, and the absence of post-processing method. In response to these problems, we introduce a novel and effective Temporal Interval Guided (TIG) framework to precisely localize specific frames. The framework is incorporated with a proposed Point-Level-Soft non-maximum suppression (PLS-NMS) post-processing algorithm which is suitable for point-level task, facilitated by the well-designed confidence score decay function. Furthermore, we propose a TIG-loss, exhibiting sensitivity to temporal interval from target frame, to optimize the two-stage framework. The proposed method can be broadly applied to key frame detection in video understanding, including action start detection and static video summarization. Extensive experimentation validates the efficacy of our approach on action start detection benchmark datasets: THUMOS’14 and Activitynet v1.3, and we have reached state-of-the-art performance. Competitive results are also demonstrated on SumMe and TVSum datasets for deep learning based static video summarization.

With the recent advances in IoT, there is a growing interest in multi-surface computing, where a mobile app can cooperatively utilize multiple devices' surfaces. We propose a novel framework that seamlessly augments mobile apps with multi-surface computing capabilities. It enables various apps to employ multiple surfaces with acceptable performance.

As a typical surface defect of tunnel lining structures, cracking disease affects the durability of tunnel structures and poses hidden dangers to tunnel driving safety. Factors such as interference from the complex service environment of the tunnel and the low signal-to-noise ratio of the crack targets themselves, have led to existing crack recognition methods based on semantic segmentation being unable to meet actual engineering needs. Based on this, this paper uses the Unet network as the basic framework for crack identification and proposes to construct a multi-kernel convolution cascade enhancement (MKCE) model to achieve accurate detection and identification of crack diseases. First of all, to ensure the performance of crack feature extraction, the model modified the main feature extraction network in the basic framework to ResNet-50 residual network. Compared with the VGG-16 network, this modification can extract richer crack detail features while reducing model parameters. Secondly, considering that the Unet network cannot effectively perceive multi-scale crack features in the skip connection stage, a multi-kernel convolution cascade enhancement module is proposed by combining a cascaded connection of multi-kernel convolution groups and multi-expansion rate dilated convolution groups. This module achieves a comprehensive perception of local details and the global content of tunnel lining cracks. In addition, to better weaken the effect of tunnel background clutter interference, a convolutional block attention calculation module is further introduced after the multi-kernel convolution cascade enhancement module, which effectively reduces the false alarm rate of crack recognition. The algorithm is tested on a large number of subway tunnel crack image datasets. The experimental results show that, compared with other crack recognition algorithms based on deep learning, the method in this paper has achieved the best results in terms of accuracy and intersection over union (IoU) indicators, which verifies the method in this paper has better applicability.

To achieve the extreme high data rate and extreme coverage extension requirements of 6G wireless communication, new spectrum in sub-THz (100-300GHz) and non-terrestrial network (NTN) are two of the macro trends of 6G candidate technologies, respectively. However, non-linearity of power amplifiers (PA) is a critical challenge for both sub-THz and NTN. Therefore, high power efficiency (PE) or low peak to average power ratio (PAPR) waveform design becomes one of the most significant 6G research topics. Meanwhile, high spectral efficiency (SE) and low out-of-band emission (OOBE) are still important key performance indicators (KPIs) for 6G waveform design. Single-carrier waveform discrete Fourier transform spreading orthogonal frequency division multiplexing (DFT-s-OFDM) has achieved many research interests due to its high PE, and it has been supported in 5G New Radio (NR) when uplink coverage is limited. So DFT-s-OFDM can be regarded as a candidate waveform for 6G. Many enhancement schemes based on DFT-s-OFDM have been proposed, including null cyclic prefix (NCP)/unique word (UW), frequency-domain spectral shaping (FDSS), and time-domain compression and expansion (TD-CE), etc. However, there is no unified framework to be compatible with all the enhancement schemes. This paper firstly provides a general description of the 6G candidate waveforms based on DFT-s-OFDM enhancement. Secondly, the more flexible TD-CE supporting methods for unified non-orthogonal waveform (uNOW) are proposed and discussed. Thirdly, a unified waveform framework based on DFT-s-OFDM structure is proposed. By designing the pre-processing and post-processing modules before and after DFT in the unified waveform framework, the three technical methods (NCP/UW, FDSS, and TD-CE) can be integrated to improve three KPIs of DFT-s-OFDM simultaneously with high flexibility. Then the implementation complexity of the 6G candidate waveforms are analyzed and compared. Performance of different DFT-s-OFDM enhancement schemes is investigated by link level simulation, which reveals that uNOW can achieve the best PAPR performance among all the 6G candidate waveforms. When considering PA back-off, uNOW can achieve 124% throughput gain compared to traditional DFT-s-OFDM.

This paper proposes individual computation processes of the partial demodulation reference signal (DM-RS) sequence in a synchronization signal (SS)/physical broadcast channel (PBCH) block to be used to detect the radio frame timing based on SS/PBCH block index detection for New Radio (NR) initial access. We present the radio frame timing detection probability using the proposed partial DM-RS sequence detection method that is applied subsequent to the physical-layer cell identity (PCID) detection in five tapped delay line (TDL) models in both non-line-of-sight (NLOS) and line-of-sight (LOS) environments. Computer simulation results show that by using the proposed method, the radio frame timing detection probabilities of almost 100% and higher than 90% are achieved for the LOS and NLOS channel models, respectively, at the average received signal-to-noise power ratio (SNR) of 0dB with the frequency stability of a local oscillator in a set of user equipment (UE) of 5ppm at the carrier frequency of 4GHz.

High frame rate and ultra-low delay vision system, which can finish reading and processing of 1000fps sequence within 1ms/frame, draws increasing attention in the field of robotics that requires immediate feedback from image process core. Meanwhile, tracking task plays an important role in many computer vision applications. Among various tracking algorithms, Lucas Kanade (LK)-based template tracking, which tracks targets with high accuracy over the sub-pixel level, is one of the keys for robotic applications, such as factory automation (FA). However, the substantial spatial iterative processing and complex computation in the LK algorithm, make it difficult to achieve a high frame rate and ultra-low delay tracking with limited resources. Aiming at an LK-based template tracking system that reads and processes 1000fps sequences within 1ms/frame with small resource costs, this paper proposes: 1) High temporal resolution-based temporal iterative tracking, which maps the spatial iterations into the temporal domain, efficiently reduces resource cost and delay caused by spatial iterative processing. 2) Label scanner-based multi-stream spatial processing, which maps the local spatial processing into the labeled input pixel stream and aggregates them with a label scanner, makes the local spatial processing in the LK algorithm possible be implemented with a small resource cost. Algorithm evaluation shows that the proposed temporal iterative tracking performs dynamic tracking, which tracks object with coarse accuracy when it's moving fast and achieves higher accuracy when it slows down. Hardware evaluation shows that the proposed label scanner-based multi-stream architecture makes the system implemented on FPGA (zcu102) with resource cost less than 20%, and the designed tracking system supports to read and process 1000fps sequence within 1ms/frame.

Pedestrian detection is an essential but challenging task in computer vision, especially in crowded scenes due to heavy intra-class occlusion. In human visual system, head information can be used to locate pedestrian in a crowd because it is more stable and less likely to be occluded. Inspired by this clue, we propose a dual-task detector which detects head and human body simultaneously. Concretely, we estimate human body candidates from head regions with statistical head-body ratio. A head-body alignment map is proposed to perform relational learning between human bodies and heads based on their inherent correlation. We leverage the head information as a strict detection criterion to suppress common false positives of pedestrian detection via a novel pull-push loss. We validate the effectiveness of the proposed method on the CrowdHuman and CityPersons benchmarks. Experimental results demonstrate that the proposed method achieves impressive performance in detecting heavy-occluded pedestrians with little additional computation cost.

This paper describes a novel 4K 120fps (frames per second) real-time HEVC (High Efficiency Video Coding) encoder for high-frame-rate video encoding and transmission. Motion portrayal problems such as motion blur and jerkiness may occur in video scenes containing fast-moving objects or quick camera panning. A high-frame-rate solves such problems and provides a more immersive viewing experience that can express even the fast-moving scenes without discomfort. It can also be used in remote operation for scenes with high motion, such as VAR (Video Assistant Referee) systems in sports. Real-time encoding of high-frame-rate videos with low latency and temporal scalability is required for providing such high-frame-rate video services. The proposed encoder achieves full 4K/120fps real-time encoding, which is twice the current 4K service frame rate of 60fps, by multichip configuration with two encoder LSI. Exchange of reference picture data near a spatially divided slice boundary provides cross-chip motion estimation, and maintains the coding efficiency. The encoder supports temporal-scalable coding mode, in which it output stream with temporal scalability transmitted over one or two transmission paths. The encoder also supports the other mode, low-delay coding mode, in which it achieves 21.8msec low-latency processing through motion vector restriction. Evaluation of the proposed encoder's multichip configuration shows that the BD-bitrate (the average rate of bitrate increase), compared to simple slice division without inter-chip transfer, is -2.86% at minimum and -2.41% on average in temporal-scalable coding mode. The proposed encoder system will open the door to the next generation of high-frame-rate UHDTV (ultra-high-definition television) services.

High frame rate (HFR) video is attracting strong interest since it is considered as a next step toward providing Ultra-High Definition video service. For instance, the Association of Radio Industries and Businesses (ARIB) standard, the latest broadcasting standard in Japan, defines a 120 fps broadcasting format. The standard stipulates temporally scalable coding and hierarchical transmission by MPEG Media Transport (MMT), in which the base layer and the enhancement layer are transmitted over different paths for flexible distribution. We have developed the first ever MMT transmitter/receiver module for 4K/120fps temporally scalable video. The module is equipped with a newly proposed encapsulation method of temporally scalable bitstreams with correct boundaries. It is also designed to be tolerant to severe network constraints, including packet loss, arrival timing offset, and delay jitter. We conducted a hierarchical transmission experiment for 4K/120fps temporally scalable video. The experiment demonstrated that the MMT module was successfully fabricated and capable of dealing with severe network constraints. Consequently, the module has excellent potential as a means to support HFR video distribution in various network situations.

This paper describes designing a new pedestrian navigation system using a comprehensive framework called the pedestrian navigation concept reference model (PNCRM). We implement this system as a publicly-available smartphone application and evaluate its positioning performance near Omiya station's western entrance. We also evaluate users' subjective impressions of the system using a questionnaire. In both cases, promising results are obtained, showing that the PNCRM can be used as a tool for designing pedestrian navigation systems, allowing such systems to be created systematically.

High frame rate and ultra-low delay are the most essential requirements for building excellent human-machine-interaction systems. As a state-of-the-art local keypoint detection and feature extraction algorithm, A-KAZE shows high accuracy and robustness. Nonlinear scale space is one of the most important modules in A-KAZE, but it not only has at least one frame delay and but also is not hardware friendly. This paper proposes a hardware oriented nonlinear scale space for high frame rate and ultra-low delay A-KAZE matching system. In the proposed matching system, one part of nonlinear scale space is temporally forward and calculated in the previous frame (proposal #1), so that the processing delay is reduced to be less than 1 ms. To improve the matching accuracy affected by proposal #1, pre-adjustment of nonlinear scale (proposal #2) is proposed. Previous two frames are used to do motion estimation to predict the motion vector between previous frame and current frame. For further improvement of matching accuracy, pixel-level pre-adjustment (proposal #3) is proposed. The pre-adjustment changes from block-level to pixel-level, each pixel is assigned an unique motion vector. Experimental results prove that the proposed matching system shows average matching accuracy higher than 95% which is 5.88% higher than the existing high frame rate and ultra-low delay matching system. As for hardware performance, the proposed matching system processes VGA videos (640×480 pixels/frame) at the speed of 784 frame/second (fps) with a delay of 0.978 ms/frame.

High frame rate and ultra-low delay matching system plays an increasingly important role in human-machine interactions, because it guarantees high-quality experiences for users. Existing image matching algorithms always generate mismatches which heavily weaken the performance the human-machine-interactive systems. Although many mismatch removal algorithms have been proposed, few of them achieve real-time speed with high frame rate and low delay, because of complicated arithmetic operations and iterations. This paper proposes a temporal constraints and block weighting judgement based high frame rate and ultra-low delay mismatch removal system. The proposed method is based on two temporal constraints (proposal #1 and proposal #2) to firstly find some true matches, and uses these true matches to generate block weighting (proposal #3). Proposal #1 finds out some correct matches through checking a triangle route formed by three adjacent frames. Proposal #2 further reduces mismatch risk by adding one more time of matching with opposite matching direction. Finally, proposal #3 distinguishes the unverified matches to be correct or incorrect through weighting of each block. Software experiments show that the proposed mismatch removal system achieves state-of-the-art accuracy in mismatch removal. Hardware experiments indicate that the designed image processing core successfully achieves real-time processing of 784fps VGA (640×480 pixels/frame) video on field programmable gate array (FPGA), with a delay of 0.858 ms/frame.

Massive open online course (MOOC) is an online course aimed at unlimited participation and open access via the web. Although there are many MOOC providers, they typically focus on the online course providing and typically do not link with traditional education and business sector requirements. This paper presents a MOOC service framework that focuses on adopting MOOC to provide additional services to support students in traditional education and to provide credit bank consisting of student academic credentials for business sector demand. Particularly, it extends typical MOOC to support academic/ credential record and transcript issuance. The MOOC service framework consists of five layers: authentication, resources, learning, assessment and credential layers. We discuss the adoption of the framework in Thai MOOC, the national MOOC system for Thai universities. Several main issues related to the framework adoption are discussed, including the service strategy and model as well as infrastructure design for large-scale MOOC service.

In this paper, we describe a novel low-delay 4K 120-fps real-time HEVC decoder with a parallel processing architecture that conforms to the HEVC main 4:2:2 10 profile. It supports the hierarchical temporal scalable streams required for Ultra High Definition high-frame-rate broadcasting and also supports low-delay and high-bitrate decoding for video transmission uses. To achieve this support, the decoding processes are parallelized and pipelined at the frame level, slice level, and coding tree unit row level. The proposed decoder was implemented on three FPGAs operated at 133 and 150 MHz, and it achieved 300-Mbps stream decoding and 37-msec end-to-end delay with our concurrently developed 4K 120-fps encoder.

Volleyball video analysis plays important roles in providing data for TV contents and developing strategies. Among all the topics of volleyball analysis, qualitative player action recognition is essential because it potentially provides not only the action that being performed but also the quality, which means how well the action is performed. However, most action recognition researches focus on the discrimination between different actions. The quality of an action, which is helpful for evaluation and training of the player skill, has only received little attention so far. The vital problems in qualitative action recognition include occlusion, small inter-class difference and various kinds of appearance caused by the player change. This paper proposes a 3D global and multi-view local features combination based recognition framework with global team formation feature, ball state feature and abrupt pose features. The above problems are solved by the combination of 3D global features (which hide the unstable and incomplete 2D motion feature caused by occlusion) and the multi-view local features (which get detailed local motion features of body parts in multiple viewpoints). Firstly, the team formation extracts the 3D trajectories from the whole team members rather than a single target player. This proposal focuses more on the entire feature while eliminating the personal effect. Secondly, the ball motion state feature extracts features from the 3D ball trajectory. The ball motion is not affected by the personal appearance, so this proposal ignores the influence of the players appearance and makes it more robust to target player change. At last, the abrupt pose feature consists of two parts: the abrupt hit frame pose (which extracts the contour shape of the player's pose at the hit time) and abrupt pose variation (which extracts the pose variation between the preparation pose and ending pose during the action). These two features make difference of each action quality more distinguishable by focusing on the motion standard and stability between different quality actions. Experiments are conducted on game videos from the Semifinal and Final Game of 2014 Japan Inter High School Games of Men's Volleyball in Tokyo Metropolitan Gymnasium. The experimental results show the accuracy achieves 97.26%, improving 11.33% for action discrimination and 91.76%, and improving 13.72% for action quality evaluation.

High frame rate and ultra-low delay matching system plays an important role in various human-machine interactive applications, which demands better performance in matching deformable and out-of-plane rotating objects. Although many algorithms have been proposed for deformation tracking and matching, few of them are suitable for hardware implementation due to complicated operations and large time consumption. This paper proposes a hardware-oriented template update and recovery method for high frame rate and ultra-low delay deformation matching system. In the proposed method, the new template is generated in real time by partially updating the template descriptor and adding new keypoints simultaneously with the matching process in pixels (proposal #1), which avoids the large inter-frame delay. The size and shape of region of interest (ROI) are made flexible and the Hamming threshold used for brute-force matching is adjusted according to pixel position and the flexible ROI (proposal #2), which solves the problem of template drift. The template is recovered by the previous one with a relative center-shifting vector when it is judged as lost via region-wise difference check (proposal #3). Evaluation results indicate that the proposed method successfully achieves the real-time processing of 784fps at the resolution of 640×480 on field-programmable gate array (FPGA), with a delay of 0.808ms/frame, as well as achieves satisfactory deformation matching results in comparison with other general methods.

A 3Gbps/lane transmission buffer chip including a high-speed mode detector is proposed for a field-programmable gate array (FPGA)-based frame generator supporting the mobile industry processor interface (MIPI) D-PHY version 1.2. It performs 1-to-3 repeat while buffering low voltage differential signaling (LVDS) or scalable low voltage signaling (SLVS) to SLVS.

This paper presents a concept of a domain-specific framework for software analytics by enabling querying, modeling, and integration of heterogeneous software repositories. The framework adheres to a multi-layered abstraction mechanism that consists of domain-specific operators. We showcased the potential of this approach by employing a case study.

Collocation is a ubiquitous phenomenon in languages and accurate collocation recognition and extraction is of great significance to many natural language processing tasks. Collocations can be differentiated from simple bigram collocations to collocation frames (referring to distant multi-gram collocations). So far little focus is put on collocation frames. Oriented to translation and parsing, this study aims to recognize and extract the longest possible collocation frames from given sentences. We first extract bigram collocations with distributional semantics based method by introducing collocation patterns and integrating some state-of-the-art association measures. Based on bigram collocations extracted by the proposed method, we get the longest collocation frames according to recursive nature and linguistic rules of collocations. Compared with the baseline systems, the proposed method performs significantly better in bigram collocation extraction both in precision and recall. And in extracting collocation frames, the proposed method performs even better with the precision similar to its bigram collocation extraction results.

In GNSS (Global Navigation Satellite System) Distributed Simulation Environment (GDSE), the simulation task could be designed with the sharing models on the Internet. However, too much information and relation of model need to be managed in GDSE. Especially if there is a large quantity of sharing models, the model retrieval would be an extremely complex project. For meeting management demand of GDSE and improving the model retrieval efficiency, the characteristics of service simulation model are analysed firstly. A semantic management method of simulation model is proposed, and a model management architecture is designed. Compared with traditional retrieval way, it takes less retrieval time and has a higher accuracy result. The simulation results show that retrieval in the semantic management module has a good ability on understanding user needs, and helps user obtain appropriate model rapidly. It improves the efficiency of simulation tasks design.