This investigation proposes a fast wavelet-based color segmentation (FWCS) technique and a modified directional region-growing (DRG) technique for semantic image segmentation. The FWCS is a subsequent combination of progressive color truncation and histogram-based color extraction processes for segmenting color regions in images. By exploring specialized centroids of segmented fragments as initial growing seeds, the proposed DRG operates a directional 1-D region growing on pairs of color segmented regions based on those centroids. When the two examined regions are positively confirmed by DRG, the proposed framework subsequently computes the texture features extracted from these two regions to further check their relation using texture similarity testing (TST). If any pair of regions passes double checking with both DRG and TST, they are identified as associated regions. If two associated regions/areas are connective, they are unified to a union area enclosed by a single contour. On the contrary, the proposed framework merely acknowledges a linking relation between those associated regions/areas highlighted with any linking mark. Particularly, by the systematic integration of all proposed processes, the critical issue to decide the ending level of wavelet decomposition in various images can be efficiently solved in FWCS by a quasi-linear high-frequency analysis model newly proposed. The simulations conducted here demonstrate that the proposed segmentation framework can achieve a quasi-semantic segmentation without priori a high-level knowledge.