Search

Abstract Training defect detection algorithms for visual surface inspection systems requires a large and representative set of training data. Often there is not enough real data available which additionally cannot cover the variety of possible defects. Synthetic data generated by a synthetic visual surface inspection environment can overcome this problem. Therefore, a digital twin of the object is needed, whose micro-scale surface topography is modeled by texture synthesis models. We develop stochastic texture models for sandblasted and milled surfaces based on topography measurements of such surfaces. As the surface patterns differ significantly, we use separate modeling approaches for the two cases. Sandblasted surfaces are modeled by a combination of data-based texture synthesis methods that rely entirely on the measurements. In contrast, the model for milled surfaces is procedural and includes all process-related parameters known from the machine settings.

Training defect detection algorithms for visual surface inspection systems requires a large and representative set of training data. Often there is not enough real data available which additionally cannot cover the variety of possible defects. Synthetic data generated by a synthetic visual surface inspection environment can overcome this problem. Therefore, a digital twin of the object is needed, whose micro-scale surface topography is modeled by texture synthesis models. We develop stochastic texture models for sandblasted and milled surfaces based on topography measurements of such surfaces. As the surface patterns differ significantly, we use separate modeling approaches for the two cases. Sandblasted surfaces are modeled by a combination of data-based texture synthesis methods that rely entirely on the measurements. In contrast, the model for milled surfaces is procedural and includes all process-related parameters known from the machine settings. [ABSTRACT FROM AUTHOR]

For accurate urban planning, three-dimensional (3D) building models with a high level of detail (LOD) must be developed. However, most large-scale 3D building models are limited to a low LOD of 1–2, as the creation of higher LOD models requires the modeling of detailed building elements such as walls, windows, doors, and roof shapes. This process is currently not automated and is performed manually. In this study, an end-to-end framework for the creation of 3D building models was proposed by integrating multi-source data such as omnidirectional images, building footprints, and aerial photographs. These different data sources were matched with the building ID considering their spatial location. The building element information related to the exterior of the building was extracted, and detailed LOD3 3D building models were created. Experiments were conducted using data from Kobe, Japan, yielding a high accuracy for the intermediate processes, such as an 86.9% accuracy in building matching, an 88.3% pixel-based accuracy in the building element extraction, and an 89.7% accuracy in the roof type classification. Eighty-one LOD3 3D building models were created in 8 h, demonstrating that our method can create 3D building models that adequately represent the exterior information of actual buildings. [ABSTRACT FROM AUTHOR]

The Metaverse is regarded as a brand-new virtual society constructed by deep media, and the new media art produced by new media technology will gradually replace the traditional art form and play an important role in the infinite Metaverse in the future. The maturity of the new media art creation mechanism must also depend on the help of artificial intelligence (AI) and Internet of Things (IoT) technology. The purpose of this study is to explore the image style transfer of digital painting art in new media art, that is, to reshape the image style by neural network technology in AI based on retaining the semantic information of the original image. Based on neural style transfer, an image style conversion method based on feature synthesis is proposed. Using the feature mapping of content image and style image and combining the advantages of traditional texture synthesis, a richer multi-style target feature mapping is synthesized. Then, the inverse transformation of target feature mapping is restored to an image to realize style transformation. In addition, the research results are analyzed. Under the background of integrating AI and IoT, the creation mechanism of new media art is optimized. Regarding digital art style transformation, the Tensorflow program framework is used for simulation verification and performance evaluation. The experimental results show that the image style transfer method based on feature synthesis proposed in this study can make the image texture more reasonably distributed, and can change the style texture by retaining more semantic structure content of the original image, thus generating richer artistic effects, and having better interactivity and local controllability. It can provide theoretical help and reference for developing new media art creation mechanisms. [ABSTRACT FROM AUTHOR]

Abstract This paper investigates deep learning (DL)-based semantic segmentation of textured mosaics. Existing popular datasets for mosaic texture segmentation, designed prior to the DL era, have several limitations: (1) training images are single-textured and thus differ from the multi-textured test images; (2) training and test textures are typically cut out from the same raw images, which may hinder model generalization; (3) each test image has its own limited set of training images, thus forcing an inefficient training of one model per test image from few data. We propose two texture segmentation datasets, based on the existing Outex and DTD datasets, that are suitable for training semantic segmentation networks and that address the above limitations: SemSegOutex focuses on materials acquired under controlled conditions, and SemSegDTD focuses on visual attributes of textures acquired in the wild. We also generate a synthetic version of SemSegOutex via texture synthesis that can be used in the same way as standard random data augmentation. Finally, we study the performance of the state-of-the-art DeepLabv3+ for textured mosaic segmentation, which is excellent for SemSegOutex and variable for SemSegDTD. Our datasets allow us to analyze results according to the type of material, visual attributes, various image acquisition artifacts, and natural versus synthetic aspects, yielding new insights into the possible usage of recent DL technologies for texture analysis.

For accurate urban planning, three-dimensional (3D) building models with a high level of detail (LOD) must be developed. However, most large-scale 3D building models are limited to a low LOD of 1–2, as the creation of higher LOD models requires the modeling of detailed building elements such as walls, windows, doors, and roof shapes. This process is currently not automated and is performed manually. In this study, an end-to-end framework for the creation of 3D building models was proposed by integrating multi-source data such as omnidirectional images, building footprints, and aerial photographs. These different data sources were matched with the building ID considering their spatial location. The building element information related to the exterior of the building was extracted, and detailed LOD3 3D building models were created. Experiments were conducted using data from Kobe, Japan, yielding a high accuracy for the intermediate processes, such as an 86.9% accuracy in building matching, an 88.3% pixel-based accuracy in the building element extraction, and an 89.7% accuracy in the roof type classification. Eighty-one LOD3 3D building models were created in 8 h, demonstrating that our method can create 3D building models that adequately represent the exterior information of actual buildings.

Three-dimensional solid computational representations of natural heterogeneous materials are challenging to generate due to their high degree of randomness and varying scales of patterns, such as veins and cracks, in different sizes and directions. In this regard, this paper introduces a new architecture to synthesise 3D solid material models by using encoding deep convolutional generative adversarial networks (EDCGANs). DCGANs have been useful in generative tasks in relation to image processing by successfully recreating similar results based on adequate training. While concentrating on natural heterogeneous materials, this paper uses an encoding and a decoding DCGAN combined in a similar way to auto-encoders to convert a given image into marble, based on patches. Additionally, the method creates an input dataset from a single 2D high-resolution exemplar. Further, it translates of 2D data, used as a seed, into 3D data to create material blocks. While the results on the Z-axis do not have size restrictions, the X- and Y-axis are constrained by the given image. Using the method, the paper explores possible ways to present 3D solid textures. The modelling potentials of the developed approach as a design tool is explored to synthesise a 3D solid texture of leaf-like material from an exemplar of a leaf image.

Image-guided drawing can compensate for a lack of skill but often requires a significant number of repetitive strokes to create textures. Existing automatic stroke synthesis methods are usually limited to predefined styles or require indirect manipulation that may break the spontaneous flow of drawing. We present an assisted drawing system to autocomplete repetitive short strokes during a user's normal drawing process. Users draw over a reference image as usual; at the same time, our system silently analyzes the input strokes and the reference to infer strokes that follow the user's input style when certain repetition is detected. Users can accept, modify, or ignore the system's predictions and continue drawing, thus maintaining fluid control over drawing. Our key idea is to jointly analyze image regions and user input history to detect and predict repetition. The proposed system can effectively reduce the user's workload when drawing repetitive short strokes, helping users to create results with rich patterns. [ABSTRACT FROM AUTHOR]

This paper investigates deep learning (DL)-based semantic segmentation of textured mosaics. Existing popular datasets for mosaic texture segmentation, designed prior to the DL era, have several limitations: (1) training images are single-textured and thus differ from the multi-textured test images; (2) training and test textures are typically cut out from the same raw images, which may hinder model generalization; (3) each test image has its own limited set of training images, thus forcing an inefficient training of one model per test image from few data. We propose two texture segmentation datasets, based on the existing Outex and DTD datasets, that are suitable for training semantic segmentation networks and that address the above limitations: SemSegOutex focuses on materials acquired under controlled conditions, and SemSegDTD focuses on visual attributes of textures acquired in the wild. We also generate a synthetic version of SemSegOutex via texture synthesis that can be used in the same way as standard random data augmentation. Finally, we study the performance of the state-of-the-art DeepLabv3+ for textured mosaic segmentation, which is excellent for SemSegOutex and variable for SemSegDTD. Our datasets allow us to analyze results according to the type of material, visual attributes, various image acquisition artifacts, and natural versus synthetic aspects, yielding new insights into the possible usage of recent DL technologies for texture analysis. Article highlights: We propose two texture segmentation datasets that address the limitations of existing texture segmentation datasets. Experiments with materials and attributes shed a new light on recent deep learning technologies for texture analysis. Our results also suggest that synthetic textures can be used for data augmentation to improve segmentation results. [ABSTRACT FROM AUTHOR]

In this article, we introduce a new mathematical functional whose minimization determines the quality of the solution for the exemplar-based inpainting-by-patch problem. The new functional expression includes finite difference terms in a similar fashion to what happens in the theoretical Sobolev spaces: its use reduces the uncertainty in the choice of the most suitable values for each point to inpaint. Moreover, we introduce a probabilistic model by which we prove that the usual principal directions, generally employed for continuous problems, are not enough to achieve consistent reconstructions in the discrete inpainting asset. Finally, we formalize a new priority index and new rules for its dynamic update. The quality of the reconstructions, achieved using a reduced neighborhood size of more than 95 % with respect to the current state-of-the-art algorithms based on the same inpainting approach, further provides the experimental validation of the method. [ABSTRACT FROM AUTHOR]

Example-based texture synthesis is a fundamental topic of many image analysis and computer vision applications. Consequently, its representation is one of the most critical and challenging topics in computer vision and pattern recognition, attracting much academic interest throughout the years. In this paper, a new statistical method to synthesize textures is proposed. It consists in using two indexed random coefficients autoregressive (2D-RCA) models to deal with this problem. These models have a good ability to well detect neighborhood information. Simulations have demonstrated that the 2D-RCA models are very suitable to represent textures. So, in this work, to generate textures from an example, each original image is splitted into blocks which are modeled by the 2D-RCA. The proposed algorithm produces approximations of the obtained blocks images from the original image using the generalized method of moments (GMM). Different sizes of windows have been used. This study offers some important insights into the newly generated image. Satisfying obtained results have been compared to those given by well-established methods. The proposed algorithm outperforms the state-of-the-art approaches.

Abstract Image-guided drawing can compensate for a lack of skill but often requires a significant number of repetitive strokes to create textures. Existing automatic stroke synthesis methods are usually limited to predefined styles or require indirect manipulation that may break the spontaneous flow of drawing. We present an assisted drawing system to autocomplete repetitive short strokes during a user’s normal drawing process. Users draw over a reference image as usual; at the same time, our system silently analyzes the input strokes and the reference to infer strokes that follow the user’s input style when certain repetition is detected. Users can accept, modify, or ignore the system’s predictions and continue drawing, thus maintaining fluid control over drawing. Our key idea is to jointly analyze image regions and user input history to detect and predict repetition. The proposed system can effectively reduce the user’s workload when drawing repetitive short strokes, helping users to create results with rich patterns.

Abstract Additive manufacturing (AM) technologies have been recognized for their capability to build complex components and hence have offered more freedom to designers for a long time. The ability to directly use a computer-aided design (CAD) model has allowed for fabricating and realizing complicated components, monolithic design, reducing the number of components in an assembly, decreasing time to market, and adding performance or comfort-enhancing functionalities. One of the features that can be introduced for boosting a component functionality using AM is the inclusion of surface texture on a given component. This inclusion is usually a difficult task as creating a CAD model resolving fine details of a given texture is difficult even using commercial software packages. This paper develops a methodology to include texture directly on the CAD model of a target surface using a patch-based sampling texture synthesis algorithm, which can be manufactured using AM. Input for the texture generation algorithm can be either a physical sample or an image with heightmap information. The heightmap information from a physical sample can be obtained by 3D scanning the sample and using the information from the acquired point cloud. After obtaining the required inputs, the patches are sampled for texture generation according to non-parametric estimation of the local conditional Markov random field (MRF) density function, which helps avoid mismatched features across the patch boundaries. While generating the texture, a design constraint to ensure AM producibility is considered, which is essential when manufacturing a component using, e.g., Fused Deposition Melting (FDM) or Laser Powder Bed Fusion (LPBF). The generated texture is then mapped onto the surface using the developed distance and angle preserving mapping algorithms. The implemented algorithms can be used to map the generated texture onto a mathematically defined surface. This paper maps the textures onto flat, curved, and sinusoidal surfaces for illustration. After the texture mapping, a stereolithography (STL) model is generated with the desired texture on the target surface. The generated STL model is printed using FDM technology as a final step.

Aiming at the problems of patch-based synthesis algorithm of structured texture images,such as discontinuity of structure,distortion of boundary,seam misalignment,and low synthesis speed,a new fast non-overlapping synthesis algorithm of texture images is proposed based on the consistency criterion of double-seam lines,thereby effectively improving the synthesis quality and speed of structured texture images.Firstly,the seamline consistency criterion considering hue,saturation,brightness and edge characteristics simultaneously is established in HSI color space that is more consistent with human visual characteristic.Then,a sub-block search strategy and a new non-overlapping splicing algorithm based on the consistency criterion of double-seam line are proposed and implemented.The experiment results show that the proposed algorithm can significantly improve the synthesis quality and speed of structured texture images in comparison with the traditional algorithms.

Capturing non-local, long range features present in non-homogeneous textures is difficult to achieve with existing techniques. We introduce a new training method and architecture for single-exemplar texture synthesis that combines a Generative Adversarial Network (GAN) and a Variational Autoencoder (VAE). In the proposed architecture, the combined networks share information during training via structurally identical, independent blocks, facilitating highly diverse texture variations from a single image exemplar. Supporting this training method, we also include a similarity loss term that further encourages diverse output while also improving the overall quality. Using our approach, it is possible to produce diverse results over the entire sample size taken from a single model that can be trained in approximately 15 min. We show that our approach obtains superior performance when compared to SOTA texture synthesis methods and single image GAN methods using standard diversity and quality metrics. [Display omitted] • Synthesizing long-range features in varied, non-homogeneous textures is challenging • Our method combines two models to create high quality, diverse texture variations • Combining VAE and GAN, we leverage probabilistic sampling and fine-detail generation • We estimate feature diversity during training with a sampling loss for regularization [ABSTRACT FROM AUTHOR]

In this paper, we introduced a novel voxel-wise UV parameterization and view-dependent texture synthesis for the immersive rendering of a truncated signed distance field (TSDF) scene model. The proposed UV parameterization delegates a precomputed UV map to each voxel using the UV map lookup table and consequently, enabling efficient and high-quality texture mapping without a complex process. By leveraging the convenient UV parameterization, our view-dependent texture synthesis method extracts a set of local texture maps for each voxel from the multiview color images and separates them into a single view-independent diffuse map and a set of weight coefficients for an orthogonal specular map basis. Furthermore, the view-dependent specular maps for an arbitrary view are estimated by combining the specular weights of each source view using the location of the arbitrary and source viewpoints to generate the view-dependent textures for arbitrary views. The experimental results demonstrate that the proposed method effectively synthesizes texture for an arbitrary view, thereby enabling the visualization of view-dependent effects, such as specularity and mirror reflection.

Example-based texture synthesis is a fundamental topic of many image analysis and computer vision applications. Consequently, its representation is one of the most critical and challenging topics in computer vision and pattern recognition, attracting much academic interest throughout the years. In this paper, a new statistical method to synthesize textures is proposed. It consists in using two indexed random coefficients autoregressive (2D-RCA) models to deal with this problem. These models have a good ability to well detect neighborhood information. Simulations have demonstrated that the 2D-RCA models are very suitable to represent textures. So, in this work, to generate textures from an example, each original image is splitted into blocks which are modeled by the 2D-RCA. The proposed algorithm produces approximations of the obtained blocks images from the original image using the generalized method of moments (GMM). Different sizes of windows have been used. This study offers some important insights into the newly generated image. Satisfying obtained results have been compared to those given by well-established methods. The proposed algorithm outperforms the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Abstract As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over texture appearance. To encourage fidelity to the input condition, we introduce a novel differentiable co-occurrence loss that is integrated seamlessly into our framework in an end-to-end fashion. We demonstrate that our solution offers a stable, intuitive, and interpretable latent representation for texture synthesis, which can be used to generate smooth texture morphs between different textures. We further show an interactive texture tool that allows a user to adjust local characteristics of the synthesized texture by directly using the co-occurrence values.

In this article, we introduce a new mathematical functional whose minimization determines the quality of the solution for the exemplar-based inpainting-by-patch problem. The new functional expression includes finite difference terms in a similar fashion to what happens in the theoretical Sobolev spaces: its use reduces the uncertainty in the choice of the most suitable values for each point to inpaint. Moreover, we introduce a probabilistic model by which we prove that the usual principal directions, generally employed for continuous problems, are not enough to achieve consistent reconstructions in the discrete inpainting asset. Finally, we formalize a new priority index and new rules for its dynamic update. The quality of the reconstructions, achieved using a reduced neighborhood size of more than 95% with respect to the current state-of-the-art algorithms based on the same inpainting approach, further provides the experimental validation of the method.

It has been recognized that texture patterns with abundant high-frequency components, such as grass and water, produce visual masking effects, and the distortion in textures is hard to be perceived by human eyes than structure regions. However, modern video codecs in a rate-distortion optimized manner usually consume a lot of bits to encode textures, leading to the insufficiency in perceptual coding performance. Nowadays, with the rapid development of deep learning, learning based texture synthesis methods have been proposed to replace the coding process of prediction residuals to reduce the rate cost. In this paper, we present a deep texture synthesizer named edge-based texture synthesis framework (ETSF). At encoder side, the framework detects texture regions by semantic and fidelity classification criteria, and the detected regions are quantized coarsely by the hybrid coding framework. In texture characterization, ETSF extracts low-level edge features representing pixel intensity variation. Feature processing tools are developed to remove the spatiotemporal redundancy of edges. The processed edge information is compressed and transmitted. To effectively recover textures, we design an edge-based texture synthesis generative adversarial network (ETSGAN) at the decoder of ETSF, which can incorporate edge information into convolutional layers and generate realistic textures. Experimental results on a collected texture dataset show that the proposed ETSF can achieve an average of -12.8%, -14.2% and -9.6% MOS BD-rate under lowdelay_B, lowdelay_P and random_access configurations of VVC coding, respectively. [ABSTRACT FROM AUTHOR]

Abstract A mural is any piece of artwork sculpted or applied directly on a wall, ceiling or other permanent surface. This artwork symbolizes various culture’s, traditions, historical events, spiritual stories, and civilizations of respective societies of ancient times. But these mural paintings are subjected to degradation either by various natural causes as well as pollution or by human beings without knowing their value. Restoring these paintings requires skilled artisans who are hard to find these days. Consequently, an efficient image restoration technique is required to meet the particular needs of the paintings. Existing in-painting algorithms largely use pixel-based textural reconstruction. The technique, however, does not work well for images with large, degraded portions. and also fails in the restoration of the structure. To resolve these drawbacks, we propose a combined technique for the textural and structural reconstruction of ancient murals. The proposed Extended Exemplar-based Region-Filling Algorithm uses a patch-based reconstruction procedure and masked images are created automatically using the Dynamic Mask Generation Algorithm. The deteriorated portions are identified by creating masks, and masks are created in such a way that degraded portions have a pixel intensity value of one and the remaining part has a value of zero, and filling is done by analyzing the surrounding pixel values of the degraded pixel. The algorithm reconstructs the structure of the paintings efficiently by generating sketches. The proposed technique reconstructs both the structure and textural information, and ensures efficient reconstructed results, compared to existing in-painting techniques. Performance is evaluated by metrics such as the Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM).

Additive manufacturing (AM) technologies have been recognized for their capability to build complex components and hence have offered more freedom to designers for a long time. The ability to directly use a computer-aided design (CAD) model has allowed for fabricating and realizing complicated components, monolithic design, reducing the number of components in an assembly, decreasing time to market, and adding performance or comfort-enhancing functionalities. One of the features that can be introduced for boosting a component functionality using AM is the inclusion of surface texture on a given component. This inclusion is usually a difficult task as creating a CAD model resolving fine details of a given texture is difficult even using commercial software packages. This paper develops a methodology to include texture directly on the CAD model of a target surface using a patch-based sampling texture synthesis algorithm, which can be manufactured using AM. Input for the texture generation algorithm can be either a physical sample or an image with heightmap information. The heightmap information from a physical sample can be obtained by 3D scanning the sample and using the information from the acquired point cloud. After obtaining the required inputs, the patches are sampled for texture generation according to non-parametric estimation of the local conditional Markov random field (MRF) density function, which helps avoid mismatched features across the patch boundaries. While generating the texture, a design constraint to ensure AM producibility is considered, which is essential when manufacturing a component using, e.g., Fused Deposition Melting (FDM) or Laser Powder Bed Fusion (LPBF). The generated texture is then mapped onto the surface using the developed distance and angle preserving mapping algorithms. The implemented algorithms can be used to map the generated texture onto a mathematically defined surface. This paper maps the textures onto flat, curved, and sinusoidal surfaces for illustration. After the texture mapping, a stereolithography (STL) model is generated with the desired texture on the target surface. The generated STL model is printed using FDM technology as a final step. [ABSTRACT FROM AUTHOR]

Existing deep-network based texture synthesis approaches all focus on fine-grained control of texture generation by synthesizing images from exemplars. Since the networks employed by most of these methods are always tied to individual exemplar textures, a large number of individual networks have to be trained when modeling various textures. In this paper, we propose to generate textures directly from coarse-grained control or high-level guidance, such as texture categories, perceptual attributes and semantic descriptions. We fulfill the task by parsing the generation process of a texture into the three-level Bayesian hierarchical model. A coarse-grained signal first determines a distribution over Markov random fields. Then a Markov random field is used to model the distribution of the final output textures. Finally, an output texture is generated from the sampled Markov random field distribution. At the bottom level of the Bayesian hierarchy, the isotropic and ergodic characteristics of the textures favor a construction that consists of a fully convolutional network. The proposed method integrates texture creation and texture synthesis into one pipeline for real-time texture generation, and enables users to readily obtain diverse textures with arbitrary scales from high-level guidance only. Extensive experiments demonstrate that the proposed method is capable of generating plausible textures that are faithful to user-defined control, and achieving impressive texture metamorphosis by interpolation in the learned texture manifold. [ABSTRACT FROM AUTHOR]

The issue of image completion has been extensively developed for a few dozen years and many computational methods exist to tackle various image completion problems. When only one incomplete image is available, the missing regions are relatively large but their texture can be recovered from surrounding regions, then the texture synthesis methods are usually the most efficient. If an incomplete image contains a large number of missing pixels (e.g. 90%) or small missing regions uniformly distributed across it, low-rank approximation or interpolation-based methods seem to be the best choice. However, when the image of interest contains both types of incompleteness, none of the mentioned methods can be applied. In this study, we propose a hybrid computational method combining the radial basis function interpolation with the texture synthesis to tackle the mentioned problem. The experiments, carried out on various image completion problems, demonstrate that our hybrid strategy considerably outperforms the baseline and selected low-rank completion methods. [ABSTRACT FROM AUTHOR]

As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over texture appearance. To encourage fidelity to the input condition, we introduce a novel differentiable co-occurrence loss that is integrated seamlessly into our framework in an end-to-end fashion. We demonstrate that our solution offers a stable, intuitive, and interpretable latent representation for texture synthesis, which can be used to generate smooth texture morphs between different textures. We further show an interactive texture tool that allows a user to adjust local characteristics of the synthesized texture by directly using the co-occurrence values. [ABSTRACT FROM AUTHOR]

Objects in clutter are difficult to recognize, a phenomenon known as crowding. There is little consensus on the underlying mechanisms of crowding, and a large number of models have been proposed. There have also been attempts at unifying the explanations of crowding under a single model, such as the weighted feature model of Harrison and Bex (2015) and the texture synthesis model of Rosenholtz and colleagues (Balas, Nakano, & Rosenholtz, 2009; Keshvari & Rosenholtz, 2016). The goal of this work was to test various models of crowding and to assess whether a unifying account can be developed. Adopting Harrison and Bex's (2015) experimental paradigm, we asked observers to report the orientation of two concentric C-stimuli. Contrary to the predictions of their model, observers' recognition accuracy was worse for the inner C-stimulus. In addition, we demonstrated that the stimulus paradigm used by Harrison and Bex has a crucial confounding factor, eccentricity, which limits its usage to a very narrow range of stimulus parameters. Nevertheless, reporting the orientations of both C-stimuli in this paradigm proved very useful in pitting different crowding models against each other. Specifically, we tested deterministic and probabilistic versions of averaging, substitution, and attentional resolution models as well as the texture synthesis model. None of the models alone was able to explain the entire set of data. Based on these findings, we discuss whether the explanations of crowding can (should) be unified.

Classification of remote sensing scene image (RSSI) has been broadly applied and has attracted increasing attention. However, scene classification methods based on convolutional neural networks (CNNs) require a large number of manually labeled samples as training data, which is time-consuming and costly. Therefore, generating labeled data becomes a practical approach. However, conventional scene generation based on generative adversarial networks (GANs) involve some significant limitations, such as distortion and limited size. To solve the mentioned problems, herein, we propose a method of RSSI generation using element geometric transformation and GAN-based texture synthesis. Firstly, we segment the RSSI, extracting the element information in the RSSI. Then, we perform geometric transformations on the elements and extract the texture information in them. After that, we use the GAN-based method to model and generate the texture. Finally, we fuse the transformed elements with the generated texture to obtain the generated RSSI. The geometric transformation increases the complexity of the scene. The GAN-based texture synthesis ensures the generated scene image is not distorted. Experimental results demonstrate that the RSSI generated by our method achieved a better visual effect than a GAN model. In addition, the performance of CNN classifiers was reduced by 0.44–3.41% on the enhanced data set, which is partly attributed to the complexity of the generated samples. The proposed method was able to generate diverse scene data with sufficient fidelity under conditions of small sample size and solve the accuracy saturation issues of the public scene data sets. [ABSTRACT FROM AUTHOR]

In this paper, we introduced a novel voxel‐wise UV parameterization and view‐dependent texture synthesis for the immersive rendering of a truncated signed distance field (TSDF) scene model. The proposed UV parameterization delegates a precomputed UV map to each voxel using the UV map lookup table and consequently, enabling efficient and high‐quality texture mapping without a complex process. By leveraging the convenient UV parameterization, our view‐dependent texture synthesis method extracts a set of local texture maps for each voxel from the multiview color images and separates them into a single view‐independent diffuse map and a set of weight coefficients for an orthogonal specular map basis. Furthermore, the view‐dependent specular maps for an arbitrary view are estimated by combining the specular weights of each source view using the location of the arbitrary and source viewpoints to generate the view‐dependent textures for arbitrary views. The experimental results demonstrate that the proposed method effectively synthesizes texture for an arbitrary view, thereby enabling the visualization of view‐dependent effects, such as specularity and mirror reflection. [ABSTRACT FROM AUTHOR]

Multiple research studies have recently demonstrated deep networks can generate realistic‐looking textures and stylised images from a single texture example. However, they suffer from some drawbacks. Generative adversarial networks are in general difficult to train. Multiple feature variations, encoded in their latent representation, require a priori information to generate images with specific features. The auto‐encoders are prone to generate a blurry output. One of the main reasons is the inability to parameterise complex distributions. The authors present a novel texture generative model architecture extending the variational auto‐encoder approach. It gradually increases the accuracy of details in the reconstructed images. Thanks to the proposed architecture, the model is able to learn a higher level of details resulting from the partial disentanglement of latent variables. The generative model is also capable of synthesising complex real‐world textures. The model consists of multiple separate latent layers responsible for learning the gradual levels of texture details. Separate training of latent representations increases the stability of the learning process and provides partial disentanglement of latent variables. The experiments with proposed architecture demonstrate the potential of variational auto‐encoders in the domain of texture synthesis and also tend to yield sharper reconstruction as well as synthesised texture images.

A mural is any piece of artwork sculpted or applied directly on a wall, ceiling or other permanent surface. This artwork symbolizes various culture's, traditions, historical events, spiritual stories, and civilizations of respective societies of ancient times. But these mural paintings are subjected to degradation either by various natural causes as well as pollution or by human beings without knowing their value. Restoring these paintings requires skilled artisans who are hard to find these days. Consequently, an efficient image restoration technique is required to meet the particular needs of the paintings. Existing in-painting algorithms largely use pixel-based textural reconstruction. The technique, however, does not work well for images with large, degraded portions. and also fails in the restoration of the structure. To resolve these drawbacks, we propose a combined technique for the textural and structural reconstruction of ancient murals. The proposed Extended Exemplar-based Region-Filling Algorithm uses a patch-based reconstruction procedure and masked images are created automatically using the Dynamic Mask Generation Algorithm. The deteriorated portions are identified by creating masks, and masks are created in such a way that degraded portions have a pixel intensity value of one and the remaining part has a value of zero, and filling is done by analyzing the surrounding pixel values of the degraded pixel. The algorithm reconstructs the structure of the paintings efficiently by generating sketches. The proposed technique reconstructs both the structure and textural information, and ensures efficient reconstructed results, compared to existing in-painting techniques. Performance is evaluated by metrics such as the Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM). [ABSTRACT FROM AUTHOR]

Texture synthesis technology has the advantages of repairing texture and structure at the same time. However, during the filling process, the size of the patch is fixed, and the content of the filling is not fully considered. In order to be able to adaptively change the patch size, we used the exemplar-based inpainting technique as the test algorithm, considering the image structure and texture, calculated the image structure patch size and texture patch size, and comprehensively determined the image patch size. This can adaptively change the patch size according to the filling content. In addition, we use multi-layer images to calculate the priority, so that the order of image repair was more stable. The proposed repair algorithm is compared with other image repair algorithms. The experimental results showed that the proposed adaptive image repair algorithm can better repair the texture and structure of the image, which proved the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Considering the problem of discrete texture synthesis and the time for texturing, this paper proposes a novel framework for synthesizing texture images based on discrete example-based elements. We start with extracting texture feature distribution from exemplars and then produce discrete elements based on the cluster algorithm. After initializing a texture image, we propose a texture optimization algorithm based on heuristic searching to improve the quality of the texture image. Final, we use a texture transfer method based on Convolutional Neural Network (CNN) to stylize the optimized texture image. Our results show that the proposed texture synthesis method can significantly improve the quality of discrete texture synthesis and effectively shorten the time for texture generation.

The goal of exemplar-based texture synthesis is to generate texture images that are visually similar to a given exemplar. Recently, promising results have been reported by methods relying on convolutional neural networks (ConvNets) pretrained on large-scale image datasets. However, these methods have difficulties in synthesizing image textures with non-local structures and extending to dynamic or sound textures. In this article, we present a conditional generative ConvNet (cgCNN) model which combines deep statistics and the probabilistic framework of generative ConvNet (gCNN) model. Given a texture exemplar, cgCNN defines a conditional distribution using deep statistics of a ConvNet, and synthesizes new textures by sampling from the conditional distribution. In contrast to previous deep texture models, the proposed cgCNN does not rely on pre-trained ConvNets but learns the weights of ConvNets for each input exemplar instead. As a result, cgCNN can synthesize high quality dynamic, sound and image textures in a unified manner. We also explore the theoretical connections between our model and other texture models. Further investigations show that the cgCNN model can be easily generalized to texture expansion and inpainting. Extensive experiments demonstrate that our model can achieve better or at least comparable results than the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• Compact polynomial representation of spatio-temporal BRDFs. • Closed-form evaluation of distance between time-varying BRDFs. • Patch-based synthesis of spatio-temporal BRDFs. • Applications: spatio-temporal BRDFs representation, synthesis, and transfer. [Display omitted] We propose a generalization of example-based texture synthesis to spatio-temporal BRDFs. A key component of our method is a novel representation of time-varying materials using polynomials describing time-varying BRDF parameters. Our representation allows efficient fitting of measured data into a compact spatio-temporal BRDF representation, and it allows an efficient analytical evaluation of distances between spatio-temporal BRDF parameters. We show that even polynomials of low degree are sufficient to represent various time-varying phenomena and provide more accurate results than the previously proposed representation. We are the first who applied the example-based texture synthesis on abstract structures such as polynomial functions. We present two applications of synthesizing spatio-temporal BRDFs using our method: material enlargement and transfer of time-varying phenomenon from an example to a given static material. We evaluated the synthesized BRDFs in the context of realistic rendering and real-time rendering. [ABSTRACT FROM AUTHOR]

Classification of remote sensing scene image (RSSI) has been broadly applied and has attracted increasing attention. However, scene classification methods based on convolutional neural networks (CNNs) require a large number of manually labeled samples as training data, which is time-consuming and costly. Therefore, generating labeled data becomes a practical approach. However, conventional scene generation based on generative adversarial networks (GANs) involve some significant limitations, such as distortion and limited size. To solve the mentioned problems, herein, we propose a method of RSSI generation using element geometric transformation and GAN-based texture synthesis. Firstly, we segment the RSSI, extracting the element information in the RSSI. Then, we perform geometric transformations on the elements and extract the texture information in them. After that, we use the GAN-based method to model and generate the texture. Finally, we fuse the transformed elements with the generated texture to obtain the generated RSSI. The geometric transformation increases the complexity of the scene. The GAN-based texture synthesis ensures the generated scene image is not distorted. Experimental results demonstrate that the RSSI generated by our method achieved a better visual effect than a GAN model. In addition, the performance of CNN classifiers was reduced by 0.44–3.41% on the enhanced data set, which is partly attributed to the complexity of the generated samples. The proposed method was able to generate diverse scene data with sufficient fidelity under conditions of small sample size and solve the accuracy saturation issues of the public scene data sets.

Multiple research studies have recently demonstrated deep networks can generate realistic‐looking textures and stylised images from a single texture example. However, they suffer from some drawbacks. Generative adversarial networks are in general difficult to train. Multiple feature variations, encoded in their latent representation, require a priori information to generate images with specific features. The auto‐encoders are prone to generate a blurry output. One of the main reasons is the inability to parameterise complex distributions. The authors present a novel texture generative model architecture extending the variational auto‐encoder approach. It gradually increases the accuracy of details in the reconstructed images. Thanks to the proposed architecture, the model is able to learn a higher level of details resulting from the partial disentanglement of latent variables. The generative model is also capable of synthesising complex real‐world textures. The model consists of multiple separate latent layers responsible for learning the gradual levels of texture details. Separate training of latent representations increases the stability of the learning process and provides partial disentanglement of latent variables. The experiments with proposed architecture demonstrate the potential of variational auto‐encoders in the domain of texture synthesis and also tend to yield sharper reconstruction as well as synthesised texture images. [ABSTRACT FROM AUTHOR]

The thangka image inpainting method based on wavelet transform is not ideal for contour curves when the high frequency information is repaired. In order to solve the problem, a new image inpainting algorithm is proposed based on edge structural constraints and wavelet transform coefficients. Firstly, a damaged thangka image is decomposed into low frequency subgraphs and high frequency subgraphs with different resolutions using wavelet transform. Then, the improved fast marching method is used to repair the low frequency subgraphs which represent structural information of the image. At the same time, for the high frequency subgraphs which represent textural information of the image, the extracted and repaired edge contour information is used to constrain structure inpainting in the proposed algorithm. Finally, the texture part is repaired using texture synthesis based on the wavelet coefficient characteristic of each subgraph. In this paper, the improved method is compared with the existing three methods. It is found that the improved method is superior to them in inpainting accuracy, especially in the case of contour curve. The experimental results show that the hierarchical method combined with structural constraints has a good effect on the edge damage of thangka images. [ABSTRACT FROM AUTHOR]

We present a technique to infer a model of the spatio-temporal statistics of a collection of images of dynamic scenes seen from a moving camera. We use a time-variant linear dynamical system to jointly model the statistics of the video signal and the moving vantage point. We propose three approaches to inference, the first based on the plenoptic function, the second based on interpolating linear dynamical models, the third based on approximating the scene as piecewise planar. For the last two approaches, we also illustrate the potential of the proposed techniques with a number of experiments. The resulting algorithms could be useful for video editing where the motion of the vantage point can be controlled interactively, as well as to perform stabilized synthetic generation of video sequences.

Transferring the style of an example image to a content image opens the door of artistic creation for end users. However, it is especially challenging for portrait photos since human vision system is sensitive to the slight artifacts on portraits. Previous methods use facial landmarks to densely align the content face with the style face to reduce the artifacts. However, they can only handle the facial region. As for the whole image, building the dense correspondence is difficult and may easily introduce errors. In this paper, we propose a robust approach for portrait style transfer that gets rid of dense correspondence. Our approach is based on the guided image synthesis framework. We propose three novel guidance maps for the synthesis process. Contrary to former methods, these maps do not require the dense correspondence between content image and style image, which allows our method to handle the whole portrait photo instead of facial region only. In comparison with recent neural style transfer methods, our method achieves more pleasing results and preserves more texture details. Extensive experiments demonstrate our advantage over former methods on portrait style transfer.

The calculation of the priority of the pixel to be repaired and the determination of the best matching block are based on two basic steps of the texture synthesis image inpainting method.The traditional method is not suitable because it is difficult to determine the confidence in the priority calculation,and it is difficult to search for the best matching block.An image inpainting method based on weighted priority and classification matching was proposed.In the priority model,the exponential function and the normalization function were introduced to optimize the confidence and data items separately,the further benefit of which was a more objective priority of calculation and a more reasonable repair order.Based on the proposed functions,the structure information was used as a measure factor of the search matching block,and the best matching block was selected by the classification and screening method.Experimental results show that the proposed method can shorten the repair time under the condition of obtaining good repair effect.

Abstract This paper surveys the state-of-the-art of research in patch-based synthesis. Patch-based methods synthesize output images by copying small regions from exemplar imagery. This line of research originated from an area called “texture synthesis”, which focused on creating regular or semi-regular textures from small exemplars. However, more recently, much research has focused on synthesis of larger and more diverse imagery, such as photos, photo collections, videos, and light fields. Additionally, recent research has focused on customizing the synthesis process for particular problem domains, such as synthesizing artistic or decorative brushes, synthesis of rich materials, and synthesis for 3D fabrication. This report investigates recent papers that follow these themes, with a particular emphasis on papers published since 2009, when the last survey in this area was published. This survey can serve as a tutorial for readers who are not yet familiar with these topics, as well as provide comparisons between these papers, and highlight some open problems in this area.

Facial aging simulation is one of the most challenging issues in automatic machine based face analysis, where the most essential requirements are (i) human identity should remain stable in texture synthesis and (ii) the texture synthesised is expected to accord with human cognitive perception in aging. In this study, the authors propose a tensor completion based method to transform the simulation task to a standard matrix completion one. To protect human dependent characteristics during texture synthesis, the proposed method processes the two major components, i.e. identity and age, in different channels. Furthermore, they incorporate prior information in such a process, assuming that the textures of different subjects in the same age group are similar and similar looking people tend to age in similar ways, and the metric learning technique is adopted to measure the similarity between identities so that the faces that have the highest similarities with the one in the test image are assigned bigger weights in texture generation. In addition, shape deformation is also considered to make the synthesised images more natural. Experimental results achieved on the FG‐NET database demonstrate the effectiveness of the proposed method.

• Novel strategy for the measurement of mutual perceptual texture similarity. • Generation of continuous transitions between textures via texture interpolation. • User-based localization of intermediate interpolated specimen in the continuum. • Localization uncertainty is related with the perceived pairwise texture similarity. • Suitability for addressing fine-grained similarities. The perceptual similarity of textures has gained considerable attention in the computer vision and graphics communities. Here, we focus on the challenging task of estimating the mutual perceptual similarity between two textures from materials on a consistent scale. Unlike previous studies that more or less directly queried pairwise similarity from human subjects, we propose an indirect approach that is inspired by the notion of just-noticeable differences (JND). Similar metrics are common in imaging and color science, but so far have not been directly transferred to textures, since they require the generation of intermediate stimuli. Using patch-based statistical texture synthesis, we produce continuous transitions between pairs of textures. In a user experiment, participants are then asked to locate an interpolated specimen in the linear continuum. Our intuition is that the JND, defined as the uncertainty with which participants perform this task, is closely related with the perceived pairwise texture similarity. Using a dataset of fabric textures, we show that this metric is particularly suitable to address fine-grained similarities, produces approximately interval-scale measurements and is additionally convenient for crowdsourcing. We further validate our approach by comparing it with a well-established data collection technique using the same dataset. [ABSTRACT FROM AUTHOR]

Texture synthesis methods based on patch sampling and pasting can generate realistic textures with a similar appearance to a small sample. However, the sample usually has to be used throughout the synthesis stage. In contrast, the learned representation of the textures is more compact and discriminative, and can also yield good synthesis results. In this paper, we introduce a learned approach for texture synthesis based on support vector machines (SVM). This approach benefits from the merit of SVM that the sample texture pattern is learned using a model, and the sample itself can be discarded during the synthesis stage; the approach is also used to synthesize three-dimensional surface textures. Experimental results show that our approach is particularly effective in modeling and synthesizing near-regular or regular textures, which are difficult to achieve using traditional parametric texture synthesis methods. We further apply the proposed approach to constrained texture synthesis, image extrapolation, and texture inpainting. For texture inpainting, we develop a new method for automatically detecting holes in textures without the requirement of human intervention. Our approach yields promising results for the three tasks. [ABSTRACT FROM AUTHOR]

Nonlocal texture similarity and local intensity smoothness are both essential for solving most image inpainting problems. In this paper, we propose a novel image inpainting algorithm that is capable of reproducing the underlying textural details using a nonlocal texture measure and also smoothing pixel intensity seamlessly in order to achieve natural-looking inpainted images. For matching texture, we propose a Gaussian-weighted nonlocal texture similarity measure to obtain multiple candidate patches for each target patch. To compute the pixel intensity, we apply the $\alpha $ -trimmed mean filter to the candidate patches to inpaint the target patch pixel-by-pixel. The proposed algorithm is compared with four current image inpainting algorithms under different scenarios, including object removal, texture synthesis, and error concealment. Experimental results show that the proposed algorithm outperforms the existing algorithms when inpainting large missing regions in images with texture and geometric structures. [ABSTRACT FROM AUTHOR]