key: cord-0296410-u5bddzkz authors: Khanolkar, Pranav Milind; Abraham, Aaron; McComb, Christopher; Basu, Saurabh title: Using Deep Image Colorization to Predict Microstructure-Dependent Strain Fields date: 2020-12-31 journal: Procedia Manufacturing DOI: 10.1016/j.promfg.2020.05.138 sha: 511785ae4f2ff5a326d64327633d3d4d794e65af doc_id: 296410 cord_uid: u5bddzkz Abstract The microstructure of a material governs mechanical properties such as strength and toughness. Various finite element analysis (FEA) software packages are used to perform structural analyses such as predicting the flow of strain or strain fields in a microstructure. Engineers frequently operate these software packages to evaluate mechanical behavior and predict failure. Even though these FEA software packages provide highly accurate analyses, they are computationally intensive, taking a significant amount of time to produce a solution. The time required by the FEA software packages to achieve accurate results largely depends on microstructure details and mesh resolution, thus providing a trade-off between fidelity and computation time. This research proposes the use of Deep Learning algorithms to achieve a significant reduction in the time required to predict high-accuracy strain fields in a two-dimensional microstructure with defects. This work presents a foundation for developing deep neural networks to conduct structural analyses, thus reducing the exclusive use of computationally demanding FEA software and augmenting the analytical capabilities of scientists and engineers. Structural analysis is an important tool for investigating the integrity of a structure by evaluating its stress and strain response under specific boundary conditions. This is often performed using finite element analysis to pinpoint locations that may experience maximum stress and strain that could lead to failure. In this regard, structural analysis helps to assess whether a given structure is able to withstand required loading conditions adequately during its intended life. Finite Element Analysis (FEA), simulations are performed using software such as ANSYS Workbench, COMSOL Multiphysics, ABAQUS, Autodesk Simulation, and FEA Multiphysics and these software packages also facilitate changes in the design might improve functionality. Microscale structural analysis can be performed to study the effects of microstructure defects in accelerating failure. Although the aforementioned software packages are able to conduct such analysis with high accuracy, the time taken to predict the results largely depend on the computational power of the device as well as the complexity of the physics governing behavior of the microstructure. As this complexity increases, the time to predict accurate results increases as well. Predicting the performance of extremely complex microstructures becomes burdensome, requiring a substantial amount of time to complete a single prediction/iteration. Therefore, there is a need to develop suitable methods that compute and predict results in a shorter time without significant loss of accuracy. Machine learning algorithms have proven quite useful to predict and classify various entities based on available data in relatively short time and will be used here to accomplish rapid FEA. Deep learning [1] , a subset of machine learning is a powerful tool in feature recognition that has various applications [2] such as handwriting detection [3, 4] , face Structural analysis is an important tool for investigating the integrity of a structure by evaluating its stress and strain response under specific boundary conditions. This is often performed using finite element analysis to pinpoint locations that may experience maximum stress and strain that could lead to failure. In this regard, structural analysis helps to assess whether a given structure is able to withstand required loading conditions adequately during its intended life. Finite Element Analysis (FEA), simulations are performed using software such as ANSYS Workbench, COMSOL Multiphysics, ABAQUS, Autodesk Simulation, and FEA Multiphysics and these software packages also facilitate changes in the design might improve functionality. Microscale structural analysis can be performed to study the effects of microstructure defects in accelerating failure. Although the aforementioned software packages are able to conduct such analysis with high accuracy, the time taken to predict the results largely depend on the computational power of the device as well as the complexity of the physics governing behavior of the microstructure. As this complexity increases, the time to predict accurate results increases as well. Predicting the performance of extremely complex microstructures becomes burdensome, requiring a substantial amount of time to complete a single prediction/iteration. Therefore, there is a need to develop suitable methods that compute and predict results in a shorter time without significant loss of accuracy. Machine learning algorithms have proven quite useful to predict and classify various entities based on available data in relatively short time and will be used here to accomplish rapid FEA. Deep learning [1] , a subset of machine learning is a powerful tool in feature recognition that has various applications [2] such as handwriting detection [3, 4] , face Structural analysis is an important tool for investigating the integrity of a structure by evaluating its stress and strain response under specific boundary conditions. This is often performed using finite element analysis to pinpoint locations that may experience maximum stress and strain that could lead to failure. In this regard, structural analysis helps to assess whether a given structure is able to withstand required loading conditions adequately during its intended life. Finite Element Analysis (FEA), simulations are performed using software such as ANSYS Workbench, COMSOL Multiphysics, ABAQUS, Autodesk Simulation, and FEA Multiphysics and these software packages also facilitate changes in the design might improve functionality. Microscale structural analysis can be performed to study the effects of microstructure defects in accelerating failure. Although the aforementioned software packages are able to conduct such analysis with high accuracy, the time taken to predict the results largely depend on the computational power of the device as well as the complexity of the physics governing behavior of the microstructure. As this complexity increases, the time to predict accurate results increases as well. Predicting the performance of extremely complex microstructures becomes burdensome, requiring a substantial amount of time to complete a single prediction/iteration. Therefore, there is a need to develop suitable methods that compute and predict results in a shorter time without significant loss of accuracy. Machine learning algorithms have proven quite useful to predict and classify various entities based on available data in relatively short time and will be used here to accomplish rapid FEA. Deep learning [1] , a subset of machine learning is a powerful tool in feature recognition that has various applications [2] such as handwriting detection [3, 4] , face 48th SME North American Manufacturing Research Conference, NAMRC 48 (Cancelled due to detection or recognition [5, 6] , and natural language processing [7, 8] across different fields of science [9] [10] [11] , engineering [12] and design [13] [14] [15] . Convolutional Neural Networks (CNNs) [16, 17] facilitate deep learning that are particularly useful for the analysis of data that can be represented as images. Shea and Nash [17] have clearly depicted the use of CNNs and their uniqueness in recognizing patterns through series of layers, that makes CNNs suitable for image-oriented applications. Recent research in the field of machine learning has focused on the prediction of material behavior and properties based on microstructural characteristics [18] [19] [20] . The current research focusses on the prediction of strain fields due to a uniform displacement boundary condition imposed on a microstructure with defects. Specifically, this work will predict strain fields associated with a two-dimensional (2-D) microstructures with defects. To accomplish this analysis using deep learning, we will test the efficacy of image colorization CNNs [21, 22] . CNNs of this type take a greyscale image as input (here, the microstructure), and provide three images on red, green, and blue channels as output. We hypothesize that problem of prediction of strain fields can be treated as an image colorization problem, wherein the microstructure image can be treated as the original gray scale image (e.g. in conventional image colorization) and strain components (e.g. ε11, ε22 and ε12) are predicted as output color channels. This problem is motivated from challenges in advanced manufacturing of complex geometries and microstructures. Volumetric microstructure defects, e.g. porosity, are often produced in these scenarios and must be mitigated before the fabricated component is deployed. However, routes for their mitigation are challenging and require understanding of the behavior of the advanced geometry + microstructure + defect triad under various loading conditions. Occasionally, these defects are too challenging to mitigate and hence, components may be designed and fabricated with the assumption that they will contain tolerable defects, thus also necessitating an understanding of their behavior in complex loading conditions. However, traditional computational routes, e.g. FEA might be too resource intensive to address this requirement. The remainder of the paper is organized as follows. The background section provides a brief overview of the current research and developments made in the field of material sciences using FEA along with a brief review of the basics of deep learning. The next section describes the methodology of this work including data generation and the application of the deep learning algorithm. In the next section, the CNN is trained and tested on data generated using a Finite Element Analysis software. Several experimental analyses highlight the effect of changing the number and size of defects in the microstructures on accuracy and computation time performance of the CNN (and FEA, as a baseline comparison). The final section ends with discussion of conclusions and future work. Modern advanced materials are characterized by a wide range of mechanical, optical or thermodynamic properties those of which are frequently attributed to a microstructure [23] . Herein, the advent of computational modelling has enabled researchers to conduct various parametric studies on a microstructure and improve coupled processes. Computational modelling is extremely useful in both obtaining specific material properties and also in designing new microstructures [23] . To this end, FEA software is useful for analyzing the properties of novel microstructures. FEA software has become a very important aspect of industrial-level and academic research. Initially researchers used to study macroscopic behavior of microstructures like chemical properties, crystal structure and crystal orientation by using programs like Object-Oriented Finite (OOF) elements and Portable Pixel Map to OOF format translator (ppm2oof) [24] . Programs like OOF enabled the use of mean-field and other spatial averaging methods to produce approximate models for materials behavior [24] . Before the advent of OOF programs, it was significantly more challenging to correlate microstructures with important properties that affected the integrity of the structure. With the development of such a software, a finite element mesh and an easy-to-use graphical interface became the standard for structural analysis of any component. The purpose of such software is to assign properties to features in a material's image and generate a finite-element mesh representing the material useful for conducting desired analysis. This work specifically makes use of ABAQUS for FEA simulations. ABAQUS is one widely-used software for performing detailed simulations [25] . In ABAQUS several parameter configurations with different computational costs are utilized for finite element analysis such as stress-strain prediction, flow of temperature etc. [26] . The various types of algorithms present in ABAQUS provides different types of advantages and disadvantages, based on the problem that needs to be solved. Although ABAQUS is a very useful software for finite element analysis especially for choosing the correct algorithm for a problem, researchers often face difficulties when performing analysis since the speed and performance level of the software is limited by the processing power of the hardware used. FEA software can take a significant amount of time to yield a converging solution, and this time increases substantially when detailed, e.g. high spatial resolution results are necessary or when a complex microstructure is analyzed. These shortcomings of FEA software highlight the need for better analysis tool to provide results with high accuracy in relatively short time. With the increasing applications of machine learning algorithms in the field of material sciences [19, 20, [27] [28] [29] , both prediction [30] [31] [32] and classification [33] [34] [35] of mechanical properties and material parameters can be done more quickly and with less computation power than traditional analyses . Work by others has effectively used neural networks to predict shrinkage in molds [30] , analyze properties of composites [31, 32] , provide classification of steel microstructures [33] and perform microstructure-recognition through classification [34] . The utilization of neural networks provided motivation for the work performed here to predict strain fields in microstructures using CNNs. Generally, deep learning is used to model and understand that contain complex relationships [1, 36] . Deep learning algorithms extract patterns often termed as features in the data, and through a process of self-learning provide a prediction such that the final error computed between the actual output and the predictions is minimum. Many deep learning algorithms entail the use of artificial neural networks. These are computing systems that can solve complex real-world problems through layers of interconnected nodes or neurons that perform parallel computations [37] . The reader can find additional background in [2] . Image recognition is one of the most popular applications of deep learning algorithms. Manufacturing applications can benefit from utilizing CNNs, where imaging is primarily used for analyzing the material properties through examining the effect of the certain boundary conditions on the given material. Furthermore, works involving surveying three-dimensional structures [12, 15] as well as additive manufacturing [13] have successfully incorporated the use of CNNs to achieve their desired results. There are numerous deep neural networks that are used for image recognition, classification and manipulation [34, [38] [39] [40] [41] . One type of image manipulation involves changing the color scheme of the input image. An extremely common type of image manipulation is colorization [22, 42] in which a colored version of a greyscale image is inferred. The research documented in this paper will highlight a colorization approach to predict strain fields in two-dimensional microstructures with specific boundary conditions. The data generated and use of CNN for this purpose are depicted in the next section. Synthetic data was generated in the ABAQUS PDE (Python Development Environment). Specifically, a 2-D microstructure was generated with Aluminum (Al6061 -T6), rectangular dimensions, 38.1 mm as length and 6 mm as breadth, for planestrain simulation of elastic deformation, according to the ASTM E8/E8M standards [43] . For initial training and testing portion, each of the microstructures were modelled with 100 circular pores with radii randomly selected between 100 and 500 µm with their locations within the microstructures set according to uniform distribution. The finite element analysis conducted involved uniform uniaxial displacement of 0.1 mm in the plane-strain mode to obtain strain fields. Three plane-strain fields obtained were ε11, ε22 and ε12 as displayed in Fig. 1 . Traditionally, deep learning algorithms tend to require a substantial amount of data in order to perform well [29] . To that end, a data generation pipeline was developed to produce 1000 samples. After the samples were generated, a MATLAB code was developed to convert the 2-D microstructures and their respective strain fields into images. The code converted the microstructure model and its resultant strain fields into images with a resolution of 1 pixel equivalent to 0.06 mm, such that the images consist of 101 pixels in the vertical direction and 636 pixels in the horizontal direction. This resolution was chosen to preserve the major features of the microstructure with the minimum number of pixels. The inputs to the CNN were the 2-D defects-ridden microstructures whose pixel values were either 0 (indicating a defect) or 1 (indicating material), depicted in Fig. 2 . The strain fields from the ABAQUS simulations were also converted to images with pixels similar as the input (101 × 636) using the MATLAB code as displayed in These samples served as training and testing data for the development of the CNN. A Convolutional Neural Network (CNN) was developed to perform an image colorization as a means of strain field inference. Several CNN architectures, with a varying number of convolutional layers and corresponding parameters such as number of filters, kernel size, activation functions, and strides were evaluated in terms of prediction time and accuracy. In general, increases in the number of convolutional layers, filters sizes and number of filters resulted in increase in time required for training the corresponding CNN and subsequent increase in time to predict strain fields. Ultimately, an 18-layer CNN consisting 2-D convolutional layers with filters, kernel sizes, padding, activation functions and up-sampling layers, as shown in the Table 1 , was developed and selected for this work, based on the accuracy and prediction time of the results. The CNN starts with the input layer defined by the shape obtained in terms of pixels (101×636). The following layers are specified with number of filters to capture maximum detail within the images formed in the layers, kernel size within each layer to set the dimensions of the weights, padding to account for the edges of each images so that all pixels are effectively computed and dimensionality of the images is maintained, appropriate activation functions to provide outputs to every following layer and strides to move the filters across the images to account for every pixel. These layers ultimately converge to the output layer consisting of 3 images each denoting strain fields ε11, ε22 and ε12 as colored images. Coefficient of determination (R-squared) was used to assess the accuracy of the CNN. The R-squared value was obtained by comparing the predicted output and actual output, both represented by 3 strain field images each. Of the 1000 samples created using the data generation pipeline, 500 samples were used for training and 500 samples were used for testing. This division in datasets is done in order to estimate how the model is expected to perform on samples not used during the training of the model. The CNN was trained for 100 iterations. The Adam optimizer [44] was used for training the CNN with default parameters of 0.001 for learning rate, β1 = 0.9 and β2 = 0.999. ReLU activation functions were utilized throughout the CNN except for the final two layers where TanH activation function were used to provide appropriate pixel values in accordance to the actual strain values. These activation functions [45] ReLU and TanH are illustrated in Fig. 4 . A device with GeForce 940M GPU and 6th Gen Intel Core i7-6500U processor as CPU was used for CNN development and simulation. With GPU usage, the trained CNN model provided results with an R-squared values 96.25% obtained from the dataset used for testing. Fig. 5 displays an example of the input data (black and white 2-D microstructure) along with the output data (colored strain fields) obtained from the CNN. The strain fields which are displayed in Fig 5, were predicted by the CNN with R-squared value of 95.33%. Further analysis was conducted for 50 samples selected randomly from the 1000 samples by comparing the time required by the CNN prediction and the original ABAQUS simulation time. The ABAQUS simulation time considered in this work includes the problem solving time, subsequent to the microstructure modeling and meshing stages. The results of the mentioned computation time comparison are displayed in Fig. 6 . The mean computation time required for ABAQUS was found to be 2.23 ± 0.0433 seconds, while the CNN predicted the results in 0.13 ± 0.0171 seconds. This serves as a strong indication that CNNs have efficacy for computing material properties in a rapid fashion that enables quick iteration. Two one-factor experiments were conducted to analyze the effects of varying size and number of defects in microstructures to assess their impact on computation time and accuracy of the CNN in comparison to ABAQUS. In the first experiment, the size of the pores in a microstructure was randomly varied from 10 µm to 500 µm in increments of 50 µm, each range distributed uniformly, with the number of pores kept constant at 50. In the second experiment the size range of the pores was kept constant between 100 -300 µm and the number of pores incremented from 20 till 200 in uniform increments of 20. For both of these experiments, the original CNN (trained with images of microstructures having 100 pores of size range 100-500 µm) was used to make predictions. Fig. 7 and 8 present the results of these experiments. The computation time of the CNN and the corresponding accuracy in terms of R-squared values are largely independent of the size and number of the defects. Hence the graph for the prediction times by CNN is always (almost) uniform around 0.1 -0.2 seconds. In contrast, these values greatly influenced the prediction time exhibited by ABAQUS. The computation time required by ABAQUS to predict strain fields is displayed in Figs. 7 and 8, and summarized in Table 2 and Table 3 , along with the number of elements created during the FEA mesh for the respective microstructures, respectively. The computation time is dependent on the number of elements created for the model during the meshing as depicted in Table 2 and Table 3 . As more defects are generated, or as these defects become smaller, a larger number of elements are required in the mesh, which greatly increases the computational time. As the radii of the pores increases, the mesh becomes coarser which requires that fewer elements for respective microstructure model for FEA. Precise values for the number of elements in these meshes are tabulated in Table 2 . The increase in the radii-range of the pores results in decrease in curvature as well as the decrease in proximity of these pores, which is why the number of elements decrease as the radii range increase. The parabolic shape of FEA prediction times with maximum at 150 µm can be justified based on this reasoning. In contrast, the number of elements in the CNN is constant by definition. In the case of a fixed radii range (100-300 µm), increase in the number of pores resulted in increase in the number of elements created for the particular samples considered in Fig 8, as displayed in Table 3 . Thus, the computation time required the FEA software largely and proportionally depends on the features present in the given model, which in this case are the number of pores and radii-range of pores along with the proximity of these defects/pores that define the number of elements required for meshing. Since the weights of the CNN were finalized by training on 500 samples, the computation time required is almost uniform and only dependent on the pixel size of the input image of the microstructure whose strain fields are to be predicted. These weights are set by capturing maximum details and features such as curvature/radii of the pores as well as the proximity of these pores within the trained microstructures. In such manner, the accuracy of predicted strain fields of the microstructures largely depends on the finalized weights set by the training dataset of 500 samples of microstructures each having different defect layout and size. Thus, different microstructures, with distinct range of radii coupled with their locations within the microstructure, will be predicted with different accuracies as depicted in Figs. 7 and 8. As displayed in Figs. 7 and 8, the accuracies of predicted microstructures, with varied radii range and number of pores along with their locations set randomly following a uniform distribution, are obtained within the range 92-97%, which is remarkably high, proving that the CNN works for wide range of defect layout and pore size. Furthermore, strain fields of microstructures with defects in shape of ellipses were predicted using the CNN. These predictions were made with mean accuracy of 95.20%. FEA software, such as ABAQUS, requires a considerable amount of time to predict results of structural analysis. Changes in the complexity of the structural model, such as increasing the number or decreasing the size of microstructural defects, may lead to increase in the computation time. This highlights a need to reduce the computation time in FEA analysis in order to enable more rapid iteration by engineers. This work attempts to meet that need by developing an image-colorization CNN to predict strain fields in microstructures. The overall accuracy of the CNN developed was approximately 96%. The computation time performance of the CNN developed in predicting the strain fields was almost uniform in the range 0.1-0.2 seconds, irrespective of the shape, number and position of the defects within the microstructure, whereas the computation time required by FEA software fluctuated between 1 to 5 seconds based on the size and number of defects present. Furthermore, it was observed that changes in the number, size and position of the defects in the microstructures had a significant effect on the computation time of the FEA software because these variables resulted in changes to the number of elements created in the FEA mesh. To summarize the main outcome of this project, the time required to predict strain fields by CNN based colorization algorithm was significantly less than the time required by FEA software (ABAQUS), as displayed in Figs. 7, 8 and 9. There are certain limitations to this research. The CNN developed was trained on synthetic data, generated specifically for this work. Any microstructure with a complex shape and size of defects other than circular pores may not be predicted accurately. Moreover, non-obvious characteristics that are present in empirically-measured data may lead to low prediction accuracy, as CNNs are often not robust to variability in data. Even though the CNN predictions require less time, the training of the CNN requires considerable amount of time and computation power, often meaning a GPU for effective learning. Therefore, it is likely that CNNs such as that developed here will be most useful in applications where a large number of simulations will be performed for a consistent set of input geometries. Otherwise, the large amount of training time required will not be outweighed by a comparable amount of analysis time. In addition to that, this CNN is only capable of predicting strain fields of microstructures with the input image shape, boundary conditions, and defect characteristics on which it was trained. It is important to note that the CNN must be retrained for other microstructure images. Future work can be done to address these issues by including various shapes of defects and their locations, set through different distributions such as normal, exponential along with clustering these defects, to address a wide range of microstructures. Furthermore, research work using CNNs can be done to predict mechanical behavior within microstructures with non-homogenous material properties, including inclusions of varying sizes and compositions. In addition to that, microstructure strain fields due to different boundary conditions can be labelled based on their boundary conditions and utilized for training the CNN to predict strain fields for microstructures with specified boundary conditions within those labelled ones. A tutorial survey of architectures, algorithms, and applications for deep learning The MNIST database of handwritten digit images for machine learning research CNN based common approach to handwritten character recognition of multiple scripts Face detection using convolutional neural networks and gabor filters Face recognition using the nearest feature line method Unsupervised feature learning for audio classification using convolutional deep belief networks Recent trends in deep learning based natural language processing Deep learning for computational chemistry Opportunities and obstacles for deep learning in biology and medicine Searching for exotic particles in high-energy physics with deep learning Toward the Rapid Design of Engineered Systems Through Deep Neural Networks Design Repository Effectiveness for 3D Convolutional Neural Networks: Application to Additive Manufacturing Learning to Design From Humans: Imitating Human Designers Through Deep Learning Predicting Part Mass, Required Support Material, and Build Time via Autoencoded Voxel Patterns Introduction to convolutional neural networks An Introduction to Convolutional Neural Networks n.d Machine learning for molecular and materials science Performance of neural networks in materials science Neural networks in materials science Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2 Colorful image colorization Computational materials engineering: an introduction to microstructure evolution Oof: An image-based finiteelement analysis of material microstructures Comparative Study of Finite Element Simulation Software. Consort Deburring Edge Finish Comparison of implicit and explicit finite element methods for dynamic problems Parameter identification of a mechanical ductile damage using Artificial Neural Networks in sheet metal forming Neural networks in computational materials science: Training algorithms The use of artificial neural networks in materials science based research Shrinkage Analysis of Molded Parts Using Neural Network Artificial neural networks applied to polymer composites: A review Prediction of density, porosity and hardness in aluminum-copper-based composite materials using artificial neural network Advanced steel microstructural classification by deep learning methods Image driven machine learning methods for microstructure recognition Insightful classification of crystal structures using deep learning Deep machine learning-A new frontier in artificial intelligence research Artificial neural networks: A tutorial. Computer (Long Beach Calif) Deep learning of feature representation with multiple instance learning for medical image analysis. ICASSP, IEEE Int PCANet: A Simple Deep Learning Baseline for Image Classification? Deep learning for medical image processing: Overview, challenges and the future Learning a deep compact image representation for visual tracking Deep colorization Conf. Comput. Vis ASTM E8M-13a A method for stochastic optimization. 3rd Int. Conf. Learn. Represent. ICLR 2015 -Conf. Track Proc This material is based upon work supported by the National Science Foundation under Grant No. 1825686. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.