Additive manufacturing, or three-dimensional (3D) printing, is a promising technology that enables the direct fabrication of complex shapes and eliminates the waste associated with traditional manufacturing. The wide adoption of 3D printing for functional parts is hampered by the poor understanding of the process-quality causal relationship and the absence of an online data-driven prediction method for part quality. This study aims to develop methods for extracting spatiotemporal patterns from real-time, in-process thermal images and then correlating the patterns with part quality. Possible solutions include statistical methods in spatio-temporal modeling such as the Gaussian processes, and deep learning methods such as CNN and RNN. A desirable novelty in the method is how to incorporate physics knowledge about the manufacturing process into deep learning.
Project Introduction Slide Deck
For Week 1, the REU put on a great WebEx conference with a lot of people from near and far! This conference helped discuss the expectations of the summer and become aquatinted with some of the fellow undergraduate researchers in the program!
I also began communication with my mentor for the summer, Dr. Guo (Grace), where we discussed what papers I would need to get up to speed with the project ideas. After reading the papers, I was confused at first of the goal of the project. From what I had read, it had seemed that there were current machine learning techniques that were already deployed that had high success in determining when additive engineered parts would have a defect. Luckily, I was able to discuss this issue with my mentor and the main goal of the project was further clarified. In the past, the neural networks that yielded high success had issues in reproducing the same results when ran in succession. Thus, the problem to tackle for the summer is to find and test different types of neural network architectures that produce both accurate and reliable results when predicting defects in additive engineering processes.
The week also consisted of building this website to provide information about me, my project for the summer, references and contact information. Dr. Guo and I also decided that a good goal for the summer is to have a paper written by the end of the program.
Looking forward to a great and productive summer!
Week 2 Progress Report Slide Deck
For Week 2, I began setting up a jupyter notebook with python 3, tensorflow, keras, and numpy to begin creating a neural network to begin constructing a deep learning model for the melt pool data set.
A lot of preprocessing with files was required to get the model in a state where it could begin to train. All the files in the are separated into values of t-time_x-location_y-location_z-location_layer#.csv. All the values describing defects (1 corresponding to defects and 0 corresponding to no defects) have a corresponding title in a different csv file. Thus, a mapping had to be created between the filename’s datasets and their corresponding defect value (1 or 0). Unfortunately, there may be some errors in naming, because around 1200 files were successfully mapped to their defect value; however, around 400 files did not have an exact match for their title, which caused their data to be disregarded.
An initial neural network architecture was made to begin training the model. However, because of the bulk of data, my laptop was unable to successfully begin to train the model, which resulted in my laptop crashing many times.
I also discussed my progress with my mentor, Dr. Guo (Grace), and a couple of solutions and goals were discussed for the following week. First to cut down on the sheer amount of data values, is to do some pre-processing on the data and create cropped versions of the image matrices that hone in on the melt pool exactly, which will cut down on unnecessary data points. Also, we began to discuss alternative platforms to train the model on. Dr. Guo guided me in the direction of The Office of Advanced Research Computing (ORAC) in Rutgers, which would give me access to GPUs which could speed up training the model significantly, and I submitted a form to create an account. Alternatively, google colab was discussed, a cloud-based server that is often used to speed up machine learning models, to develop on.
Week 3 Progress Report Slide Deck
For Week 3, I first began to migrate existing code to Google Colab so I could have access to GPUs to speed up training. This migration went smoothly because only code on how to read the data files’ paths changed. Google Colab also has a relatively short timeout period, which would cause me to re-read the data files if I did not have constant interaction with it; however, I was able to fix this issue by using javascript in the Google Chrome console to simulate myself clicking on the connect button on Google Colab.
This week also involved creating a more encompassing algorithm to read files. Many files ended up having incorrect titles due to rounding or truncation errors in the files. After including these cases, the new algorithm was able to include 354 of the 358 files that were not included (the remaining four were added manually).
Cropping the datafiles was also an effort to reduce the amount of data points to speed up training and remove redundant data points that were the same in each file. Heatmaps of the data matrices were create using MatPlotLib which was important to determine where to crop the datafiles to fully capture the melt pool.
A re-sampling method to increase the instances of data files that induced a bad status (1 for bad, 0 for good). The purpose of this was to give around a 50/50 split of good and bad instances for the Neural Network to be created.
A simple Neural Network was also created to test the new files. Not many tests with the architecture of the network were used to optimize the model. For the next week, different types of neural networks and architecture will be used to improve the accuracy of the model.
Week 4 Progress Report Slide Deck
For Week 4, the major bulk of the week was attempting different deep learning models and machine learning methods to get a high accuracy score for correctly classifying datafiles and their corresponding status.
I first drew inspiration from existing deep learning models that train on images. Images are composed of 3-layer matrices values from values of 0-255; however, the difference between the images and our dataset is that our dataset contains matrices that have one layer and have values ranging from 0-1750, but the general premise was to be able to train matrices. After modifying an existing image classifier model to fit our dataset, the model was able to get approximately a 79.03% accuracy on test data after 10 trails with the highest accuracy of 98.99% on trial and the lowest accuracy on trial 9 with an accuracy on 46.31%.
Hyperparameter tuning was also explored to try and increase model accuracy and reliability. Unfortunately, after trying to tune batch sizes and epochs for the existing model, it resulted consistently giving an accuracy of approximately 50%.
Ensemble methods were also explored (Logistic Regression, Decision Tree Classifiers, Random Forest Classifiers, XGB Classifiers, Gradient Boost Classifiers, LGBM Classifiers). All of these methods were able to get an accuracy over 99%. The only issue with the ensemble methods is that requires the data to come in as a vector, which requires our current data matrices to be flattened. Despite the good accuracy results, it might not be able to explain the spatial relationship in temperatures to the defects caused by porosity (because a vector cannot display an image like a matrix can).
After meeting with my mentor Dr. Guo, the plan for next week is to keep exploring deep learning models and ensemble methods, but create better justification for using them, rather than solely a high accuracy score. Other ideas to explore was also discussed like Principle Competent Analysis and combining a deep learning model with ensemble methods.
Week 5 Progress Report Slide Deck
For Week 5, the goal was to continue the search for a successful model that could have a high-test accuracy. The main three entities that were investigated this week were Principal Component Analysis (PCA), Random Forest Classifier, and more Neural Networks.
The primary reason PCA was investigated is to try and find the most important pixels that help determine if the following image will correspond to a good or bad status. Ultimately, by using PCA, I was able to get metrices regarding the most influential pixels in terms of the mode and mean.
Random Forest Classifier was explored last week, which produced promising results, so this week more rigorous testing of the model was conducted. 100 trails were conducted (25 each with a 50/50, 60/40, 70/30 and 80/20 train/test split ratio). All training splits received a test accuracy score over 99%, with the 60/40 split being the highest at 99.8%.
Late in the week, a promising result was received for a new neural network model. After 5 trials of the model, it received 100% test accuracy for each trial, which improved from last week which had an average of 79%.
For the upcoming week, algorithms for automatic cropping for the PCA data metrics could also be explored. More rigorous testing of the neural network model needs to be conducted to ensure validity and consistency (like the one performed on the random forest model this week). Possibilities of combining a random forest model with a neural network could also be investigated.
Week 6 Progress Report Slide Deck
For Week 6, the aim of the week was to further test the Neural Network test accuracy and develop two different algorithms to crop datafiles based on Principal Component Analysis (PCA) and the hottest pixel.
The first aim was to test the Neural Network model developed last week. Over 50 trials of various training/test split ratios (50/50, 60/40, 70/30, 80/20, 90/20 train/test splits) were tested, which resulted in a max test accuracy of 87% accuracy with the 80/20 split and a low test accuracy of 72% with a 70/30 train/test split.
Efforts to increase this accuracy was made shortly after. By looking at how the model trains, it was discovered that there was a high correlation of the validation accuracy of the last epoch and the overall test accuracy results. Occasionally, the last epoch’s validation accuracy would plummet due to overfitting of the data. To help combat this phenomenon, a Keras early stopping method API was implemented. By increasing the number of epochs, setting a patience value (the amount of epochs a validation score could decrease before the model would halt training) and setting the model to revert back to the epoch with the best validation score, ensured the model would not finish training on a bad validation score and also return to an epoch with the highest validation score. After testing the model with the early stopping API, all the previous train/test ratios resulted in test accuracies above 99%, which was a very pleasing result.
Last week, PCA was used to find the most important values based on finding the highest magnitude eigenvalue out of all the pixels. This data was then used to create a 100x100 grid with the most important pixel (or average of multiple pixels) being at the center of the grid. By visualizing these new images, it was seen that it did not fully capture the center of the melt pool but rather an area to the left of it. This new cropping algorithm was then put through the two existing successful models, Random Forest and Neural Network, with both models consistently getting test accuracies around 99%.
Finding the hottest pixel was also an algorithm explored, which can be used to automatically find the melt pool without having to use visualization and then a manual crop. After finding the hottest pixel, a 200x200 grid was created with the hottest pixel at the center. These newly cropped images were also put through the random forest and neural network model, where every train/test split received an accuracy over 99%.
For the next week, more testing with lower training percentages will be explored. Also trying to identify which images are being incorrectly identified and seeing if both the random forest and neural network are incorrectly identifying them. Lastly, a rough draft for the methodology and results section of the final paper will be written.
Week 7 Progress Report Slide Deck
For Week 7, the main efforts of the week included testing both the current Random Forest Model and Early Stopping Neural Network with lower training ratios; finding which datasets are being misidentified by the models; and beginning to write the final report.
The first aspect of the week was testing both the Random Forest and Early Stopping Neural Network with lower training ratios, specifically 10/90, 20/80, 30/70, 40/60 training/testing ratios, for all three cropping methods (manual crop, principle component analysis crop, and hottest pixel crop). From the gathered results, the 10/90 split did not perform as well as higher training ratios; however, for both models and all three types of crops the 40/60 crop received similar accuracy scores to higher training ratios. The results from these tests are linked above in Week 7 Slide Deck.
Another effort this week was trying to find commonalities from misidentified data files from both models (i.e. their title, layer, or visual aspects). Unexpectedly, this turned out to be a more difficult task than first anticipated because of the way data is inputted into the model. Titles are first mapped to their corresponding ‘good’ or ‘bad’ status. Then for each title in the map, its corresponding dataset is added to a list and its status is added to another list, thereby each dataset is matched to its status by index location. This turned into a difficultly because the model shuffles the data each time it trains, thus there was no easy way to figure out which datasets match to which titles because the indices are no longer in their original locations. Despite not currently being able to determine which datasets match to which title, visualizations were made about misidentified datasets, which can be seen in the Week 7 Slide Deck.
Lastly, I began to write the final paper for the research, starting with the outline and both the methodology and results sections. For next week, I will continue to work on the paper and find a way to see which titles are being misidentified from the models.
Week 8 Progress Report Slide Deck
For Week 8, results concerning which layers being identified were collected and a rough draft for the final paper was made.
After being unable to find what layers were being misidentified last week, this week there was some success in finding them. By mapping titles to their data matrix and then by adding all the misidentified data matrices to a list after training, a linear search was conducted through the map and for every data matrix in the map, it was checked point-wise with the misidentified data matrices to see if they matched. If they did match, the corresponding titles were added to a list which represents the titles that have been misidentified. This list then gave insights on what layers were being misidentified through some string parsing. These results were promising and gave insight that layer 2 and layer 1 were being incorrectly classified more than others. With some more in-depth test, these results could also be added to the final paper.
This week, I also met with my mentor, Dr. Guo (Grace), to discuss certain aspects of the rough draft of my final paper. Many important aspects of a successful paper were discussed including section layout, what images to include, how to pander to a certain publishing journal, which and how many references to include and more. With this exceptionally helpful advice, I will continue to improve my paper.
For the next and final week of the REU program, I will refine results conducting which titles and layers are being misidentified and continue to work on the final paper. I will also need to prepare for my final presentation on Wednesday.
Week 9 Progress Report Slide Deck
For the final week of the program, I gathered metrics regarding which layers might be incorrectly classified by the models, I prepared and gave my final presentation, and I continued to review my paper with my mentor.
Regarding collecting results for incorrectly predicted data files, I used similar methodologies when test accuracies methods were collected. The methodology included 25 trials, each trial receiving a random split of a 50/50 training/testing ratio. All files that are incorrectly predicted are added to a list. Without counting file more than once, it was seen that Layer 1, 2, and 3 contributed 29 of 75 total incorrectly predicted layers, which can be attributed to the nature of LBAM process. Since in the early layers the LBAM machine is in a cooler state compared to other layers, it makes predicting earlier layers correctly more difficult because they do not fully relate to later layers where the machine is at a consistent heat.
Final presentations were also given this week from Wednesday to Friday. I presented on Wednesday (7/22) and the link to my final presentation can be found above this week’s progress report.
Finally, I met with my mentor, Dr. Guo, again to discuss the progress of the paper. Notes from last week’s meeting were incorporated into the paper. More discussion revolving what journal to submit to was also discussed and a plan was developed regarding how to go about publishing to a journal.
This concludes the progress reports for this summer. I would like to give thanks to Lazaros Gallos and The National Science Foundation for the opportunity to conduct this research. I would also like to give special thanks to my mentor, Dr. Guo (Grace), who helped me tremendously this summer with her knowledge and guidance, it was a pleasure to work with her and continue working with her to pursue a publication.
Acknowledgement of funding goes to the NSF through NSF grant CCF-1852215.
Thank you to the Rutgers DIMACS Team and NSF for the oppurtunity!