During week 1, I began my research into the visual system and its connection to apical amplification. I read a slew of papers on apical amplification in general, such as
this paper, which was an in vivo study on the effect of apical amplification on weak dendritic inputs, and
this paper, which describes how we can look at the effects of apical amplification through the lens of a Bayesian calculation. One of the primary papers of interest on apical amplification is
the one co-authored by Dr. Silverstein from the Division of Schizophrenia Research at Rutgers.
Additionally, I did some research into the programming libraries that I will be using for the project. The majority of the project is being done in Python with which I have a lot of familiarity. The main package that I need to familiarize myself with is PyTorch. This library offers many options for machine learning including deep neural networks along with a lot of flexibility and control in how these models are implemented. This is important as we need to train a spiking neural network (SNN) model with various biological aspects such as multiple dendritic compartments and somatic voltage, which is not seen in the neurons of artificial neural networks (ANNs).
During week 2, I got in contact with one of the members of Professor Michmizos's group since he has been working on the model for visual impairments in schizophrenia as well. He sent me a series of papers that describe implementations of spiking neural networks with apical amplification feedback for the purpose of classifciation tasks like in the class MNIST image classifciation and decorrupting image data through reverse visual input (i.e. the output feeds to the input to help us predict what the input was that lead to a certain output).
It seems to be that the fundamental issue that needs to be addressed for construction of the model is understanding how apical amplifciation may help solve the credit assignment problem in hierarchical SNNs. The credit assignment problem is a problem in reinforcement learning first described by Marvin Minsky in
this paper. Simply put, how do we decide the value of each neuron's contribution to the success of the network and therefore assign it an appropriate weight. It is possible that within our own brains, this is being done with error feedback mechanisms such as apical amplification from later layers to earlier layers.
Additionally, I was also able to get some code that the same member of Professor Michmizos's group has been working that somewhat implements the model similarly to how it is described in the papers. I have devoted the rest of this week to debugging and tuning the model and making adjustments to see if we can get results similar to those in the paper. Once this is established we can make further adjustments by training the network to hopefully mimic schizophrenic visual impairments. We have some data of actual visal learning tasks done by individuals with schizophrenia so we should be able to use this as our training data.
This week, I began to invest more time into documenting and understanding the code of the model that I received last week. There was once recently published paper upon which most of the model is based on. This paper has describes a hierarchical neural network that includes the mechanism of apical amplification. This additional mechanism is supposed to aid in the tuning of weights to be able to better adjust to new input as it is fed in. This draws a parallel to the idea described in
this paper, which is that the apical compartment essentially acts as a form of context sensitivity.
When there is no change in context over a reasonable time scale, all we need is the changing sensory input in order to understand the holistic event which we are perceiving. In our model, this is represented as a silencing of the apical comparment. This is due to the fact that the sensory and contextual inputs are modeled as competing mechanisms and if they are equal, they cancel out the input at the apical site. However, if the context suddenly changes, i.e. we move to a different room than we were in previously or the people around us are no longer there, we need the understanding that the stimuli we are perceiving are occuring through a different contextual system. As such, apical amplification allows the difference in context and sensory input to be registered. When apical amplification occurs, it begins a feedback mechanism that drives the neuron's behavior, i.e. its firing, in the direction that tries to match the context.
This week, the PhD student from which I received the first draft of the model returned to the lab. We began working to finalize the model and have it properly trained on the MNIST image dataset. We were able to get through debugging and running our first successful iteration of the model by the end of the week. There does however appear to be a bottleneck in speed. This is mostly because our model requires a lot of sequential steps, which are difficult to parallelize. Additionally, our data structures are potentially not the optimal choice for this model, so we may have to see if there are better approaches to speeding up the running time. Overall, this week was very productive. We set a goal and were able to follow through with it faster than expected.
This week, I and the team I am working with at the lab had to prepare a presentation for Dr. Silverstein's group in order to present the results of our model and to discuss further directions for our research. Most of the week was spent cleaning up the model, making plots of the data, and preparing our presentation.
Cleaning up the model wasn't too difficult as most of the work for that had been done last week. We spent a bit of time trying to speed up the model as described in last week's update, but this proved very difficult, unruly, and eventually, not fruitful. After timing the runtime of the parallelized model, we found that it worked slower than our original version by a hefty margin. So, instead, we opted to simply stick with our current implementation for the time being.
Much of my time at the lab this week was spent helping to prepare the presentation. For the presentation, we prepared a slideshow detailing our implementation and results and I also worked on making a visual representation of the model. The visual representation of the model was done in JavaScript and HTML. I generated an animation which showed the input, hidden, and output neurons along with all the connections between them. Additionally, each equation in our model was placed next to a relevant location in the visualization to show which equation applies to what. This idea for creating a visual representation of our model using HTML and JavaScript was inspired by our visit to Bell Labs from a few weeks ago. One of the presentations was done entirely in JavaScript and HTML and I found the flexibility and possibilities of the format to be very attractive.
A lot of time was also devoted to preparing nice and understandable plots for the presentation. These plots were made to show how our model learns over time and the different phases of training it undergoes. I also realized during this process how important aesthetic clarity is for a plot to make sense. Our first plots were much less readable and only after some tweaks to the font sizes, labels, line colors, and widths, did the plots start to look really nice.
The week ended on Friday with us going to Dr. Silverstein's lab and presenting our findings to him. Our presentation got a very good reception from him and now we are discussing how to move forward. Next week, we should have Dr. Silverstein's dataset from and actual visual task done on individuals with schizophrenia and we should be able to start using this data with our existing model. What will be a challenging task is the fact that the input images are basically a bunch of small gratings arranged in such a way that you should be able to make out a contour. However, this is not the standard format for image classification tasks. So we intend to perform some image preprocessing that will hopefully help us train the model with much less headache.
This week, we received the dataset from Dr. Silverstein as promised. Our main objective is basically to train our model on the dataset that we have received and have it perform with an accuracy similar to that of the control group of the task (the individuals who do not have schizophrenia). Once we can replicate this accuracy, we want to modify the parameters of the trained model so that its accuracy now matches the Schizophrenia group. Then we will examine the network architecture and see what has changed. Hopefully, the change will have something to do with the apical compartment, supporting the idea of apical amplification.
We began by first doing some analysis on the data. The first part of the analysis was on the input images. They were of the format described at the end of last weeks post. We decided we need to find a way to isolate relevant features of the image so that the model has an easier time finding the contour in the image. I tried one approach while the PhD student I am working with tried a different one. My approach was based on an implementation of a paper written by Professor Michmizos from a few years ago. We were successful in reproducing the results of that paper with an implementation in Python, however, it didn't seem like the output of this model would be useful in the end. So we decided to continue with the other approach. This involved an implementation of convolutional neural network-based architecture to extract features. However, even this proved to be difficult. So we decided to see if we could first get features by training some algorithm that does contour integration. We were unable to find something that specifically is made to integrate contours so instead, we decided to look at image segmentation networks and see if this would be helpful. For this to work, we need to manually trace out the contour for each image and hopefully the model will learn what the contour is for images without the directly traced line.
The other sort of analysis that we did on the data was on the actual results of the participants of the experiments. We needed to get statistics of success and failure rates in the control and schizophrenia groups to see what sort of accuracy in classification we want from our model and whether there is a big enough difference between the control and schizophrenia groups to be able to duplicate it in our model. Another difficulty is that the task of the experiment was to label whether the contour in the image is facing left or right, but there is always a contour present in the image. This is a more difficult task for our model than simply seeing whether there is a contour or not since the model needs to infer some information on the orientation of the contour and not just its existence. So we are trying to see from the statistics whether we can reformulate this task so that whenever the success of classification is no better than random chance, we classify the contour as not there and otherwise the classify the contour as there. Hopefully, this modification should help the model perform better.
This week, we spent a lot of time generating the training dataset for our chosen image segmentation network- the u-net. The u-net was originally meant for image segmentation for medical images so, for example, identifying a tumor from the surrounding organs and tissues or differentiating between different types of cells in a zoomed in image of tissues. Our purpose for it is very different, we want it to segment the contour from the rest of the surrounding image. This is a good test of the generalizability of the network to arbitrary image types. Generating the training dataset for this network basically consisted of going through each image and tracing out the contour by hand and then saving the image with the contour as one color and the background as another. Then, we began the task of fitting the u-net architecture to our purpose and ensuring it worked with our input images and desired outputs.
Here's where things got a bit problematic. Our initial problem began with issues with our hand-generated target masks that I described in the paragraph above. Firstly, we figured out that we had to binarize our images i.e. ensure that the traced out contour was one consistent solid color and the background was also one solid color. Additionally, we confronted the problem of some of the images not being the same size as others. So we first deleted the images that didn't match the scale of most of the other images. Next, we binarized the remaining images. Then, all that was left was to train our u-net on this dataset. We found that our way of saving or viewing the network was not working for some reason. It appeared like all of the weights were very low, making the output all black, with none of the contour to be seen. As of the end of this week, this problem persists. We will continue to look into this next week.
Another thing I invested a lot of time into this week was my final presentation. Here, I brought together a cohesive update on my work starting from the initial idea for the project I and the group at the lab have been working on and going up to my most recent progress. Since this presentation was meant for people with very little expertise in my particular field of interest, I made sure to try to make the presentation as approachable and understandable as possible for someone without much prior knowledge in the field of brain inspired computing or machine learning. Additionally, I was able to integrate the visual representation of the dendritic model that I had made for the presentation for Dr. Silverstein into this presentation as well.
Overall, this week was a mixed bag of failures and successes. But, generally it feels like we are moving in the right direction.
This week, the PhD student that I was working with was on leave for surgery for much of the week and afterwards was in and out. This left figuring out what was wrong with the u-net to me. I was excited to finally take on a task entirely by myself, but also a bit concerned that this might also be a bit out of my reach. With only two weeks left, I wanted to make sure I wasn't hitting my head against a brick wall all week. So I set out to do three things: first was to get the full statistics we would need from the dataset we were given so that we could see what our target accuracy should look like for the dendritic model. This was first because I had a clear cut and necessary goal which was easy to acheive and just required the time to be put into it. Next, I wanted to clean up some of the code and data from our codebase. This was important because as we were iterating with new attempts at approaching the project, we had been amassing a lot of cluttered and undocumented code along with fragmented data. Finally, I wanted to go deep into the various bits of the u-net architecture and look into what it was doing in more detail.
Generating the statistics on the data was pretty straightforward and I confronted very few issues there. Afterwards, cleaning up the codebase wasn't too difficult either. I just wanted to make sure that I wasn't breaking anything we had along the way and that my organization was always an improvement and not a further obstacle. Finally, I came upon the u-net. This was where I had to get creative.
Since the PhD student had done most of the u-net implementation, I had to take some time to first understand how the architecture worked. Next, I started to unpack what the network was doing layer by layer. The u-net works with two streams. First, it performs a bunch of convolutions and downscaling by max-pooling. This is called the contraction stream. This generates a bunch of small features that should represent the most important features of the input to the network. Next, the features get upsampled in a way that the network can learn through transposed convolutions and then those upsampled features are concatenated with the features in the corresponding layer in the contraction stream. Then they get convolved again. This is called the expansion stream. The contraction stream was where I focused my attention first since the last layer there was where we wanted to get our features from for the dendritic model. I started by displaying all of the features for each layer side by side. These features actually seemed to hold some meaningful information. Next, I observed the output layer as the network trained. To my surprise, the network seemed to be doing exactly what we wanted it to do: it was learning where the contour was in each image. The mistake was only in how we were later loading and viewing the trained network. This was extremely exciting news and a welcome change from the drudgery of the week. This experience was maybe my favorite of the entire experience in the program so far. It showed me that with a combination of detailed work and creative thinking, I could solve a seemingly difficult and unapproachable problem.
This week was the final and possibly most hectic week of the program for me. I had to begin writing my final paper on my research subject. However, most of the results that I wanted were not yet acheived. The u-net was finally working, but we didn't even know if we could properly integrate the features from the u-net into the dendritic model and train it successfully. Thankfully, the PhD student had finally returned and was coming into the lab more frequently.
We began working to ensure that we had the optimal u-net architecture first. This meant finding the smallest architecture that still yielded meaningful results. This was because we wanted our input features to be as small as possible to the network. After we accomplished this, it was time to go back through the code for the dendritic model and clean it up so that it was modular and usable for our new inputs. First, however, we needed to generate our new labels for each image, i.e. whether a contour was easily discernible from the background or not. For this, we used the statistics I had generated and chose a threshold of percent of correct choices by the test subjects to dictate that the contour was easily discernible or not. This gave us our new labels. Next, I went into the dendritic model code and made some functions for saving weights and caching previous iterations of the model. This was important for rapid iteration and testing of the model.
In tandem with this, I kept writing my paper. I found that as I wrote the paper, it became more apparent to me how much we had actually accomplished in these two months. We had done meaningful work that was genuinely new and could stand out from all the other work that had been done in the field. In the final days of the week, I had to do most of the new work. I ran the model on the new dataset and got the relevant results. This almost took until the last minute since I also had to take some time to move out of my apartment on Busch. Then, I had to clean up the plots which I had generated and put them together in a meaningful way. I also used the plots from our presentation to Dr. Silverstein. Being able to bring together all of this work into one coherent paper was really rewarding in the end. It was really amazing to see what only two months of research could yield.
Being in this program has been an invaluable experience that most undergraduate students don't get to experience. I am thankful to have had this opportunity and I am really appreciative for all of the effort put in by the DIMACS REU staff and faculty for making this possible. I am also lucky to be able to continue this research past the duration of the program. I am excited to see what this work will yield in the future.
