General Information

Student: David Zeng
Office: Rm 343 CoRE
School: The City College of New York
E-mail: david82794 (at) yahoo (dot) com
Project: Crowd Analysis From Video Feeds

Project Description

This summer I am working with Professor Hao Tang and Greg Olmschenk on Crowd Detection and Analysis. We are working with video surveillance data from the New York Port Authority Bus Terminal to try to analyze the crowds that form at the bus terminal. Port Authority has been having trouble with unexpectedly large crowds that arise from a halt in bus operations or during special days of the year. They hope that our research and analysis can help them understand the ways crowds react to specific situations and help them develop new and more efficient ways to manage crowds. There is also a CCICADA team at Rutgers that has been working on computer simulations of crowds at the Port Authority Bus terminal for a while. This research project involves using computer vision techniques and algorithms in order to analyze the video data we have. One important piece of information to note is that the video feeds provided are of extremely poor quality of which is perfectly fine for normal surveillance but not suited for regular computer vision.

Weekly Log

Before Start of Program
Started summer internship notes on google docs Read through the Crowd Analysis survey article from the 2010 IEEE magazine and searched for some of the references for source code and datasets. Went through the OpenCV python tutorial to read up on computer vision techniques before meeting Professor Hao Tang. Met with Professor Hao Tang to talk about the summer project and about any concerns or recommendations and additional tasks. Looked at some public datasets from old research papers to see if they were useful for our specific case. Completed python tutorial from code academy to prepare for the summer project.
Week 1:
Slowly settling in the Rutgers Silver Apartments since I am dorming for the first time. Mingled with other REU students, discovered and discussed about other research projects taking place in the REU summer program. Found out more details about the DHS project and created presentation slides. Presented on my summer research project as part of the REU program. Attended several meetings and workshops for the REU program. Python 3.4, PyCharm, OpenCV and Numpy properly installed and working on laptop with much help from Greg. Helped analyze and give input on the video data from Port Authority. Helped annotate the floor plans for Port Authority for important and interesting or relevant paths that commuters may take. Created a simple working OpenCV program to manually place dots and bounding boxes over an image file while printing out relevant coordinates. Included a permanent undo button event for accidental drawings. Fixed bugs and debugged the program. Converted video data from .avi to .mp4 and tested their compatibility with OpenCV.
Week 2:
Added extra features to my program: make program run for any image path, outputting coordinates to text file. Saving cropped areas of bounding box. Set up WinSCP, with directions from Greg. Attempted installation of Ubuntu in terms of dual booting and virtual machine setup Encountered many difficulties due to specific problems stemming from my HP laptop’s set up. Finally settled with partitioning my external hard drive and installing on there Attempted installation of Vatic. Lots of difficulty due to poor documentation and outdated scripts forcing us to do several workarounds. Vatic is installed but offline mode is not working properly. Finished ground truthing labeling on a 20 second clip and writing a script to automatically tally up my manual labeling. Ground truthing done was on people counting using red and blue dots on the video surveillance frames. Red dots represented people in the close range and blue represented people farther into the horizon.
Week 3:
Worked on creating ground truth bounding boxes for close range people. Then I worked on a program to find if a dot coordinate was within a detected bounding box. Looked up how to find the intersecting coordinates for two overlapping rectangles in order to compute the area. To do this, I had to create a boolean function that would test if two axis-aligned rectangles were indeed overlapping using the separating axis theorem(SAT). Once we know that the rectangles are indeed overlapping we can apply a quick formula to gather the coordinates of the intersecting region. With the coordinates of the intersecting region, we can compute the area of intersection, which is then used to compute the percentage of overlap. Once I was able to compute the percentage of overlap, I then worked on writing a python script to check for the number of overlapping bounding boxes from a list of ground truth and a list of detected boxes from Greg's algorithm. This was then used to determine the accuracy and effectiveness of the algorithm. This involved several steps of parsing input files for the box coordinates and saving them to a list then running them through the overlap functions. I also attended a tutorial talk on "Math Modeling of Crowd Dynamics" by Benedetto Piccoli. Next I worked on writing a script to compile data on our algorithm's precision and recall statistics in order to graph the success and failures of the current algorithm. After that I worked on getting a height based bounding box filtering to get rid of erroneous false positives that are too big or too small at higher thresholds.
Week 4:
Worked on fixing my bounding box filtering and collecting new data for graphs for to compare the effects the filtering has on our results. Worked on getting positive and negative images of people to use for the OpenCV Harr Classifier training. Attended a talk by David Molnar on "Connection Games and Sperner's Lemma". Successfully installed VATIC. Took a break on Thursday and Friday to attend my younger brother's high school graduation in New York City
Week 5:
Since we got the VATIC program working, I pretty much spent the entire week working on the tedious task of creating ground truth data for a number of five second video segments. With the new versions of ground truths, we were able to get 10 times as many positive ground truth images since we were able to get positives from every frame instead of every 10 frames. With the new images, we retrained the Harr Classifier using the new set of images. The new detections were much better than before and showed more promise. Attended a Fourth of July pinic hosted by Eugene Fiorini on July 3rd.
Week 6:
I was given a task of writing a program that would parse two input files containing motion cue coordinates and bounding box coordinates that the detection algorithm got. Once the coordinates are parse, the data is stored in data structures and we use the data to try to compute tracks. By collecting series of bounding box coordinates that are close to each other and contain a similar amount of specific features from the motion cue data, we can obtain "tracks". The last step of the track program was to color each track a different color. Since the data structure I started off with was not idea, debugging became a huge pain and I ended up having to redo my program in a more object oriented approach. Once the tracking program was completed, we tested it on various video segments to make sure it was working properly. Greg helpped me set up a git repository for my tracking project. I looked into the Port Authority Bus Terminal live camera feeds for any cameras of interest or ones with overlapping areas. I compiled a list of cameras for future reference and downloaded some of the videos. Attended a Graduate school workshop. Lastly, once I was done with the tracking program, I worked on refining my VATIC ground truth data to make them more accurate, fix incorrect sections, and adding new people.
Week 7:
This week, I worked on refining the VATIC ground truth for a few days. We also received a new stack of DVD's from Port Authority. They contained new video sequences. I added a small part of my tracking program to output track points for Hao to work with for his new idea. Hao used the track points to connect broken tracks and to show an example of a rough top down view of the camera area. I also rewrote the first stage of my tracking program to incorporate classes from the start and spent time debugging my new code. I also tried different parameters for filtering data before the tracking to see if we got better results as well as speed improvements. Slowly put together a presentation for friday. The presentation is a summary the progress we have made for our research projects. Since I am not allowed show any data from the Port Authority I had to find and use stock footage and diagrams for my slides. Presented my slides on friday. Over the weekend I was asked to write up some documentation drafts for the ground truth data and for my tracking program.
Week 8:
Added diagrams to my documentation drafts. Worked on more ground truth for a face detector that Hao found. We trained the new data and tested it out. We discovered that ffmpeg was compressing our video frame extractions and lowering the quality of the images. This time I created categories and looked for ground truth images that would fit into that category. Updated my previous slides for a CRG meeting. We were asked to create a 1 minute video about our research project.


Additional Information

References & Links

Here is REU website: