Computer Vision
We propose an automated system to crop out certain objects, especially human, in a given photo and fuse the object into a new background to form a new photo. The first part of our work will focus on building and train a semantic segmentation model that features in detecting and recognizing the human figure in a photo, and cropping out according to its outline. For this part, our supervised semantic segmentation algorithm will be based on Fully Convolutional Networks (FCN). The second part of our focus of work, also the FUN part, will be stitching the cropped part into a new background image and try to create a new scene. Poisson Image Editing will be the main technique for this part, while some other blending methods might also be used, e.g. Feathering, The Laplacian Pyramid, Alpha Blending. This is useful in multiple photo editing scenarios, especially in portrait photo editing. An easy inverse of this method can be used to erase the unwanted objects from a photo.
With semantic segmentation, we will be able to separate human portrait from image and substitute the background easily. A simple use case would be allowing users to choose new background scene by preferences. Also, we can do portrait editing since we have already detected where the people are in an image. For instance, we will be able to adjust color, illumination of the object of interest without changing the rest of the background, which is sort of similar to what smart phones cameras can do nowadays. We can also use an inverse version of this function to eliminate undesired portions of an image.
We included all of our exploration methods, model-building steps and results in a finalized paper: Seamless Image Cloning with Semantic Segmentation
./FCN32.py --save 32.pkl --enable_testing<br />
To train FCN16:
a. Training using 32.pkl: ./FCN16.pkl —load 32.pkl —save 16.pkl —enable_testing
b. Training only with pretrained vgg: ./FCN.py —save 16.pkl —enable_testing
To train FCN8:
a. Training using 16.pkl: ./FCN8.pkl —load 16.pkl —save 8.pkl —enable_testing
b. Training only with pretrained vgg: ./FCN.py —save 8.pkl —enable_testing
http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
https://drive.google.com/open?id=1CTulFL9A-VyQ7br_4GMxJ4ZRNvym5Pwj