PhotoSketch: Internet Image Montage Software Creates Custom Images

Written by Christian Milsom on October 8, 2009 · Filed under Internet, Software 

Occasionally something will come around, often a piece of software, which will just amaze you and leave you marvelling at the capabilities of today’s technology. Not wanting to build it up too much, but in my opinion Photosketch is one of those.

So what actually is it? Well in the words of the people behind it (five students from Tsinghua University, China) it’s an ‘internet image montage’ which is capable of compositing a realistic picture from a user provided sketch with text labels.

Ok, that may not sound much, but it is more impressive than it first appears. If, for example, I wished to have a picture of a man parachuting out of a hot air balloon I would draw the rough outline of the hot air balloon and the man, label them as such, label the background as ‘sky’ and then set it off. What Photosketch then does is trawl through the internet looking for pictures that look vaguely like your little sketch and then stitch them together to make a realistic picture.

However obviously it’s a little more complicated than that and there are quite a few neat tricks that the system uses. Once you’ve ‘drawn’ your picture Photosketch with search the internet using your labels, and removes incorrect and unwanted results by clustering and discarding unusual ones leaving itself with a host of options for each item and background.

All of these ‘candidates’ are then assessed with each other to see which ones match, and a few optimal options are chosen which are then shown to the user for them to choose. The accuracy to which the system can find suitable pictures and then seamlessly blend them together it pretty incredible, and there are a few key tricks which the programme uses.

Firstly, having searched for images and removed incorrect ones, the system uses ‘Salient region detection’ to remove the images with complicated backgrounds that would be hard to work with and identify the regions within which the desired image (in my case a man) lie – something refined using a ‘grab-cut’ algorithm.

Then the programme uses shape matching to identify the images which are of a similar shape to the ones drawn by you originally in the sketch. The final stage involves assessing the feasibility of the combination of the background and the individual images found (using texture and colour similarity and matting feasibility) and using a whole load of complicated stages combines the images to form a (hopefully) realistic interpretation of your original sketch.

You would be forgiven for thinking that there aren’t really many real world uses for this (because as far as I can see there aren’t!) but you can’t deny that this is a pretty neat piece of software and that if it works as well as the demonstrations suggest Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir and Shi-Min Hu (the students behind this) have done a good job.

The software is available to download, but at the moment the site is down (or may as well be seeing as it fails to load correctly) due to the volumes of traffic so unfortunately we haven’t been able to have a play around but do check out the samples which give a good idea of what this system is capable of, and the results are quite impressive!

Via – PhotoSketch: Internet Image Montage