The BRADLEY DEPARTMENT of ELECTRICAL and COMPUTER ENGINEERING

ECE NEWS

Coding jokes: Virginia Tech research team tackles the algorithm of humor

Originally funny scene (left) and it's object-replaced counterpart (right).

Originally funny scene (left) and it's object-replaced counterpart (right).

Are you sad because no one laughs at your jokes? Take heart. The day is coming when a robot will! Or, then again, maybe it won't -- even the computer thinks you're corny.

Members of a Virginia Tech research team are harbingers of this humor apocalypse. Arjun Chandrasekaran, a Ph.D. student in the College of Engineering, and Ashwin Kalyan, a visiting student from the National Institute of Technology Karnataka in India, are the lead authors on a paper entitled "We are humor beings: Understanding and predicting visual humor," which has gotten press in magazines like the MIT Technology Review and Newsweek.

The project investigates how a computer could learn to recognize and replicate humor in visual scenes. This is a collaboration between the Electrical and Computer Engineering Department's Computer Vision Lab and the Machine Learning and Perception Group, led by assistant professors Devi Parikh and Dhruv Batra, respectively. Parikh and Batra are co-authors of this study, along with Mohit Bansal from Toyota Technological Institute, and Larry Zitnick from Facebook AI Research.

"Humor is such an important part of the human experience," said Parikh. "But, surprisingly, we still don't have a grasp on why one scene is funny while another is not especially when it comes to visual humor."

Equipping a machine with a sense of humor may be a long way off, but the collaboration is approaching this task by presenting the computer with examples of visual scenes that fit the bill. They are accumulating a large database of images that run the gamut from mundane to hilarious.

The research team employs workers from Amazon Mechanical Turk, an online marketplace for work, to create their own funny scenes from clip art and include a short description of why they think the scenes are funny. Along with some regular scenes that have no comedic aspects, there are 6,400 images in the database.

The team calibrated the database by asking workers to rate the funniness of each scene and found that, for the most part, everyone agreed on which images were actually funny. They also found that an unfunny scene could take on a comedic edge when the objects were switched around substituting a raccoon for a man, for instance, or a piece of cheese for a cat.

Humor is a fickle thingIt's hard to pin it down. But, as they reported in their paper, the team used these exercises to understand the fine-grained semantics that cause a specific object category to contribute to humor.

To train an algorithm to spot the difference between funny and unfunny images, the team gave the machine two tasks: judge the funniness of a scene, and then alter the funniness of a scene by replacing an object within it.

After training on the dataset and learning which scenes were designated funny and unfunny by humans, the algorithm was able to compute a funniness score for each new image it encountered. By familiarizing itself with individual features, such as animate or inanimate objects, and the interaction between features, the computer used neural networks to classify an image as funny or not.

The machine's algorithm beat the baseline, which means the computer had a general notion of what was supposed to be funny certainly better than a random guess. And, while it didn't function as well in creating its own funny scenes, its performance did churn out some interesting results.

The model learned that, in general, animate objects like humans and animals are more likely to be sources of humor than inanimate objects, and therefore tended to replace these objects, the team reported.

Chandrasekaran and Kalyan will be presenting their paper at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in June 2016.

So, while Virginia Tech robots won't be performing stand-up comedy routines anytime soon, Chandrasekaran and the team have made progress in outfitting an algorithm with an inkling of humor, which is a vital facet of common sense, and human nature as a whole.