Social Media Image Mining Based on Deep Learning

Posting images in social media have been a common activity in the mobile internet era. Many research has been conducted on social media text mining, but few studies focus on social media image mining.
We believe that image mining can provide insights about society from another perspective, so we developed a GeoAI-based platform to collect, store, and analyze millions of social media images (Twitter) in real-time. This platform is designed to be extensible and flexible so that various image analysis models and APIs( e.g., a trained CNN) can be easily plugged.

Currently, we have added flooding photo detector, object detector, and face detector to the platform. This platform now has the ability to analysis over 4 million tweets per day. Flooding photos will be collected and analyzed automatically during flood events. Objects, such as cars, books, and cakes are labeled. Faces appear in the social media images are detected along with ethnicity, gender, and age. Meanwhile, image mining is language-free, which means cross-cultural research is relatively easy to be conducted than text mining. To better serve cross-cultural research, we also added a translation module to the platform.

The flooding photo detection result obtained
accuracy of 63% and recall of 95% in a highly imbalanced dataset of Houston Flood 2017 which have only 3.6% flooding photo.
Object detection results of social media images show the pet preferences.
Tweets translation for cross-cultural studies.
Translate ยป