Posting images on social media has been a common activity in the mobile internet era. Many research has been conducted on social media text mining, but few studies focus on social media image mining.
We believe that image mining can provide insights about society from another perspective, so we developed a GeoAI-based system to collect, store, and analyze millions of social media images (Twitter) in real-time. This platform is designed to be extensible and flexible so that various image analysis models and APIs( e.g., a trained CNN) can be easily plugged.
Currently, we have added a flooding photo detector, an object detector, and a face detector to the platform. This platform now has the ability to analysis over 4 million tweets per day. Flooding photos will be collected and analyzed automatically during flood events. Objects, such as cars, books, and cakes are labeled. Faces that appear in the social media images are detected along with ethnicity, gender, and age. Meanwhile, image mining is language-free, which means cross-cultural research is relatively easy to be conducted than text mining. To better serve cross-cultural research, we also added a translation module to the platform.
platform
accuracy of 63% and recall of 95% in a highly imbalanced dataset of Houston Flood 2017 which has only 3.6% flooding photo.
Publication: Ning, Huan, Zhenlong Li, Michael E. Hodgson, and Cuizhen Susan Wang. “Prototyping a social media flooding photo screening system based on deep learning.” ISPRS International Journal of Geo-Information 9, no. 2 (2020): 104.
https://doi.org/10.3390/ijgi9020104