In this study, we investigate the potential driving factors that lead to the disparity in the time-series of home dwell time, aiming to provide fundamental knowledge that benefits policy-making for better mitigation strategies of future pandemics. Taking Metro Atlanta as a study case, we perform a trend-driven analysis by conducting Kmeans time-series clustering using fine-grained home dwell time records from SafeGraph, and further assess the statistical significance of sixteen demographic/socioeconomic variables from five major categories. We find that demographic/socioeconomic variables can explain the disparity in home dwell time in response to the stay-at-home order, which potentially leads to disparate exposures to the risk from the COVID-19. The results further suggest that socially disadvantaged groups are less likely to follow the order to stay at home, pointing out the extensive gaps in the effectiveness of social distancing measures exist between socially disadvantaged groups and others. Our study reveals that the long-standing inequity issue in the U.S. stands in the way of the effective implementation of social distancing measures. Policymakers need to carefully evaluate the inevitable trade-off among different groups, making sure the outcomes of their policies reflect interests of the socially disadvantaged groups.