- Evaluating Differentially Private Synthetic Data Generators in Social Science Settings
- Project Year:
2024
- REU Student (s):
Thomas Chen | University of California-Berkeley CA
- Student 1 Institution:
University of California-Berkeley
- Project Mentor:
Ruobin Gong
- Project Mentor Area:
Statistics
- Project Abstract:
As we continue to use large public datasets for studies and models, there are increasing concerns over the risk of sharing public data. Many studies have shown that potential malevolent agents can cross-reference public datasets to find out sensitive information about individuals in public datasets. One such theoretical remedy that has been suggested is to use differentially private synthetic datasets in place of the original dataset. However, no studies so far have evaluated these synthetic data generators on actual real-life studies and measured how effective they are at replicating results. In this work, we look at the current differentially private synthetic data generators and evaluate them on recent social science studies that use large public datasets. We focus on the synthetic data generator called DataSynthesizer and analyze how faithful it is in replicating results from the study using the Panel Study of Income Dynamics. We find that Datasynthesizer struggles with complex queries that consider more than just marginal distributions. Additionally, we examine how the quality of the synthetic data varies across different settings to give future guidance on how to provide quality differentially private synthetic datasets.
