7.2 Machine Learning for Data Repair
7.3 Data Cleaning for Analytics and Machine Learning
Chapter 8 Conclusion and Future Thoughts
Preface
Data quality is one of the most important problems in data management, since dirty data often leads to inaccurate data analytics results and incorrect business decisions. Poor data across businesses and the U.S. government are reported to cost trillions of dollars a year. Multiple surveys show that dirty data is the most common barrier faced by data scientists. Not surprisingly, developing effective and efficient data cleaning solutions is challenging and is rife with deep theoretical and engineering problems.
Data cleaning is used to refer to all kinds of tasks and activities to detect and repair errors in the data. Rather than focus on a particular data cleaning task, in this book, we give an overview of the end-to-end data cleaning process, describing various error detection and repair methods, and attempt to anchor these proposals with multiple taxonomies and views. Specifically, we cover four of the most common and important data cleaning tasks, namely, outlier detection, data transformation, error repair (including imputing missing values), and data deduplication. Furthermore, due to the increasing popularity and applicability of machine learning techniques, we include a chapter that specifically explores how machine learning techniques are used for data cleaning, and how data cleaning is used to improve machine learning models.
This book is intended to serve as a useful reference for researchers and practitioners who are interested in the area of data quality and data cleaning. It can also be used as a textbook for a graduate course. Although we aim at covering state-of-the-art algorithms and techniques, we recognize that data cleaning is still an active field of research and therefore provide future directions of research whenever appropriate.
Ihab Ilyas
Xu Chu
March 2019
Figure and Table Credits
Figures
Figure 2.6 Charu C. Aggarwal. Outlier Analysis. Springer, 2013.
Figure 3.3 Based On: Jiannan Wang, Guoliang Li, and Jianhua Feng. 2012. Can we beat the prefix filtering?: an adaptive framework for similarity join and search. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD ’12). ACM, New York, NY, USA, 85–96. DOI: 10.1145/2213836.2213847.
Figure 3.6 Jens Bleiholder and Felix Naumann. 2009. Data fusion. ACM Comput. Surv. 41, 1, Article 1 (January 2009), 41 pages. DOI: 10.1145/1456650.1456651.
Figure 3.7 Based On: George Beskales, Mohamed A. Soliman, Ihab F. Ilyas, and Shai Ben-David. Modeling and querying possible repairs in duplicate detection. Proc. VLDB Endowment, 2(1): 598–609, (August 2009), 598–609. DOI: 10.14778/1687627.1687695.
Figure 3.8 Based On: George Beskales, Mohamed A. Soliman, Ihab F. Ilyas, and Shai Ben-David. Modeling and querying possible repairs in duplicate detection. Proc. VLDB Endowment, 2(1): 598–609, (August 2009), 598–609. DOI: 10.14778/1687627.1687695.
Figure 3.11 Jiannan Wang, Tim Kraska, Michael J. Franklin, and Jianhua Feng. Crowder: Crowdsourcing entity resolution. Proc. VLDB Endowment, 5(11): 1483–1494, DOI: 10.14778/2350229.2350263.
Figure 3.12 Jiannan Wang, Tim Kraska, Michael J. Franklin, and Jianhua Feng. Crowder: Crowdsourcing entity resolution. Proc. VLDB Endowment, 5(11): 1483–1494, DOI: 10.14778/2350229.2350263.
Figure 3.13 Chaitanya Gokhale, Sanjib Das, AnHai Doan, Jeffrey F. Naughton, Narasimhan Rampalli, Jude Shavlik, and Xiaojin Zhu. 2014. Corleone: hands-off crowdsourcing for entity matching. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SIGMOD ’14). ACM, New York, NY, USA, 601–612. DOI: 10.1145/2588555.2588576.
Figure 3.14 Pradap Konda, Sanjib Das, Paul Suganthan GC, AnHai Doan, Adel Ardalan, Jeffrey R. Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeff Naughton, et al. Magellan: Toward building entity matching management systems. Proc. VLDB Endowment, 9(12): 1197–1208, 2016.
Figure 3.15 Based on: Michael Stonebraker, Daniel Bruckner, Ihab F. Ilyas, George Beskales, Mitch Cherniack, Stanley B. Zdonik, Alexander Pagan, and Shan Xu. Data curation at scale: The data tamer system. In Proc. 6th Biennial Conf. on Innovative Data Systems Research, 2013. http://cidrdb.org/
Figure 4.3 Vijayshankar Raman and Joseph M. Hellerstein. 2001. Potter’s Wheel: An Interactive Data Cleaning System. In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB ’01), Peter M. G. Apers, Paolo Atzeni, Stefano Ceri, Stefano Paraboschi, Kotagiri Ramamohanarao, and Richard Thomas Snodgrass (Eds.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 381–390.
Figure 4.4 Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011. Proactive wrangling: mixed-initiative end-user programming of data transformation scripts. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ’11). ACM, New York, NY, USA, 65–74. DOI: 10.1145/2047196.2047205. and Jeffrey Heer, Joseph Hellerstein, and Sean Kandel. Predictive interaction for data transformation. In Proc. 7th Biennial Conf. on Innovative Data Systems Research, 2015. and Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wrangler: interactive visual specification of data transformation scripts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, USA, 3363–3372. DOI: 10.1145/1978942.1979444.
Figure 4.5 Copyright © 2007 Free Software Foundation, Inc. http://fsf.org/, (http://fsf.org/)
Figure 4.6 Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. In Proceedings of the 38th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages (POPL ’11). ACM, New York, NY, USA, 317–330. DOI: 10.1145/1926385.1926423.