Our Zectonal research team recently identified an attack vector that could be used in an AI Poisoning attack. Establishing threshold benchmarks that alert you when missing values exceed a specific minimum or maximum metric is one way to ensure that too much missing data unintentionally ends up in your data lake, thereby polluting it when used for training accurate AI models.ĭata & AI Poisoning Can Spread Rapidly and is Far Reaching One would have to assume a portion of the features were empty.īut when data, specifically training data for AI, starts to contain an unexpected number of missing values, unforeseen behavior could start to occur with the performance and outcomes of the AI model. In one extreme case, Google is said to have one of the largest training data sets for a natural language model using trillions of features. It is not uncommon for large datasets to have a portion of their data empty or missing. We have tested other types of tools that monitor data quality after ETL processes are executed, and in our opinion, its too late - your data lake is already contaminated.Ĭatch Missing Data Before It Pollutes Your Data Lake The more complicated the ETL process, the more difficult it becomes to detect and enforce quality standards. If the data is combined with other data as part of the ETL process, quality defects and malicious payloads are incorporated further upstream into the data manufacturing process. Monitoring your data for quality metrics or embedded threats before it is extracted and ultimately loaded into your data lake is imperative to maintaining a high-quality data repository. ETL processes can be simple, or when used to combine, or fuse, multiple data sets together, can become more complex. One of the most fundamental processes that occurs to data before it is ultimately ingested into a data lake is the Extract-Transform-Load (“ETL”) process. The optimal step in the data lifecycle for data observability monitoring is not always obvious. We focus first on monitoring the data from the point it originates, through its flow to the data lake via data pipelines.Ī Good Place to Start – Monitor Before “E” In this article, we describe how Zectonal’s software can benchmark the quality of data supply chains, and how you can maintain a more pristine, higher quality data lake as a result. Raw Materials -> Fleet Transportation -> Factories -> Manufacturing Equipment -> Finished Goods We defined the 5 data supply chain components as:ĭata -> Data Pipelines -> Data Lake -> Data Analytic Software -> AI-driven InsightsĪnd compared them to the 5 Industrial Age components: In a previous blog Don’t Underestimate the Importance of Characterizing Your Data Supply Chain, we defined the 5 components the data supply chain while also making a comparison to an Industrial Age physical supply chain. Zectonal offers 4 ways to prevent your Data Lake from becoming polluted resulting in more impactful AI driven outcomes. Monitoring for established data quality standards and protecting from emerging AI Security threats are critical for ensuring the optimal use of your Data Lake. Polluted Data Lakes and Data Warehouses are becoming an increasingly significant problem resulting in AI Bias and Data & AI Poisoning security vulnerabilities.Ĭounterfeit data, in both non-deceptive and deceptive forms is entering the global data supply chain at an alarming rate.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |