New customer case study with DataStax! Read More

Telmai Partners with Google Cloud to Bring ML-Driven Anomaly Detection and Data Quality Monitoring to BigQuery

Farnaz Erfan

On April 25, 2023, Telmai announced its partnership with Google Cloud to empower Data Observability and Quality Monitoring on BigQuery and Google Cloud Storage. This partnership is built on product integrations between Telmai and Google Cloud. It combines the speed and scale of Google Cloud BigQuery with the ML-driven anomaly detection and data quality monitoring of Telmai.

Integrations between Telmai and Google Cloud BigQuery enable data teams to detect, investigate, and remediate their data quality issues at scale and with a low-code, no-code, and API-first platform. Enterprises have growing amounts of data available to them and are constantly leveraging that data for data-driven decision-making and building data products. Today, data quality issues and the time to detect and remediate those issues have become a roadblock for many – a gap that Telmai is closing.

Today Telmai customers like DataStax and Clearbit use Telmai's Data Observability to detect, investigate, and monitor their data in BigQuery and various other sources.

"We're thrilled to be partnering with Google Cloud," said Mona Rakibe, CEO and Co-Founder of Telmai. "This partnership will open up opportunities for GCP customers to accelerate their advanced analytics and machine learning initiatives without worrying about the reliability of underlying data."

Companies that use Telmai can monitor the quality of their data in BigQuery in addition to various other data sources, analytic databases, semi-structured datasets, and event streaming sources. Telmai proactively detects and investigates anomalies and data drifts in real-time using ML.

"Trusted data is essential for decision-making and to power digital transformation," said Manvinder Singh, Managing Director, Partnerships at Google Cloud. "We're pleased that BigQuery and Google Cloud infrastructure will support Telmai's solution, and help bring new capabilities and products to market from innovative, fast-growing organizations like Telmai."

"Our customers deserve the best possible data to make their business grow," said Harlow Ward, CTO, Clearbit. "Telmai's Data Observability enables us to show the quality of the data that we are delivering to our customers. We use Telmai to monitor and quality check the data that is coming in, the data that goes through our proprietary algorithms, and the data we package for our customers. Telmai is in every step of our data pipeline including BigQuery to ensure that the data we deliver to our customers is credible and reliable."

"To prepare data for product usage analysis, we needed data quality metrics beyond monitoring operational data pipelines and job status checks," said Raghu Nadiger, Data & Analytics Leader, DataStax. "We chose Telmai to detect the quality and accuracy of the data in our BigQuery data warehouse. Telmai's ML and no-code product makes it easy for us to monitor drifts in our data and get notifications of any gaps in real time."

"Migrating to a modern cloud data warehouse has become more challenging than ever due to data's increasing volumes and complexity," said Darius Kemeklis, EVP & CTO, Myers-Holum. "However, with our consulting expertise and cutting-edge products like Telmai, we can efficiently migrate legacy systems to Google Cloud BigQuery while staying within our client's time and budget. Telmai's data profiling and anomaly detection enable pre-migration data quality assessment of legacy SQL or Hadoop environments. Telmai allows us to validate migrated data within Big Query and compare it to the source system data and value distribution, ensuring a successful migration. Telmai's ongoing monitoring and observability of data assures long solution health."

In partnership, Telmai and Google have jointly developed this video to showcase Telmai's capabilities on Google Cloud.

To learn more about Telmai's capabilities on BigQuery, visit Telmai's integrations page.
To learn more, request a demo.

‍

Data profiling helps organizations understand their data, identify issues and discrepancies, and improve data quality. It is an essential part of any data-related project and without it data quality could impact critical business decisions, customer trust, sales and financial opportunities.

‍

To get started, there are four main steps in building a complete and ongoing data profiling process:

Data Collection
Discovery & Analysis
Documenting the Findings
Data Quality Monitoring

We'll explore each of these steps in detail and discuss how they contribute to the overall goal of ensuring accurate and reliable data. Before we get started, let's remind ourself of what is data profiling.

What are the different kinds of data profiling?

Data profiling falls into three major categories: structure discovery, content discovery, and relationship discovery. While they all help in gaining more understanding of the data, the type of insights they provide are different:

Structure discovery analyzes that data is consistent, formatted correctly, and well structured. For example, if you have a ‘Date’ field, structure discovery helps you see the various patterns of dates (e.g., YYYY-MM-DD or YYYY/DD/MM) so you can standardize your data into one format.

Structure discovery also examines simple and basic statistics in the data, for example, minimum and maximum values, means, medians, and standard deviations.

Content discovery looks more closely into the individual attributes and data values to check for data quality issues. This can help you find null values, empty fields, duplicates, incomplete values, outliers, and anomalies.

For example, if you are profiling address information, content discovery helps you see whether your ‘State’ field contains the two-letter abbreviation or the fully spelled out city names, both, or potentially some typos.

Content discovery can also be a way to validate databases with predefined rules. This process helps find ways to improve data quality by identifying instances where the data does not conform to predefined rules. For example, a transaction amount should never be less than $0.

Relationship discovery discovers how different datasets are related to each other. For example, key relationships between database tables, or lookup cells in a spreadsheet. Understanding relationships is most critical in designing a new database schema, a data warehouse, or an ETL flow that requires joining tables and data sets based on those key relationships.

‍

1. Data Collection

Start with data collection. Gather data from various sources and extract it into a single location for analysis. If you have multiple sources, choose a centralized data profiling tool (see our recommendation in the conclusion) that can easily connect and analyze all your data without having you do any prep work.

‍

2. Discovery & Analysis

Now that you have collected your data for analysis, it's time to investigate it. Depending on your use case, you may need structure discovery, content discovery, relationship discovery, or all three. If data content or structure discovery is important for your use case, make sure that you collect and profile your data in its entirety and do not use samples as it will skew your results.

Use visualizations to make your discovery and analysis more understandable. It is much easier to see outliers and anomalies in your data using graphs than in a table format.

‍

3. Documenting the Findings

Create a report or documentation outlining the results of the data profiling process, including any issues or discrepancies found.

Use this step to establish data quality rules that you may not have been aware of. For example, a United States ZIP code of 94061 could have accidentally been typed in as 94 061 with a space in the middle. Documenting this issue could help you establish new rules for the next time you profile the data.

‍

4. Data Quality Monitoring

Now that you know what you have, the next step is to make sure you correct these issues. This may be something that you can correct or something that you need to flag for upstream data owners to fix.

After your data profiling is done and the system goes live, your data quality assurance work is not done – in fact, it's just getting started.

‍

Data constantly changes. If unchecked, data quality defects will continue to occur, both as a result of system and user behavior changes.

‍

Build a platform that can measure and monitor data quality on an ongoing basis.

‍

Take Advantage of Data Observability Tools

Automated tools can help you save time and resources and ensure accuracy in the process.

‍

Unfortunately, traditional data profiling tools offered by legacy ETL and database vendors are complex and require data engineering and technical skills. They also only handle data that is structured and ready for analysis. Semi-structured data sets, nested data formats, blob storage types, or streaming data do not have a place in those solutions.

‍

Today organizations that deal with complex data types or large amounts of data are looking for a newer, more scalable solution.

That’s where a data observability tool like Telmai comes in. Telmai is built to handle the complexity that data profiling projects are faced with today. Some advantages include centralized profiling for all data types, a low-code no-code interface, ML insights, easy integration, and scale and performance.