Announcing Data Observability for Firebolt

Announcing Data Observability for Firebolt
Mona Rakibe

Overview

Firebolt is the cloud data warehouse designed for next-gen analytics experiences. The platform's ease of use of modern architecture with a sub-second performance at a terabyte-scale has enabled many companies to adopt it in the recent past.

So when a  months back, a channel partner company said: You got to build an integration with Firebolt! I almost felt like Firebolt was the coolest kid on the block with whom we had to make friends :-) 

After reading their white papers, the data geek in me felt even more strongly that we had to make friends with this cool kid, and Max couldn't agree more.

So here we go. As a complete data pipeline Observability tool, Telmai now supports Firebolt along with many other DataWarehouse and datalake integrations like Snowflake, GCS, S3, AzureBlog, and BigQuery.

Telmai + Firebolt 

As a low-code, no-code data observability product Telmai helps data teams proactively detect and investigate data issues within Firebolt before having a downstream impact. 

Data teams struggle to detect data quality issues as soon as they occur — these issues are often first experienced by consumers of ML and Analytics and cause long and tedious investigation cycles.  Telmai helps reduce the mean time to detect (MTTD) and mean time to resolve(MTTR) data quality issues.

Benefits for Firebolt users :

  • Automatic monitoring on 40+ predefined data metrics like schema change, row count, completeness, uniqueness, patterns, distribution change, freshness, accuracy, etc
  • Simple UI for expectations/rules - no more hand-coded rules using Great Expectation or DBT expectations.
  • Support for structured & semi-structured data, i.e., nested and multi-valued attributes.
  • Change data capture so users can independently monitor the entire dataset and changes alone.
  • Support multiple formats like JSON, Parquet, CSV, and Avro
  • Intuitive human-in-loop model for fine-tuning thresholds, policies, or writing expectations
  • Spark processing to enable infinite scale without performance impact on Firebolt queries.

How does this work

How to get started ?

  1. Create a free Telmai account https://www.telm.ai/get-started
  2. Within minutes you will get your account with onboarding instructions
  3. Connect to Firebolt data source
  4. You are all set. You automatically get monitored.

Summary

Firebolt users can now continuously monitor their data in Firebolt for Key Data quality metrics without impacting performance. This no-code integration ensures that the data inside Firebolt is always trustworthy for Analytics.
Feel free to get your trial account or schedule a demo @ www.telm.ai

Data profiling helps organizations understand their data, identify issues and discrepancies, and improve data quality. It is an essential part of any data-related project and without it data quality could impact critical business decisions, customer trust, sales and financial opportunities. 

To get started, there are four main steps in building a complete and ongoing data profiling process:

  1. Data Collection
  2. Discovery & Analysis
  3. Documenting the Findings
  4. Data Quality Monitoring

We'll explore each of these steps in detail and discuss how they contribute to the overall goal of ensuring accurate and reliable data. Before we get started, let's remind ourself of what is data profiling.

What are the different kinds of data profiling?

Data profiling falls into three major categories: structure discovery, content discovery, and relationship discovery. While they all help in gaining more understanding of the data, the type of insights they provide are different:

 

Structure discovery analyzes that data is consistent, formatted correctly, and well structured. For example, if you have a ‘Date’ field, structure discovery helps you see the various patterns of dates (e.g., YYYY-MM-DD or YYYY/DD/MM) so you can standardize your data into one format.

 

Structure discovery also examines simple and basic statistics in the data, for example, minimum and maximum values, means, medians, and standard deviations.

 

Content discovery looks more closely into the individual attributes and data values to check for data quality issues. This can help you find null values, empty fields, duplicates, incomplete values, outliers, and anomalies.

 

For example, if you are profiling address information, content discovery helps you see whether your ‘State’ field contains the two-letter abbreviation or the fully spelled out city names, both, or potentially some typos.

 

Content discovery can also be a way to validate databases with predefined rules. This process helps find ways to improve data quality by identifying instances where the data does not conform to predefined rules. For example, a transaction amount should never be less than $0.

 

Relationship discovery discovers how different datasets are related to each other. For example, key relationships between database tables, or lookup cells in a spreadsheet. Understanding relationships is most critical in designing a new database schema, a data warehouse, or an ETL flow that requires joining tables and data sets based on those key relationships.

Stay in touch

Stay updated with our progress. Sign up now

By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Stay in touch

Stay updated with our progress. Sign up now

By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Data Observability
Data Quality

Leverages ML and statistical analysis to learn from the data and identify potential issues, and can also validate data against predefined rules

Uses predefined metrics from a known set of policies to understand the health of the data

Detects, investigates the root cause of issues, and helps remediate

Detects and helps remediate.

Examples: continuous monitoring, alerting on anomalies or drifts, and operationalizing the findings into data flows

Examples: data validation, data cleansing, data standardization

Low-code / no-code to accelerate time to value and lower cost

Ongoing maintenance, tweaking, and testing data quality rules adds to its costs

Enables both business and technical teams to participate in data quality and monitoring initiatives

Designed mainly for technical teams who can implement ETL workflows or open source data validation software

Stay in touch

Stay updated with our progress. Sign up now

By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Start your data observibility today

Connect your data and start generating a baseline in less than 10 minutes. 

No sales call needed

Stay in touch

Stay updated with our progress. Sign up now

By subscribing you agree to with our Privacy Policy.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Start your data observability today

Connect your data and start generating a baseline in less than 10 minutes. 

Telmai is a platform for the Data Teams to proactively detect and investigate anomalies in real-time.
© 2023 Telm.ai All right reserved.