Engineering Insights: StarTree ThirdEye — Revolutionizing Time Series Anomaly Detection

6 min readFeb 13, 2024

Traditional solutions often fail in high-velocity data, where insights are measured in milliseconds and decisions depend on real-time anomalies. StarTree ThirdEye, a game-changer in the anomaly detection space, is built on the rock-solid foundation of Apache Pinot.

This concise overview highlights how the integration of Apache Pinot with StarTree ThirdEye transforms time series anomaly detection, ensuring businesses can preemptively address potential issues.

Anomaly Detection is Getting Progressively Harder

With rapidly growing data volumes, it’s nearly impossible for insight producers and consumers to keep track of all the real-time changes occurring in the data that can produce abnormal events. Thus, detecting interesting patterns, deriving meaningful insights, and making them actionable is a growing problem for many businesses.

The expansion of data results in a long time to detect and resolve outliers in that data and greater difficulty improving follow-on decision-making via actionable insights from detected anomalous events.

Limitation of traditional anomaly detection solution

Building Anomaly Detection software is time-consuming and can take years. Some tools are available to address this problem but come with limitations that make them unfit for monitoring time series anomaly detection.

Solution: StarTree ThirdEye Automates and Monitors

How does StarTree ThirdEye work?

Connect to data, visualize recommendations, select an algorithm, “auto-tune”, apply filters to eliminate noise, create an alert, schedule, and notify.

Create alert

Review anomalies

Root-cause Analysis (Compare, Analyze, Spot Events)

(1. Week over week comparison)

(2. Top contributors)

(3. Heatmap)

(4. Spot Events caused an issue)

Unique advantages of StarTree ThirdEye

Fast-track problem-solving with applied Anomaly Detection on time-series data at scale.

Core Architecture of StarTree ThirdEye

The Infrastructure

StarTree ThirdEye is built on top of StarTree Cloud (managed services for Apache Pinot). It natively supports k8s and works seamlessly with AWS, GCP, or Azure.

This distributed system acts as the foundation for StarTree ThirdEye, ingesting and storing real-time and historical time-series data with lightning-fast efficiency. Here are the engineering insights:

Beyond the Infrastructure: Building Blocks For Engineering Brilliance

Scalability and Independence

One of the key features of StarTree ThirdEye’s architecture is the ability to scale UI, Coordinator, and Worker components independently based on their respective loads. This flexibility ensures the system can adapt to varying demands without compromising performance. The Scheduler component, critical for task management, is designed for high availability but not replicated, maintaining system integrity and coordination.

StarTree ThirdEye Entities

StarTree ThirdEye introduces a structured approach to managing data, alerts, and notifications:

Workflows & Concepts: Orchestrating the Anomaly-Detection

Key Features

StarTree Cloud empowers StarTree ThirdEye with a rich set of features, making it an anomaly-detection ninja:

Key Use Cases

These features democratize metrics monitoring for a broad range of industries through an easy-to-use, no-code/low-code interface. The product has shown early success in Fintech, Logistics, Retail, and E-commerce by offering real-time KPI monitoring, from sales and customer satisfaction to data quality and system health.

Democratizing Metrics Monitoring

Self-service No-code/Low-code UI: No-code/low-code UI enables users of all skill levels to easily monitor and analyze data.
Scalability: Capable of handling large datasets for any organization size.

Industry Applications

Fintech: Instant fraud detection and customer satisfaction tracking.
Logistics: Real-time order tracking/anomaly detection and route optimization.
Retail/E-commerce: Conversion rate optimization and efficient inventory management.

KPIs Monitored

Sales revenue: Track trends and identify growth opportunities.
Customer satisfaction: Understand customer sentiment and take corrective actions.
Data quality: Ensure data accuracy and integrity for reliable decision-making.
System health: Monitor system performance and prevent outages.

Evolution and Early Wins

Product maturity: StarTree ThirdEye has progressed beyond its initial MVP stage and demonstrated effectiveness in real-world scenarios.
Customer success: Companies in various industries have achieved positive results using ThirdEye for real-time monitoring and analysis.

Engineering and Innovation Opportunity

For engineers and data scientists, StarTree ThirdEye presents a unique opportunity to work on cutting-edge technology that’s shaping the future of data analytics. It’s a chance to tackle complex challenges, innovate within a scalable architecture, and contribute to a platform at the forefront of digital transformation.

StarTree Cloud (managed services for Apache Pinot) and StarTree ThirdEye together offer a revolutionary approach to time series anomaly detection, combining real-time data processing with advanced analytics. This synergy not only enhances business operations but also inspires a new generation of engineering talent to explore the depths of data analytics innovation.

What’s in store in the future

The team is considering evolving the product further to bring in more innovation:

Automate data quality checks and model drift detection.
Enhance alerts with real-time, seasonality-based recommendations.
Support advanced algorithms for complex data monitoring.
Use GenerativeAI for pattern detection and analysis.
Scale for enterprise with detailed, instant monitoring.
Upgrade user interface for complex workflows.

If you are interested to learn more and want to explore StarTree ThirdEye, you can:

Credits: Startree ThirdEye Team at Startree (Suvodeep Pyne, Cyril de Catheu, Tushar Thole) and R&D leaders (Jitender Aswani , Chinmay Soman), Founder CEO (Kishore Gopalakrishna) and Co-founder Xiang Fu)