# DataHub Sentinel 🚨

**Metadata-Aware Anomaly Detection Agent for the DataHub Agent Hackathon**

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Hackathon](https://img.shields.io/badge/Hackathon-DataHub%20Agent-blue)](https://devpost.com/challenges/72982)

## 🎯 Problem

Data pipelines fail silently every day. A schema changes upstream, data quality degrades, or latency spikes—but downstream teams don't know until users complain. Existing monitoring tools show *what* broke, but not *why* or *who to notify*.

## ✨ Solution

**DataHub Sentinel** is an autonomous agent that:
1. **Detects anomalies** in real-time metrics (latency, error rates, schema drift)
2. **Queries DataHub** for lineage context (what's upstream? what's downstream?)
3. **Identifies root cause** using metadata (recent changes, ownership, schema history)
4. **Generates incident reports** with actionable recommendations
5. **Writes back to DataHub** to enrich the knowledge graph

Unlike passive MCP servers that only answer queries, Sentinel is **active**—it monitors, detects, investigates, and reports.

## 🏗️ Architecture

```
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Data Sources  │────▶│  Sentinel Agent  │────▶│   DataHub MCP   │
│  (Metrics, APIs)│     │  (Anomaly Detect)│     │   (Context)     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                               │
                               ▼
                        ┌──────────────────┐
                        │  Incident Report │
                        │  (Auto-generated)│
                        └──────────────────┘
```

## 🛠️ Tech Stack

- **MCP Server:** Python + [FastMCP](https://github.com/modelcontextprotocol/python-sdk)
- **Anomaly Detection:** Statistical (z-score, IQR) + rule-based
- **DataHub Integration:** DataHub REST API + GraphQL
- **Deployment:** Vercel serverless functions (free tier)
- **Demo Dashboard:** HTML/JS (vanilla, no framework bloat)

## 🚀 Quick Start

### Prerequisites
- Python 3.11+
- Node.js 18+ (for Vercel CLI)
- Vercel account (free)

### Local Development

```bash
# Clone the repo
git clone https://github.com/0xConsole/datahub-sentinel-agent
cd datahub-sentinel-agent

# Install dependencies
pip install -r requirements.txt

# Run MCP server locally
python mcp_server.py

# Test with MCP inspector
npx @modelcontextprotocol/inspector python mcp_server.py
```

### Deploy to Vercel

```bash
# Install Vercel CLI
npm i -g vercel

# Deploy
vercel --prod

# Your MCP server is now live at: https://datahub-sentinel-agent.vercel.app
```

## 📡 MCP Tools

Sentinel exposes these tools via MCP:

| Tool | Description |
|------|-------------|
| `query_lineage` | Get upstream/downstream dependencies for a dataset |
| `get_schema` | Fetch schema definition (fields, types, descriptions) |
| `search_metadata` | Search DataHub catalog |
| `get_ownership` | Get dataset owners and stewards |
| `detect_anomaly` | Statistical anomaly detection (z-score based) |
| `generate_incident_report` | Combine anomaly + metadata into incident report |
| `get_trust_score` | Calculate dataset trust score (0-100) |

## 🎬 Demo Flow

1. **Anomaly detected:** Sentinel monitors a Kafka topic's error rate
2. **Context query:** Agent asks DataHub: "What's the lineage? Who owns this?"
3. **Root cause:** Upstream Postgres table had a schema change 2 hours ago
4. **Incident report:** Auto-generated with:
   - Severity: HIGH (z-score: 4.2)
   - Upstream: `urn:li:dataset:(postgres,users,PROD)`
   - Downstream: 3 dashboards affected
   - Owners to notify: `data-engineering`, `jane.doe`
   - Recommended actions: Check schema history, notify dashboard owners

## 📊 Example Usage

```python
from mcp import Client

async with Client("http://localhost:8000") as client:
    # Detect anomaly
    anomaly = await client.call_tool("detect_anomaly", {
        "metric_name": "kafka_error_rate",
        "value": 185.0,
        "threshold": 3.0
    })
    
    if anomaly["is_anomaly"]:
        # Generate incident report
        report = await client.call_tool("generate_incident_report", {
            "anomaly_result": anomaly,
            "dataset_urn": "urn:li:dataset:(kafka,transactions,PROD)"
        })
        
        print(f"Incident: {report['incident_id']}")
        print(f"Severity: {report['severity']}")
        print(f"Owners to notify: {report['owners_to_notify']}")
```

## 🏆 Hackathon Track

**Track:** "Agents That Do Real Work" (Track 1)

This project fits the track because:
- ✅ **Real problem:** Data pipeline failures cost companies millions
- ✅ **Working agent:** Not just a demo—actual anomaly detection + metadata integration
- ✅ **Uses DataHub:** MCP server queries lineage, schema, ownership
- ✅ **Writes back:** Incident reports enrich DataHub's knowledge graph

## 📝 What's Real vs Mocked

| Component | Status |
|-----------|--------|
| MCP server | ✅ Fully functional |
| Anomaly detection | ✅ Real statistical methods (z-score) |
| DataHub API calls | ⚠️ Mocked (using DataHub demo instance pattern) |
| Incident generation | ✅ Real logic, mock data |
| Dashboard | ⚠️ Basic HTML demo (expandable) |

## 🎯 Next Steps

- [ ] Integrate with real DataHub instance (production or self-hosted)
- [ ] Add more anomaly detection methods (Isolation Forest, Prophet)
- [ ] Build real-time dashboard with WebSocket updates
- [ ] Add Slack/Teams notifications for incidents
- [ ] Implement feedback loop: learn from resolved incidents

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- DataHub team for the open-source Context Platform
- Model Context Protocol (MCP) for the agent framework
- Hackathon organizers for the opportunity to build

---

**Built with ❤️ for the DataHub Agent Hackathon (Aug 2026)**

**Repo:** [github.com/0xConsole/datahub-sentinel-agent](https://github.com/0xConsole/datahub-sentinel-agent)  
**Demo:** [datahub-sentinel-agent.vercel.app](https://datahub-sentinel-agent.vercel.app) (coming soon)