Cursor: Add Enterprise API key support and analytics dashboards

backend/plugins/cursor/README.md

@@ -31,6 +31,15 @@ It follows the same structure and patterns as other DevLake AI usage plugins (no
 | `/teams/filtered-usage-events` | POST | Event-level usage and charges |
 | `/teams/daily-usage-data` | POST | Per-user per-day adoption metrics |
 
+**Cursor Enterprise API endpoints (Enterprise keys only -- skipped for Team keys):**
+
+| Endpoint | Method | Data |
+|----------|--------|------|
+| `/analytics/ai-code/commits` | GET | Per-commit AI line attribution (TAB vs Composer vs non-AI) |
+| `/analytics/ai-code/commits/:commitHash` | GET | Commit-detail file blame and conversation metadata (limited alpha) |
+| `/analytics/ai-code/changes` | GET | Granular accepted AI changes with per-file metadata |
+| `/analytics/team/conversation-insights` | GET | Aggregate conversation work classifications (intents, categories, complexity) |
+
 **Stored data (tool layer):**
 
 | Table | Description |
@@ -39,6 +48,11 @@ It follows the same structure and patterns as other DevLake AI usage plugins (no
 | `_tool_cursor_usage_events` | Billable usage events with model, tokens, and charged amounts |
 | `_tool_cursor_user_spend` | Per-user spend for the current billing cycle (on-demand and included) |
 | `_tool_cursor_daily_usage` | Daily adoption metrics: completions, requests by feature, tab acceptance, line edits |
+| `_tool_cursor_ai_code_commits` | Per-commit AI line attribution: TAB, Composer, and non-AI lines (Enterprise only) |
+| `_tool_cursor_ai_code_range_annotations` | File-level blame rows from commit-detail API with conversation and model (Enterprise alpha) |
+| `_tool_cursor_ai_code_conversations` | Conversation metadata referenced by commit-detail range annotations (Enterprise alpha) |
+| `_tool_cursor_ai_code_changes` | Accepted AI changes with source, model, and per-file metadata (Enterprise only) |
+| `_tool_cursor_conversation_insights` | Flattened Conversation Insights metrics: intents, categories, complexity, guidance, work types (Enterprise only) |
 
 Data is collected in the **Raw → Tool** layers only. There is no domain-layer converter in this plugin; Grafana dashboards query `_tool_cursor_*` tables directly.
 
@@ -60,6 +74,10 @@ flowchart LR
 2. `collectUsageEvents` → `extractUsageEvents`
 3. `collectUserSpend` → `extractUserSpend`
 4. `collectDailyUsage` → `extractDailyUsage`
+5. `collectAiCodeCommits` → `extractAiCodeCommits` *(Enterprise only -- skipped for Team keys)*
+6. `collectAiCodeCommitDetails` → `extractAiCodeCommitDetails` *(Enterprise alpha -- skipped when commit-details inaccessible)*
+7. `collectAiCodeChanges` → `extractAiCodeChanges` *(Enterprise only -- skipped for Team keys)*
+8. `collectConversationInsights` → `extractConversationInsights` *(Enterprise only -- skipped when insights disabled or inaccessible)*
 
 ## Repository layout
 
@@ -124,17 +142,19 @@ Run the blueprint on a daily schedule to keep usage and cost data current.
 - **Date range chunking**: both `/teams/daily-usage-data` and `/teams/filtered-usage-events` requests are split into **30-day** chunks (API limit for daily usage; applied to usage events for resilience).
 - **Extract**: `extractUsageEvents` and `extractDailyUsage` use a cursor-local stateful extractor (incremental by default). Incremental runs also promote any raw rows with `id > MAX(_raw_data_id)` already in the tool table, so collected-but-unpromoted raw data is healed without a full refresh. A config version bump (`extractorVersion`) triggers a one-time full re-extract after upgrade.
 - **Dashboards**: query `_tool_cursor_*` tables only. Legacy domain tables (`cursor_usage`, `cursor_team_events`) are unrelated and must not be used.
-- **Rate limiting**: collectors honor `Retry-After` response headers and respect the configured `rateLimitPerHour`.
+- **Rate limiting**: collectors honor `Retry-After` response headers and respect the configured `rateLimitPerHour`. The Admin API documents **20 requests/minute**; the connection default of **1,200/hour** (~20/min) matches that limit. Enterprise Analytics endpoints use separate quotas — **AI Code Tracking** (`/analytics/ai-code/*`) is **20 requests/minute**; team-level analytics (`/analytics/team/*`) is **100 requests/minute**. If enterprise collection triggers throttling, lower `rateLimitPerHour` on the connection.
 
 ## Dashboards
 
-Grafana dashboard JSON lives under `grafana/dashboards/mysql/`:
+Grafana dashboard JSON lives under both `grafana/dashboards/mysql/` and `grafana/dashboards/postgresql/`. PostgreSQL UIDs add a `-pg` suffix.
 
-| Dashboard | File | UID |
-|-----------|------|-----|
-| Cursor Usage & Cost | `cursor-usage.json` | `cursor_usage` |
-| AI Cost Efficiency (Cursor panels) | `ai-cost-efficiency.json` | — |
-| Multi-AI Comparison (Cursor panels) | `multi-ai-comparison.json` | — |
+| Dashboard | File | MySQL UID | PostgreSQL UID |
+|-----------|------|-----------|----------------|
+| Cursor Usage & Cost | `cursor-usage.json` | `cursor_usage` | `cursor_usage-pg` |
+| Cursor Enterprise AI Code & Insights | `cursor-enterprise.json` | `cursor_enterprise` | `cursor_enterprise-pg` |
+| Cursor BugBot Review Analytics | `cursor-bugbot.json` | `cursor_bugbot` | `cursor_bugbot-pg` |
+| AI Cost Efficiency (Cursor panels) | `ai-cost-efficiency.json` | — | — |
+| Multi-AI Comparison (Cursor panels) | `multi-ai-comparison.json` | — | — |
 
 ## Error handling
 
@@ -149,30 +169,52 @@ Grafana dashboard JSON lives under `grafana/dashboards/mysql/`:
 
 Tokens are sanitized before persisting. Connection test results include a `permissions` object showing which Admin API endpoints succeeded.
 
-## Not collected (Enterprise AI Code Tracking)
+## Enterprise AI Code Tracking
 
-The following endpoints from the [AI Code Tracking API](https://cursor.com/docs/account/teams/ai-code-tracking-api) are **Enterprise plan only** and are **not implemented** in this plugin (no collector, extractor, or `_tool_cursor_*` table):
+The following endpoints from the [AI Code Tracking API](https://cursor.com/docs/account/teams/ai-code-tracking-api) are **Enterprise plan only** and are collected when the connection uses an Enterprise Admin API key:
 
-| Endpoint | Purpose |
-|----------|---------|
-| `GET /analytics/ai-code/commits` | Per-commit AI line attribution (TAB vs Composer vs non-AI) |
-| `GET /analytics/ai-code/changes` | Granular accepted AI changes |
-| `GET /analytics/ai-code/commits.csv` / `changes.csv` | Bulk CSV exports of the above |
+| Endpoint | Purpose | Tool table |
+|----------|---------|------------|
+| `GET /analytics/ai-code/commits` | Per-commit AI line attribution (TAB vs Composer vs non-AI) | `_tool_cursor_ai_code_commits` |
+| `GET /analytics/ai-code/commits/:commitHash` | File-level blame and conversation metadata (limited alpha) | `_tool_cursor_ai_code_range_annotations`, `_tool_cursor_ai_code_conversations` |
+| `GET /analytics/ai-code/changes` | Granular accepted AI changes | `_tool_cursor_ai_code_changes` |
+
+Team/Business Admin API keys receive **401** on these routes. `TestConnection` probes `/analytics/team/dau`, `/analytics/ai-code/commits`, and `/analytics/team/conversation-insights` to detect Enterprise access. The detected `KeyTier` (`personal`, `team`, or `enterprise`) is stored on the connection when you create, update the token, or test an existing connection. **Optional endpoints** (`hasBugbotReviews`, `hasConversationInsights`, `hasAiCodeCommitDetails`) are re-probed at the start of each pipeline run and persisted on the connection, so changes in the Cursor dashboard are picked up without a manual Test Connection. Enterprise collectors check `KeyTier` at runtime and skip silently for non-enterprise keys. Commit-detail collectors require `hasAiCodeCommitDetails` (limited alpha — probed via first commit hash from the list endpoint). Conversation Insights collectors also require `hasConversationInsights` on the connection (false when insights are disabled in Cursor team settings).
+
+## Conversation Insights
 
-Team/Business Admin API keys typically receive **401** on these routes. Connection test optionally probes `GET /analytics/team/dau` (`probeEnterpriseAnalytics`) to detect Enterprise access; it does **not** ingest analytics data.
+The following endpoint from the [Analytics API](https://cursor.com/docs/account/teams/analytics-api) is **Enterprise plan only** and is collected when the connection uses an Enterprise Admin API key with Conversation Insights enabled:
 
-**Metrics this would unlock** (shown on the Cursor native dashboard but absent from DevLake today):
+| Endpoint | Purpose | Tool table |
+|----------|---------|------------|
+| `GET /analytics/team/conversation-insights` | Aggregate work classifications (intents, complexity, categories, guidance levels, work types) | `_tool_cursor_conversation_insights` |
 
-- AI share of **committed** code (not editor line acceptance)
-- Per-commit `commitSource`: IDE, CLI, or cloud
-- Repo name and primary-branch filters
-- TAB vs Composer line attribution on commits
+Returns **aggregate** insights only — no raw conversation content or conversation IDs. Collectors request all five `include` slices per date chunk. Optional access is re-probed at pipeline start; enable Conversation Insights in Cursor team settings and re-run the pipeline (Test Connection is optional but shows the current status in Config UI).
 
-The **Cursor Usage** Grafana dashboard (`grafana/dashboards/mysql/cursor-usage.json`) uses Admin API proxies from `_tool_cursor_daily_usage` and `_tool_cursor_usage_events` instead; panel descriptions note where metrics are approximate.
+## BugBot Review Analytics
+
+The following endpoint from the [BugBot Analytics API](https://cursor.com/docs/bugbot) is collected when `TestConnection` confirms access via `GET /analytics/team/bugbot-reviews`:
+
+| Endpoint | Purpose | Tool tables |
+|----------|---------|-------------|
+| `GET /analytics/team/bugbot-reviews` | Completed BugBot reviews with findings, cost, and resolution status | `_tool_cursor_bugbot_reviews`, `_tool_cursor_bugbot_findings` |
+
+Access is probed during connection test and persisted as `hasBugbotReviews` on the connection (may be available on Team or Enterprise keys). Collectors skip silently when the flag is false.
+
+**Not yet collected:**
+
+| Endpoint | Purpose |
+|----------|---------|
+| `GET /analytics/ai-code/commits.csv` / `changes.csv` | Bulk CSV exports (not needed; JSON pagination covers the same data) |
+| `GET /analytics/team/agent-edits` | Agent edit metrics (suggested vs accepted diffs) |
+| `GET /analytics/team/tabs` | Tab suggestion/accept/reject with line counts |
+| `GET /analytics/team/dau` | CLI/Cloud/BugBot DAU breakdown |
+| `GET /analytics/team/models` | Model usage by message count per day |
+| `GET /analytics/by-user/*` | Per-user breakdowns of the above |
 
 ## Limitations
 
-- **Team/Business Admin API only** — Enterprise-only Analytics API endpoints (`/analytics/*`) are not collected in this plugin (see [Not collected (Enterprise AI Code Tracking)](#not-collected-enterprise-ai-code-tracking) above).
+- **Enterprise API endpoints are conditional** — AI Code Tracking (`/analytics/ai-code/*`) is collected only with Enterprise Admin keys. Conversation Insights runs when `hasConversationInsights` is true. BugBot review analytics (`/analytics/team/bugbot-reviews`) runs when `hasBugbotReviews` is true on the connection. Other Analytics endpoints (`/analytics/team/*`) are not yet collected (see sections above).
 - **Tool layer only** — no domain-layer tables; cross-plugin joins (Jira, GitHub PRs, etc.) are done in Grafana SQL or separate tooling.
 - **Team-level scope** — one scope per connection represents the whole team; per-team multi-tenant collection is not supported.
 - **Beta** — the plugin is marked beta in Config UI while the Admin API surface continues to evolve.

backend/plugins/cursor/api/connection.go

@@ -18,12 +18,14 @@ limitations under the License.
 package api
 
 import (
+	gocontext "context"
 	"strings"
 
 	"github.com/apache/devlake/core/errors"
 	"github.com/apache/devlake/core/plugin"
 	helper "github.com/apache/devlake/helpers/pluginhelper/api"
 	"github.com/apache/devlake/plugins/cursor/models"
+	"github.com/apache/devlake/plugins/cursor/service"
 )
 
 // PostConnections creates a new Cursor connection.
@@ -37,6 +39,9 @@ func PostConnections(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput,
 	if err := validateConnection(connection); err != nil {
 		return nil, err
 	}
+	if err := service.PopulateKeyTier(gocontext.Background(), basicRes, connection); err != nil {
+		return nil, err
+	}
 
 	if err := connectionHelper.Create(connection, input); err != nil {
 		return nil, err
@@ -49,13 +54,19 @@ func PatchConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput,
 	if err := connectionHelper.First(connection, input.Params); err != nil {
 		return nil, err
 	}
+	originalToken := connection.Token
 	if err := (&models.CursorConnection{}).MergeFromRequest(connection, input.Body); err != nil {
 		return nil, errors.Convert(err)
 	}
 	connection.Normalize()
 	if err := validateConnection(connection); err != nil {
 		return nil, err
 	}
+	if connection.Token != originalToken {
+		if err := service.PopulateKeyTier(gocontext.Background(), basicRes, connection); err != nil {
+			return nil, err
+		}
+	}
 	if err := connectionHelper.SaveWithCreateOrUpdate(connection); err != nil {
 		return nil, err
 	}

backend/plugins/cursor/api/test_connection.go

@@ -49,22 +49,29 @@ func TestConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput,
 
 // TestExistingConnection validates a stored Cursor connection with optional overrides.
 func TestExistingConnection(input *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
-	connection := &models.CursorConnection{}
-	if err := connectionHelper.First(connection, input.Params); err != nil {
+	stored := &models.CursorConnection{}
+	if err := connectionHelper.First(stored, input.Params); err != nil {
 		return nil, plugin.WrapTestConnectionErrResp(basicRes, errors.BadInput.Wrap(err, "find connection from db"))
 	}
-	if err := (&models.CursorConnection{}).MergeFromRequest(connection, input.Body); err != nil {
+	connection := *stored
+	if err := (&models.CursorConnection{}).MergeFromRequest(&connection, input.Body); err != nil {
 		return nil, plugin.WrapTestConnectionErrResp(basicRes, errors.Convert(err))
 	}
 
 	connection.Normalize()
-	if err := validateConnection(connection); err != nil {
+	if err := validateConnection(&connection); err != nil {
 		return nil, plugin.WrapTestConnectionErrResp(basicRes, err)
 	}
 
-	result, err := service.TestConnection(gocontext.Background(), basicRes, connection)
+	result, err := service.TestConnection(gocontext.Background(), basicRes, &connection)
 	if err != nil {
 		return nil, plugin.WrapTestConnectionErrResp(basicRes, err)
 	}
+	if result != nil && result.KeyTier != "" {
+		service.ApplyTestResultToConnection(stored, result)
+		if err := connectionHelper.SaveWithCreateOrUpdate(stored); err != nil {
+			return nil, plugin.WrapTestConnectionErrResp(basicRes, err)
+		}
+	}
 	return &plugin.ApiResourceOutput{Body: result, Status: http.StatusOK}, nil
 }

backend/plugins/cursor/e2e/ai_code_commit_details_extractors_test.go

@@ -0,0 +1,118 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"testing"
+	"time"
+
+	"github.com/apache/devlake/core/config"
+	"github.com/apache/devlake/core/models/common"
+	"github.com/apache/devlake/core/runner"
+	"github.com/apache/devlake/helpers/e2ehelper"
+	helper "github.com/apache/devlake/helpers/pluginhelper/api"
+	"github.com/apache/devlake/plugins/cursor/impl"
+	"github.com/apache/devlake/plugins/cursor/models"
+	"github.com/apache/devlake/plugins/cursor/tasks"
+)
+
+func TestCursorAiCodeCommitDetailsExtractorsDataFlow(t *testing.T) {
+	cfg := config.GetConfig()
+	dbUrl := cfg.GetString("E2E_DB_URL")
+	if dbUrl == "" {
+		t.Skip("skipping e2e test: E2E_DB_URL is not set")
+	}
+	if err := runner.CheckDbConnection(dbUrl, 10*time.Second); err != nil {
+		t.Skipf("skipping e2e test: cannot connect to E2E_DB_URL: %v", err)
+	}
+
+	var cursorPlugin impl.Cursor
+	dataflowTester := e2ehelper.NewDataFlowTester(t, "cursor", cursorPlugin)
+
+	taskData := &tasks.CursorTaskData{
+		Options: &tasks.CursorOptions{
+			ConnectionId: 1,
+			ScopeId:      "team",
+		},
+		Connection: &models.CursorConnection{
+			CursorConn: models.CursorConn{
+				RestConnection:         helper.RestConnection{Endpoint: models.DefaultEndpoint},
+				KeyTier:                models.KeyTierEnterprise,
+				HasAiCodeCommitDetails: true,
+			},
+		},
+	}
+
+	dataflowTester.ImportCsvIntoRawTable("./raw_tables/_raw_cursor_ai_code_commit_details.csv", "_raw_cursor_ai_code_commit_details")
+
+	dataflowTester.FlushTabler(&models.CursorAiCodeConversation{})
+	dataflowTester.FlushTabler(&models.CursorAiCodeRangeAnnotation{})
+
+	dataflowTester.Subtask(tasks.ExtractAiCodeCommitDetailsMeta, taskData)
+
+	dataflowTester.VerifyTableWithOptions(&models.CursorAiCodeConversation{}, e2ehelper.TableOptions{
+		CSVRelPath:  "./snapshot_tables/_tool_cursor_ai_code_conversations.csv",
+		IgnoreTypes: []interface{}{common.NoPKModel{}},
+	})
+
+	dataflowTester.VerifyTableWithOptions(&models.CursorAiCodeRangeAnnotation{}, e2ehelper.TableOptions{
+		CSVRelPath:  "./snapshot_tables/_tool_cursor_ai_code_range_annotations.csv",
+		IgnoreTypes: []interface{}{common.NoPKModel{}},
+	})
+}
+
+func TestCursorAiCodeCommitDetailsExtractorsSkipWhenInaccessible(t *testing.T) {
+	cfg := config.GetConfig()
+	dbUrl := cfg.GetString("E2E_DB_URL")
+	if dbUrl == "" {
+		t.Skip("skipping e2e test: E2E_DB_URL is not set")
+	}
+	if err := runner.CheckDbConnection(dbUrl, 10*time.Second); err != nil {
+		t.Skipf("skipping e2e test: cannot connect to E2E_DB_URL: %v", err)
+	}
+
+	var cursorPlugin impl.Cursor
+	dataflowTester := e2ehelper.NewDataFlowTester(t, "cursor", cursorPlugin)
+
+	taskData := &tasks.CursorTaskData{
+		Options: &tasks.CursorOptions{
+			ConnectionId: 1,
+			ScopeId:      "team",
+		},
+		Connection: &models.CursorConnection{
+			CursorConn: models.CursorConn{
+				RestConnection: helper.RestConnection{Endpoint: models.DefaultEndpoint},
+				KeyTier:        models.KeyTierEnterprise,
+			},
+		},
+	}
+
+	dataflowTester.ImportCsvIntoRawTable("./raw_tables/_raw_cursor_ai_code_commit_details.csv", "_raw_cursor_ai_code_commit_details")
+	dataflowTester.FlushTabler(&models.CursorAiCodeConversation{})
+	dataflowTester.FlushTabler(&models.CursorAiCodeRangeAnnotation{})
+
+	dataflowTester.Subtask(tasks.ExtractAiCodeCommitDetailsMeta, taskData)
+
+	var count int64
+	if err := dataflowTester.Db.Model(&models.CursorAiCodeRangeAnnotation{}).Count(&count).Error; err != nil {
+		t.Fatalf("count range annotations: %v", err)
+	}
+	if count != 0 {
+		t.Fatalf("expected 0 range annotations without commit-details access, got %d", count)
+	}
+}