Own the behavioral truth
Who activated, which workflows they used, where they stopped, what they repeated, what changed after a release, and which cohorts behave differently.
PostHog Agent vs. AlexAI Product Manager
What perfect product tracking would unlock—and what still needs our own intelligence layer.
Research current Aug 6
Bottom line
PostHog can be the analytics engine for the AI Product Manager, but it is not the whole Product Manager.
With perfect tracking, PostHog AI could answer most factual questions about who used what, where they got stuck, what changed, and which behaviors predict repeat use. It can already generate SQL, insights, and native dashboards. Our layer should turn that evidence into clear priorities, add product-quality and project context, and present Alex with a stable executive view.
The recommendation
Use PostHog. Do not rebuild its analytics agent. Build the AlexAI Product Manager above it.
PostHog should answer the behavioral questions. The custom layer should decide what matters and what to do next.
The clean division of labor
Three layers, each doing the work it is best suited to do.
Add outcome and quality signals
Track whether work finished, was copied or saved, needed an immediate redo, received positive feedback, or passed an evaluation—not merely whether a button was clicked.
Make the product judgment
Combine behavior with feedback, traces, evaluations, roadmap constraints, and Alex’s priorities to recommend the next action in plain English.
The important implication: perfect PostHog tracking would make the evidence far better, but it would not automatically produce the exact executive Product Manager Alex described. That final step is a product we build.
What this means for the prototype
| Need | PostHog’s role | Our Product Manager’s role |
|---|---|---|
| Reliable numbers | Query events, people, funnels, paths, retention, experiments, replays, errors, and connected warehouse data. | Use the numbers without exposing query mechanics or internal data-source names to Alex. |
| Custom dashboards | Generate and maintain native PostHog dashboards, insights, and text tiles. | Render the branded, narrative, decision-oriented experience Alex reviewed. |
| Priorities | Surface patterns, anomalies, affected cohorts, and likely causes. | Judge value, urgency, effort, strategic fit, and what Tom should actually do next. |
| Product quality | Measure tracked outcomes, feedback, regressions, and model/mode performance if instrumented. | Interpret traces and evaluations to explain why an answer failed and whether a fix is safe. |
Verified current capability
Yes: PostHog AI can already answer data questions and build dashboards.
PostHog says its AI can answer questions in plain English, generate SQL, build dashboards, and surface insights. Its current MCP exposes 18 dashboard tools, including creating dashboards, adding widgets, updating them, running them, and rearranging tiles.
What “custom” means
Custom analytics dashboard: yes.
Custom Alex-facing product: build our layer.
PostHog can create the analytical artifact. It will not automatically reproduce our branded narrative interface and product-management rules.
What the current PostHog agent stack offers
This combines PostHog AI, its MCP interface, context warehouse, and current self-driving features.
Natural-language product analysis
Ask questions about signups, funnels, paths, retention, cohorts, and other tracked behavior; receive an answer backed by a real query.
Official examples ↗SQL, insights, and dashboards
Generate SQL and native visualizations, create dashboards, add or update widgets and text tiles, and run saved insights.
Dashboard tools ↗Replay and friction analysis
Retrieve and summarize recordings, use Replay Vision scanners, identify dead clicks or stalls, and save affected users as cohorts.
Signal sources ↗Errors and operational diagnosis
Rank errors by frequency and users affected, inspect stack traces and logs, diagnose ingestion problems, and propose fixes.
Debugging workflows ↗Take product actions
Create cohorts, surveys, flags, experiments, workflows, and notifications. Mutating actions can use confirmation and approval flows.
Read/write scope ↗Join product and business context
Use the context warehouse to query product events alongside connected systems such as databases, Stripe, CRMs, and support tools.
Context warehouse ↗Run autonomous scouts
Background agents can gather signals, deduplicate them into reports, and—in the self-driving setup—produce code changes for human review.
Self-driving overview ↗Work through AI clients
The hosted MCP lets Codex, Claude, Cursor, and other compatible agents query and act on PostHog without a separate analytics UI.
Official AI plugin ↗
Boundary: these are product capabilities, not proof that every capability is enabled or correctly instrumented in AlexAI today. This page is not an audit of our current PostHog setup.
Hypothetical: perfect instrumentation
“Perfect tracking” means following the entire path from invitation to useful outcome—not capturing every private word.
The product needs a coherent event and outcome model. Every important question should be answerable from stable identities, task metadata, product behavior, quality signals, and release exposure.
Success criterion
For every important workflow, we can tell who tried it, what happened, whether it helped, and what they did next.
The eight layers of complete tracking
Each layer answers a different class of product question.
1. Identity and account
User, organization, role, internal/external status, invitation source, permissions, and acquisition cohort.
2. Onboarding
Invitation delivered, accepted, account created, first task started, first useful result, and time between each step.
3. Task intent
Privacy-safe workflow classification: research, writing, policy analysis, summarization, brainstorming, or another job.
4. Execution
Model, mode, tool, duration, sources consulted, completion, cancellation, interruption, retry, and errors.
5. Outcome
Copied, downloaded, saved, shared, continued, rerun, abandoned, or followed by corrective instructions.
6. Quality
Explicit feedback, evaluation scores, citation coverage, instruction-following, human review, and known failure labels.
7. Retention and value
Repeat use by workflow, depth of use, return intervals, habitual jobs, expansion to new features, and team adoption.
8. Change exposure
Release version, experiment group, configuration, model policy, feature flags, and the exact fixes active for each task.
Privacy rule: “perfect” does not mean copying raw private chats into PostHog. Prefer content-safe metadata, derived classifications, evaluation results, and permissioned pointers to detailed evidence. Raw content should remain governed by the product’s retention and access rules.
What complete tracking changes
“Five users sent messages.”
This is activity, but it does not show whether AlexAI Pro created value.
“Three users completed a repeatable research job.”
This begins to identify a real product behavior and a potential flagship workflow.
“Fix source coverage, then recruit these users as design partners.”
This is the executive recommendation the Product Manager layer should make.
Questions the agent could answer with perfect tracking
Select a question to see the form of answer and the evidence it would require. All example answers are illustrative, not current AlexAI measurements.
Direct answer
Yes, we could get custom dashboards—and there are two different versions worth separating.
PostHog AI can create native PostHog dashboards today. A branded Alex-facing dashboard with narrative judgment, cross-source context, permissions, and dynamically generated tabs would be our custom application powered by PostHog.
Best arrangement
Native dashboards for analysis. Custom dashboard for decisions.
Use PostHog’s artifacts as the dependable analytical substrate, not as the final executive experience.
AlexAI Pro — product health
Hypothetical dashboard generated from complete trackingActivated invitees50%
Repeat users11
Useful outcome rate72%
| Capability | Native PostHog | Custom AlexAI layer |
|---|---|---|
| Generated charts and dashboards | Yes — built in. | Yes — can reuse PostHog queries or APIs. |
| Stable executive narrative | Partial — insights and reports, but analytics-first. | Yes — designed around Alex’s questions and decisions. |
| Dynamic tabs from chat | Partial — create dashboards, insights, and notebooks. | Yes — any generated analysis can become a temporary or saved view. |
| Cross-source strategy context | Possible when sources are connected and modeled. | Purpose-built for feedback, evaluations, roadmap, and meeting context. |
| Alex-specific product judgment | Not automatic. | The central job. |
Recommended architecture
Put the custom Product Manager above PostHog instead of building a second analytics system beside it.
PostHog supplies the product telemetry, querying, native artifacts, and agent tools. The AlexAI layer adds governed context, persistent product rules, and executive-quality recommendations.
Why this is better
We inherit a mature analytics agent while retaining control of the experience Alex actually wants.
Layer 1
AlexAI instrumentation
Identity, onboarding, workflow, model, mode, outcome, quality, and release events.
→
Layer 2
PostHog context
Events, cohorts, replays, errors, surveys, experiments, and connected business data.
→
Layer 3
PostHog AI and MCP
Queries, SQL, insights, native dashboards, reports, cohorts, and safe product actions.
→
Layer 4
AlexAI Product Manager
Cross-source synthesis, strategic judgment, recommended action, permissions, and Alex-facing UI.
What the custom layer still needs beyond ordinary analytics
Traces, evaluations, and failure analysis
A clickstream cannot determine whether a Canadian infrastructure answer was substantively wrong or whether a generated article falsely attributed authorship. Those require governed trace and evaluation evidence.
Feedback, roadmap, effort, and Alex’s priorities
“What should Tom do next?” requires current commitments, engineering effort, user feedback, and strategic priorities—not only product behavior.