Technical Walkthrough
Galileo: AI Marketing Agent with Observations & Evals
A professional, plain-English walkthrough of your real multi-agent codebase: framework choices, node-level graph flow, state behavior, tool usage, observability, and evaluation pipeline.
This multi-agent system is built to handle two real business workflows in one place: GTM product assistance and outreach content execution. It works by first routing each user request to the right specialist branch, then running branch-specific nodes that gather context, apply business gates (like pricing email verification or send approval), and return a final answer. Every step is traced with Galileo so you can inspect exactly what happened and evaluate quality over time.
LangGraph
Supervisor Routing Agent + GTM + Outreach
Qdrant + Gemini Embeddings
Anthropic LLM
Galileo Tracing + Evals
Step 1: Architecture Overview
The assistant behaves like a small team with one coordinator and two specialists:
- Supervisor Routing Agent: reads the user request and picks the best branch.
- GTM Agent: handles product, features, comparisons, and pricing logic.
- Outreach Agent: handles content generation, lead research, and optional email sending.
All branches use one shared state object, so every node updates the same request context consistently.
Step 2: Project Map (What Each File Does)
| File | What it does |
|---|---|
| app.py | Main Streamlit entrypoint that boots the UI and wires sidebar + chat loop. |
| ui/ui.py | UI behavior: styles, upload widgets, chat history rendering, prompt handling, and trace display. |
| services/agent_service.py | Service adapter for `ask()`, `load_graph_image()` (PNG), and `load_graph_ascii()` (fallback). |
| services/vector_db_service.py | Thin service wrapper for adding docs/PDFs and checking vector DB counts. |
| agents/__init__.py | Main runtime API (`ask`), `get_graph_image()` (PNG), `get_graph_ascii()` (fallback when PNG fails). |
| agents/graph.py | Defines LangGraph nodes and all edges/conditional routes. |
| agents/state.py | Shared `AgentState` schema and `steps` merge behavior across nodes. |
| agents/router_agent/nodes.py | Intent classification (`gtm` vs `outreach`) using LLM. |
| agents/gtm_agent/nodes.py | GTM branch: retrieve context, pricing gate, email gate, final GTM generation. |
| agents/outreach_agent/nodes.py | Outreach branch: research, draft generation, send gate, optional SendGrid send. |
| agents/tools.py | All 4 tools: search_knowledge_base (Qdrant), web_search (DuckDuckGo), apollo_search (Apollo), send_email (SendGrid) + tool-calling loop. |
| vector_db/database.py | Qdrant client setup, collection validation, add/search/count methods. |
| vector_db/chunker.py | Text chunking and PDF extraction for indexing documents. |
| vector_db/embeddings.py | Gemini embedding model factory used by vector store operations. |
| llm.py | Anthropic LLM loader with config/env validation. |
| config.py | Central configuration and environment variable loading. |
| observability/galileo.py | Galileo SDK integration: init, callbacks, spans, sessions, logger helpers. |
| evals/run_galileo_evals.py | Evaluation harness with sessions mode, experiment mode, and coverage checks. |
| evals/README.md | Eval-specific usage guide, criteria, and Galileo integration notes. |
| README.md | Top-level architecture, setup/run steps, and project navigation docs. |
Step 3: Framework and Runtime Stack
| Layer | Technology | How it is used in this codebase |
|---|---|---|
| Orchestration | LangGraph | Node graph in agents/graph.py with conditional edges for routing. |
| Generation LLM | ChatAnthropic | Used in supervisor routing, gates, and generation nodes via llm.py. |
| Embeddings | Google Gemini Embeddings | Converts knowledge docs into vectors for Qdrant retrieval. |
| Vector Store | Qdrant Cloud | Stores chunks and returns similarity-scored context for tool calls. |
| Web Research | DuckDuckGo | web_search — live web for market/competitor context. |
| Leads + Email | Apollo + SendGrid | apollo_search (leads by job/industry), send_email (outreach send path). |
| Observability | Galileo | Spans, traces, sessions, and experiment logging. |
| UI | Streamlit | Chat UX, graph preview (PNG with ASCII/Mermaid fallback), upload flow, and step trace display. |
Step 4: Node Graph (Current Runtime)
Node (function)
Edge (always)
Branch (conditional)
The graph is tree-shaped from a single classifier root, with conditional branch points for pricing and send intent. The Streamlit UI shows PNG when available; falls back to ASCII or Mermaid if Mermaid.INK is unreachable.
Step 5: State Contract and How It Evolves
| State field | Who writes it | Purpose | Typical value |
|---|---|---|---|
| question | ask() | Original user request | "What does Galileo cost?" |
| agent_type | classify | Branch selector | gtm or outreach |
| context | gtm_retrieve / outreach_research | Tool-generated evidence context | KB snippets, web notes, leads |
| is_pricing | pricing_gate | Controls pricing branch behavior | true / false |
| user_email | collect_email | Required for gated pricing output | ritvik@dauji.ai |
| send_requested | send_gate | Controls send vs review routing | true / false |
| answer | gtm_generate / outreach_generate / outreach_send | Final user-facing response | Generated answer or send summary |
| steps | Every node | Trace log shown in UI and eval parsing | ["Supervisor Routing Agent -> GTM", "..."] |
Step 6: Node-by-Node Execution Framework
GTM branch logic
- gtm_retrieve gathers internal + web context.
- pricing_gate decides if email verification is required.
- collect_email extracts/validates email from question text.
- gtm_generate composes final grounded answer.
Outreach branch logic
- outreach_research selects Apollo+KB or KB+web tool mix.
- outreach_generate drafts post/email content.
- send_gate resolves "send now" vs "review only".
- outreach_send performs SendGrid delivery if approved.
Step 6b: All 4 Tools
| Tool | Backend | Used by | Purpose |
|---|---|---|---|
| search_knowledge_base | Qdrant | GTM, Outreach | Product docs from your indexed knowledge base |
| web_search | DuckDuckGo | GTM, Outreach | Live web for market/competitor/industry info |
| apollo_search | Apollo.io | Outreach | Find leads by job title, location, industry |
| send_email | SendGrid | Outreach | Deliver personalized emails when user asks to send |
Step 7: Galileo SDK (Tracing + Evaluations)
Tracing in app runtime
- observability/galileo.py is the integration hub for initialization, callbacks, spans, and logger access.
- ensure_galileo_initialized() connects project + log stream once.
- get_langchain_config() injects GalileoCallback into LLM/tool invokes.
- @log_span(...) decorates nodes/tools so every step is captured as a named span.
- agents/__init__.py::ask() starts top-level trace (start_trace) and concludes/flushes it.
- ui/ui.py starts one Galileo session per fresh chat thread.
Evaluation logging flow
- evals/run_galileo_evals.py uses the same ask() runtime path (no eval/prod drift).
- Sessions mode: each dataset row opens its own Galileo session via logger.start_session(...).
- Experiment mode: uses galileo.experiments.run_experiment(...) for experiment tracking.
- Expected tool coverage is inferred from trace steps and validated at run end.
- Key env vars: GALILEO_API_KEY, GALILEO_PROJECT, GALILEO_LOG_STREAM.
Step 8: Examples (End-to-End Requests)
Example A - Pricing
User: "What does Galileo cost?"
Path: classify -> gtm_retrieve -> pricing_gate -> collect_email
Outcome: asks for verified email before full pricing details.
Example B - Content draft
User: "Write a LinkedIn post for AI platform teams."
Path: classify -> outreach_research -> outreach_generate -> send_gate(review)
Outcome: returns draft only, no email sent.
Example C - Send request
User: "Send this to ritvik@dauji.ai now."
Path: classify -> outreach_research -> outreach_generate -> send_gate(send) -> outreach_send
Outcome: sends via SendGrid and returns send status summary.
Step 10: Official Resource Links
| Service | Why it is used here | Link |
|---|---|---|
| Galileo | Tracing, session logs, and experiment/eval tracking | app.galileo.ai |
| Anthropic | Primary chat model runtime for routing/generation logic | console.anthropic.com |
| Google AI Studio | API key source for Gemini embedding model | makersuite.google.com/app/apikey |
| Qdrant Cloud | Vector storage and semantic retrieval | cloud.qdrant.io |
| LangGraph | State graph orchestration and conditional routing | LangGraph docs |
| Streamlit | Application UI and interactive chat view | docs.streamlit.io |
| Apollo | Lead search and enrichment in outreach flow | apollo.io |
| SendGrid | Email delivery for approved send actions | sendgrid.com |