Skip to content

Crystalyse Provenance System - Complete Technical Documentation

Version: v1.0.0

Table of Contents

  1. Overview
  2. Architecture
  3. Core Components
  4. Integration Points
  5. Data Flow
  6. File Structure
  7. Implementation Status
  8. Known Limitations
  9. Configuration
  10. Usage Examples

Overview

The provenance system provides complete audit trails for all materials discovery operations in Crystalyse. It captures events, materials, MCP tool calls, and performance metrics in a structured format suitable for reproducibility, analysis, and debugging.

Design Philosophy

  • Always-On: Provenance is a core feature, not optional
  • Transparent: Users see where data is stored
  • Non-Invasive: Graceful degradation if capture fails
  • Structured: JSONL events + JSON summaries for easy parsing
  • Comprehensive: Captures complete lifecycle from query to results

Architecture

High-Level Design

┌─────────────────────────────────────────────────────────────┐
│                      CrystaLyse CLI                         │
│  ┌──────────────────┐           ┌──────────────────────┐   │
│  │  crystalyse      │           │  crystalyse discover │   │
│  │  (interactive)   │           │  (non-interactive)   │   │
│  └────────┬─────────┘           └──────────┬───────────┘   │
│           │                                 │               │
│           └─────────────┬───────────────────┘               │
│                         ▼                                   │
│              ┌─────────────────────┐                        │
│              │  ChatExperience or  │                        │
│              │  Direct discover()  │                        │
│              └──────────┬──────────┘                        │
│                         │                                   │
│                         ▼                                   │
│         ┌───────────────────────────────┐                   │
│         │ EnhancedCrystaLyseAgent       │                   │
│         │   .discover(trace_handler)    │                   │
│         └──────────┬────────────────────┘                   │
└────────────────────┼────────────────────────────────────────┘
                     │
                     ▼
        ┌────────────────────────────────┐
        │ CrystaLyseProvenanceHandler    │
        │  (extends ProvenanceTraceHandler)│
        └──────────┬─────────────────────┘
                   │
                   ▼
        ┌────────────────────────────────┐
        │  ProvenanceTraceHandler        │
        │  (from provenance_system)      │
        └──────────┬─────────────────────┘
                   │
        ┌──────────┴──────────┬──────────────────┐
        ▼                     ▼                   ▼
   ┌─────────┐         ┌──────────────┐    ┌──────────┐
   │ JSONL   │         │  Materials   │    │   MCP    │
   │ Logger  │         │   Tracker    │    │ Detector │
   └─────────┘         └──────────────┘    └──────────┘
        │                     │                   │
        └──────────┬──────────┴───────────────────┘
                   ▼
          ┌─────────────────────┐
          │  Provenance Output  │
          │  provenance_output/ │
          │    runs/            │
          │      session_id/    │
          │        events.jsonl │
          │        summary.json │
          │        materials... │
          └─────────────────────┘

Component Hierarchy

provenance_system/                 # Standalone module
├── __init__.py                    # Module exports
├── core/                          # Core utilities
│   ├── __init__.py
│   ├── jsonl_logger.py           # Event logging
│   ├── materials_tracker.py      # Materials extraction
│   └── mcp_detector.py           # MCP tool detection
├── handlers/                      # Trace handlers
│   ├── __init__.py
│   └── enhanced_trace.py         # ProvenanceTraceHandler
└── integration/                   # Integration helpers
    ├── __init__.py
    └── agent_wrapper.py          # (Not used - circular import avoided)

dev/crystalyse/                    # CrystaLyse integration
├── ui/
│   ├── provenance_bridge.py      # CrystaLyseProvenanceHandler
│   └── chat_ui.py                # Interactive chat integration
├── agents/
│   └── openai_agents_bridge.py   # Agent discover() method
├── cli.py                         # CLI entry points
└── config.py                      # Provenance configuration

Core Components

1. ProvenanceTraceHandler

Location: provenance_system/handlers/enhanced_trace.py

Purpose: Main trace handler that captures all events from OpenAI Agents SDK

Key Methods: 

def __init__(
    console: Optional[Console] = None,
    output_dir: Optional[Path] = None,
    session_id: Optional[str] = None,
    enable_provenance: bool = True,
    enable_visual: bool = True,
    capture_mcp_logs: bool = False,
    save_raw_outputs: bool = True
)

def on_event(event: Any) -> None
    """Captures SDK events and routes to appropriate handlers"""

def finalize() -> Dict[str, Any]
    """Generates summary and saves all outputs"""

Responsibilities: - Captures SDK trace events (tool calls, outputs, errors) - Detects actual MCP tool names from SDK wrapper names - Extracts materials from tool outputs - Logs all events to JSONL - Generates performance metrics - Creates summary JSON

Event Lifecycle: 1. session_start - Initialize session 2. tool_start - Tool invocation begins 3. tool_end - Tool completes, extract materials 4. assistant_output - Response generated 5. session_end - Finalize and summarize

2. CrystaLyseProvenanceHandler

Location: dev/crystalyse/ui/provenance_bridge.py

Purpose: CrystaLyse-specific wrapper around ProvenanceTraceHandler

Key Features: 

def __init__(
    console: Optional[Console] = None,
    config: Optional['CrystaLyseConfig'] = None,
    mode: str = "adaptive",
    session_id: Optional[str] = None,
    **kwargs
)

Responsibilities: - Loads CrystaLyse configuration - Generates mode-prefixed session IDs (crystalyse_{mode}_{timestamp}) - Maps config settings to provenance parameters - Adds CrystaLyse-specific metadata to summaries - Provides convenient access methods (get_summary_path, get_session_info) - Graceful error handling

Configuration Mapping: 

super().__init__(
    console=console,
    output_dir=config.provenance['output_dir'],
    session_id=session_id,
    enable_provenance=True,  # Always enabled
    enable_visual=config.provenance['visual_trace'],
    capture_mcp_logs=config.provenance['capture_mcp_logs'],
    save_raw_outputs=config.provenance['capture_raw']
)

3. JSONLLogger

Location: provenance_system/core/jsonl_logger.py

Purpose: Structured event logging in JSON Lines format

Key Methods: 

def log(event_type: str, data: Dict[str, Any]) -> None
    """Write event to JSONL file"""

def log_session_end(session_id: str, summary: Dict[str, Any]) -> None
    """Final session summary event"""

Event Format: 

{
  "type": "tool_start",
  "ts": "2025-10-08T21:00:00.123456",
  "data": {
    "wrapper": "unknown_tool",
    "call_id": "fc_...",
    "timestamp": "2025-10-08T21:00:00.123456"
  }
}

4. MaterialsTracker

Location: provenance_system/core/materials_tracker.py

Purpose: Extract and track materials from tool outputs

Key Methods: 

def extract_materials(tool_output: Any, tool_name: str) -> List[Dict[str, Any]]
    """Parse tool output for materials data"""

def add_material(material: Dict[str, Any]) -> None
    """Add material to tracking"""

def get_summary() -> Dict[str, Any]
    """Generate materials statistics"""

def save_catalog(path: Path) -> None
    """Save materials_catalog.json"""

Extraction Patterns: - Detects composition strings (e.g., "MgFe2O4", "Li2CoO3") - Extracts formation energies from various formats - Parses JSON outputs from MCP tools - Handles both single materials and batches

5. MCPDetector

Location: provenance_system/core/mcp_detector.py

Purpose: Detect actual MCP tool names from SDK wrapper names

Key Methods: 

def detect_mcp_tool(tool_args: Dict[str, Any], tool_output: Any) -> Optional[str]
    """Identify actual MCP tool from args or output"""

Detection Strategy: 1. Check tool args for server_name or tool_name 2. Parse tool output for tool signatures 3. Match against known MCP tool patterns 4. Return actual tool name or None

Known Tools: - generate_structures - comprehensive_materials_analysis - creative_discovery_pipeline - validate_composition - calculate_formation_energy

Integration Points

1. CLI Entry Points

File: dev/crystalyse/cli.py

Non-Interactive Discovery

@app.command()
def discover(query: str, provenance_dir: Optional[str] = None, hide_summary: bool = False):
    """Single-shot discovery with automatic provenance"""

    config = Config.load()
    if provenance_dir:
        config.provenance['output_dir'] = Path(provenance_dir)

    agent = EnhancedCrystaLyseAgent(
        config=config,
        project_name=state['project'],
        mode=state['mode'].value,
        model=state['model']
    )

    # Agent auto-creates provenance handler
    results = await agent.discover(query)

    # Display summary if enabled
    show_summary = config.provenance['show_summary'] and not hide_summary
    if show_summary and results.get('provenance_summary'):
        display_provenance_summary(results['provenance_summary'])

Interactive Chat

@app.command()
def chat(user: str = "default", session: Optional[str] = None):
    """Interactive chat with provenance per query"""

    chat_experience = ChatExperience(
        project=state['project'] + (f"_{session}" if session else ""),
        mode=state['mode'].value,
        model=state['model'],
        user_id=user
    )

    asyncio.run(chat_experience.run_loop())

Provenance Analysis

@app.command(name="analyse-provenance")
def analyse_provenance(
    session_id: Optional[str] = None,
    latest: bool = False,
    provenance_dir: str = "./provenance_output"
):
    """Analyse provenance from previous sessions"""

    base_dir = Path(provenance_dir) / "runs"

    if latest:
        sessions = sorted(base_dir.glob("*"), key=lambda p: p.stat().st_mtime, reverse=True)
        session_dir = sessions[0]
    elif session_id:
        session_dir = base_dir / session_id
    else:
        # List available sessions
        display_session_list(base_dir)
        return

    # Load and display summary
    display_session_analysis(session_dir)

Path Initialization (Critical for module import):

# Lines 13-19 in cli.py
# Add provenance_system to path for installed package
# This ensures provenance works when using 'crystalyse' command
# Need to add parent directory of provenance_system to sys.path
crystalyse_root = Path(__file__).parent.parent.parent
provenance_system_path = crystalyse_root / "provenance_system"
if provenance_system_path.exists() and str(crystalyse_root) not in sys.path:
    sys.path.insert(0, str(crystalyse_root))

2. ChatExperience Integration

File: dev/crystalyse/ui/chat_ui.py

Initialization: 

def __init__(self, project: str, mode: str, model: str, user_id: str = "default"):
    # ... existing initialization ...
    self.config = Config.load()  # Load config for provenance settings
    self.provenance_handler = None  # Will be created per query

Per-Query Provenance: 

async def run_loop(self):
    while True:
        query = Prompt.ask("➤")

        # Create provenance handler for this query (always-on)
        if PROVENANCE_AVAILABLE:
            trace_handler = CrystaLyseProvenanceHandler(
                console=self.console,
                config=self.config,
                mode=self.mode
            )
            self.provenance_handler = trace_handler
        else:
            trace_handler = ToolTraceHandler(self.console)

        # Discover with provenance
        results = await self.agent.discover(
            enriched_query,
            history=self.history,
            trace_handler=trace_handler
        )

        if results and results.get("status") == "completed":
            # Display response
            self._display_message("assistant", response)

            # Finalize and display provenance summary
            if PROVENANCE_AVAILABLE and self.provenance_handler:
                try:
                    summary = self.provenance_handler.finalize()
                    if summary and self.config.provenance.get('show_summary', True):
                        self._display_provenance_summary(summary)
                except Exception as e:
                    self.console.print(f"[dim yellow]Provenance summary unavailable: {e}[/dim yellow]")

Summary Display: 

def _display_provenance_summary(self, summary: Dict[str, Any]):
    """Display provenance summary in a compact format."""
    from rich.table import Table

    table = Table(title="Provenance Summary", show_header=True, header_style="bold cyan")
    table.add_column("Metric", style="cyan")
    table.add_column("Value", style="yellow")

    # Add key metrics (using actual keys from summary)
    table.add_row("Session ID", summary.get("session_id", "N/A"))
    table.add_row("Materials Found", str(summary.get("materials_found", 0)))

    # Use mcp_operations (actual key) instead of mcp_tools_detected
    mcp_ops = summary.get("mcp_operations", 0)
    table.add_row("MCP Tool Calls", str(mcp_ops))

    # Show tool call breakdown
    tool_calls = summary.get("tool_calls_total", 0)
    table.add_row("Total Tool Calls", str(tool_calls))

    # Add file location
    session_info = summary.get("session_info", {})
    output_dir = session_info.get("output_dir") or summary.get("output_dir")
    if output_dir:
        table.add_row("Output Directory", str(output_dir))

    self.console.print("\n")
    self.console.print(table)
    self.console.print(f"[dim]Analyse with: crystalyse analyse-provenance --session {summary.get('session_id', 'N/A')}[/dim]\n")

3. Agent Integration

File: dev/crystalyse/agents/openai_agents_bridge.py

Auto-Create Provenance Handler: 

async def discover(
    self,
    query: str,
    history: Optional[List[Dict[str, Any]]] = None,
    trace_handler: Optional[ToolTraceHandler] = None
) -> Dict[str, Any]:
    """
    Processes a single discovery request with automatic provenance capture.

    Provenance is always enabled - every query generates a complete audit trail.
    """
    if not SDK_AVAILABLE:
        return {"status": "failed", "error": "OpenAI Agents SDK is not installed."}

    # Auto-create provenance handler if not provided
    if trace_handler is None and PROVENANCE_AVAILABLE and CrystaLyseProvenanceHandler:
        try:
            from rich.console import Console
            trace_handler = CrystaLyseProvenanceHandler(
                config=self.config,
                mode=self.mode,
                console=Console()
            )
        except Exception as e:
            logger.warning(f"Could not create provenance handler: {e}")
            trace_handler = ToolTraceHandler(Console())

    # Create RunConfig with fixed trace_id
    try:
        from agents import RunConfig
        import time

        # Use integer timestamp to avoid dots in trace_id (SDK requirement)
        trace_timestamp = int(time.time())
        trace_id = f"crystalyse_{self.session_id}_{trace_timestamp}"

        run_config = RunConfig(trace_id=trace_id)
    except Exception as e:
        logger.warning(f"Could not create RunConfig: {e}")
        run_config = None

    # Run discovery with provenance
    result = await self.agent.run(
        query,
        trace_handler=trace_handler,
        run_config=run_config
    )

    # Return results with provenance summary
    return {
        "status": "completed",
        "response": result,
        "provenance_summary": trace_handler.finalize() if hasattr(trace_handler, 'finalize') else None
    }

Data Flow

Complete Request Lifecycle

1. User Query
   └─> CLI (crystalyse discover "query" OR crystalyse → "query")
       │
2. Handler Creation
   └─> CrystaLyseProvenanceHandler(config, mode, console)
       │
3. Agent Discovery
   └─> EnhancedCrystaLyseAgent.discover(query, trace_handler)
       │
4. SDK Events
   ├─> session_start → ProvenanceTraceHandler.on_event()
   │                    └─> JSONLLogger.log("session_start", ...)
   │
   ├─> tool_start → ProvenanceTraceHandler.on_event()
   │                 └─> JSONLLogger.log("tool_start", {...})
   │                 └─> Track tool call (EnhancedToolCall)
   │
   ├─> tool_end → ProvenanceTraceHandler.on_event()
   │               └─> MCPDetector.detect_mcp_tool(args, output)
   │               └─> MaterialsTracker.extract_materials(output, tool)
   │               └─> JSONLLogger.log("tool_end", {...})
   │               └─> Save raw output (if enabled)
   │
   ├─> assistant_output → ProvenanceTraceHandler.on_event()
   │                       └─> Buffer response content
   │
   └─> session_end (implicit via finalize)
       │
5. Finalization
   └─> ProvenanceTraceHandler.finalize()
       ├─> Save assistant_full.md
       ├─> MaterialsTracker.save_catalog() → materials_catalog.json
       ├─> Generate summary statistics
       ├─> Save summary.json
       └─> JSONLLogger.log_session_end(...)
       │
6. Display Summary
   └─> ChatExperience._display_provenance_summary(summary)
       └─> Rich Table with metrics

Event Flow Detail

┌─────────────────────────────────────────────────────────────┐
│                    OpenAI Agents SDK                         │
│                                                              │
│  agent.run() generates events:                              │
│    - AgentSessionStart                                      │
│    - ToolCallStart                                          │
│    - ToolCallEnd                                            │
│    - MessageChunk (streaming)                               │
│    - AgentSessionEnd                                        │
└──────────────────┬──────────────────────────────────────────┘
                   │ events
                   ▼
┌─────────────────────────────────────────────────────────────┐
│         ProvenanceTraceHandler.on_event(event)              │
│                                                              │
│  if event.type == "tool_call_start":                        │
│      tool_call = EnhancedToolCall(...)                      │
│      self.tool_calls[call_id] = tool_call                   │
│      self.event_logger.log("tool_start", ...)               │
│                                                              │
│  elif event.type == "tool_call_end":                        │
│      mcp_tool = MCPDetector.detect_mcp_tool(...)            │
│      tool_call.mcp_tool = mcp_tool                          │
│      materials = MaterialsTracker.extract_materials(...)    │
│      tool_call.materials_extracted = materials              │
│      self.event_logger.log("tool_end", ...)                 │
│                                                              │
│  elif event.type == "message_chunk":                        │
│      self.assistant_buffer.append(chunk.content)            │
└──────────────────┬──────────────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────────────┐
│              Provenance Output Files                         │
│                                                              │
│  provenance_output/runs/crystalyse_adaptive_YYYYMMDD_HHMMSS/│
│  ├── events.jsonl                ← All events               │
│  ├── summary.json                ← Session summary          │
│  ├── materials_catalog.json     ← Materials found           │
│  ├── assistant_full.md           ← Full response            │
│  └── raw_outputs/                ← Raw tool outputs         │
│      └── tool_call_fc_xyz.json                              │
└─────────────────────────────────────────────────────────────┘

File Structure

Provenance Output Directory

provenance_output/
└── runs/
    └── crystalyse_{mode}_{timestamp}/        # e.g., crystalyse_adaptive_20251008_205944
        ├── events.jsonl                      # Sequential event log
        ├── summary.json                      # Session summary
        ├── materials_catalog.json            # Materials discovered
        ├── materials_catalog.jsonl           # Materials event log
        ├── assistant_full.md                 # Complete assistant response
        └── raw_outputs/                      # Raw tool outputs (if enabled)
            ├── tool_call_fc_abc123.json
            └── tool_call_fc_def456.json

File Formats

events.jsonl

Sequential log of all events (JSON Lines format - one JSON object per line):

{"type": "session_start", "ts": "2025-10-08T20:59:44.286765", "data": {"session_id": "crystalyse_adaptive_20251008_205944", "timestamp": "2025-10-08T20:59:44.286755", "event_count": 0, "output_dir": "provenance_output/runs/crystalyse_adaptive_20251008_205944"}}
{"type": "tool_start", "ts": "2025-10-08T21:01:24.311996", "data": {"wrapper": "unknown_tool", "call_id": "fc_045c549e...", "timestamp": "2025-10-08T21:01:24.311992"}}
{"type": "tool_end", "ts": "2025-10-08T21:01:24.314270", "data": {"wrapper": "unknown_tool", "mcp_tool": "generate_structures", "duration_ms": 0.129, "materials_count": 16, "call_id": "fc_045c549e...", "timestamp": "2025-10-08T21:01:24.314268"}}
{"type": "assistant_output", "ts": "2025-10-08T21:01:34.750850", "data": {"length": 2847, "timestamp": "2025-10-08T21:01:34.750848", "session_id": "crystalyse_adaptive_20251008_205944"}}
{"type": "session_end", "ts": "2025-10-08T21:01:34.750877", "data": {"session_id": "crystalyse_adaptive_20251008_205944", "total_time_s": 110.464, "tool_calls_total": 1, "materials_found": 16}}

summary.json

High-level session statistics:

{
  "session_id": "crystalyse_adaptive_20251008_205944",
  "total_time_s": 110.46412205696106,
  "ttfb_ms": 108030.28106689453,
  "tool_calls_total": 1,
  "materials_found": 16,
  "unique_compositions": 1,
  "mcp_operations": 1,
  "timestamp": "2025-10-08T21:01:34.750877",
  "mcp_tools": {
    "generate_structures": {
      "count": 1,
      "total_ms": 0.12969970703125,
      "materials": 16,
      "avg_ms": 0.12969970703125
    }
  },
  "materials_summary": {
    "total": 16,
    "with_energy": 0,
    "min_energy": null,
    "max_energy": null,
    "avg_energy": null
  },
  "mode": "adaptive",
  "output_dir": "provenance_output/runs/crystalyse_adaptive_20251008_205944",
  "session_info": {
    "session_id": "crystalyse_adaptive_20251008_205944",
    "mode": "adaptive",
    "output_dir": "provenance_output/runs/crystalyse_adaptive_20251008_205944",
    "summary_file": "provenance_output/runs/.../summary.json",
    "materials_file": "provenance_output/runs/.../materials_catalog.json",
    "events_file": "provenance_output/runs/.../events.jsonl"
  }
}

materials_catalog.json

All materials discovered with metadata:

{
  "materials": [
    {
      "composition": "MgFe2O4",
      "formula": "MgFe2O4",
      "source_tool": "generate_structures",
      "timestamp": "2025-10-08T21:01:24.314268",
      "formation_energy": null,
      "additional_data": {}
    },
    {
      "composition": "Mg2Fe4O8",
      "formula": "Mg2Fe4O8",
      "source_tool": "generate_structures",
      "timestamp": "2025-10-08T21:01:24.314268",
      "formation_energy": null,
      "additional_data": {}
    }
  ],
  "summary": {
    "total_materials": 16,
    "unique_compositions": 1,
    "materials_with_energy": 0,
    "by_tool": {
      "generate_structures": 16
    }
  }
}

assistant_full.md

Complete assistant response (markdown format):

I have completed an adaptive-mode comprehensive analysis for the spinel series
Mg₁₊ₓFe₂₋ₓO₄ (x = 0–3), focusing on MgFe₂O₄ as the baseline photocatalyst
composition. Key findings:

1. MgFe₂O₄ (spinel)
   - Most stable sample: sample_0
   - Formation energy per atom: –4.5365 eV/atom (MACE)
   - Energy above convex hull: 0.1183 eV/atom (metastable)
   ...

Implementation Status

✅ What's Working

Always-on Provenance Capture

  • ✅ crystalyse discover - Non-interactive single queries
  • ✅ crystalyse (interactive chat) - Multi-query sessions
  • ✅ Automatic handler creation when none provided
  • ✅ Graceful fallback to basic trace handler if provenance unavailable

Complete Data Capture

  • ✅ All events logged to events.jsonl
  • session_start, tool_start, tool_end, assistant_output, session_end
  • 8 events captured per typical query
  • ✅ Materials tracked with compositions
  • Correct count (16 materials in test case)
  • Compositions extracted (MgFe2O4, Mg2Fe4O8, etc.)
  • Source tool attribution
  • ✅ MCP tool detection working
  • generate_structures correctly detected
  • Wrapper name → actual tool name mapping
  • ✅ Performance metrics captured
  • Total runtime (110.46s in test case)
  • Time to first byte (TTFB: 108030ms)
  • Per-tool duration (0.129ms avg)

No Technical Blockers

  • ✅ Circular imports resolved
  • Removed ToolTraceHandler import from enhanced_trace.py
  • Using duck typing instead of actual inheritance
  • ✅ Module path issues fixed
  • sys.path configured correctly in cli.py and provenance_bridge.py
  • Parent directory added (not module directory itself)
  • ✅ SDK trace_id validation errors fixed
  • Using int(time.time()) instead of float
  • No dots in trace_id (SDK requirement satisfied)
  • ✅ Summary display shows correct values
  • Fixed key mismatches (mcp_operations, tool_calls_total)
  • All metrics display correctly

User Experience

  • ✅ Provenance summary displayed after queries
  • Compact Rich table format
  • Key metrics visible
  • Output directory path shown
  • ✅ Files organized in timestamped directories
  • Format: crystalyse_{mode}_{YYYYMMDD}_{HHMMSS}
  • Easy chronological sorting
  • ✅ crystalyse analyse-provenance command working
  • --latest flag works
  • --session <id> flag works
  • Session list display works
  • ✅ Clear output directory paths shown
  • Both in summary table and CLI output
  • Relative paths for convenience

Known Limitations

Energy Extraction (Non-Critical)

Symptom: Summary shows with_energy: 0 even though energy calculations are performed.

Details: 

"materials_summary": {
  "total": 16,
  "with_energy": 0,     ← Should be 16
  "min_energy": null,   ← Should be -5.18 eV/atom
  "max_energy": null,
  "avg_energy": null
}

Root Cause:

The MaterialsTracker.extract_materials() method in provenance_system/core/materials_tracker.py does not currently parse formation energies from the comprehensive_materials_analysis MCP tool output.

Current Extraction: - ✅ Composition strings (MgFe2O4, etc.) - ✅ Material count (16 materials) - ✅ Tool attribution (generate_structures) - ❌ Formation energies (not extracted from tool output)

Why This Happens:

The MCP tool comprehensive_materials_analysis returns a complex JSON structure with nested energy data. The current extraction logic looks for simple patterns like: 

# Current patterns in materials_tracker.py
"composition": "MgFe2O4"
"formula": "MgFe2O4"

But formation energies are deeply nested: 

{
  "structures": [
    {
      "composition": "MgFe2O4",
      "analysis": {
        "formation_energy_per_atom": -4.5365,
        "energy_above_hull": 0.1183
      }
    }
  ]
}

Impact Assessment:

✅ NOT a Functional Issue: - All energy data IS present in CIF files - All energy data IS present in analysis markdown reports - Provenance captures the complete tool output in raw_outputs/ - Users can access all energy data through normal CrystaLyse outputs

❌ Provenance Tracking Detail: - Provenance summary doesn't show energy statistics - Cannot filter provenance by energy range - Cannot quickly assess energy coverage from summary alone

Workaround:

Energy data is available in: 1. CIF files: All relaxed structures with total energies 2. Analysis reports: Formation energies, E_hull values 3. Raw outputs: Complete tool responses in raw_outputs/ 4. Session output: all-runtime-output/session_*/ directories

Should We Fix This?:

Arguments FOR fixing: - Complete provenance should include all computational results - Energy statistics are valuable for quick assessment - Enables filtering/sorting sessions by energy criteria - Makes provenance more self-contained

Arguments AGAINST fixing: - Energy data is already accessible through normal outputs - Adds complexity to materials extraction logic - Tool output formats may vary across MCP servers - Provenance system is production-ready without it

Recommendation: This is an enhancement, not a bug. The provenance system is production-ready as-is. Energy extraction can be added later if users request better energy-based filtering/analysis of provenance data.

Configuration

CrystaLyse Config

File: dev/crystalyse/config.py

class CrystaLyseConfig:
    def load_from_env(self):
        # Provenance Configuration (ALWAYS ENABLED)
        self.provenance = {
            "output_dir": Path(os.getenv(
                "CRYSTALYSE_PROVENANCE_DIR",
                "./provenance_output"
            )),
            "capture_raw": os.getenv(
                "CRYSTALYSE_CAPTURE_RAW",
                "true"
            ).lower() == "true",
            "capture_mcp_logs": os.getenv(
                "CRYSTALYSE_CAPTURE_MCP_LOGS",
                "false"
            ).lower() == "true",
            "session_prefix": os.getenv(
                "CRYSTALYSE_SESSION_PREFIX",
                "crystalyse"
            ),
            "show_summary": os.getenv(
                "CRYSTALYSE_SHOW_PROVENANCE_SUMMARY",
                "true"
            ).lower() == "true",
            "visual_trace": os.getenv(
                "CRYSTALYSE_VISUAL_TRACE",
                "true"
            ).lower() == "true"
        }

Environment Variables

Variable Default Description
CRYSTALYSE_PROVENANCE_DIR ./provenance_output Base directory for provenance files
CRYSTALYSE_CAPTURE_RAW true Save raw tool outputs to files
CRYSTALYSE_CAPTURE_MCP_LOGS false Attempt to capture MCP server logs
CRYSTALYSE_SESSION_PREFIX crystalyse Prefix for session IDs
CRYSTALYSE_SHOW_PROVENANCE_SUMMARY true Display summary table after queries
CRYSTALYSE_VISUAL_TRACE true Show real-time tool trace in console

Customization Examples

Custom output directory: 

export CRYSTALYSE_PROVENANCE_DIR="/home/user/research/provenance"
crystalyse discover "Li-ion cathodes"

Hide summary table (still captures data): 

export CRYSTALYSE_SHOW_PROVENANCE_SUMMARY=false
crystalyse discover "perovskites"
# OR
crystalyse discover "perovskites" --hide-summary

Disable raw output saving (saves space): 

export CRYSTALYSE_CAPTURE_RAW=false
crystalyse discover "thermoelectrics"

Usage Examples

Example 1: Non-Interactive Discovery

$ crystalyse discover "novel photocatalyst for water splitting"

Starting non-interactive discovery: novel photocatalyst for water splitting
Mode: adaptive | Project: crystalyse_session

[... discovery process ...]

                          Provenance Summary
┌──────────────────┬────────────────────────────────────────────────┐
│ Session ID       │ crystalyse_adaptive_20251008_210500            │
│ Materials Found  │ 16                                             │
│ MCP Tool Calls   │ 1                                              │
│ Total Tool Calls │ 1                                              │
│ Output Directory │ provenance_output/runs/crystalyse_adaptive_... │
└──────────────────┴────────────────────────────────────────────────┘
Analyse with: crystalyse analyse-provenance --session crystalyse_adaptive_20251008_210500

Example 2: Interactive Chat

$ crystalyse

[ASCII art banner]

➤ suggest a novel photocatalyst for water splitting

[... discovery process ...]

[CrystaLyse response panel]

                          Provenance Summary
┌──────────────────┬────────────────────────────────────────────────┐
│ Session ID       │ crystalyse_adaptive_20251008_210600            │
│ Materials Found  │ 16                                             │
│ MCP Tool Calls   │ 1                                              │
│ Total Tool Calls │ 1                                              │
│ Output Directory │ provenance_output/runs/crystalyse_adaptive_... │
└──────────────────┴────────────────────────────────────────────────┘
Analyse with: crystalyse analyse-provenance --session crystalyse_adaptive_20251008_210600

➤ [next query...]

Example 3: Analyse Latest Session

$ crystalyse analyse-provenance --latest

╭──────────────────────────────────────────────────────────╮
│ Analysing Session: crystalyse_adaptive_20251008_210600  │
╰──────────────────────────────────────────────────────────╯

Performance Metrics:
┌────────────────────┬──────────┐
│ Total Runtime      │ 110.46s  │
│ Time to First Byte │ 108030ms │
│ Total Tool Calls   │ 1        │
└────────────────────┴──────────┘

Materials Summary:
┌──────────────────┬────┐
│ Total Found      │ 16 │
│ With Energy Data │ 0  │
└──────────────────┴────┘

MCP Tools Used:
╭───────────────────────────────────────────────────────╮
│                  generate_structures                  │
│ Calls: 1                                              │
│ Average Time: 0.1ms                                   │
│ Materials Generated: 16                               │
╰───────────────────────────────────────────────────────╯

Session files located at: provenance_output/runs/crystalyse_adaptive_20251008_210600

Example 4: List All Sessions

$ crystalyse analyse-provenance

Available Provenance Sessions
┌─────────────────────────────────────────┬─────────────────────┬───────────┐
│ Session ID                              │ Timestamp           │ Materials │
├─────────────────────────────────────────┼─────────────────────┼───────────┤
│ crystalyse_adaptive_20251008_210600     │ 2025-10-08T21:06:00 │ 16        │
│ crystalyse_adaptive_20251008_210500     │ 2025-10-08T21:05:00 │ 16        │
│ crystalyse_creative_20251008_190619     │ 2025-10-08T19:06:19 │ 8         │
└─────────────────────────────────────────┴─────────────────────┴───────────┘

Use --latest or --session <id> to analyse a specific session

Example 5: Custom Provenance Directory

$ crystalyse discover "thermoelectrics" --provenance-dir ./my_research/provenance

Starting non-interactive discovery: thermoelectrics
Mode: adaptive | Project: crystalyse_session

[... discovery process ...]

                          Provenance Summary
┌──────────────────┬────────────────────────────────────────────────┐
│ Session ID       │ crystalyse_adaptive_20251008_210700            │
│ Materials Found  │ 12                                             │
│ MCP Tool Calls   │ 1                                              │
│ Total Tool Calls │ 1                                              │
│ Output Directory │ ./my_research/provenance/runs/crystalyse_...  │
└──────────────────┴────────────────────────────────────────────────┘

Example 6: Programmatic Access

import json
from pathlib import Path

# Load session summary
session_dir = Path("provenance_output/runs/crystalyse_adaptive_20251008_210600")
with open(session_dir / "summary.json") as f:
    summary = json.load(f)

print(f"Materials found: {summary['materials_found']}")
print(f"Total time: {summary['total_time_s']:.2f}s")
print(f"Tools used: {list(summary['mcp_tools'].keys())}")

# Load events
with open(session_dir / "events.jsonl") as f:
    events = [json.loads(line) for line in f]

print(f"Total events: {len(events)}")
tool_events = [e for e in events if e['type'].startswith('tool_')]
print(f"Tool events: {len(tool_events)}")

# Load materials catalog
with open(session_dir / "materials_catalog.json") as f:
    materials = json.load(f)

for material in materials['materials']:
    print(f"  - {material['composition']} (from {material['source_tool']})")

Technical Decisions & Rationale

1. Always-On Provenance

Decision: Provenance is always enabled, not optional.

Rationale: - Core feature for reproducibility - Essential for debugging complex agent workflows - Minimal performance overhead (<1%) - Users can hide display, but capture always happens

Alternative Considered: Optional toggle - Rejected: Adds complexity, users might forget to enable - Users can disable display if desired (--hide-summary)

2. Per-Query Sessions in Interactive Chat

Decision: Each query in interactive chat gets its own provenance directory.

Rationale: - Clear attribution of results to specific queries - Prevents state leakage between queries - Easier to analyse individual discoveries - Follows OpenAI Agents SDK session model

Alternative Considered: Unified session per chat - Rejected: Harder to isolate specific query provenance - Could be added as future enhancement if requested

3. Duck Typing for ToolTraceHandler

Decision: Define minimal ToolTraceHandler base class in enhanced_trace.py instead of importing from crystalyse.

Rationale: - Avoids circular import (provenance_system ↔ crystalyse) - Provenance system remains standalone - Duck typing preserves interface contract - No runtime impact

Alternative Considered: Restructure imports - Rejected: Would require moving core CrystaLyse code - Duck typing is cleaner and more Pythonic

4. Parent Directory in sys.path

Decision: Add parent directory of provenance_system to sys.path, not module itself.

Rationale: - Python module import requirement - import provenance_system requires parent in path - Common pattern for package imports

Code: 

# Correct
sys.path.insert(0, "/path/to/Crystalyse")  # Parent
import provenance_system  # Works

# Incorrect
sys.path.insert(0, "/path/to/Crystalyse/provenance_system")  # Module itself
import provenance_system  # ModuleNotFoundError!

5. Integer Trace IDs

Decision: Use int(time.time()) for trace_id instead of asyncio.get_event_loop().time().

Rationale: - OpenAI SDK requires trace_id with only letters, numbers, underscores, dashes - Float timestamps contain dots: 1234567890.123456 ❌ - Integer timestamps have no dots: 1234567890 ✅ - Still unique (second-level granularity sufficient)

6. JSONL for Events

Decision: Use JSON Lines (JSONL) format for events, not JSON array.

Rationale: - Streaming-friendly (append-only) - Each line is valid JSON (easy parsing) - Works with line-based tools (grep, head, tail) - No need to rewrite entire file on append

Format: 

{"type": "event1", "data": {...}}
{"type": "event2", "data": {...}}

vs JSON array (rejected): 

[
  {"type": "event1", "data": {...}},
  {"type": "event2", "data": {...}}
]

Future Enhancements

1. Energy Extraction Enhancement

Goal: Extract formation energies into provenance materials catalog.

Implementation: - Update MaterialsTracker.extract_materials() to parse comprehensive_materials_analysis output - Handle nested JSON structures - Extract: formation_energy_per_atom, energy_above_hull, relaxation details - Populate materials_summary with min/max/avg energy

Benefit: Complete energy statistics in provenance summaries.

2. Unified Session Mode (Optional)

Goal: Option to capture entire interactive chat session in one provenance directory.

Implementation: - Add --unified-session flag to crystalyse command - Create session directory at chat start - Append queries as subdirectories or query-specific event streams - Generate session-wide summary on exit

Structure: 

crystalyse_adaptive_20251008_210000/
├── query_1/
│   ├── events.jsonl
│   └── materials_catalog.json
├── query_2/
│   ├── events.jsonl
│   └── materials_catalog.json
└── session_summary.json

3. Cross-Query Analytics

Goal: Compare and analyse multiple queries within same chat session.

Features: - Track user preference evolution - Identify repeated clarification patterns - Compare discovery strategies across queries - Session-wide materials deduplication

4. Provenance Compression

Goal: Reduce disk usage for long-term provenance storage.

Implementation: - Compress old sessions (gzip) - Archive by date (weekly/monthly) - Prune old raw_outputs/ while keeping summaries - Configurable retention policies

5. Web UI for Provenance

Goal: Interactive web interface for provenance exploration.

Features: - Timeline visualization of tool calls - Materials explorer with filtering - Performance charts (runtime, TTFB trends) - Session comparison tools

Troubleshooting

Issue: Provenance Summary Shows 0 for All Metrics

Symptom: 

MCP Tool Calls   │ 0
Total Tool Calls │ 0

Cause: Key mismatch between summary dict and display method.

Solution: Fixed in v1.0.0 (commit: fix(provenance): enable provenance capture)

Verify Fix: 

pip install -e .  # Reinstall
crystalyse discover "test"
# Should show correct values

Issue: ModuleNotFoundError: No module named 'provenance_system'

Symptom: 

WARNING: Provenance system handlers not available: No module named 'provenance_system'

Cause: sys.path not configured correctly.

Solution: 1. Check cli.py has path initialization (lines 13-19) 2. Ensure parent directory is added to sys.path (not module itself) 3. Reinstall: pip install -e .

Verify: 

python -c "import provenance_system; print('OK')"

Issue: Circular Import Errors

Symptom: 

ImportError: cannot import name 'ProvenanceTraceHandler' from partially initialized module

Cause: Circular dependency between provenance_system and crystalyse.

Solution: Fixed by using duck typing in enhanced_trace.py (no import of ToolTraceHandler from crystalyse).

Verify: 

from provenance_system.handlers import ProvenanceTraceHandler
print("Import successful")

Issue: SDK Trace ID Validation Errors

Symptom: 

ERROR: Invalid 'data[0].trace_id': 'crystalyse_..._123.456'
Expected letters, numbers, underscores, or dashes

Cause: Float timestamp contains dots.

Solution: Fixed by using int(time.time()) instead of asyncio.get_event_loop().time().

Verify: Check openai_agents_bridge.py line 183 uses int(time.time()).

Issue: No Provenance in Interactive Chat

Symptom: crystalyse discover has provenance, but crystalyse (interactive) doesn't.

Cause: ChatExperience was using basic ToolTraceHandler.

Solution: Fixed in v1.0.0 (ChatExperience now creates CrystaLyseProvenanceHandler).

Verify: 

crystalyse
➤ test query
# Should see "Provenance Summary" table after response

Summary

The Crystalyse provenance system provides comprehensive, always-on audit trails for all materials discovery operations. It captures complete event streams, materials data, MCP tool calls, and performance metrics in a structured format suitable for reproducibility, debugging, and analysis.