# dtctl Quick Start Guide

This guide provides practical examples for using dtctl to manage your Dynatrace environment. It covers configuration, common workflows, and all resource types with hands-on examples.

> **Note**: This guide assumes dtctl is already installed. If you need to build or install dtctl, see [INSTALLATION.md](INSTALLATION.md) first.

## Table of Contents

1. [Configuration](#configuration)
2. [Workflows](#workflows)
3. [Dashboards & Notebooks](#dashboards--notebooks)
4. [DQL Queries](#dql-queries)
5. [Service Level Objectives (SLOs)](#service-level-objectives-slos)
6. [Notifications](#notifications)
7. [Grail Buckets](#grail-buckets)
8. [Lookup Tables](#lookup-tables)
9. [OpenPipeline](#openpipeline)
10. [Settings API](#settings-api)
11. [App Engine](#app-engine)
    - [List and View Apps](#list-and-view-apps)
    - [App Functions](#app-functions)
    - [App Intents](#app-intents)
12. [EdgeConnect](#edgeconnect)
13. [Davis AI](#davis-ai)
14. [Live Debugger](#live-debugger)
15. [Extensions 2.0](#extensions-20)
16. [Output Formats](#output-formats)
17. [AWS Monitoring](#aws-monitoring)
18. [Azure Monitoring](#azure-monitoring)
19. [GCP Monitoring (Preview)](#gcp-monitoring-preview)
20. [AI Agent Skills](#ai-agent-skills)
21. [Tips & Tricks](#tips--tricks)
22. [Troubleshooting](#troubleshooting)

---

## Configuration

### Initial Setup

Set up your first Dynatrace environment:

#### Option 1: OAuth Login (Recommended)

The easiest way to authenticate — uses your Dynatrace SSO credentials, no token management needed:

```bash
dtctl auth login --context my-env --environment "https://abc12345.apps.dynatrace.com"
# Opens your browser for Dynatrace SSO login
# Tokens are stored securely and refreshed automatically

# Verify your configuration
dtctl doctor
```

To log out:
```bash
dtctl auth logout
```

#### Option 2: Token-based Authentication

If you prefer API tokens (e.g. for CI/CD or headless environments):

```bash
# Create a context with your environment details
dtctl config set-context my-env \
  --environment "https://abc12345.apps.dynatrace.com" \
  --token-ref my-token

# Store your platform token securely
dtctl config set-credentials my-token \
  --token "dt0s16.XXXXXXXXXXXXXXXXXXXXXXXX"

# Verify your configuration
dtctl config view
```

**Creating a Platform Token:**

To create a platform token in Dynatrace:
1. Go to [https://myaccount.dynatrace.com/platformTokens](https://myaccount.dynatrace.com/platformTokens) (Account Management > **My platform tokens**)
2. Select **Platform token** and give it a descriptive name (e.g., "dtctl-token")
3. Choose expiration, account, and environments
4. Add the required scopes based on what you'll manage (see [Token Scopes](TOKEN_SCOPES.md))
5. Select **Generate** and copy the token immediately - it's only shown once!

For detailed instructions, see [Dynatrace Platform Tokens documentation](https://docs.dynatrace.com/docs/manage/identity-access-management/access-tokens-and-oauth-clients/platform-tokens).

**Required Token Scopes**: See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for a complete list of scopes for each safety level and resource type. You can copy-paste scope lists directly from that document.

### Multiple Environments

Manage multiple Dynatrace environments easily:

```bash
# Set up dev environment with unrestricted access
dtctl config set-context dev \
  --environment "https://dev.apps.dynatrace.com" \
  --token-ref dev-token \
  --safety-level dangerously-unrestricted \
  --description "Development sandbox"

dtctl config set-credentials dev-token \
  --token "dt0s16.DEV_TOKEN_HERE"

# Set up prod environment with read-only safety
dtctl config set-context prod \
  --environment "https://prod.apps.dynatrace.com" \
  --token-ref prod-token \
  --safety-level readonly \
  --description "Production - read only"

dtctl config set-credentials prod-token \
  --token "dt0s16.PROD_TOKEN_HERE"

# List all contexts (shows safety levels)
dtctl config get-contexts

# Switch between environments
dtctl config use-context dev
dtctl config use-context prod

# Or use the ctx shortcut:
dtctl ctx                    # List contexts
dtctl ctx dev                # Switch to dev
dtctl ctx prod               # Switch to prod

# Check current context
dtctl config current-context

# Delete a context you no longer need
dtctl config delete-context old-env
```

### One-Time Context Override

Use a different context without switching:

```bash
# Execute a command in prod while dev is active
dtctl get workflows --context prod
```

### Per-Project Configuration

dtctl supports per-project configuration files for team collaboration and CI/CD workflows.

#### Creating Project Config

Use `dtctl config init` to generate a `.dtctl.yaml` template:

```bash
# Create .dtctl.yaml in current directory
dtctl config init

# Create with custom context name
dtctl config init --context staging

# Overwrite existing file
dtctl config init --force
```

This generates a template with environment variable placeholders:

```yaml
# .dtctl.yaml - per-project configuration
apiVersion: dtctl.io/v1
kind: Config
current-context: production
contexts:
  - name: production
    context:
      environment: ${DT_ENVIRONMENT_URL}
      token-ref: my-token
      safety-level: readwrite-all
      description: Project environment
tokens:
  - name: my-token
    token: ${DT_API_TOKEN}
preferences:
  output: table
```

#### Environment Variable Expansion

Config files support `${VAR_NAME}` syntax for environment variables:

```yaml
contexts:
  - name: ci
    context:
      environment: ${DT_ENVIRONMENT_URL}  # Expanded from env var
      token-ref: ci-token
tokens:
  - name: ci-token
    token: ${DT_API_TOKEN}  # Expanded from env var
```

This allows teams to commit `.dtctl.yaml` files to repositories **without secrets**, while each developer or CI system provides tokens via environment variables.

**Search Order:**
1. `--config` flag (explicit path)
2. `.dtctl.yaml` in current directory or any parent directory (walks up to root)
3. Global config (`~/.config/dtctl/config`)

#### Usage Examples

```bash
# In a project directory with .dtctl.yaml
cd my-project/
export DT_ENVIRONMENT_URL="https://abc12345.apps.dynatrace.com"
export DT_API_TOKEN="dt0c01.xxx"
dtctl get workflows  # Uses .dtctl.yaml with expanded env vars

# Override with global config
dtctl --config ~/.config/dtctl/config get workflows
```

### Safety Levels

Safety levels provide **client-side** protection against accidental destructive operations:

| Level | Description |
|-------|-------------|
| `readonly` | No modifications allowed |
| `readwrite-mine` | Modify own resources only |
| `readwrite-all` | Modify all resources (default) |
| `dangerously-unrestricted` | All operations including bucket deletion |

```bash
# Set safety level when creating a context
dtctl config set-context prod \
  --environment "https://prod.apps.dynatrace.com" \
  --token-ref prod-token \
  --safety-level readonly

# View context details including safety level
dtctl config describe-context prod
```

> **Important**: Safety levels are client-side only. For actual security, configure your API tokens with minimum required scopes. See [Token Scopes](TOKEN_SCOPES.md) for scope requirements and [Context Safety Levels](dev/context-safety-levels.md) for details.

### Current User Identity

View information about the currently authenticated user:

```bash
# View current user info
dtctl auth whoami

# Output:
# User ID:     621321d-1231-dsad-652321829b50
# User Name:   John Doe
# Email:       john.doe@example.com
# Context:     prod
# Environment: https://abc12345.apps.dynatrace.com

# Get just the user ID (useful for scripting)
dtctl auth whoami --id-only

# Output as JSON
dtctl auth whoami -o json
```

**Note:** The `whoami` command requires the `app-engine:apps:run` scope for full user details. If that scope is unavailable, it falls back to extracting the user ID from the JWT token.

### Token Health

Check whether your OAuth session is still valid and when it expires:

```bash
dtctl auth status

# Output:
# Context:       prod
# Environment:   https://abc12345.apps.dynatrace.com
# Auth type:     OAuth
# Storage:       macOS Keychain
# Access token:  valid for 42m10s (expires 2026-04-17T14:32:00Z)
# Refresh token: present

# Output as JSON (includes full expiry timestamps)
dtctl auth status -o json
```

This command reads from local storage only — it makes no network call. Useful for confirming that automatic token refresh is in place before running long operations.

### Command Aliases

dtctl supports custom command aliases to create shortcuts for frequently used commands. Aliases can be simple text replacements, parameterized templates, or shell commands.

#### Simple Aliases

Create shortcuts for common commands:

```bash
# Create a simple alias
dtctl alias set wf "get workflows"

# Use the alias
dtctl wf
# Expands to: dtctl get workflows

# List all aliases
dtctl alias list

# Delete an alias
dtctl alias delete wf
```

#### Parameterized Aliases

Use positional parameters `$1-$9` for reusable command templates:

```bash
# Create an alias that takes a parameter
dtctl alias set logs-errors "query 'fetch logs | filter status=\$1 | limit 100'"

# Use with parameter
dtctl logs-errors ERROR
# Expands to: dtctl query 'fetch logs | filter status=ERROR | limit 100'

# Multiple parameters
dtctl alias set query-host "query 'fetch logs | filter host=\$1 | limit \$2'"
dtctl query-host server-01 50
# Expands to: dtctl query 'fetch logs | filter host=server-01 | limit 50'
```

#### Shell Aliases

Prefix aliases with `!` to execute them through the system shell, enabling pipes, redirection, and external tools:

```bash
# Create a shell alias with jq for JSON processing
dtctl alias set wf-names "!dtctl get workflows -o json | jq -r '.workflows[].title'"

# Use the shell alias
dtctl wf-names
# Executes through shell: dtctl get workflows -o json | jq -r '.workflows[].title'

# Shell alias with grep
dtctl alias set errors "!dtctl query 'fetch logs' -o json | grep -i error"
dtctl errors
```

#### Import and Export Aliases

Share aliases with your team by exporting and importing them:

```bash
# Export all aliases to a file
dtctl alias export -f team-aliases.yaml

# Import aliases from a file
dtctl alias import -f team-aliases.yaml

# Merge imported aliases (skip conflicts)
dtctl alias import -f team-aliases.yaml --no-overwrite
```

**Example alias file** (`team-aliases.yaml`):

```yaml
wf: get workflows
wfe: get workflow-executions
logs-error: query 'fetch logs | filter status=ERROR | limit 100'
top-errors: "!dtctl query 'fetch logs | filter status=ERROR' -o json | jq -r '.records[] | .message' | sort | uniq -c | sort -rn | head -10"
```

#### Alias Safety

Aliases cannot shadow built-in commands to prevent confusion:

```bash
# This will fail - 'get' is a built-in command
dtctl alias set get "query 'fetch logs'"
# Error: alias name "get" conflicts with built-in command

# Use a different name instead
dtctl alias set gl "query 'fetch logs'"
```

#### Practical Alias Examples

```bash
# Quick shortcuts for common operations
dtctl alias set w "get workflows"
dtctl alias set d "get dashboards"
dtctl alias set nb "get notebooks"

# Workflow shortcuts
dtctl alias set wf-run "exec workflow \$1 --wait"
dtctl alias set wf-logs "logs workflow-execution \$1 --follow"

# Query templates
dtctl alias set errors "query 'fetch logs | filter status=ERROR | limit \$1'"
dtctl alias set spans-by-trace "query 'fetch spans | filter trace_id=\$1'"

# Shell aliases for complex operations
dtctl alias set workflow-count "!dtctl get workflows -o json | jq '.workflows | length'"
dtctl alias set top-users "!dtctl query 'fetch logs' -o json | jq -r '.records[].user' | sort | uniq -c | sort -rn | head -10"

# Import team-shared aliases
dtctl alias import -f ~/.dtctl-team-aliases.yaml
```

---

## Workflows

Workflows automate tasks and integrate with Dynatrace monitoring.

### List and View Workflows

```bash
# List all workflows
dtctl get workflows

# List in table format with more details
dtctl get workflows -o wide

# Get a specific workflow by ID
dtctl get workflow workflow-123

# View detailed information
dtctl describe workflow workflow-123

# Describe by name (with fuzzy matching)
dtctl describe workflow "My Workflow"

# Output as JSON for processing
dtctl get workflow workflow-123 -o json
```

### Edit Workflows

Edit workflows directly in your preferred editor:

```bash
# Edit in YAML format (default)
dtctl edit workflow workflow-123

# Edit by name
dtctl edit workflow "My Workflow"

# Edit in JSON format
dtctl edit workflow workflow-123 --format=json

# Set your preferred editor
export EDITOR=vim
# or
dtctl config set preferences.editor vim
```

### Create Workflows

Create new workflows from YAML or JSON files:

```bash
# Create from a file
dtctl create workflow -f my-workflow.yaml

# Apply (create or update if exists)
dtctl apply -f my-workflow.yaml

# First apply: stamp the generated ID back into the file for idempotent future applies
dtctl apply -f my-workflow.yaml --write-id

# Forgot --write-id on the first run? Recover without creating another duplicate:
dtctl apply -f my-workflow.yaml --write-id --id <id-from-first-run>

# CI/scripting: apply a template file to a specific known workflow
dtctl apply -f my-workflow.yaml --id $WORKFLOW_ID
```

**Example workflow file** (`my-workflow.yaml`):

```yaml
title: Daily Health Check
description: Runs a health check every day at 9 AM
trigger:
  schedule:
    rule: "0 9 * * *"
    timezone: "UTC"
tasks:
  check_errors:
    action: dynatrace.automations:run-javascript
    input:
      script: |
        export default async function () {
          console.log("Running health check...");
          return { status: "ok" };
        }
```

### Execute Workflows

Run workflows on-demand:

```bash
# Execute a workflow
dtctl exec workflow workflow-123

# Execute with parameters
dtctl exec workflow workflow-123 \
  --params environment=production \
  --params severity=high

# Execute and wait for completion
dtctl exec workflow workflow-123 --wait

# Execute with custom timeout
dtctl exec workflow workflow-123 --wait --timeout 10m

# Execute, wait, and print each task's return value when done
dtctl exec workflow workflow-123 --wait --show-results
```

### View Executions

Monitor workflow executions:

```bash
# List all recent executions
dtctl get workflow-executions

# List executions for a specific workflow
dtctl get workflow-executions -w workflow-123

# Get details of a specific execution
dtctl describe workflow-execution exec-456
# or use short alias
dtctl describe wfe exec-456

# View execution logs
dtctl logs workflow-execution exec-456
# or
dtctl logs wfe exec-456

# Stream logs in real-time
dtctl logs wfe exec-456 --follow

# View logs for all tasks
dtctl logs wfe exec-456 --all

# View logs for a specific task
dtctl logs wfe exec-456 --task check_errors
```

### View Task Results

Retrieve the structured return value of a specific task (distinct from log output):

```bash
# Get the return value of a task
dtctl get wfe-task-result exec-456 --task my_task
dtctl get wfe-task-result exec-456 -t my_task

# Output as JSON or YAML
dtctl get wfe-task-result exec-456 --task my_task -o json
dtctl get wfe-task-result exec-456 --task my_task -o yaml
```

### Watch Workflows

Monitor workflows in real-time with watch mode:

```bash
# Watch all workflows for changes
dtctl get workflows --watch

# Watch with custom polling interval (default: 2s)
dtctl get workflows --watch --interval 5s

# Watch specific workflow
dtctl get workflow my-workflow --watch

# Watch only your workflows
dtctl get workflows --mine --watch

# Only show changes (skip initial state)
dtctl get workflows --watch --watch-only
```

**Watch mode features:**
- `+` (green) prefix for newly added workflows
- `~` (yellow) prefix for modified workflows
- `-` (red) prefix for deleted workflows
- Graceful shutdown with Ctrl+C
- Automatic retry on transient errors

### Delete Workflows

```bash
# Delete by ID
dtctl delete workflow workflow-123

# Delete by name (prompts for confirmation)
dtctl delete workflow "Old Workflow"

# Skip confirmation prompt
dtctl delete workflow "Old Workflow" -y
```

### Version History

View and restore previous versions of workflows:

```bash
# View version history
dtctl history workflow workflow-123
dtctl history workflow "My Workflow"

# Output as JSON
dtctl history workflow workflow-123 -o json
```

### Restore Previous Versions

Restore a workflow to a previous version:

```bash
# Restore to a specific version
dtctl restore workflow workflow-123 5

# Restore by name
dtctl restore workflow "My Workflow" 3

# Skip confirmation prompt
dtctl restore workflow "My Workflow" 3 --force
```

---

## Dashboards & Notebooks

Dashboards provide visual monitoring views, while notebooks enable interactive data exploration.

### List and View Documents

```bash
# List all dashboards
dtctl get dashboards

# List all notebooks
dtctl get notebooks

# Filter by name
dtctl get dashboards --name "production"
dtctl get notebooks --name "analysis"

# List only your own dashboards/notebooks
dtctl get dashboards --mine
dtctl get notebooks --mine

# Combine filters
dtctl get dashboards --mine --name "production"

# Sort results (prefix field with '-' for descending)
dtctl get dashboards --sort "name,-modificationInfo.lastModifiedTime"

# Request fields the API omits by default
dtctl get dashboards --add-fields "originExtensionId,labels,shareInfo.isShared"

# List as effective owner (requires document:documents:admin)
dtctl get dashboards --admin-access

# Raw Document API filter — sent verbatim, overrides --name/--mine
dtctl get dashboards --filter "originAppId exists"
dtctl get documents --filter "type in ('dashboard','notebook') and name contains 'report'"

# Get a specific document by ID
dtctl get dashboard dash-123
dtctl get notebook nb-456

# Describe by name
dtctl describe dashboard "Production Overview"
dtctl describe notebook "Weekly Analysis"
```

### Edit Documents

```bash
# Edit a dashboard in YAML (default)
dtctl edit dashboard dash-123

# Edit by name
dtctl edit dashboard "Production Overview"

# Edit in JSON format
dtctl edit notebook nb-456 --format=json
```

### Create and Apply Documents

Both `create` and `apply` work with dashboards and notebooks:

```bash
# Create a new dashboard (always creates new)
dtctl create dashboard -f dashboard.yaml

# Apply a dashboard (creates if new, updates if exists)
dtctl apply -f dashboard.yaml

# First apply: stamp the generated ID back into the file so future applies update in place
dtctl apply -f dashboard.yaml --write-id

# Forgot --write-id? Recover without creating another duplicate:
dtctl apply -f dashboard.yaml --write-id --id <id-from-first-run>

# CI/scripting: apply a template to a specific known dashboard
dtctl apply -f dashboard.yaml --id $DASHBOARD_ID

# Both commands show tile count and URL:
# Dashboard "My Dashboard" (abc-123) created successfully [18 tiles]
# URL: https://env.apps.dynatrace.com/ui/apps/dynatrace.dashboards/dashboard/abc-123
```

**When to use which:**
- **`create`**: Use when you want to create a new resource. Fails if the ID already exists.
- **`apply`**: Use for declarative management. Creates new resources or updates existing ones based on the ID in the file.

Both commands validate the document structure and warn about issues:
```bash
# If structure is wrong, you'll see warnings:
# Warning: dashboard content has no 'tiles' field - dashboard may be empty
```

### Round-Trip Export/Import

Export a dashboard and re-import it (works directly without modifications):

```bash
# Export existing dashboard
dtctl get dashboard abc-123 -o yaml > dashboard.yaml

# Re-apply to same or different environment
dtctl apply -f dashboard.yaml

# dtctl automatically handles the content structure
```

**Example dashboard** (`dashboard.yaml`):

```yaml
type: dashboard
name: Production Monitoring
content:
  tiles:
    - name: Response Time
      tileType: DATA_EXPLORER
      queries:
        - query: "timeseries avg(dt.service.request.response_time)"
```

### Share Documents

Share dashboards and notebooks with users and groups:

```bash
# Share with a user (read access by default)
dtctl share dashboard dash-123 --user user@example.com

# Share with write access
dtctl share dashboard dash-123 \
  --user user@example.com \
  --access read-write

# Share with a group
dtctl share notebook nb-456 --group "Platform Team"

# View sharing information
dtctl describe dashboard dash-123

# Remove user access
dtctl unshare dashboard dash-123 --user user@example.com

# Remove all shares
dtctl unshare dashboard dash-123 --all
```

### Version History (Snapshots)

View and restore previous versions of dashboards and notebooks:

```bash
# View version history
dtctl history dashboard dash-123
dtctl history notebook nb-456

# View history by name
dtctl history dashboard "Production Overview"
dtctl history notebook "Weekly Analysis"

# Output as JSON
dtctl history dashboard dash-123 -o json
```

### Restore Previous Versions

Restore a document to a previous snapshot version:

```bash
# Restore to a specific version
dtctl restore dashboard dash-123 5
dtctl restore notebook nb-456 3

# Restore by name
dtctl restore dashboard "Production Overview" 5

# Skip confirmation prompt
dtctl restore notebook "Weekly Analysis" 3 --force
```

**Notes:**
- Snapshots are created when documents are updated with the `create-snapshot` option
- Maximum 50 snapshots per document (oldest auto-deleted when exceeded)
- Snapshots auto-delete after 30 days
- Only the document owner can restore snapshots
- Restoring automatically creates a snapshot of the current state before restoring

### Watch Documents

Monitor dashboards and notebooks for changes in real-time:

```bash
# Watch all dashboards
dtctl get dashboards --watch

# Watch your own dashboards
dtctl get dashboards --mine --watch

# Watch notebooks with custom interval
dtctl get notebooks --watch --interval 10s

# Watch with name filter
dtctl get dashboards --name "production" --watch
```

### Delete Documents

```bash
# Delete a dashboard (moves to trash)
dtctl delete dashboard dash-123

# Delete by name
dtctl delete notebook "Old Analysis"

# Skip confirmation
dtctl delete dashboard dash-123 -y
```

**Note**: Deleted documents are moved to trash and kept for 30 days before permanent deletion. See [Trash Management](#trash-management) below.

### Trash Management

Deleted dashboards and notebooks are moved to trash and kept for 30 days before permanent deletion. You can list, view, restore, or permanently delete items in trash.

#### List Trash

```bash
# List all trashed documents
dtctl get trash

# List only trashed dashboards
dtctl get trash --type dashboard

# List only trashed notebooks
dtctl get trash --type notebook

# List only documents you deleted
dtctl get trash --mine

# Filter by deletion date
dtctl get trash --deleted-after 2024-01-01
dtctl get trash --deleted-before 2024-12-31

# Output in different formats
dtctl get trash -o json
dtctl get trash -o yaml
```

**Example output:**
```
ID                                    TYPE        NAME                DELETED BY    DELETED AT           EXPIRES IN
abc123-def456-ghi789-jkl012-mno345    dashboard   Prod Overview       john.doe      2024-01-15 10:30:00  29 days
xyz987-uvw654-rst321-opq098-lmn765    notebook    Debug Session       jane.smith    2024-01-20 14:45:00  24 days
```

#### View Trash Details

```bash
# Get detailed information about a trashed document
dtctl describe trash abc-123

# Shows: ID, name, type, owner, deleted by, deletion date, expiration date, size, tags, etc.
```

#### Restore from Trash

```bash
# Restore a single document
dtctl restore trash abc-123

# Restore multiple documents
dtctl restore trash abc-123 def-456 ghi-789

# Restore with a new name (to avoid conflicts)
dtctl restore trash abc-123 --new-name "Recovered Dashboard"

# Force restore (overwrite if name conflict exists)
dtctl restore trash abc-123 --force
```

#### Permanently Delete from Trash

**WARNING**: Permanent deletion cannot be undone!

```bash
# Permanently delete a single document
dtctl delete trash abc-123 --permanent

# Permanently delete multiple documents
dtctl delete trash abc-123 def-456 --permanent -y

# The --permanent flag is required to prevent accidental deletion
```

**Notes:**
- Documents remain in trash for **30 days** before automatic permanent deletion
- You can only restore documents that haven't expired yet
- Trash operations require appropriate permissions (document owner or admin)
- Use `--deleted-by` flag to filter by who deleted the documents

---

## DQL Queries

Execute Dynatrace Query Language (DQL) queries to fetch logs, metrics, events, and more.

### Simple Queries

```bash
# Execute an inline query
dtctl query "fetch logs | limit 10"

# Filter logs by status
dtctl query "fetch logs | filter status='ERROR' | limit 100"

# Query recent events
dtctl query "fetch events | filter event.type='CUSTOM_ALERT' | limit 50"

# Summarize data
dtctl query "fetch logs | summarize count(), by: {status} | sort count desc"
```

### File-Based Queries

Store complex queries in files for reusability:

```bash
# Execute from file
dtctl query -f queries/errors.dql

# Save output to file
dtctl query -f queries/errors.dql -o json > results.json
```

### Stdin Input (Avoid Shell Escaping)

For queries with special characters like quotes, use stdin to avoid shell escaping issues:

```bash
# Heredoc syntax (recommended for complex queries)
dtctl query -f - -o json <<'EOF'
metrics
| filter startsWith(metric.key, "dt")
| summarize count(), by: {metric.key}
| fieldsKeep metric.key
| limit 10
EOF

# Pipe from a file
cat query.dql | dtctl query -o json

# Pipe from echo (simple cases)
echo 'fetch logs | filter status="ERROR"' | dtctl query -o table
```

**Tip:** Using single-quoted heredocs (`<<'EOF'`) preserves all special characters exactly as written—no escaping needed.

### PowerShell Quoting Issues and Solutions

PowerShell has different quoting rules that can cause problems with inline DQL queries. Here's how to handle them:

#### The Problem

```powershell
# ❌ FAILS - PowerShell removes inner double quotes
dtctl query 'fetch logs, bucket:{"custom-logs"} | filter contains(host.name, "api")'
# Error: MANDATORY_PARAMETER_HAS_TO_BE_CONSTANT
# PowerShell passes: bucket:{custom-logs} (missing quotes around "custom-logs")

# ❌ FAILS - DQL doesn't support single quotes
dtctl query "fetch logs, bucket:{'custom-logs'} | filter contains(host.name, 'api')"
# Error: PARSE_ERROR_SINGLE_QUOTES
# Single quotes are not supported. Please use double quotes for strings.
```

#### Solution 1: Use PowerShell Here-Strings (Recommended)

PowerShell's here-string syntax (`@'...'@`) preserves all characters exactly:

```powershell
# ✅ WORKS - Use @'...'@ for verbatim strings
dtctl query -f - -o json @'
fetch logs, bucket:{"custom-logs"}
| filter contains(host.name, "api")
| limit 10
'@

# ✅ More complex example with multiple quotes
dtctl query -f - -o json @'
fetch logs, bucket:{"application-logs"}
| filter contains(log.source, "backend")
| filter status = "ERROR"
| summarize count(), by:{log.source}
| limit 100
'@

# ✅ Works with any DQL query structure
dtctl query -f - -o csv @'
timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}
| filter avg > 80
'@
```

#### Solution 2: Use a Query File

Save your query to a file and reference it:

```powershell
# Save query to file
@"
fetch logs, bucket:{"custom-logs"}
| filter contains(host.name, "api")
| limit 10
"@ | Out-File -Encoding UTF8 query.dql

# Execute from file
dtctl query -f query.dql -o json
```

#### Solution 3: Pipe from Get-Content

```powershell
# Read from file and pipe
Get-Content query.dql | dtctl query -o json

# Or use cat alias
cat query.dql | dtctl query -o json
```

#### Quick Reference: PowerShell vs Bash

| Shell | Heredoc Syntax | Example |
|-------|----------------|---------|
| **Bash/Zsh** | `<<'EOF'` | `dtctl query -f - <<'EOF'`<br>`fetch logs`<br>`EOF` |
| **PowerShell** | `@'...'@` | `dtctl query -f - @'`<br>`fetch logs`<br>`'@` |

**Why This Matters:**
- DQL requires double quotes for strings (e.g., `"custom-logs"`, `"ERROR"`, `"api"`)
- PowerShell's quote parsing can strip or convert these quotes
- Using `-f -` (stdin) with here-strings bypasses shell quote parsing entirely

**Example query file** (`queries/errors.dql`):

```dql
fetch logs
| filter status = 'ERROR'
| filter timestamp > now() - 1h
| summarize count(), by: {log.source}
| sort count desc
| limit 10
```

### Template Queries

Use templates with variables for flexible queries:

```bash
# Query with variable substitution
dtctl query -f queries/logs-by-host.dql --set host=my-server

# Override multiple variables
dtctl query -f queries/logs-by-host.dql \
  --set host=my-server \
  --set timerange=24h \
  --set limit=500
```

**Example template** (`queries/logs-by-host.dql`):

```dql
fetch logs
| filter host = "{{.host}}"
| filter timestamp > now() - {{.timerange | default "1h"}}
| limit {{.limit | default 100}}
```

**Template syntax:**
- `{{.variable}}` - Reference a variable
- `{{.variable | default "value"}}` - Provide default value

### Output Formats

```bash
# Table format (default, human-readable)
dtctl query "fetch logs | limit 5" -o table

# JSON format (for processing)
dtctl query "fetch logs | limit 5" -o json

# YAML format
dtctl query "fetch logs | limit 5" -o yaml

# CSV format (for spreadsheets and data export)
dtctl query "fetch logs | limit 5" -o csv

# Export to CSV file
dtctl query "fetch logs" -o csv > logs.csv
```

### Large Dataset Downloads

By default, DQL queries are limited to 1000 records. Use query limit flags to download larger datasets:

```bash
# Increase result limit to 5000 records
dtctl query "fetch logs" --max-result-records 5000 -o csv > logs.csv

# Download up to 15000 records
dtctl query "fetch logs | limit 15000" --max-result-records 15000 -o csv > logs.csv

# Set result size limit in bytes (100MB)
dtctl query "fetch logs" \
  --max-result-records 10000 \
  --max-result-bytes 104857600 \
  -o csv > large_export.csv

# Set scan limit in gigabytes
dtctl query "fetch logs" \
  --max-result-records 10000 \
  --default-scan-limit-gbytes 5.0 \
  -o csv > large_export.csv

# Combine with filters for targeted exports
dtctl query "fetch logs | filter status='ERROR'" \
  --max-result-records 5000 \
  -o csv > error_logs.csv
```

**Query Limit Parameters:**
- `--max-result-records`: Maximum number of result records to return (default: 1000)
- `--max-result-bytes`: Maximum result size in bytes (default: varies by environment)
- `--default-scan-limit-gbytes`: Scan limit in gigabytes (default: varies by environment)

**Query Execution Parameters:**
- `--default-sampling-ratio`: Sampling ratio for query results (normalized to power of 10 ≤ 100000)
- `--fetch-timeout-seconds`: Time limit for fetching data in seconds
- `--enable-preview`: Request preview results if available within timeout
- `--enforce-query-consumption-limit`: Enforce query consumption limit
- `--include-types`: Include type information in query results

**Timeframe Parameters:**
- `--default-timeframe-start`: Query timeframe start timestamp (ISO-8601/RFC3339, e.g., '2022-04-20T12:10:04.123Z')
- `--default-timeframe-end`: Query timeframe end timestamp (ISO-8601/RFC3339, e.g., '2022-04-20T13:10:04.123Z')

**Localization Parameters:**
- `--locale`: Query locale (e.g., 'en_US', 'de_DE')
- `--timezone`: Query timezone (e.g., 'UTC', 'Europe/Paris', 'America/New_York')

**Metadata Parameters:**
- `--metadata`, `-M`: Include query execution metadata in output. Use bare `--metadata` for all fields, or select specific fields with `--metadata=field1,field2`. Valid fields: `analysisTimeframe`, `canonicalQuery`, `contributions`, `dqlVersion`, `executionTimeMilliseconds`, `locale`, `metrics`, `query`, `queryId`, `sampled`, `scannedBytes`, `scannedDataPoints`, `scannedRecords`, `timezone`
- `--include-contributions`: Include bucket contribution details in metadata (requires API support)

**Note:** All parameters are sent in the DQL query request body and work with both immediate responses and long-running queries that require polling.

**Advanced Query Examples:**

```bash
# Query with specific timeframe
dtctl query "fetch logs" \
  --default-timeframe-start "2024-01-01T00:00:00Z" \
  --default-timeframe-end "2024-01-02T00:00:00Z" \
  -o csv

# Query with timezone and locale
dtctl query "fetch logs" \
  --timezone "Europe/Paris" \
  --locale "fr_FR" \
  -o json

# Query with sampling for large datasets
dtctl query "fetch logs" \
  --default-sampling-ratio 10 \
  --max-result-records 10000 \
  -o csv

# Query with preview mode (faster results)
dtctl query "fetch logs" \
  --enable-preview \
  -o table

# Query with type information included
dtctl query "fetch logs" \
  --include-types \
  -o json
```

**Tip:** Use CSV output with increased limits for:
- Exporting data for analysis in Excel or Google Sheets
- Creating backups of log data
- Feeding data into external analysis tools
- Generating reports from DQL query results

### Live Query Results

Monitor DQL query results in real-time with live mode:

```bash
# Live mode with periodic updates (default: 60s)
dtctl query "fetch logs | filter status='ERROR'" --live

# Live mode with custom interval
dtctl query "fetch logs" --live --interval 5s

# Live mode with charts
dtctl query "timeseries avg(dt.host.cpu.usage)" -o chart --live --interval 10s
```

### Query Warnings

DQL queries may return warnings (e.g., scan limits reached, results truncated). These warnings are printed to **stderr**, keeping stdout clean for data processing.

```bash
# Warnings appear on stderr, data on stdout
dtctl query "fetch spans, from: -10d | summarize count()"
# Warning: Your execution was stopped after 500 gigabytes of data were scanned...
# map[count():194414758]

# Pipe data normally - warnings don't interfere
dtctl query "fetch logs | limit 100" -o json | jq '.records[0]'

# Suppress warnings entirely
dtctl query "fetch spans | summarize count()" 2>/dev/null

# Save data to file (warnings still visible in terminal)
dtctl query "fetch logs" -o csv > logs.csv

# Save data and warnings separately
dtctl query "fetch logs" -o json > data.json 2> warnings.txt

# Discard warnings, save only data
dtctl query "fetch logs" -o csv 2>/dev/null > clean_data.csv
```

**Common warnings:**
- **SCAN_LIMIT_GBYTES**: Query stopped after scanning the default limit. Use `--default-scan-limit-gbytes` to adjust.
- **RESULT_TRUNCATED**: Results exceeded the limit. Use `--max-result-records` to increase.

### Query Verification

Verify DQL query syntax without executing it. This is useful for:
- Testing queries in CI/CD pipelines
- Pre-commit hooks to validate query files
- Checking query correctness before execution
- Getting the canonical (normalized) representation of queries

#### Basic Verification

```bash
# Verify inline query
dtctl verify query "fetch logs | limit 10"

# Verify query from file
dtctl verify query -f query.dql

# Read from stdin (recommended for complex queries)
dtctl verify query -f - <<'EOF'
fetch logs | filter status == "ERROR"
EOF

# Pipe query from file
cat query.dql | dtctl verify query
```

#### Verification with Options

```bash
# Get canonical query representation (normalized format)
dtctl verify query "fetch logs" --canonical

# Verify with specific timezone and locale
dtctl verify query "fetch logs" --timezone "Europe/Paris" --locale "fr_FR"

# Get structured output (JSON or YAML)
dtctl verify query "fetch logs" -o json
dtctl verify query "fetch logs" -o yaml

# Fail on warnings (strict validation for CI/CD)
dtctl verify query -f query.dql --fail-on-warn
```

#### Exit Codes

The `verify` command returns different exit codes based on the result:

| Exit Code | Meaning |
|-----------|---------|
| 0 | Query is valid |
| 1 | Query is invalid or has errors (or warnings with `--fail-on-warn`) |
| 2 | Authentication/permission error |
| 3 | Network/server error |

```bash
# Check exit code in scripts
if dtctl verify query -f query.dql --fail-on-warn; then
  echo "Query is valid"
else
  echo "Query validation failed"
  exit 1
fi
```

#### CI/CD Integration

```bash
# Validate all queries in a directory
for file in queries/*.dql; do
  echo "Verifying $file..."
  dtctl verify query -f "$file" --fail-on-warn || exit 1
done

# Pre-commit hook: Verify staged query files
git diff --cached --name-only --diff-filter=ACM "*.dql" | \
  xargs -I {} dtctl verify query -f {} --fail-on-warn

# GitHub Actions / CI pipeline
- name: Validate DQL queries
  run: |
    for file in queries/*.dql; do
      dtctl verify query -f "$file" --fail-on-warn || exit 1
    done
```

#### Template Variables

Verify queries with template variables before execution:

```bash
# Verify template query
dtctl verify query -f template.dql --set env=prod --set timerange=1h

# If valid, execute it
if dtctl verify query -f template.dql --set env=prod 2>/dev/null; then
  dtctl query -f template.dql --set env=prod -o csv > results.csv
fi
```

#### PowerShell Examples

```powershell
# Verify query using here-strings
dtctl verify query -f - @'
fetch logs, bucket:{"custom-logs"} | filter contains(host.name, "api")
'@

# Validate all queries in a directory
Get-ChildItem queries/*.dql | ForEach-Object {
  Write-Host "Verifying $_..."
  dtctl verify query -f $_.FullName --fail-on-warn
  if ($LASTEXITCODE -ne 0) { exit 1 }
}
```

#### Canonical Query Output

Get the normalized representation of your query:

```bash
# Get canonical query
dtctl verify query "fetch logs" --canonical

# Extract canonical query with jq
dtctl verify query "fetch logs" --canonical -o json | jq -r '.canonicalQuery'

# Compare original vs canonical
echo "Original:"
cat query.dql
echo ""
echo "Canonical:"
dtctl verify query -f query.dql --canonical 2>&1 | grep -A 999 "Canonical Query:"
```

---

## Service Level Objectives (SLOs)

SLOs define and track service reliability targets.

### List and View SLOs

```bash
# List all SLOs
dtctl get slos

# Filter by name
dtctl get slos --filter 'name~production'

# Get a specific SLO
dtctl get slo slo-123

# Detailed view
dtctl describe slo slo-123
```

### SLO Templates

Use templates to quickly create SLOs:

```bash
# List available templates
dtctl get slo-templates

# View template details
dtctl describe slo-template template-456

# Create SLO from template
dtctl create slo \
  --from-template template-456 \
  --name "API Availability" \
  --target 99.9
```

### Create and Apply SLOs

```bash
# Create from file
dtctl create slo -f slo-definition.yaml

# Apply (create or update)
dtctl apply -f slo-definition.yaml
```

**Example SLO** (`slo-definition.yaml`):

```yaml
name: Service Availability
description: 97% of requests are successful over the last 7 days
criteria:
  - timeframeFrom: -7d
    target: 97.0
    warning: 98.0
customSli:
  indicator: |
    timeseries { total=sum(dt.service.request.count), failures=sum(dt.service.request.failure_count) }
    , by: { dt.smartscape.service }
    | fieldsAdd sli=(((total[]-failures[])/total[])*(100))
    | fieldsAdd dt.smartscape.entity.name = getNodeName(dt.smartscape.service)
    | fieldsRemove total, failures, smartscapeIds
tags:
  - slo-type:availability
```

### Evaluate SLOs

Evaluate SLOs to get current status, values, and error budget for each criterion:

```bash
# Evaluate SLO performance
dtctl exec slo slo-123

# Evaluate with custom timeout (default: 30 seconds)
dtctl exec slo slo-123 --timeout 60

# Output as JSON for analysis
dtctl exec slo slo-123 -o json

# Extract error budget from results
dtctl exec slo slo-123 -o json | jq '.evaluationResults[].errorBudget'

# View in table format (default)
dtctl exec slo slo-123
```

### Watch SLOs

Monitor SLO status changes in real-time:

```bash
# Watch all SLOs
dtctl get slos --watch

# Watch with custom interval
dtctl get slos --watch --interval 30s

# Watch with filter
dtctl get slos --filter 'name~production' --watch
```

### Delete SLOs

```bash
# Delete an SLO
dtctl delete slo slo-123

# Skip confirmation
dtctl delete slo slo-123 -y
```

---

## Notifications

View and manage event notifications.

### List Notifications

```bash
# List all notifications
dtctl get notifications

# Filter by type
dtctl get notifications --type EMAIL

# Get a specific notification
dtctl get notification notif-123

# Detailed view
dtctl describe notification notif-123
```

### Watch Notifications

Monitor notifications in real-time:

```bash
# Watch all notifications
dtctl get notifications --watch

# Watch specific notification type
dtctl get notifications --type EMAIL --watch
```

### Delete Notifications

```bash
# Delete a notification
dtctl delete notification notif-123
```

---

## Grail Buckets

Grail buckets provide scalable log and event storage.

### List and View Buckets

```bash
# List all buckets
dtctl get buckets

# Get a specific bucket
dtctl get bucket logs-production

# Detailed view with configuration
dtctl describe bucket logs-production
```

### Create and Apply Buckets

```bash
# Create a bucket from file
dtctl create bucket -f bucket-config.yaml

# Apply (create or update)
dtctl apply -f bucket-config.yaml
```

**Example bucket configuration** (`bucket-config.yaml`):

```yaml
bucketName: logs-production
displayName: Production Logs
table: logs
retentionDays: 35
status: active
```

### Watch Buckets

Monitor bucket changes in real-time:

```bash
# Watch all buckets
dtctl get buckets --watch

# Watch with custom interval
dtctl get buckets --watch --interval 10s
```

### Delete Buckets

```bash
# Delete a bucket
dtctl delete bucket logs-staging

# Skip confirmation
dtctl delete bucket logs-staging -y
```

---

## Lookup Tables

Lookup tables enable data enrichment in DQL queries by mapping key values to additional information. They're stored in Grail and can be referenced in queries to add context like mapping error codes to descriptions, IPs to locations, or IDs to human-readable names.

### List and View Lookup Tables

```bash
# List all lookup tables
dtctl get lookups

# Get a specific lookup (shows metadata + 10 row preview)
dtctl get lookup /lookups/production/error_codes

# View detailed information
dtctl describe lookup /lookups/production/error_codes
```

### Create Lookup Tables from CSV

The easiest way to create a lookup table is from a CSV file. dtctl automatically detects the CSV structure:

```bash
# Create from CSV (auto-detects headers and format)
dtctl create lookup -f error_codes.csv \
  --path /lookups/production/error_codes \
  --display-name "Error Code Mappings" \
  --description "Maps error codes to descriptions and severity" \
  --lookup-field code

# Output:
# ✓ Created lookup table: /lookups/production/error_codes
#   Records: 150
#   File Size: 12,458 bytes
#   Discarded Duplicates: 0
```

**Example CSV file** (`error_codes.csv`):

```csv
code,message,severity
E001,Connection timeout,high
E002,Invalid credentials,critical
E003,Resource not found,medium
E004,Rate limit exceeded,low
```

### Create with Custom Parse Patterns

For non-CSV formats or custom delimiters, specify a parse pattern:

```bash
# Pipe-delimited file
dtctl create lookup -f data.txt \
  --path /lookups/custom/pipe_data \
  --parse-pattern "LD:id '|' LD:name '|' LD:value" \
  --lookup-field id \
  --skip-records 1

# Tab-delimited file
dtctl create lookup -f data.tsv \
  --path /lookups/custom/tab_data \
  --parse-pattern "LD:col1 '\t' LD:col2 '\t' LD:col3" \
  --lookup-field col1 \
  --skip-records 1
```

**Parse Pattern Syntax:**
- `LD:columnName` - Define a column
- `','` - Comma separator (single quotes required)
- `'\t'` - Tab separator
- `'|'` - Pipe separator

### Update Lookup Tables

To update an existing lookup table, you need to delete it first and then recreate it:

```bash
# Delete the existing lookup table
dtctl delete lookup /lookups/production/error_codes -y

# Create with new data
dtctl create lookup -f updated_codes.csv \
  --path /lookups/production/error_codes \
  --lookup-field code
```

**Note:** Updates completely replace the existing lookup table data.

### Using Lookup Tables in DQL Queries

Once created, use lookup tables to enrich your query results:

```bash
# Simple lookup join
dtctl query "
fetch logs
| filter status = 'ERROR'
| lookup [
    fetch dt.system.files
    | load '/lookups/production/error_codes'
  ], sourceField:error_code, lookupField:code
| fields timestamp, error_code, message, severity
| limit 100
"

# Enrich host data with location info
dtctl query "
fetch dt.entity.host
| lookup [
    load '/lookups/infrastructure/host_locations'
  ], sourceField:host.name, lookupField:hostname
| fields host.name, datacenter, region, cost_center
"

# Map user IDs to names
dtctl query "
fetch logs
| filter log.source = 'api'
| lookup [
    load '/lookups/users/directory'
  ], sourceField:user_id, lookupField:id, fields:{name, email, department}
| summarize count(), by:{name, department}
"
```

### Practical Examples

#### Error Code Enrichment

Create a lookup table for error codes:

```bash
# Create error_codes.csv
cat > error_codes.csv <<EOF
code,message,severity,documentation_url
E001,Connection timeout,high,https://docs.example.com/errors/e001
E002,Invalid credentials,critical,https://docs.example.com/errors/e002
E003,Resource not found,medium,https://docs.example.com/errors/e003
E004,Rate limit exceeded,low,https://docs.example.com/errors/e004
E005,Internal server error,critical,https://docs.example.com/errors/e005
EOF

# Upload to Dynatrace
dtctl create lookup -f error_codes.csv \
  --path /lookups/monitoring/error_codes \
  --display-name "Application Error Codes" \
  --lookup-field code

# Use in query
dtctl query "
fetch logs
| filter status = 'ERROR'
| lookup [load '/lookups/monitoring/error_codes'], 
  sourceField:error_code, lookupField:code
| fields timestamp, error_code, message, severity, documentation_url
| limit 50
"
```

#### IP to Location Mapping

Map IP addresses to geographic locations:

```bash
# Create ip_locations.csv
cat > ip_locations.csv <<EOF
ip_address,city,country,datacenter
10.0.1.50,New York,USA,DC-US-EAST-1
10.0.2.50,London,UK,DC-EU-WEST-1
10.0.3.50,Singapore,SG,DC-APAC-1
192.168.1.100,Frankfurt,Germany,DC-EU-CENTRAL-1
EOF

# Upload
dtctl create lookup -f ip_locations.csv \
  --path /lookups/infrastructure/ip_locations \
  --display-name "IP to Location Mapping" \
  --lookup-field ip_address

# Use in query to geo-locate traffic
dtctl query "
fetch logs
| filter log.source = 'nginx'
| lookup [load '/lookups/infrastructure/ip_locations'], 
  sourceField:client_ip, lookupField:ip_address
| summarize request_count=count(), by:{city, country, datacenter}
| sort request_count desc
"
```

#### Service ID to Team Mapping

Map service identifiers to team ownership:

```bash
# Create service_owners.csv
cat > service_owners.csv <<EOF
service_id,service_name,team,team_email,slack_channel
svc-001,payment-api,Payments,payments@example.com,#team-payments
svc-002,user-service,Identity,identity@example.com,#team-identity
svc-003,order-processor,Fulfillment,fulfillment@example.com,#team-fulfillment
svc-004,notification-service,Platform,platform@example.com,#team-platform
EOF

# Upload
dtctl create lookup -f service_owners.csv \
  --path /lookups/services/ownership \
  --display-name "Service Ownership" \
  --lookup-field service_id

# Find errors by team
dtctl query "
fetch logs
| filter status = 'ERROR'
| lookup [load '/lookups/services/ownership'], 
  sourceField:service, lookupField:service_id
| summarize error_count=count(), by:{team, team_email, slack_channel}
| sort error_count desc
"
```

#### Country Code Enrichment

```bash
# Create country_codes.csv
cat > country_codes.csv <<EOF
code,name,continent,currency
US,United States,North America,USD
GB,United Kingdom,Europe,GBP
DE,Germany,Europe,EUR
JP,Japan,Asia,JPY
AU,Australia,Oceania,AUD
BR,Brazil,South America,BRL
IN,India,Asia,INR
EOF

# Upload
dtctl create lookup -f country_codes.csv \
  --path /lookups/reference/countries \
  --display-name "Country Reference Data" \
  --lookup-field code

# Enrich user analytics
dtctl query "
fetch logs
| filter log.source = 'analytics'
| lookup [load '/lookups/reference/countries'], 
  sourceField:country_code, lookupField:code, 
  fields:{name, continent, currency}
| summarize users=countDistinct(user_id), by:{name, continent}
| sort users desc
"
```

### Delete Lookup Tables

```bash
# Delete a lookup table
dtctl delete lookup /lookups/production/old_data

# Skip confirmation
dtctl delete lookup /lookups/staging/test_data -y
```

### Path Requirements

Lookup table paths must follow these rules:
- Must start with `/lookups/`
- Only alphanumeric characters, hyphens (`-`), underscores (`_`), dots (`.`), and slashes (`/`)
- Must end with an alphanumeric character
- Maximum 500 characters
- At least 2 slashes (e.g., `/lookups/category/name`)

**Good paths:**
- `/lookups/production/error_codes`
- `/lookups/infrastructure/host-locations`
- `/lookups/reference/country.codes`

**Invalid paths:**
- `/data/lookup` - Must start with `/lookups/`
- `/lookups/test/` - Cannot end with slash
- `/lookups/data@prod` - Invalid character `@`
- `/lookups/name` - Must have at least 2 slashes

### Tips & Best Practices

**1. Organize with meaningful paths:**
```bash
/lookups/production/...      # Production data
/lookups/staging/...         # Staging/test data
/lookups/reference/...       # Static reference data
/lookups/infrastructure/...  # Infrastructure mappings
/lookups/applications/...    # Application-specific data
```

**2. Use descriptive display names and descriptions:**
```bash
dtctl create lookup -f data.csv \
  --path /lookups/prod/error_codes \
  --display-name "Production Error Code Mappings" \
  --description "Maps application error codes to user-friendly messages and severity levels. Updated weekly." \
  --lookup-field code
```

**3. Export for backup:**
```bash
# Export lookup metadata and data
dtctl get lookup /lookups/production/error_codes -o yaml > backup.yaml

# List all lookups for documentation
dtctl get lookups -o csv > lookup_inventory.csv
```

**4. Version your source CSV files:**
```bash
# Keep CSV files in version control
git add lookups/error_codes.csv
git commit -m "Update error code E005 description"

# Apply from repository (delete first if it exists)
dtctl delete lookup /lookups/production/error_codes -y 2>/dev/null || true
dtctl create lookup -f lookups/error_codes.csv \
  --path /lookups/production/error_codes \
  --lookup-field code
```

**5. Test before production:**
```bash
# Upload to staging first
dtctl create lookup -f new_data.csv \
  --path /lookups/staging/test_lookup \
  --lookup-field id

# Test with queries
dtctl query "fetch logs | lookup [load '/lookups/staging/test_lookup'], sourceField:id, lookupField:key"

# Promote to production (delete first if exists)
dtctl delete lookup /lookups/production/live_lookup -y 2>/dev/null || true
dtctl create lookup -f new_data.csv \
  --path /lookups/production/live_lookup \
  --lookup-field id
```

### Required Token Scopes

For lookup table management: `storage:files:read`, `storage:files:write`, `storage:files:delete`

See [TOKEN_SCOPES.md](../TOKEN_SCOPES.md) for complete scope reference.

See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for complete scope lists by safety level.

---

## OpenPipeline

OpenPipeline processes and routes observability data. As of September 2025, OpenPipeline configurations have been migrated from the direct API to the Settings API v2 for better access control and configuration management.

**Important:** The direct OpenPipeline commands (`dtctl get openpipelines`, `dtctl describe openpipeline`) have been removed. Use the Settings API instead to manage OpenPipeline configurations.

### View Pipeline Configurations via Settings API

```bash
# List OpenPipeline schemas
dtctl get settings-schemas | grep openpipeline

# View specific schema details
dtctl describe settings-schema builtin:openpipeline.logs.pipelines

# List log pipelines
dtctl get settings --schema builtin:openpipeline.logs.pipelines

# Get a specific pipeline by object ID
dtctl get settings <object-id> --schema builtin:openpipeline.logs.pipelines
```

**Note:** See the [Settings API](#settings-api) section below for full details on managing OpenPipeline configurations.

### Translate Classic Pipelines to OpenPipeline

Migrating from Classic pipelines to OpenPipeline? `dtctl get classic-pipelines-translation` converts your tenant's Classic pipeline configuration for a scope into an OpenPipeline configuration pipeline (Settings shape). It is a read-only call that returns the translated pipeline verbatim — the reliable starting point you then review and apply via the Settings API.

```bash
# Translate the logs Classic pipeline (pretty-printed pipeline document)
dtctl get classic-pipelines-translation logs

# Translate business events and export as YAML for review/editing
dtctl get classic-pipelines-translation bizevents -o yaml > reference-pipeline.yaml

# Print the translated pipeline as JSON
dtctl get classic-pipelines-translation logs -o json

# Skip disabled rules in the translation (overrides the server default)
dtctl get classic-pipelines-translation logs --skip-disabled-rules=true
```

The scope is a positional argument and must be `logs` or `bizevents`. Every output format emits the translated pipeline document directly (no `{value, withWarning}` wrapper), so the exported file is applyable as-is. The translation is deterministic where possible; when a processing rule's definition script could not be translated automatically (`withWarning=true`), a warning is printed to stderr (and carried in the agent envelope under `-A`) and that part needs a manual rewrite. Apply the reviewed result with `dtctl create settings --schema builtin:openpipeline.<scope>.pipelines -f <file>`.

**Note:** The underlying API is public but early-adopter and may change.

---

## Settings API

The Settings API provides a unified way to manage Dynatrace configurations, including OpenPipeline pipelines, ingest sources, and routing configurations. Settings are organized by schemas and scopes.

### List Settings Schemas

Discover available configuration schemas:

```bash
# List all available schemas
dtctl get settings-schemas

# Filter for OpenPipeline schemas
dtctl get settings-schemas | grep openpipeline

# Get a specific schema definition
dtctl get settings-schema builtin:openpipeline.logs.pipelines

# View detailed schema information
dtctl describe settings-schema builtin:openpipeline.logs.pipelines

# Output as JSON for processing
dtctl get settings-schemas -o json
```

**Common OpenPipeline Schemas:**
- `builtin:openpipeline.logs.pipelines` - Log processing pipelines
- `builtin:openpipeline.logs.ingest-sources` - Log ingest sources
- `builtin:openpipeline.logs.routing` - Log routing configuration
- `builtin:openpipeline.spans.pipelines` - Trace span pipelines
- `builtin:openpipeline.metrics.pipelines` - Metric pipelines
- `builtin:openpipeline.bizevents.pipelines` - Business event pipelines

### List Settings Objects

View configured settings for a schema:

```bash
# List all settings objects for a schema
dtctl get settings --schema builtin:openpipeline.logs.pipelines

# Filter by scope
dtctl get settings --schema builtin:openpipeline.logs.pipelines --scope environment

# Get a specific settings object
dtctl get settings aaaaaaaa-bbbb-cccc-dddd-000000000001

# Output as JSON
dtctl get settings --schema builtin:openpipeline.logs.pipelines -o json
```

### Create Settings Objects

Create new configuration objects from YAML or JSON files:

```bash
# Create a log pipeline
dtctl create settings -f log-pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment

# Create with template variables
dtctl create settings -f pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment \
  --set environment=production,retention=90

# Dry run to preview
dtctl create settings -f pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment \
  --dry-run
```

**Example pipeline file** (`log-pipeline.yaml`):

```yaml
customId: production-logs-pipeline
displayName: Production Log Processing Pipeline
processing:
  - processor: fields-add
    fields:
      - name: environment
        value: production
      - name: team
        value: platform
  - processor: dql
    processorDefinition:
      dpl: |
        fieldsAdd(severity: if(loglevel=="ERROR", "critical", "info"))
storage:
  table: logs
  retention: 90
routing:
  catchAll: false
  rules:
    - matcher: matchesValue(log.source, "kubernetes")
      target: builtin:storage-default
```

### Update Settings Objects

Modify existing settings:

```bash
# Update a settings object
dtctl update settings aaaaaaaa-bbbb-cccc-dddd-000000000001 \
  -f updated-pipeline.yaml

# Update with template variables
dtctl update settings aaaaaaaa-bbbb-cccc-dddd-000000000001 \
  -f pipeline.yaml \
  --set retention=120

# Dry run
dtctl update settings aaaaaaaa-bbbb-cccc-dddd-000000000001 \
  -f pipeline.yaml \
  --dry-run
```

**Note:** Updates use optimistic locking automatically - the current version is fetched before updating to prevent conflicts.

### Delete Settings Objects

Remove settings objects:

```bash
# Delete a settings object (with confirmation)
dtctl delete settings aaaaaaaa-bbbb-cccc-dddd-000000000001

# Delete without confirmation
dtctl delete settings aaaaaaaa-bbbb-cccc-dddd-000000000001 -y
```

### OpenPipeline Configuration Workflow

Complete workflow for managing OpenPipeline configurations:

```bash
# 1. Discover available pipeline schemas
dtctl get settings-schemas | grep "openpipeline.logs"

# 2. View the schema structure
dtctl describe settings-schema builtin:openpipeline.logs.pipelines

# 3. List existing pipelines
dtctl get settings --schema builtin:openpipeline.logs.pipelines

# 4. Export existing pipeline for reference
dtctl get settings <pipeline-id> -o yaml > reference-pipeline.yaml

# 5. Create your new pipeline
cat > my-pipeline.yaml <<EOF
customId: my-custom-pipeline
displayName: My Custom Pipeline
processing:
  - processor: fields-add
    fields:
      - name: source
        value: my-app
storage:
  table: logs
EOF

# 6. Create the pipeline
dtctl create settings -f my-pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment

# 7. Verify it was created
dtctl get settings --schema builtin:openpipeline.logs.pipelines | grep my-custom
```

### Multi-Environment Configuration

Deploy the same configuration across environments:

```bash
# Export from dev
dtctl --context dev get settings <pipeline-id> -o yaml > pipeline.yaml

# Review and modify for production
$EDITOR pipeline.yaml

# Deploy to staging
dtctl --context staging create settings -f pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment \
  --set environment=staging

# Deploy to production
dtctl --context prod create settings -f pipeline.yaml \
  --schema builtin:openpipeline.logs.pipelines \
  --scope environment \
  --set environment=production
```

**Required Token Scopes:**
- `settings:objects:read` - List and view settings objects (includes schema read access)
- `settings:objects:write` - Create, update, and delete settings objects

See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for complete scope lists by safety level.

---

## App Engine

Manage Dynatrace apps and their serverless functions.

### List and View Apps

```bash
# List all apps
dtctl get apps

# Filter by name
dtctl get apps --name "monitoring"

# Get a specific app
dtctl get app app-123

# Detailed view
dtctl describe app app-123
```

### App Functions

App functions are serverless backend functions exposed by installed apps. They can be invoked via HTTP to perform various operations like sending notifications, querying external APIs, or executing custom logic.

#### Discover Functions

```bash
# List all functions across all installed apps
dtctl get functions

# List functions for a specific app
dtctl get functions --app dynatrace.automations

# Show function descriptions and metadata (wide output)
dtctl get functions --app dynatrace.automations -o wide

# Get details about a specific function
dtctl get function dynatrace.automations/execute-dql-query

# Describe a function (shows usage and metadata)
dtctl describe function dynatrace.automations/execute-dql-query
```

**Example output:**
```
Function:     execute-dql-query
Full Name:    dynatrace.automations/execute-dql-query
Title:        Execute DQL Query
Description:  Make use of Dynatrace Grail data in your workflow.
App:          Workflows (dynatrace.automations)
Resumable:    false
Stateful:     true

Usage:
  dtctl exec function dynatrace.automations/execute-dql-query
```

#### Execute Functions

> **Note:** Function input schemas are not currently exposed through the API. To discover what payload a function expects, try executing it with an empty payload `{}` to see the error message listing required fields, or check the Dynatrace UI documentation for the app.

```bash
# Execute a DQL query function (requires dynatrace.automations app - built-in)
dtctl exec function dynatrace.automations/execute-dql-query \
  --method POST \
  --payload '{"query":"fetch logs | limit 5"}' \
  -o json

# Execute with payload from file
dtctl exec function dynatrace.automations/execute-dql-query \
  --method POST \
  --data @query.json

# Execute with GET method (for functions that don't require input)
dtctl exec function <app-id>/<function-name>
```

**Discovering Required Payload Fields:**

Functions don't expose their schemas via the API. To discover what fields are required, try executing the function with an empty payload and examine the error message:

```bash
# Try with empty payload to see what fields are required
dtctl exec function dynatrace.automations/execute-dql-query \
  --method POST \
  --payload '{}' \
  -o json 2>&1 | jq -r '.body' | jq -r '.error'

# Output: Error: Input fields 'query' are missing.
```

#### Tips for Working with Functions

**Discover available functions:**
```bash
# List all available functions
dtctl get functions

# Find functions by keyword
dtctl get functions | grep -i "query\|http"

# Export function inventory
dtctl get functions -o json > functions-inventory.json

# Get detailed info about a function (shows title, description, stateful)
dtctl get functions --app dynatrace.automations -o wide
```

**Find function payloads:**
```bash
# Method 1: Check the Dynatrace UI
# Navigate to Apps → [App Name] → View function documentation

# Method 2: Use error messages to discover required fields
dtctl exec function <app-id>/<function-name> \
  --method POST \
  --payload '{}' \
  -o json 2>&1 | jq -r '.body' | jq -r '.error // .logs'

# Method 3: Look at existing workflows that use the function
dtctl get workflows -o json | jq -r '.[] | select(.tasks != null)'
```

**Common Function Examples:**

```bash
# DQL Query (dynatrace.automations/execute-dql-query)
# Required: query (string)
dtctl exec function dynatrace.automations/execute-dql-query \
  --method POST \
  --payload '{"query":"fetch logs | limit 5"}' \
  -o json

# Send Email (dynatrace.email/send-email)
# Required: to, cc, bcc (arrays), subject, content (strings)
dtctl exec function dynatrace.email/send-email \
  --method POST \
  --payload '{
    "to": ["user@example.com"],
    "cc": [],
    "bcc": [],
    "subject": "Test Email",
    "content": "This is a test email from dtctl"
  }'

# Slack Message (dynatrace.slack/slack-send-message)
# Required: connection, channel, message
dtctl exec function dynatrace.slack/slack-send-message \
  --method POST \
  --payload '{
    "connection": "connection-id",
    "channel": "#alerts",
    "message": "Hello from dtctl"
  }'

# Jira Create Issue (dynatrace.jira/jira-create-issue)
# Required: connectionId, project, issueType, components, summary, description
dtctl exec function dynatrace.jira/jira-create-issue \
  --method POST \
  --payload '{
    "connectionId": "connection-id",
    "project": "PROJ",
    "issueType": "Bug",
    "components": [],
    "summary": "Issue from dtctl",
    "description": "Created via dtctl"
  }'

# AbuseIPDB Check (dynatrace.abuseipdb/check-ip)
# Required: observable (object), settingsObjectId (string)
dtctl exec function dynatrace.abuseipdb/check-ip \
  --method POST \
  --payload '{
    "observable": {"type": "IP", "value": "8.8.8.8"},
    "settingsObjectId": "settings-object-id"
  }'
```

**Required Token Scopes:**
- `app-engine:apps:run` - Execute app functions

See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for complete scope lists.

### App Intents

Intents enable deep linking and inter-app communication by defining entry points that apps expose for opening resources with contextual data. They allow you to navigate directly to specific app views with parameters.

#### Discover Intents

```bash
# List all intents across all apps
dtctl get intents

# List intents for a specific app
dtctl get intents --app dynatrace.distributedtracing

# Show full details in wide format
dtctl get intents -o wide

# Get a specific intent
dtctl get intent dynatrace.distributedtracing/view-trace

# Describe an intent (shows properties and usage)
dtctl describe intent dynatrace.distributedtracing/view-trace
```

**Example output:**
```
Intent:       view-trace
Full Name:    dynatrace.distributedtracing/view-trace
Description:  View a distributed trace
App:          Distributed Tracing (dynatrace.distributedtracing)

Properties:
  - trace_id: string (required)
    Description: The trace identifier
  - timestamp: string
    Format: date-time
    Description: When the trace occurred

Required:     trace_id

Usage:
  dtctl open intent dynatrace.distributedtracing/view-trace --data trace_id=<value>
  dtctl find intents --data trace_id=<value>
```

#### Find Matching Intents

Find which intents can handle specific data:

```bash
# Find intents that match the provided data
dtctl find intents --data trace_id=d052c9a8772e349d09048355a8891b82

# Output shows match quality (100% = all required properties provided)
MATCH%  APP                          INTENT_ID        DESCRIPTION
100%    dynatrace.distributedtracing view-trace       View a distributed trace

# Find intents with multiple properties
dtctl find intents --data trace_id=abc123,timestamp=2026-02-02T16:04:19.947Z

# Output as JSON for processing
dtctl find intents --data log_id=xyz789 -o json
```

#### Generate Intent URLs

Generate deep links to open specific resources in apps:

```bash
# Generate intent URL with data
dtctl open intent dynatrace.distributedtracing/view-trace \
  --data trace_id=d052c9a8772e349d09048355a8891b82

# Output:
# https://your-env.apps.dynatrace.com/ui/intent/dynatrace.distributedtracing/view-trace#%7B%22trace_id%22%3A%22d052c9a8772e349d09048355a8891b82%22%7D

# Generate with multiple properties
dtctl open intent dynatrace.distributedtracing/view-trace \
  --data trace_id=abc123,timestamp=2026-02-02T16:04:19.947Z

# Generate from JSON file
echo '{"trace_id":"abc123","timestamp":"2026-02-02T16:04:19.947Z"}' > data.json
dtctl open intent dynatrace.distributedtracing/view-trace --data-file data.json

# Generate from stdin
cat data.json | dtctl open intent dynatrace.distributedtracing/view-trace --data-file -

# Generate and open in browser
dtctl open intent dynatrace.distributedtracing/view-trace \
  --data trace_id=abc123 --browser
```

#### Practical Use Cases

**Use Case 1: Deep Linking from Alerts**
```bash
# Extract trace ID from alert and open in Dynatrace
TRACE_ID=$(extract_from_alert)
dtctl open intent dynatrace.distributedtracing/view-trace \
  --data trace_id=$TRACE_ID --browser
```

**Use Case 2: Scripted Navigation**
```bash
# Find which apps can handle this data, then open the best match
dtctl find intents --data log_id=xyz789 -o json | \
  jq -r '.[0].FullName' | \
  xargs -I {} dtctl open intent {} --data log_id=xyz789 --browser
```

**Use Case 3: Generate Documentation**
```bash
# Generate intent documentation for all apps
dtctl get intents -o json | \
  jq -r '.[] | "## \(.FullName)\n\(.Description)\n"'
```

**Use Case 4: Integration with External Tools**
```bash
# Generate intent URL from external system data
TRACE_DATA=$(curl -s https://external-system/api/trace/123)
TRACE_ID=$(echo $TRACE_DATA | jq -r '.traceId')
dtctl open intent dynatrace.distributedtracing/view-trace \
  --data trace_id=$TRACE_ID
```

**Required Token Scopes:**
- `app-engine:apps:run` - Required for accessing app manifests and intent data

### Delete Apps

```bash
# Delete an app
dtctl delete app app-123

# Skip confirmation
dtctl delete app app-123 -y
```

---

## EdgeConnect

EdgeConnect provides secure connectivity for ActiveGates.

### List and View EdgeConnect

```bash
# List all EdgeConnect configurations
dtctl get edgeconnects

# Get a specific configuration
dtctl get edgeconnect ec-123

# Detailed view
dtctl describe edgeconnect ec-123
```

### Create and Apply EdgeConnect

```bash
# Create from file
dtctl create edgeconnect -f edgeconnect-config.yaml

# Apply (create or update)
dtctl apply -f edgeconnect-config.yaml
```

**Example configuration** (`edgeconnect-config.yaml`):

```yaml
name: "Production EdgeConnect"
hostPatterns:
  - "*.example.com"
  - "api.production.net"
oauthClientId: "client-id"
oauthClientSecret: "client-secret"
```

### Delete EdgeConnect

```bash
# Delete a configuration
dtctl delete edgeconnect ec-123
```

---

## Davis AI

Davis AI provides predictive analytics (Analyzers) and generative AI assistance (CoPilot).

### Davis Analyzers

Analyzers perform statistical analysis on time series data for forecasting, anomaly detection, and correlation analysis.

#### List and View Analyzers

```bash
# List all available analyzers
dtctl get analyzers

# Filter analyzers by name
dtctl get analyzers --filter "name contains 'forecast'"

# Get a specific analyzer definition
dtctl get analyzer dt.statistics.GenericForecastAnalyzer

# View analyzer details as JSON
dtctl get analyzer dt.statistics.GenericForecastAnalyzer -o json
```

#### Describe an Analyzer (input/result schemas)

`describe` resolves the analyzer's JSON Schemas and shows which inputs are
required vs. optional — so you know what to pass to `exec analyzer`. Unlike
`get`, the schemas are included in JSON/YAML output too.

```bash
# Show metadata plus required/optional input fields and the result shape
dtctl describe analyzer dt.statistics.GenericForecastAnalyzer

# Include the full markdown documentation
dtctl describe analyzer dt.statistics.GenericForecastAnalyzer --doc

# Structured output includes inputSchema and resultSchema
dtctl describe analyzer dt.statistics.GenericForecastAnalyzer -o json
```

#### Execute Analyzers

Run analyzers to perform statistical analysis:

```bash
# Execute with a DQL query (shorthand for timeseries analyzers)
dtctl exec analyzer dt.statistics.GenericForecastAnalyzer \
  --query "timeseries avg(dt.host.cpu.usage)"

# Execute with inline JSON input
dtctl exec analyzer dt.statistics.GenericForecastAnalyzer \
  --input '{"timeSeriesData":"timeseries avg(dt.host.cpu.usage)","forecastHorizon":50}'

# Execute from input file
dtctl exec analyzer dt.statistics.GenericForecastAnalyzer -f forecast-input.json

# Output result as JSON
dtctl exec analyzer dt.statistics.GenericForecastAnalyzer \
  --query "timeseries avg(dt.host.cpu.usage)" -o json
```

#### Validate Input Without Executing

`verify analyzer` checks an input against the analyzer's validate endpoint
without running it. It shares the input flags with `exec analyzer` and follows
the standard `verify` exit-code contract (0 valid, 1 invalid, 2 auth, 3 network),
so it drops straight into CI/CD pipelines. (`exec analyzer --validate` remains
available and calls the same endpoint.)

```bash
# Validate input from a file
dtctl verify analyzer dt.statistics.GenericForecastAnalyzer -f forecast-input.json

# Validate a DQL query shorthand
dtctl verify analyzer dt.statistics.GenericForecastAnalyzer \
  --query "timeseries avg(dt.host.cpu.usage)"

# Structured verdict for scripts
dtctl verify analyzer dt.statistics.GenericForecastAnalyzer -f forecast-input.json -o json

# CI/CD: fail the build on invalid input
dtctl verify analyzer dt.statistics.GenericForecastAnalyzer -f forecast-input.json || exit 1
```

**Example analyzer input file** (`forecast-input.json`):

```json
{
  "timeSeriesData": "timeseries avg(dt.host.cpu.usage)",
  "forecastHorizon": 100,
  "generalParameters": {
    "timeframe": {
      "startTime": "now-7d",
      "endTime": "now"
    }
  }
}
```

#### Common Analyzers

| Analyzer | Description |
|----------|-------------|
| `dt.statistics.GenericForecastAnalyzer` | Time series forecasting |
| `dt.statistics.ChangePointAnalyzer` | Detect changes in time series |
| `dt.statistics.CorrelationAnalyzer` | Find correlations between metrics |
| `dt.statistics.TimeSeriesCharacteristicAnalyzer` | Analyze time series properties |
| `dt.statistics.anomaly_detection.StaticThresholdAnomalyDetectionAnalyzer` | Static threshold anomaly detection |

### Davis CoPilot

CoPilot provides AI-powered assistance for understanding your Dynatrace environment.

#### List CoPilot Skills

```bash
# List available CoPilot skills
dtctl get copilot-skills

# Output:
# NAME
# conversation
# nl2dql
# dql2nl
# documentSearch
```

#### Chat with CoPilot

```bash
# Ask a question
dtctl exec copilot "What is DQL?"

# Ask about your environment
dtctl exec copilot "What caused the CPU spike on my production hosts?"

# Read question from file
dtctl exec copilot -f question.txt

# Stream response in real-time (shows tokens as they arrive)
dtctl exec copilot "Explain the recent errors in my environment" --stream

# Provide additional context
dtctl exec copilot "Analyze this issue" \
  --context "Error logs show connection timeouts to database"

# Disable Dynatrace documentation retrieval
dtctl exec copilot "What is an SLO?" --no-docs

# Add formatting instructions
dtctl exec copilot "List the top 5 error types" \
  --instruction "Format as a numbered list with counts"
```

#### CoPilot Use Cases

```bash
# Get help writing DQL queries
dtctl exec copilot "Write a DQL query to find all ERROR logs from the last hour"

# Understand existing queries
dtctl exec copilot "Explain this query: fetch logs | filter status='ERROR' | summarize count()"

# Troubleshoot issues
dtctl exec copilot "Why might my service response time be increasing?"

# Learn about Dynatrace features
dtctl exec copilot "How do I set up an SLO for API availability?"
```

#### NL to DQL (Natural Language to DQL)

Generate DQL queries from natural language descriptions:

```bash
# Generate a DQL query from natural language
dtctl exec copilot nl2dql "show me error logs from the last hour"
# Output: fetch logs | filter status = "ERROR" | filter timestamp > now() - 1h

# More complex queries
dtctl exec copilot nl2dql "find hosts with CPU usage above 80%"
dtctl exec copilot nl2dql "count logs by severity for the last 24 hours"

# Read prompt from file
dtctl exec copilot nl2dql -f prompt.txt

# Get full response with messageToken (for feedback)
dtctl exec copilot nl2dql "show recent errors" -o json
```

#### DQL to NL (Explain DQL Queries)

Get natural language explanations of DQL queries:

```bash
# Explain a DQL query
dtctl exec copilot dql2nl "fetch logs | filter status='ERROR' | summarize count(), by:{host}"
# Output:
# Summary: Count error logs grouped by host
# Explanation: This query fetches logs, filters for ERROR status, and counts them by host.

# Explain a complex query
dtctl exec copilot dql2nl "timeseries avg(dt.host.cpu.usage), by:{dt.entity.host} | filter avg > 80"

# Read query from file
dtctl exec copilot dql2nl -f query.dql

# Get full response as JSON
dtctl exec copilot dql2nl "fetch logs | limit 10" -o json
```

#### Document Search

Find relevant notebooks and dashboards:

```bash
# Search for documents about CPU analysis
dtctl exec copilot document-search "CPU performance analysis" --collections notebooks

# Search across multiple collections
dtctl exec copilot document-search "error monitoring" --collections dashboards,notebooks

# Exclude specific documents from results
dtctl exec copilot document-search "performance" --exclude doc-123,doc-456

# Output as JSON for processing
dtctl exec copilot document-search "kubernetes" --collections notebooks -o json
```

### Required Token Scopes

For Davis AI features:
- **Analyzers**: `davis:analyzers:read`, `davis:analyzers:execute`
- **CoPilot** (all features): `davis-copilot:conversations:execute`

See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for complete scope lists by safety level.

---

## AWS Monitoring

This is the recommended onboarding flow for AWS with role-based authentication. The AWS connection is created first (without a role ARN), the trust policy uses the connection's `objectId` as `sts:ExternalId`, and the IAM role is created via a Dynatrace-maintained CloudFormation template.

> **Note:** The monitoring config is created in a **disabled** state. Run `dtctl enable aws monitoring` once the IAM role exists to activate it.

### 1) Create AWS connection in Dynatrace

```bash
dtctl create aws connection --name "my-aws-connection"
```

The command prints a copy-paste `aws cloudformation deploy` snippet that:
- Downloads the Dynatrace-maintained least-privilege role template.
- Creates the IAM role using the connection's `objectId` as `sts:ExternalId` and your tenant URL as `pDynatraceUrl`.
- Outputs the role ARN, ready to be patched back into the connection.

### 2) Create the IAM role in AWS (CloudShell)

Run the printed snippet in AWS CloudShell (or any environment with the `aws` CLI configured). Example shape:

```bash
STACK="dynatrace-monitoring-my-aws-connection"
curl -fsSLo da-role.yaml https://dynatrace-data-acquisition.s3.amazonaws.com/aws/deployment/cfn/latest/da-aws-nested-monitoring-role.yaml
aws cloudformation deploy \
  --stack-name "$STACK" \
  --template-file da-role.yaml \
  --parameter-overrides pDynatraceUrl=https://abc12345.apps.dynatrace.com pRoleExternalId=<connection-object-id> \
  --capabilities CAPABILITY_NAMED_IAM

ROLE_ARN=$(aws cloudformation describe-stacks --stack-name "$STACK" \
  --query "Stacks[0].Outputs[?OutputKey=='DynatraceMonitoringRoleArn'].OutputValue" --output text)
```

Re-running the same command later updates the role in place — `curl` re-downloads the latest template and CloudFormation does an in-place update.

### 3) Patch the role ARN into the AWS connection

```bash
dtctl update aws connection --name "my-aws-connection" --roleArn "$ROLE_ARN"
```

### 4) Create AWS monitoring config (created as disabled)

`--regions` is required (comma-separated AWS regions); `--featureSets` is optional and defaults to the extension's default set.

```bash
dtctl create aws monitoring --name "my-aws-monitoring" \
  --credentials "my-aws-connection" \
  --regions us-east-1,eu-central-1
```

### 5) Enable AWS monitoring config

The role ARN was already set in step 3, so no credential flags are needed here:

```bash
dtctl enable aws monitoring --name "my-aws-monitoring"
```

If you need to patch the role ARN at the same time:

```bash
dtctl enable aws monitoring --name "my-aws-monitoring" \
  --roleArn arn:aws:iam::123456789012:role/DynatraceMonitoringRole
```

Verify the config is now enabled:

```bash
dtctl get aws monitoring my-aws-monitoring
dtctl describe aws monitoring my-aws-monitoring
```

### 6) Discover available regions and feature sets

```bash
dtctl get aws monitoring-regions
dtctl get aws monitoring-feature-sets
```

### 7) Update AWS monitoring config (examples)

Change region filtering:

```bash
dtctl update aws monitoring --name "my-aws-monitoring" \
  --regions us-east-1,eu-central-1,ap-southeast-2
```

Change feature sets to a focused subset:

```bash
dtctl update aws monitoring --name "my-aws-monitoring" \
  --featureSets EC2_essential,RDS_essential
```

Create AWS monitoring config with explicit feature sets and regions in one shot:

```bash
dtctl create aws monitoring --name "my-aws-monitoring-explicit" \
  --credentials "my-aws-connection" \
  --regions us-east-1,eu-central-1 \
  --featureSets EC2_essential,RDS_essential
```

### 8) Delete by name or ID

```bash
dtctl delete aws monitoring my-aws-monitoring
dtctl delete aws connection my-aws-connection
```

---

## Azure Monitoring

This is the recommended fast flow for Azure onboarding with federated credentials.

### 1) Create Azure connection in Dynatrace

```bash
dtctl create azure connection --name "my-azure-connection" --type federatedIdentityCredential
```

Command output prints dynamic values you need for Azure setup:
- Issuer
- Subject (dt:connection-id/...)
- Audience

### 2) Create Service Principal and capture IDs

```bash
CLIENT_ID=$(az ad sp create-for-rbac --name "my-azure-connection" --create-password false --query appId -o tsv)
TENANT_ID=$(az account show --query tenantId -o tsv)
```

### 3) Assign Reader role on subscription scope

```bash
IAM_SCOPE="/subscriptions/00000000-0000-0000-0000-000000000000"
az role assignment create --assignee "$CLIENT_ID" --role Reader --scope "$IAM_SCOPE"
```

### 4) Create federated credential in Entra ID

Use Issuer/Subject/Audience exactly as printed by the create command:

```bash
az ad app federated-credential create --id "$CLIENT_ID" --parameters "{'name': 'fd-Federated-Credential', 'issuer': 'https://dev.token.dynatracelabs.com', 'subject': 'dt:connection-id/<connection-object-id>', 'audiences': ['<tenant>.dev.apps.dynatracelabs.com/svc-id/com.dynatrace.da']}"
```

### 5) Finalize Azure connection in Dynatrace

```bash
dtctl update azure connection --name "my-azure-connection" --directoryId "$TENANT_ID" --applicationId "$CLIENT_ID"
```

Note: immediately after step 4, Entra propagation can take a short time. If you see AADSTS70025, retry step 5 after a few seconds.

### 6) Create Azure monitoring config (created as disabled)

```bash
dtctl create azure monitoring --name "my-azure-monitoring" --credentials "my-azure-connection"
```

### 7) Enable Azure monitoring config

Enables the monitoring config and optionally updates the linked connection credentials in one step:

```bash
dtctl enable azure monitoring --name "my-azure-monitoring" --directoryId "$TENANT_ID" --applicationId "$CLIENT_ID"
```

Verify the config is now enabled:

```bash
dtctl get azure monitoring my-azure-monitoring
dtctl describe azure monitoring my-azure-monitoring
```

### 8) Update Azure monitoring config (examples)

Change location filtering to two regions:

```bash
dtctl update azure monitoring --name "my-azure-connection" \
  --locationFiltering "eastus,westeurope"
```

Change feature sets to Virtual Machines and Azure Functions:

```bash
dtctl update azure monitoring --name "my-azure-connection" \
  --featureSets "microsoft_compute.virtualmachines_essential,microsoft_web.sites_functionapp_essential"
```

Create Azure monitoring config with explicit feature sets and two locations:

```bash
dtctl create azure monitoring --name "my-azure-monitoring-explicit" \
  --credentials "my-azure-connection" \
  --locationFiltering "eastus,westeurope" \
  --featureSets "microsoft_compute.virtualmachines_essential,microsoft_web.sites_functionapp_essential"
```

---

## GCP Monitoring (Preview)

This is the recommended onboarding flow for GCP with service account impersonation.

> **Note:** The Dynatrace backend requires the GCP connection to have a service account
> configured before a monitoring config can be created against it. This means the connection
> must be updated with `--serviceAccountId` (step 3) **before** running `create gcp monitoring`
> (step 4). The `enable` step then only needs to toggle the enabled state — no need to repeat
> the service account.

All GCP commands in this section are `Preview`.

### 1) Create GCP connection in Dynatrace

```bash
dtctl create gcp connection --name "my-gcp-connection"
```

### 2) Use gcloud CLI to establish trust relation

Define variables used in snippets:

```bash
PROJECT_ID="my-project-id"
DT_GCP_PRINCIPAL="dynatrace-<tenant-id>@dtp-prod-gcp-auth.iam.gserviceaccount.com"
CUSTOMER_SA_NAME="dynatrace-integration"
CUSTOMER_SA_EMAIL="${CUSTOMER_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
```

Create customer service account:

```bash
gcloud iam service-accounts create "${CUSTOMER_SA_NAME}" \
  --project "${PROJECT_ID}" \
  --display-name "Dynatrace Integration"
```

Grant required viewer permissions to customer service account:

```bash
for ROLE in roles/browser roles/monitoring.viewer roles/compute.viewer roles/cloudasset.viewer; do
  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --quiet --format="none" \
    --member "serviceAccount:${CUSTOMER_SA_EMAIL}" \
    --role "${ROLE}"
done

```

Grant `Service Account Token Creator` to Dynatrace principal (service account impersonation):

```bash
gcloud iam service-accounts add-iam-policy-binding "${CUSTOMER_SA_EMAIL}" \
  --project "${PROJECT_ID}" \
  --member="serviceAccount:${DT_GCP_PRINCIPAL}" \
  --role="roles/iam.serviceAccountTokenCreator"
```

### 3) Update GCP connection in Dynatrace

Use the service account from step 2 and update connection:

```bash
dtctl update gcp connection --name "my-gcp-connection" --serviceAccountId "${CUSTOMER_SA_EMAIL}"
```

### 4) Create GCP monitoring config (created as disabled)

```bash
dtctl create gcp monitoring --name "my-gcp-monitoring" --credentials "my-gcp-connection"
```

### 5) Enable GCP monitoring config

The connection credentials were already set in step 3, so no credential flags are needed here:

```bash
dtctl enable gcp monitoring --name "my-gcp-monitoring"
```

Verify the config is now enabled:

```bash
dtctl describe gcp monitoring my-gcp-monitoring
```

### 6) Discover available locations and feature sets

```bash
dtctl get gcp monitoring-locations
dtctl get gcp monitoring-feature-sets
```

### 7) Update GCP monitoring config (examples)

Change location filtering to two regions:

```bash
dtctl update gcp monitoring --name "my-gcp-monitoring" \
  --locationFiltering "us-central1,europe-west1"
```

Change feature sets to a focused subset:

```bash
dtctl update gcp monitoring --name "my-gcp-monitoring" \
  --featureSets "compute_engine_essential,cloud_run_essential"
```

Create GCP monitoring config with explicit feature sets and locations:

```bash
dtctl create gcp monitoring --name "my-gcp-monitoring-explicit" \
  --credentials "my-gcp-connection" \
  --locationFiltering "us-central1,europe-west1" \
  --featureSets "compute_engine_essential,cloud_run_essential"
```

### 8) Delete by name or ID

```bash
dtctl delete gcp monitoring my-gcp-monitoring
dtctl delete gcp connection my-gcp-connection
```

---

## Live Debugger

> **Experimental:** Live Debugger support in `dtctl` is experimental. The underlying APIs and query behavior may change in future releases without notice.

For complete guidance, see [LIVE_DEBUGGER.md](LIVE_DEBUGGER.md).

> **Authentication note:** Live Debugger breakpoint operations currently require OAuth authentication. The `dev-obs:breakpoints:set` scope is supported with `dtctl auth login`, but is not currently supported with API token authentication (for example via `dtctl config set-credentials`).

### Configure target filters

`--filters` accepts both `key:value` and `key=value` pairs.

```bash
dtctl update breakpoint --filters k8s.namespace.name:prod
dtctl update breakpoints --filters k8s.namespace.name:prod,dt.entity.host:HOST-123
dtctl update breakpoint --filters k8s.namespace.name=prod,dt.entity.host=HOST-123
```

### Breakpoint lifecycle

```bash
# Create
dtctl create breakpoint OrderController.java:306

# List
dtctl get breakpoints

# Describe by location or mutable ID
dtctl describe OrderController.java:306
dtctl describe dtctl-rule-123

# Edit condition / enabled state
dtctl update breakpoint OrderController.java:306 --condition "orderId != null"
dtctl update breakpoint OrderController.java:306 --enabled false

# Delete by ID, by location, or all
dtctl delete breakpoint dtctl-rule-123
dtctl delete breakpoint OrderController.java:306
dtctl delete breakpoint --all -y
```

### Decoded snapshot output

```bash
# Simplified (variant wrappers flattened to plain values)
dtctl query "fetch application.snapshots | sort timestamp desc | limit 5" --decode-snapshots

# Full decoded tree with type annotations
dtctl query "fetch application.snapshots | sort timestamp desc | limit 5" --decode-snapshots=full

# Compose with any output format
dtctl query "fetch application.snapshots | limit 5" --decode-snapshots -o json
dtctl query "fetch application.snapshots | limit 5" --decode-snapshots -o yaml
```

`--decode-snapshots` enriches each record with `parsed_snapshot` decoded from `snapshot.data` and `snapshot.string_map`. By default, variant wrappers are simplified to plain values; use `--decode-snapshots=full` to preserve type annotations.

---

## Extensions 2.0

Extensions 2.0 manages installed extension packages and their monitoring configurations.

### List and View Extensions

```bash
# List all installed extensions
dtctl get extensions

# Filter extensions by name
dtctl get extensions --name "com.dynatrace"

# Get versions of a specific extension
dtctl get extension com.dynatrace.extension.postgres

# Wide output (shows author, feature sets, data sources)
dtctl get extension com.dynatrace.extension.postgres -o wide

# Describe an extension (schema, feature sets, data sources)
dtctl describe extension com.dynatrace.extension.postgres

# Describe a specific version
dtctl describe extension com.dynatrace.extension.postgres --version 2.9.3

# Get the monitoring configuration JSON Schema for a version
dtctl describe extension com.dynatrace.extension.postgres --version 2.9.3 --monitoring-configuration-schema

# Same, with documentation noise stripped
dtctl describe extension com.dynatrace.extension.postgres --version 2.9.3 --monitoring-configuration-schema --no-fluff

# List ActiveGate groups available for a version
dtctl describe extension com.dynatrace.extension.postgres --version 2.9.3 --active-gate-groups
```

### Installing Extensions

```bash
# Install an extension from the Dynatrace Hub (latest version)
dtctl create extension --hub-extension com.dynatrace.extension.host-monitoring

# Install a specific Hub version
dtctl create extension --hub-extension com.dynatrace.extension.host-monitoring --version 1.2.3

# Upload a custom extension from a local zip file
dtctl create extension -f my-extension.zip

# Dry run to preview without applying
dtctl create extension --hub-extension com.dynatrace.extension.host-monitoring --dry-run
```

### Monitoring Configurations

```bash
# List monitoring configurations for an extension
dtctl get extension-configs com.dynatrace.extension.postgres

# Filter by version
dtctl get extension-configs com.dynatrace.extension.postgres --version 2.9.3

# Describe a specific monitoring configuration
dtctl describe extension-config com.dynatrace.extension.postgres --config-id <object-id>
```

### Apply Monitoring Configuration

```bash
# Create a new monitoring configuration
dtctl apply extension-config com.dynatrace.extension.postgres -f config.yaml

# Create with a specific scope
dtctl apply extension-config com.dynatrace.extension.postgres -f config.yaml --scope HOST-1234

# Update an existing configuration (objectId in file)
dtctl apply extension-config com.dynatrace.extension.postgres -f config.yaml

# Apply with template variables
dtctl apply extension-config com.dynatrace.extension.postgres -f config.yaml --set env=prod

# Dry run to preview
dtctl apply extension-config com.dynatrace.extension.postgres -f config.yaml --dry-run
```

**Example monitoring configuration** (`config.yaml`):

```yaml
scope: environment
value:
  enabled: true
  description: "Host monitoring"
  featureSets:
    - host_performance
```

---

## Output Formats

All `get` and `query` commands support multiple output formats.

### Table Format (Default)

Human-readable table output:

```bash
dtctl get workflows

# Output:
# ID            TITLE              OWNER          UPDATED
# wf-123        Health Check       me             2h ago
# wf-456        Alert Handler      team-sre       1d ago
```

### JSON Format

Machine-readable JSON:

```bash
dtctl get workflow wf-123 -o json

# Output:
# {
#   "id": "wf-123",
#   "title": "Health Check",
#   "owner": "me",
#   ...
# }

# Pretty-print with jq
dtctl get workflows -o json | jq '.'
```

### YAML Format

Kubernetes-style YAML:

```bash
dtctl get workflow wf-123 -o yaml

# Output:
# id: wf-123
# title: Health Check
# owner: me
# ...
```

### Wide Format

Table with additional columns:

```bash
dtctl get workflows -o wide

# Shows more details in table format
```

### CSV Format

Spreadsheet-compatible comma-separated values output:

```bash
# Export workflows to CSV
dtctl get workflows -o csv > workflows.csv

# Export DQL query results to CSV
dtctl query "fetch logs | limit 100" -o csv > logs.csv

# Download large datasets (up to 10000 records)
dtctl query "fetch logs" --max-result-records 5000 -o csv > large_export.csv

# Import into Excel, Google Sheets, or other tools
```

**CSV Features:**
- Proper escaping for special characters (commas, quotes, newlines)
- Alphabetically sorted columns for consistency
- Handles missing values gracefully
- Compatible with all spreadsheet applications
- Perfect for data export and offline analysis

### TOON Format

[TOON (Token-Oriented Object Notation)](https://github.com/toon-format/toon) is a compact, human-readable format optimised for LLM token efficiency. It uses CSV-style tabular layout for uniform arrays and YAML-like indentation for nested objects, achieving ~40-60% fewer tokens than JSON:

```bash
# Get workflows in TOON format
dtctl get workflows -o toon

# Output:
# [#3]{id,title,owner,lastModifiedAt}:
#   wf-123,Health Check,me,2025-03-15T10:00:00Z
#   wf-456,Alert Handler,team-sre,2025-03-14T08:30:00Z
#   wf-789,Deploy Pipeline,platform,2025-03-13T14:15:00Z

# Use TOON format in agent mode for token efficiency
dtctl get workflows --agent -o toon
```

**TOON Features:**
- ~40-60% fewer tokens than JSON for tabular data
- Lossless round-trip fidelity with JSON data model
- Available in agent mode via `-A -o toon`
- Handles nested objects and arrays (unlike CSV)

### Plain Output

No colors, no interactive prompts (for scripts):

```bash
dtctl get workflows --plain
```

### Command Catalog (`dtctl commands`)

AI agents can discover all available dtctl commands, flags, and resources at runtime:

```bash
# Full catalog in JSON
dtctl commands -o json

# Compact catalog (no descriptions, no global flags)
dtctl commands --brief -o json

# Filter to a specific resource
dtctl commands workflow -o json

# Generate a Markdown how-to guide
dtctl commands howto
```

This is especially useful for agent bootstrap — run `dtctl commands --brief -o json` at the start of a session to learn what dtctl can do.

### Agent Mode (`--agent` / `-A`)

Wraps all output in a structured JSON envelope designed for AI agents and automation:

```bash
dtctl get workflows --agent

# Output:
# {
#   "ok": true,
#   "result": [...],
#   "context": {
#     "total": 5,
#     "has_more": false,
#     "verb": "get",
#     "resource": "workflow",
#     "suggestions": [
#       "Run 'dtctl describe workflow <id>' for details",
#       "Run 'dtctl exec workflow <id>' to trigger a workflow"
#     ]
#   }
# }
```

Agent mode is auto-detected when running inside an AI agent environment (e.g., GitHub Copilot, Claude Code). To opt out, pass `--no-agent`. Agent mode implies `--plain`.

```bash
# Force agent mode off in an auto-detected environment
dtctl get workflows --no-agent

# Errors are also structured
# {
#   "ok": false,
#   "error": {
#     "code": "auth_required",
#     "message": "Authentication failed",
#     "suggestions": ["Run 'dtctl auth login' to refresh your token"]
#   }
# }
```

### Pagination (--chunk-size)

Like kubectl, dtctl automatically paginates through large result sets:

```bash
# Default: fetch all results in chunks of 500 (like kubectl)
dtctl get notebooks

# Disable chunking (return only first page from API)
dtctl get notebooks --chunk-size=0

# Use smaller chunks (useful for slow connections)
dtctl get notebooks --chunk-size=100
```

---

## AI Agent Skills

dtctl includes skill files that teach AI coding assistants how to use the CLI effectively. Skills follow the [agentskills.io](https://agentskills.io) open standard and are installed as directories containing a `SKILL.md` document and a `references/` folder.

### Install Skills

```bash
# Auto-detect agent from environment and install
dtctl skills install

# Install for a specific agent
dtctl skills install --for claude
dtctl skills install --for copilot
dtctl skills install --for opencode

# Install globally (user-wide, not just this project)
dtctl skills install --for claude --global

# Overwrite an existing installation
dtctl skills install --for claude --force
```

### Cross-Client Installation

The `--cross-client` flag installs to the shared `.agents/skills/` directory defined by the agentskills.io convention. Skills installed here are automatically discovered by any compatible agent without needing per-agent installation.

```bash
# Install to <project>/.agents/skills/dtctl/
dtctl skills install --cross-client

# Install globally to ~/.agents/skills/dtctl/
dtctl skills install --cross-client --global
```

> **Note**: `--cross-client` and `--for` cannot be used together on install/uninstall. For status checks, use `--for cross-client`.

### Check Installation Status

```bash
# Show status for all agents (including cross-client)
dtctl skills status

# Check a specific agent
dtctl skills status --for claude

# Check cross-client directory
dtctl skills status --for cross-client
```

### Uninstall Skills

```bash
# Auto-detect and uninstall
dtctl skills uninstall

# Uninstall for a specific agent
dtctl skills uninstall --for copilot

# Remove from cross-client directory
dtctl skills uninstall --cross-client
```

### List Supported Agents

```bash
# Show all supported agents and their installation paths
dtctl skills install --list
```

Supported agents: **claude**, **copilot**, **cursor**, **junie**, **kiro**, **opencode**, **openclaw**.

---

## Tips & Tricks

### Name Resolution

Use resource names instead of memorizing IDs:

```bash
# Works with any command that accepts an ID
dtctl describe workflow "My Workflow"
dtctl edit dashboard "Production Overview"
dtctl delete notebook "Old Analysis"

# If multiple resources match, you'll be prompted to select
# Use --plain to require exact matches only
```

### Shell Completion

Enable tab completion for faster workflows:

**Bash:**
```bash
source <(dtctl completion bash)

# Make it permanent:
sudo mkdir -p /etc/bash_completion.d
dtctl completion bash | sudo tee /etc/bash_completion.d/dtctl > /dev/null
```

**Zsh:**
```bash
mkdir -p ~/.zsh/completions
dtctl completion zsh > ~/.zsh/completions/_dtctl
echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc
rm -f ~/.zcompdump* && autoload -U compinit && compinit
```

**Fish:**
```bash
mkdir -p ~/.config/fish/completions
dtctl completion fish > ~/.config/fish/completions/dtctl.fish
```

### Query Libraries

Organize your DQL queries in a directory:

```bash
# Create a directory for your queries (using XDG data home)
mkdir -p ~/.local/share/dtctl/queries

# Create reusable queries
cat > ~/.local/share/dtctl/queries/errors-last-hour.dql <<EOF
fetch logs
| filter status = 'ERROR'
| filter timestamp > now() - 1h
| limit {{.limit | default 100}}
EOF

# Use them easily
dtctl query -f ~/.local/share/dtctl/queries/errors-last-hour.dql
```

**Note**: dtctl follows the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) and adapts to platform conventions:

**Linux:**
- Config: `$XDG_CONFIG_HOME/dtctl` (default: `~/.config/dtctl`)
- Data: `$XDG_DATA_HOME/dtctl` (default: `~/.local/share/dtctl`)
- Cache: `$XDG_CACHE_HOME/dtctl` (default: `~/.cache/dtctl`)

**macOS:**
- Config: `~/Library/Application Support/dtctl`
- Data: `~/Library/Application Support/dtctl`
- Cache: `~/Library/Caches/dtctl`

**Windows:**
- Config: `%LOCALAPPDATA%\dtctl`
- Data: `%LOCALAPPDATA%\dtctl`
- Cache: `%LOCALAPPDATA%\dtctl`

### Export and Backup

Backup your resources regularly:

```bash
# Export all workflows
dtctl get workflows -o yaml > workflows-backup.yaml

# Export all dashboards
dtctl get dashboards -o json > dashboards-backup.json

# Export with timestamp
dtctl get workflows -o yaml > "workflows-$(date +%Y%m%d).yaml"
```

### Dry Run

Preview changes before applying:

```bash
# See what would be created/updated (shows create vs update, validates structure)
dtctl apply -f workflow.yaml --dry-run

# For dashboards/notebooks, dry-run shows:
# - Whether it will create or update
# - Document name and ID
# - Tile/section count
# - Structure validation warnings
dtctl apply -f dashboard.yaml --dry-run

# Example output:
# Dry run: would create dashboard
#   Name: SRE Service Health Overview
#   Tiles: 18
#
# Document structure validated successfully

# If there are issues, you'll see warnings:
# Warning: detected double-nested content (.content.content) - using inner content
# Warning: dashboard content has no 'tiles' field - dashboard may be empty

# See what would be deleted
dtctl delete workflow "Test Workflow" --dry-run
```

### Diff Command

Compare resources before applying changes:

```bash
# Compare local file with remote resource (auto-detects type and ID from file)
dtctl diff -f workflow.yaml

# Compare two local files
dtctl diff -f workflow-v1.yaml -f workflow-v2.yaml

# Compare two remote resources
dtctl diff workflow prod-workflow staging-workflow

# Different output formats
dtctl diff -f dashboard.yaml --semantic          # Human-readable with impact analysis
dtctl diff -f workflow.yaml -o json-patch        # RFC 6902 JSON Patch format
dtctl diff -f dashboard.yaml --side-by-side      # Split-screen comparison

# Ignore metadata changes (timestamps, versions)
dtctl diff -f workflow.yaml --ignore-metadata

# Ignore array order (useful for tasks, tiles, etc.)
dtctl diff -f dashboard.yaml --ignore-order

# Quiet mode (exit code only, for CI/CD)
dtctl diff -f workflow.yaml --quiet
# Exit codes: 0 = no changes, 1 = changes found, 2 = error

# Works with all resource types
dtctl diff -f dashboard.yaml                     # Dashboards
dtctl diff -f notebook.yaml                      # Notebooks
dtctl diff -f workflow.yaml                      # Workflows
```

### Show Diff in Apply

See exactly what changes when updating resources:

```bash
# Show diff when updating a dashboard
dtctl apply -f dashboard.yaml --show-diff

# Output shows:
# --- existing dashboard
# +++ new dashboard
# - "title": "Old Title"
# + "title": "New Title"
```

### Verbose Output

Debug issues with verbose mode:

```bash
# See API calls and responses (auth headers redacted)
dtctl get workflows -v

# Full debug output including auth headers (use with caution!)
dtctl get workflows -vv
```

### Environment Variables

Set default preferences:

```bash
# Set default output format
export DTCTL_OUTPUT=json

# Set default context
export DTCTL_CONTEXT=production

# Override with flags
dtctl get workflows -o yaml
```

### Pipeline Commands

Combine dtctl with standard Unix tools:

```bash
# Count workflows
dtctl get workflows -o json | jq '. | length'

# Find workflows by owner
dtctl get workflows -o json | jq '.[] | select(.owner=="me")'

# Extract workflow IDs
dtctl get workflows -o json | jq -r '.[].id'

# Filter and format
dtctl query "fetch logs | limit 100" -o json | \
  jq '.records[] | select(.status=="ERROR")'
```

### Large Dataset Exports

Export large datasets from DQL queries for offline analysis:

```bash
# Export up to 5000 records to CSV
dtctl query "fetch logs | filter status='ERROR'" \
  --max-result-records 5000 \
  -o csv > error_logs.csv

# Export multiple datasets with timestamps
dtctl query "fetch logs" --max-result-records 10000 -o csv > "logs-$(date +%Y%m%d-%H%M%S).csv"

# Process large CSV exports with Unix tools
dtctl query "fetch logs" --max-result-records 5000 -o csv | \
  grep "ERROR" | \
  wc -l

# Split large exports into smaller files
dtctl query "fetch logs" --max-result-records 10000 -o csv | \
  split -l 1000 - logs_part_

# Import into databases
dtctl query "fetch logs" --max-result-records 5000 -o csv > logs.csv
# Then use database import tools:
# psql -c "\COPY logs FROM 'logs.csv' CSV HEADER"
# mysql -e "LOAD DATA LOCAL INFILE 'logs.csv' INTO TABLE logs FIELDS TERMINATED BY ',' ENCLOSED BY '\"' IGNORE 1 ROWS"
```

**Performance Tips:**
- Use filters in your DQL query to reduce dataset size
- Request only the columns you need
- Consider time-based filtering for incremental exports
- CSV format is more compact than JSON for large datasets

---

## Troubleshooting

### Quick Diagnostics with `dtctl doctor`

Before diving into manual troubleshooting, run the built-in health check:

```bash
dtctl doctor
```

This runs 6 sequential checks — version, config, context, token, connectivity, and authentication — and reports pass/fail with actionable suggestions for each.

### Understanding Error Messages

dtctl provides contextual error messages with troubleshooting suggestions. When an operation fails, you'll see:

```
Failed to get workflows (HTTP 401): Authentication failed

Request ID: abc-123-def-456

Troubleshooting suggestions:
  • Token may be expired or invalid. Run 'dtctl config get-context' to check your configuration
  • Verify your API token has not been revoked in the Dynatrace console
  • Try refreshing your authentication with 'dtctl context set' and a new token
```

Common HTTP status codes and their meanings:

- **401 Unauthorized**: Token is invalid, expired, or missing
- **403 Forbidden**: Token lacks required permissions/scopes
- **404 Not Found**: Resource doesn't exist or wrong ID/name
- **429 Rate Limited**: Too many requests (dtctl auto-retries)
- **500/502/503/504**: Server error (dtctl auto-retries)

### Using Debug Mode

For detailed HTTP request/response logging, use the `--debug` flag:

```bash
# Enable full debug mode with HTTP details
dtctl get workflows --debug

# Output shows:
# ===> REQUEST <===
# GET https://abc12345.apps.dynatrace.com/platform/automation/v1/workflows
# HEADERS:
#     User-Agent: dtctl/0.12.0
#     Authorization: [REDACTED]
#     ...
# 
# ===> RESPONSE <===
# STATUS: 200 OK
# TIME: 234ms
# HEADERS:
#     Content-Type: application/json
#     ...
# BODY:
# {"workflows": [...]}
```

The `--debug` flag is equivalent to `-vv` and shows:
- Full HTTP request URL and method
- Request and response headers (auth tokens are always redacted)
- Response body
- Response time

This is useful for:
- Diagnosing API errors
- Verifying request parameters
- Checking response format
- Troubleshooting performance issues

### "config file not found"

This means you haven't set up your configuration yet. Run:

```bash
dtctl config set-context my-env \
  --environment "https://YOUR_ENV.apps.dynatrace.com" \
  --token-ref my-token

dtctl config set-credentials my-token --token "dt0s16.YOUR_TOKEN"
```

### "failed to execute workflow" or "failed to list workflows"

Check:
1. Your token has the correct permissions
2. Your environment URL is correct
3. You're using the right context

Enable debug mode to see detailed HTTP interactions:
```bash
dtctl get workflows --debug
```

### Platform Token Scopes

Your platform token needs appropriate scopes for the resources you want to manage. See [TOKEN_SCOPES.md](TOKEN_SCOPES.md) for:
- Complete scope lists for each safety level (copy-pasteable)
- Detailed breakdown by resource type
- Token creation instructions

### AI Agent Detection

If you're using dtctl through an AI coding assistant (like Claude Code, GitHub Copilot, Cursor, OpenClaw, etc.), dtctl automatically detects this and includes it in the User-Agent header for telemetry purposes. This helps improve the CLI experience for AI-assisted workflows.

The detection is automatic and doesn't affect functionality. Supported AI agents:
- Claude Code (`CLAUDECODE` env var)
- OpenCode (`OPENCODE` env var)
- GitHub Copilot (`GITHUB_COPILOT` env var)
- Cursor (`CURSOR_AGENT` env var)
- Kiro (`KIRO` env var)
- Junie (`JUNIE` env var)
- OpenClaw (`OPENCLAW` env var)
- Codeium (`CODEIUM_AGENT` env var)
- TabNine (`TABNINE_AGENT` env var)
- Amazon Q (`AMAZON_Q` env var)

---

## Next Steps

- **API Reference**: See [dev/API_DESIGN.md](dev/API_DESIGN.md) for complete command reference
- **Architecture**: Read [dev/ARCHITECTURE.md](dev/ARCHITECTURE.md) to understand how dtctl works
- **Implementation Status**: View [dev/IMPLEMENTATION_STATUS.md](dev/IMPLEMENTATION_STATUS.md) for roadmap

## Getting Help

```bash
# General help
dtctl --help

# Command-specific help
dtctl get --help
dtctl query --help

# Resource-specific help
dtctl get workflows --help

# Machine-readable command catalog (for AI agents)
dtctl commands --brief -o json
```

### Debugging Issues

Use the `--debug` flag to see detailed HTTP request/response logs:

```bash
# Full debug output
dtctl get workflows --debug

# Alternative: use -vv for the same effect
dtctl get workflows -vv
```

The debug output includes:
- HTTP method and URL
- Request/response headers (sensitive headers are redacted)
- Response body and status
- Response time

### Verbose Levels

- No flag: Normal output
- `-v`: Verbose output with operation details
- `-vv` or `--debug`: Full HTTP debug mode with request/response details

For issues and feature requests, visit the [GitHub repository](https://github.com/dynatrace/dtctl).
