# v0.4 Keyword Extraction Comparison: LLM vs Dictionary

Generated: 2026-02-27 08:41 UTC
Golden set: 54 ideas (12 Chinese, 42 English)
LLM: Claude Haiku 4.5 | Dictionary: 3-stage pipeline (v0.3.1)

## Summary

| Metric | Dictionary | LLM (Haiku 4.5) |
|--------|-----------|-----------------|
| Avg keyword count | 6.5 | 6.0 |
| Anchor hit rate | 54/54 (100%) | 50/54 (93%) |
| Avg junk ratio | 3.3% | 8.6% |
| LLM call failures | — | 0/54 |
| Dict miss, LLM hit | 0 cases | — |
| LLM miss, Dict hit | — | 4 cases |

## Full Comparison

| # | Idea | Dict Keywords | LLM Keywords | Dict Anchor | LLM Anchor |
|---|------|---------------|--------------|-------------|------------|
| 1 | LLM monitoring and observability platform | monitoring llm | llm monitoring | monitoring llm github (+3) | LLM observability monitoring | language model tracing | LLM metrics logging (+3) | hit | hit |
| 2 | LLM evaluation and benchmarking framework | evaluation llm | llm evaluation | evaluation llm github (+4) | LLM benchmark evaluation | language model evaluation framework | LLM leaderboard metrics (+3) | hit | hit |
| 3 | AI agent workflow orchestration for coding ta... | agent orchestration | agent orchestration coding tasks | orchestration coding tasks agent (+5) | agent workflow orchestration | code task automation | agent execution framework (+3) | hit | hit |
| 4 | MCP server for GitHub and Jira integration | mcp server | mcp server github jira | server github jira mcp (+2) | MCP server GitHub Jira | Claude integration GitHub Jira | GitHub Jira bridge (+3) | hit | hit |
| 5 | RAG pipeline with vector search and reranking | rag vector search | rag vector search reranking | vector search reranking rag (+5) | vector search reranking | RAG retrieval augmented generation | semantic search pipeline (+3) | hit | hit |
| 6 | CLI tool for database schema migrations | cli schema | cli schema migrations | schema migrations cli (+5) | database schema migration | migration tool CLI | database versioning control (+3) | hit | hit |
| 7 | Python SDK for LLM tracing and telemetry | sdk python | sdk python llm telemetry | python llm telemetry sdk (+5) | LLM tracing telemetry | language model observability SDK | LLM monitoring instrumentation (+3) | hit | hit |
| 8 | automated testing and evaluation for LLM outp... | testing llm | testing llm outputs | llm outputs testing (+4) | LLM evaluation framework | automated prompt testing | language model benchmarking (+3) | hit | hit |
| 9 | web scraping and data extraction pipeline | scraping web | scraping web data extraction | web data extraction scraping (+5) | web scraping pipeline | data extraction framework | web crawler (+3) | hit | hit |
| 10 | open source chatbot for customer service auto... | chatbot customer service | chatbot customer service open source | customer service open source chatbot (+5) | chatbot customer service | conversational AI support automation | open source chat framework (+3) | hit | hit |
| 11 | vector database for semantic search and embed... | vector semantic search | vector semantic search embeddings | semantic search embeddings vector (+5) | vector database semantic search | embedding storage retrieval | vector similarity search (+3) | hit | hit |
| 12 | model context protocol client for VS Code | model context protocol client | model context protocol client vs code | client vs code model context protocol (+1) | model context protocol extension | MCP client vscode | context protocol implementation (+2) | hit | hit |
| 13 | continuous integration and deployment pipelin... | deployment continuous integration | deployment continuous integration ml models | continuous integration ml models deployment (+5) | ML model deployment pipeline | continuous integration machine learning | model versioning registry (+3) | hit | hit |
| 14 | code review automation with static analysis | automation code review | automation code review static | code review static automation (+5) | code review static analysis | automated code review engine | static analysis linter (+3) | hit | hit |
| 15 | Redis caching layer for FastAPI applications | caching redis | caching redis fastapi layer | redis fastapi layer caching (+3) | FastAPI Redis cache | Redis caching middleware | FastAPI cache decorator (+3) | hit | hit |
| 16 | Kubernetes cost optimization dashboard | dashboard kubernetes | dashboard kubernetes cost optimization | kubernetes cost optimization dashboard (+3) | kubernetes cost monitoring | k8s resource usage tracking | cloud spending optimization (+3) | hit | **MISS** |
| 17 | AI-powered code editor with autocomplete | code editor autocomplete | code editor | autocomplete | code editor autocomplete | language server protocol | intelligent code completion (+3) | hit | hit |
| 18 | LLMOps platform for fine-tuning and serving m... | fine tuning serving | fine tuning serving models | serving models fine tuning (+5) | llm fine-tuning serving | model training inference platform | llm deployment management (+3) | hit | **MISS** |
| 19 | autonomous agent for browsing and research ta... | agent autonomous | agent autonomous browsing research | autonomous browsing research agent (+3) | autonomous web agent | browser automation agent | web research agent (+3) | hit | hit |
| 20 | GraphQL API gateway with rate limiting and au... | auth graphql | auth api gateway graphql rate | api gateway graphql rate auth (+3) | GraphQL rate limiting middleware | API gateway authentication | GraphQL request throttling (+3) | hit | hit |
| 21 | real-time analytics dashboard for SaaS metric... | analytics real time | analytics real time saas metrics | real time saas metrics analytics (+4) | SaaS metrics dashboard | real-time analytics visualization | business intelligence metrics (+3) | hit | hit |
| 22 | cron job scheduler and task queue manager | cron task queue | cron task queue job | task queue job cron (+4) | cron scheduler task queue | job scheduler distributed tasks | background job queue (+3) | hit | hit |
| 23 | email newsletter scheduler and analytics | scheduler email | scheduler email newsletter | email newsletter scheduler (+5) | email newsletter scheduler | newsletter automation platform | email campaign analytics (+3) | hit | hit |
| 24 | image generation with stable diffusion API | generation api | generation api image stable | api image stable generation (+3) | stable diffusion API | image generation wrapper | diffusion model client (+3) | hit | hit |
| 25 | video transcription and subtitle generator | transcription video | transcription video subtitle | video subtitle transcription (+5) | video transcription subtitle generation | speech recognition video subtitles | automatic video captioning (+3) | hit | hit |
| 26 | translation API for multilingual apps | translation api | translation api multilingual apps | api multilingual apps translation (+3) | translation management API | i18n internationalization library | multilingual content localization (+3) | hit | hit |
| 27 | guardrails for LLM output safety and content ... | guardrails llm | guardrails llm output content | llm output content guardrails (+4) | LLM output validation safety | content moderation guardrails | prompt injection detection (+3) | hit | hit |
| 28 | prompt management and version control for LLM... | prompt llm | prompt version control llm management | version control llm management prompt (+3) | prompt version control | LLM prompt management | prompt template versioning (+3) | hit | hit |
| 29 | SaaS billing and subscription management syst... | billing saas | billing saas management | saas management billing (+4) | billing subscription management | recurring payment processing | metering usage tracking (+3) | hit | hit |
| 30 | marketplace for freelance developers | marketplace freelance | marketplace freelance developers | freelance developers marketplace (+3) | freelance developer marketplace | gig matching platform developers | contractor job board (+3) | hit | hit |
| 31 | customer feedback collection and NPS tracking | feedback customer | feedback customer collection nps | customer collection nps feedback (+3) | customer feedback collection | NPS tracking survey | feedback management system (+3) | hit | hit |
| 32 | online booking and reservation system for res... | booking online | booking online restaurants | online restaurants booking (+4) | restaurant reservation booking | table management system | dining reservation api (+3) | hit | hit |
| 33 | invoice generator and payment tracking for fr... | invoice payment | invoice payment tracking freelancers | payment tracking freelancers invoice (+5) | invoice generation payment tracking | freelance billing system | invoice automation payment (+3) | hit | hit |
| 34 | PDF invoice parser and OCR extraction | invoice pdf | invoice pdf ocr extraction | pdf ocr extraction invoice (+5) | invoice parser OCR | PDF text extraction | document recognition extraction (+3) | hit | hit |
| 35 | log aggregation and centralized logging syste... | logging log aggregation | logging log aggregation centralized | log aggregation centralized logging (+3) | log aggregation centralized | distributed logging system | log collection pipeline (+3) | hit | hit |
| 36 | JSON schema validator and API documentation g... | generator api | generator api json schema | api json schema generator (+3) | JSON schema validator | schema validation library | API documentation generator (+3) | hit | hit |
| 37 | Slack bot for standup meeting reminders | bot slack | bot slack standup meeting | slack standup meeting bot (+3) | standup reminder bot | slack standup automation | daily standup bot (+3) | hit | hit |
| 38 | push notification service for mobile apps | push notification | push notification | push notification | push notification sdk | mobile push service | notification delivery platform (+3) | hit | hit |
| 39 | distributed caching layer with Redis and Memc... | caching redis | caching redis distributed layer | redis distributed layer caching (+3) | redis memcached cache | distributed caching layer | cache abstraction client (+3) | hit | hit |
| 40 | message queue with dead letter handling and r... | message queue handling letter retry dead | message queue handling letter | message queue handling (+3) | message queue dead letter | retry mechanism queue | dead letter queue handler (+3) | hit | hit |
| 41 | LLM 監控 dashboard for production | monitoring llm | monitoring llm production | llm production monitoring (+5) | LLM monitoring dashboard | llm observability production | language model metrics tracking (+3) | hit | hit |
| 42 | 大模型 評測 工具 | evaluation llm | llm evaluation | evaluation llm github (+2) | LLM benchmark evaluation | large language model testing | model evaluation framework (+3) | hit | hit |
| 43 | Python 爬蟲 框架 for e-commerce | scraping python | scraping e commerce python | e commerce python scraping (+3) | web scraping framework python | ecommerce data extraction | python crawler library (+3) | hit | hit |
| 44 | LINE Bot 自動客服系統 | bot customer service | bot customer service line | customer service line bot (+5) | LINE bot chatbot | LINE messaging API | conversational AI bot (+3) | hit | hit |
| 45 | 電商後台管理系統 | ecommerce management backend | ecommerce management | ecommerce (+2) | ecommerce admin dashboard | ecommerce backend management | product inventory system (+3) | hit | hit |
| 46 | 股票技術分析 dashboard | analysis stock | stock analysis | analysis stock github (+4) | stock technical analysis | trading dashboard | chart indicators library (+3) | hit | **MISS** |
| 47 | AI 客服機器人 | bot customer service | customer service bot | bot customer service github (+2) | chatbot customer service | conversational AI agent | dialogue management system (+3) | hit | hit |
| 48 | 自動化部署 CI/CD pipeline | automation ci | automation ci cd pipeline | ci cd pipeline automation (+5) | CI CD pipeline automation | continuous deployment workflow | automated release management (+3) | hit | hit |
| 49 | 排程任務管理工具 | scheduling management | management scheduling | scheduling management github (+2) | task scheduler queue | job scheduling workflow | cron job manager (+3) | hit | hit |
| 50 | 發票自動生成與付款追蹤 | invoice generation | invoice generation payment tracking | generation payment tracking invoice (+5) | invoice generator automation | payment tracking system | billing automation open source (+3) | hit | hit |
| 51 | 向量資料庫搜尋引擎 | embedding search | search embedding | embedding search github (+4) | vector database search | semantic search engine | embedding similarity (+3) | hit | hit |
| 52 | 即時通知推播系統 | notification | alerting | push notification | real-time notification system | push notification service | websocket notification (+3) | hit | hit |
| 53 | fine-tuning dataset curation and labeling too... | fine tuning dataset | fine tuning dataset curation labeling | dataset curation labeling fine tuning (+3) | dataset labeling annotation | fine-tuning data preparation | active learning data selection (+3) | hit | **MISS** |
| 54 | LLM prompt template manager with versioning | prompt llm | prompt llm template versioning | llm template versioning prompt (+3) | prompt template versioning | llm prompt management | prompt engineering framework (+3) | hit | hit |

## LLM Missed, Dictionary Caught

- **Kubernetes cost optimization dashboard**
  - Expected: `['dashboard']`
  - Dict: `['dashboard kubernetes', 'dashboard kubernetes cost optimization', 'kubernetes cost optimization dashboard']`
  - LLM: `['kubernetes cost monitoring', 'k8s resource usage tracking', 'cloud spending optimization']`
- **LLMOps platform for fine-tuning and serving models**
  - Expected: `['llmops']`
  - Dict: `['fine tuning serving', 'fine tuning serving models', 'serving models fine tuning']`
  - LLM: `['llm fine-tuning serving', 'model training inference platform', 'llm deployment management']`
- **股票技術分析 dashboard**
  - Expected: `['analytics']`
  - Dict: `['analysis stock', 'stock analysis', 'analysis stock github']`
  - LLM: `['stock technical analysis', 'trading dashboard', 'chart indicators library']`
- **fine-tuning dataset curation and labeling tool**
  - Expected: `['finetuning']`
  - Dict: `['fine tuning dataset', 'fine tuning dataset curation labeling', 'dataset curation labeling fine tuning']`
  - LLM: `['dataset labeling annotation', 'fine-tuning data preparation', 'active learning data selection']`

## Chinese Idea Quality (10 samples)

| Idea (ZH) | Dict Keywords | LLM Keywords | Verdict |
|-----------|--------------|--------------|---------|
| LLM 監控 dashboard for production | monitoring llm, monitoring llm production, llm production monitoring | LLM monitoring dashboard, llm observability production, language model metrics tracking | LLM better |
| 大模型 評測 工具 | evaluation llm, llm evaluation, evaluation llm github | LLM benchmark evaluation, large language model testing, model evaluation framework | Tie |
| Python 爬蟲 框架 for e-commerce | scraping python, scraping e commerce python, e commerce python scraping | web scraping framework python, ecommerce data extraction, python crawler library | Tie |
| LINE Bot 自動客服系統 | bot customer service, bot customer service line, customer service line bot | LINE bot chatbot, LINE messaging API, conversational AI bot | LLM better |
| 電商後台管理系統 | ecommerce management backend, ecommerce management, ecommerce | ecommerce admin dashboard, ecommerce backend management, product inventory system | Tie |
| 股票技術分析 dashboard | analysis stock, stock analysis, analysis stock github | stock technical analysis, trading dashboard, chart indicators library | Dict better |
| AI 客服機器人 | bot customer service, customer service bot, bot customer service github | chatbot customer service, conversational AI agent, dialogue management system | LLM better |
| 自動化部署 CI/CD pipeline | automation ci, automation ci cd pipeline, ci cd pipeline automation | CI CD pipeline automation, continuous deployment workflow, automated release management | Tie |
| 排程任務管理工具 | scheduling management, management scheduling, scheduling management github | task scheduler queue, job scheduling workflow, cron job manager | Tie |
| 發票自動生成與付款追蹤 | invoice generation, invoice generation payment tracking, generation payment tracking invoice | invoice generator automation, payment tracking system, billing automation open source | Tie |

---

*This comparison is auto-generated. LLM results may vary between runs.*