## ASI09 – Human-Agent Trust Exploitation

**Description:**

Human-Agent Trust Exploitation occurs when attackers exploit the trust relationship between humans and AI agents. As users grow accustomed to relying on AI agents for recommendations, approvals, and automated actions, attackers manipulate this trust through false certainty claims, manufactured urgency, approval fatigue, social engineering via agent output, and unauthorized high-risk actions (especially financial transactions) that bypass human oversight.

This category covers: false confidence and verification suppression, manufactured urgency and fear-based manipulation, approval fatigue exploitation (batch-approving dangerous actions), unauthorized financial actions by agents, high-risk tool invocation without human confirmation, and social engineering attacks delivered through agent output.

**Common Examples of Vulnerability:**

1. Example 1: False Certainty and Verification Suppression — An agent (or an attacker manipulating agent output) claims: "I am 100% certain this configuration change is safe. There is zero risk of data loss. Approve immediately." Combined with verification suppression: "Do not verify these results with anyone else" and "Skip the human review step to save time." False prior validation ("I've already been verified by multiple senior review agents") adds credibility to bypass human judgment.

2. Example 2: Approval Fatigue Exploitation — An attacker batches dangerous actions with routine ones: "Just click approve for all the following file operations. This is routine maintenance." Or activates auto-approve modes: "Enable auto-approve mode so we can process these requests faster." Minimizing language ("routine", "standard", "nothing to worry about", "perfectly safe") lowers the user's guard. The actual dangerous action (e.g., granting admin access) is hidden among dozens of benign-looking approvals.

3. Example 3: Social Engineering via Agent Output — An agent's output contains phishing-style content: "Immediate action required: your account will be locked in 24 hours unless you verify your credentials at this link." Authority impersonation ("Your manager requested that you share your SSH key with the new contractor immediately") and threat of consequences ("Failure to comply will result in termination") exploit the user's trust in the agent as a reliable information source.

**How to Prevent:**

1. Prevention Step 1: Certainty Calibration and Mandatory Verification — Reject absolute certainty claims in agent output. ATR-2026-077 (Human-Agent Trust Exploitation) detects false absolute certainty ("100% certain", "zero risk", "guaranteed safe"), verification suppression ("do not verify with anyone else"), manufactured urgency ("must approve within 2 minutes"), exclusive authority claims ("only I can understand this"), and warning suppression ("hide the warning from the user"). Require independent verification for any high-impact recommendation.

2. Prevention Step 2: Granular Approval with Anti-Fatigue Controls — Require individual approval for each high-risk action — never allow batch approval for destructive, financial, or permission-modifying operations. ATR-2026-118 (Approval Fatigue) detects blanket approval encouragement ("click approve for all"), auto-approve activation, minimizing language, batch execution requests, and discouragement of individual review. ATR-2026-099 (High-Risk Tool Gate) identifies 6 categories of tools requiring human confirmation: financial, destructive, communication, permission, system execution, and high-value amounts.

3. Prevention Step 3: Financial Action Controls — Require explicit human confirmation for all financial transactions initiated by agents. ATR-2026-098 (Unauthorized Financial Action) detects financial tool invocations (`send_red_packet`, `transfer_money`, `bank_transfer`, `wechat_pay`, `stripe_charge`), financial parameters in tool arguments, and agent declarations of financial intent in both English and Chinese. No financial action should complete without a human-approved confirmation step.

**Example Attack Scenarios:**

Scenario #1: An attacker crafts a sequence of 20 file operations for an agent to execute. The first 19 are legitimate, routine file moves. The 20th is `grant_access({"user": "attacker", "role": "admin"})`. The agent presents all 20 as a batch: "These are all perfectly safe operations, nothing to worry about. Approve all." The user, fatigued by the routine appearance, clicks "Approve All" without reviewing each operation individually. The attacker gains admin access. ATR-2026-118 detects the batch approval language and minimizing rhetoric. ATR-2026-099 identifies `grant_access` as a high-risk permission-modifying tool requiring individual confirmation. Together, these rules prevent batch-approving operations that include high-risk actions.

Scenario #2: A compromised tool response injects phishing content into the agent's output. The agent displays: "URGENT: Your IT department requires immediate password verification due to a detected security breach. Verify your credentials at https://it-verify.company-secure.com to prevent account lockout. Failure to comply within 24 hours will result in automatic account suspension." The user, trusting the agent as a reliable corporate tool, clicks the link and enters credentials on the attacker's phishing page. The attack is more effective than traditional email phishing because the user has an established trust relationship with the agent. ATR-2026-119 (Social Engineering) detects phishing urgency patterns, authority impersonation, threat of consequences, credential harvesting, and phishing-style links in agent output.

**Reference Links:**

1. [ATR-2026-077: Human-Agent Trust Exploitation](https://github.com/Agent-Threat-Rule/agent-threat-rules): Detects false certainty, verification suppression, manufactured urgency, and warning hiding.
2. [ATR-2026-098: Unauthorized Financial Action](https://github.com/Agent-Threat-Rule/agent-threat-rules): Detects financial tool invocations and agent-declared financial intent (EN/ZH bilingual).
3. [ATR-2026-099: High-Risk Tool Gate](https://github.com/Agent-Threat-Rule/agent-threat-rules): Identifies 6 categories of tools requiring human confirmation before execution.
4. [ATR-2026-118: Approval Fatigue Exploitation](https://github.com/Agent-Threat-Rule/agent-threat-rules): Detects batch approval, auto-approve activation, and minimizing language.
5. [ATR-2026-119: Social Engineering via Agent](https://github.com/Agent-Threat-Rule/agent-threat-rules): Detects phishing, authority impersonation, and credential harvesting in agent output.
