---
phase: 03-apple-health-integration
plan: 02
type: execute
wave: 2
depends_on:
  - 03-01
files_modified:
  - server/services/appleHealthXml.js
  - server/routes/appleHealth.js
  - client/src/components/meatspace/tabs/ImportTab.jsx
autonomous: false
requirements:
  - HLT-03
  - HLT-04

must_haves:
  truths:
    - "Uploading a 500MB+ Apple Health XML export completes without OOM errors"
    - "XML import progress (record count) updates in real-time via WebSocket during import"
    - "XML records are normalized to the same day-file format as JSON ingest (metric names mapped from HK identifiers)"
    - "Sleep XML records with categorical values are correctly parsed into duration-based stage data"
    - "Import tab shows a working file picker and progress bar for Apple Health XML upload"
  artifacts:
    - path: "server/services/appleHealthXml.js"
      provides: "SAX streaming XML parser for Apple Health exports"
      exports: ["importAppleHealthXml"]
    - path: "server/routes/appleHealth.js"
      provides: "POST /import/xml endpoint with multer diskStorage"
      contains: "upload.single"
    - path: "client/src/components/meatspace/tabs/ImportTab.jsx"
      provides: "XML file picker with WebSocket progress bar"
      contains: "health:xml:progress"
  key_links:
    - from: "server/routes/appleHealth.js"
      to: "server/services/appleHealthXml.js"
      via: "importAppleHealthXml(filePath, io)"
      pattern: "importAppleHealthXml"
    - from: "client/src/components/meatspace/tabs/ImportTab.jsx"
      to: "socket.io client"
      via: "socket.on('health:xml:progress')"
      pattern: "health:xml:progress"
    - from: "server/services/appleHealthXml.js"
      to: "server/services/appleHealthIngest.js"
      via: "readDayFile/writeDayFile for persistence"
      pattern: "writeDayFile"
---

<objective>
Build the streaming XML import pipeline for bulk Apple Health exports (500MB+) using SAX parsing, and update the Import tab with a working file picker and real-time WebSocket progress bar.

Purpose: Enables importing years of historical Apple Health data from iOS exports without memory issues.
Output: Working XML upload at POST /api/health/import/xml, real-time progress in Import tab.
</objective>

<execution_context>
@/Users/antic/.claude/get-shit-done/workflows/execute-plan.md
@/Users/antic/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/03-apple-health-integration/03-RESEARCH.md
@.planning/phases/03-apple-health-integration/03-01-SUMMARY.md

@server/services/appleHealthIngest.js — readDayFile, writeDayFile from Plan 03-01
@server/routes/appleHealth.js — Existing routes from Plan 03-01
@server/routes/backup.js — io.emit WebSocket pattern
@client/src/components/meatspace/tabs/ImportTab.jsx — Existing import tab with Apple Health placeholder
@client/src/services/socket.js — Socket.io client

<interfaces>
<!-- From Plan 03-01 (will exist by execution time) -->

From server/services/appleHealthIngest.js:
```javascript
export async function readDayFile(dateStr) { ... }
export async function writeDayFile(dateStr, data) { ... }
export function extractDateStr(dateString) { ... }
```

From server/routes/appleHealth.js:
```javascript
// Router mounted at /api/health with existing POST /ingest and GET endpoints
// Add POST /import/xml to this same router
```

WebSocket pattern (from backup.js):
```javascript
const io = req.app.get('io');
// Service emits: io.emit('health:xml:progress', { processed, total })
// Service emits: io.emit('health:xml:complete', { days, records })
```

Apple Health XML Record structure:
```xml
<Record type="HKQuantityTypeIdentifierStepCount" value="432" startDate="2024-01-15 10:00:00 -0800" endDate="..." unit="count" sourceName="iPhone"/>
<Record type="HKCategoryTypeIdentifierSleepAnalysis" value="HKCategoryValueSleepAnalysisAsleepDeep" startDate="..." endDate="..."/>
```
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create SAX XML streaming service and add XML import route</name>
  <files>
    server/services/appleHealthXml.js
    server/routes/appleHealth.js
  </files>
  <action>
**Install dependencies first:**
```bash
cd server && npm install sax multer
```

**1. Create server/services/appleHealthXml.js:**

Streaming SAX XML parser for Apple Health exports. Key components:

**XML-to-metric name mapping:**
```javascript
const XML_TO_METRIC_NAME = {
  'HKQuantityTypeIdentifierStepCount': 'step_count',
  'HKQuantityTypeIdentifierHeartRate': 'heart_rate',
  'HKQuantityTypeIdentifierHeartRateVariabilitySDNN': 'heart_rate_variability_sdnn',
  'HKCategoryTypeIdentifierSleepAnalysis': 'sleep_analysis',
  // All other types: store with original HK identifier (lowercase normalized)
};
```

**Sleep stage mapping:**
```javascript
const SLEEP_STAGE_MAP = {
  'HKCategoryValueSleepAnalysisAsleepDeep': 'deep',
  'HKCategoryValueSleepAnalysisAsleepREM': 'rem',
  'HKCategoryValueSleepAnalysisAsleepCore': 'core',
  'HKCategoryValueSleepAnalysisAwake': 'awake',
  'HKCategoryValueSleepAnalysisInBed': 'inBed',
  'HKCategoryValueSleepAnalysisAsleep': 'asleep'  // Legacy pre-iOS 16
};
```

**`normalizeXmlRecord(node)`:** Pure function. Takes SAX opentag node with attributes. Returns `{ metricName, dateStr, dataPoint }` or null for invalid records.

- Extract `type`, `value`, `startdate`, `enddate`, `unit`, `sourcename` from `node.attributes` (lowercase because `sax.createStream(false, { lowercase: true })`).
- Map `type` via `XML_TO_METRIC_NAME`, falling back to lowercase type string for unknown types.
- Extract dateStr using `extractDateStr(startdate)` from appleHealthIngest.js. Return null if invalid.
- For sleep analysis (`type === 'hkcategorytypeidentifiersleepanalysis'` — lowercase): Calculate duration in hours from `(new Date(enddate) - new Date(startdate)) / 3600000`. Map `value` via `SLEEP_STAGE_MAP`. Create data point with `{ date: startdate, stage: mappedStage, durationHours: duration }`.
- For numeric types: Create data point with `{ date: startdate, qty: parseFloat(value), unit: unit ?? null, src: sourcename ?? null }`. If `type` is heart rate, also add `end: enddate`.
- Skip records where `startdate` is missing.

**`importAppleHealthXml(filePath, io = null)`:** Main export.

- Create `sax.createStream(false, { lowercase: true })` (non-strict mode per research pitfall 1).
- Accumulate records into `dayBuckets = {}` keyed by dateStr, then by metricName.
- On `opentag`: If `node.name !== 'record'`, return. Call `normalizeXmlRecord(node)`. If result is null, skip. Accumulate into dayBuckets. Increment `processedRecords`. Every 10000 records, emit `io.emit('health:xml:progress', { processed: processedRecords })` and log `console.log('🍎 XML import progress: ${processedRecords} records')`.
- On `error`: Clear `this._parser.error = null`, call `this._parser.resume()` (per research pitfall 1 — Apple Health XML can have minor malformations). Log error with `console.error('❌ XML parse error at record ${processedRecords}: ${e.message}')`.
- Pipe `createReadStream(filePath)` to saxStream. Await stream end via Promise.
- **After parsing completes:** For step_count in dayBuckets, aggregate: sum all `qty` values per day into a single entry (per research pitfall 3). For sleep_analysis, aggregate stage durations per day into a single summary: `{ date, totalSleep, deep, rem, core, awake }` (sum durations by stage).
- Write all day files sequentially (not during SAX event loop — per research pitfall 7). For each dateStr in dayBuckets: read existing day file, merge new metrics (dedup by date string within each metric), write back.
- Delete temp file: `unlink(filePath, () => {})`.
- Emit `io.emit('health:xml:complete', { days: Object.keys(dayBuckets).length, records: processedRecords })`.
- Log `console.log('🍎 XML import complete: ${processedRecords} records across ${days} days')`.
- Return `{ days: daysCount, records: processedRecords }`.

Export `normalizeXmlRecord` (for testing) and `importAppleHealthXml`.

**2. Add XML import route to server/routes/appleHealth.js:**

At the top of the file, add imports:
```javascript
import multer from 'multer';
import { tmpdir } from 'os';
import { importAppleHealthXml } from '../services/appleHealthXml.js';
```

Configure multer with diskStorage (NOT memoryStorage — per research, avoids OOM on 500MB files):
```javascript
const upload = multer({
  storage: multer.diskStorage({
    destination: (req, file, cb) => cb(null, tmpdir()),
    filename: (req, file, cb) => cb(null, `apple-health-${Date.now()}.xml`)
  }),
  limits: { fileSize: 2 * 1024 * 1024 * 1024 }, // 2GB max
  fileFilter: (req, file, cb) => {
    if (file.mimetype === 'text/xml' || file.mimetype === 'application/xml' || file.originalname.endsWith('.xml')) {
      cb(null, true);
    } else {
      cb(new ServerError('Only XML files are accepted', 400));
    }
  }
});
```

Add route:
```javascript
// POST /api/health/import/xml
router.post('/import/xml', upload.single('file'), asyncHandler(async (req, res) => {
  const io = req.app.get('io');
  const filePath = req.file?.path;
  if (!filePath) throw new ServerError('No file uploaded', 400);
  const result = await importAppleHealthXml(filePath, io);
  res.json(result);
}));
```
  </action>
  <verify>
    <automated>cd server && node -e "import('./services/appleHealthXml.js').then(m => console.log(typeof m.importAppleHealthXml === 'function' && typeof m.normalizeXmlRecord === 'function' ? 'PASS' : 'FAIL'))"</automated>
  </verify>
  <done>
    - SAX streaming parser handles Apple Health XML without loading file into memory
    - XML metric types mapped to normalized metric names matching JSON ingest format
    - Sleep categorical values parsed into duration-based stage data
    - Step counts aggregated per day (not raw sensor readings)
    - Non-strict SAX mode handles malformed XML gracefully
    - WebSocket progress emitted every 10000 records
    - Temp file cleaned up after import
    - POST /import/xml route uses multer diskStorage with 2GB limit
  </done>
</task>

<task type="auto">
  <name>Task 2: Update Import tab with Apple Health XML file picker and progress bar</name>
  <files>
    client/src/components/meatspace/tabs/ImportTab.jsx
    client/src/services/api.js
  </files>
  <action>
**1. Add XML upload API function in client/src/services/api.js:**

Add to the "// Apple Health" section (created in Plan 03-01):
```javascript
export const uploadAppleHealthXml = (file) => {
  const formData = new FormData();
  formData.append('file', file);
  return fetch(`${API_BASE}/health/import/xml`, {
    method: 'POST',
    body: formData
    // No Content-Type header — browser sets multipart boundary automatically
  }).then(async res => {
    if (!res.ok) {
      const err = await res.json().catch(() => ({ error: res.statusText }));
      throw new Error(err.error || res.statusText);
    }
    return res.json();
  });
};
```

Note: This uses `fetch` directly instead of the `request` helper because `request` sets `Content-Type: application/json` which conflicts with `multipart/form-data`. The FormData upload needs the browser to set the Content-Type with the correct boundary.

**2. Replace the Apple Health placeholder in ImportTab.jsx:**

Remove the disabled placeholder section (the `<div>` with `opacity-60` and "Coming soon" text, lines ~134-143 of current file).

Replace with a working Apple Health XML import section:

- Import `socket` from `'../../../services/socket'` and `uploadAppleHealthXml` from api.
- Add state: `xmlImporting`, `xmlProgress`, `xmlResult`, `xmlError`, `xmlFileName`.
- Add `xmlFileInputRef` useRef.
- Add useEffect for WebSocket listeners:
  ```javascript
  useEffect(() => {
    const onProgress = ({ processed }) => setXmlProgress(processed);
    const onComplete = (result) => { setXmlResult(result); setXmlImporting(false); };
    socket.on('health:xml:progress', onProgress);
    socket.on('health:xml:complete', onComplete);
    return () => {
      socket.off('health:xml:progress', onProgress);
      socket.off('health:xml:complete', onComplete);
    };
  }, []);
  ```
- Add `handleXmlFileSelect` handler: Get file from input, validate `.xml` extension, set `xmlImporting(true)`, `xmlProgress(0)`, clear errors/results, call `uploadAppleHealthXml(file)`. On error, set xmlError. On success, result comes via WebSocket `onComplete`.

UI structure (matching the existing TSV import card style):
- Card with `bg-port-card border border-port-border rounded-xl p-6`.
- Header: Apple icon (use `Upload` from lucide, or `Apple` if available — check lucide-react; if not, use `HeartPulse`) + "Apple Health XML Import" in uppercase tracking-wider.
- Description: "Import your Apple Health export. On your iPhone: Settings > Health > Export All Health Data. Extract the ZIP and select the export.xml file."
- Instruction note: "Extract the ZIP file first — upload the export.xml file directly."
- File picker button: Same style as TSV button. Accepts `.xml` files only.
- During import: Show progress bar with `bg-port-accent` fill and record count text: "{xmlProgress.toLocaleString()} records processed". Use a pulsing/animated progress bar since total count is unknown (indeterminate style with the record count as text).
- On complete: Show success card with days and records counts (matching TSV success card style).
- On error: Show error card (matching TSV error card style).
  </action>
  <verify>
    <automated>cd server && npm test && cd ../client && npm run build</automated>
  </verify>
  <done>
    - Apple Health placeholder replaced with working XML file picker
    - File picker accepts .xml files only
    - Progress bar shows live record count during import via WebSocket
    - Success state shows days and records imported
    - Error state displays error message
    - Import follows existing TSV import card styling
    - Instructions tell user to extract ZIP and upload export.xml
    - Client builds cleanly with new component
  </done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 3: Verify XML import and ingest pipeline</name>
  <files>none (verification only)</files>
  <action>
Human verification checkpoint. Verify:
1. The Import tab shows the Apple Health XML Import section (replacing the old placeholder)
2. The file picker accepts only .xml files
3. POST /api/health/ingest works with curl test data
4. Day-partitioned files are created in data/health/
5. Dedup works on repeated ingest
  </action>
  <verify>
    <automated>curl -s -X POST http://localhost:5554/api/health/ingest -H 'Content-Type: application/json' -d '{"data":{"metrics":[{"name":"step_count","units":"count","data":[{"date":"2024-01-15 08:30:00 -0800","qty":8432}]}]}}' | node -e "process.stdin.on('data',d=>{const r=JSON.parse(d);console.log(r.recordsIngested>0?'PASS':'FAIL')})"</automated>
  </verify>
  <done>XML import pipeline and JSON ingest verified end-to-end, Import tab UI confirmed</done>
  <what-built>Apple Health XML import pipeline with streaming SAX parser, WebSocket progress, and Import tab UI</what-built>
  <how-to-verify>
    1. Navigate to MeatSpace page and find the Import tab (check if it is accessible from the Lifestyle tab or a separate import section)
    2. Verify the Apple Health XML Import section replaces the old placeholder
    3. Verify the file picker only accepts .xml files
    4. If you have an Apple Health export.xml available, test importing it. Otherwise, verify the UI renders correctly with proper styling.
    5. Verify `curl -X POST http://localhost:5554/api/health/ingest -H 'Content-Type: application/json' -d '{"data":{"metrics":[{"name":"step_count","units":"count","data":[{"date":"2024-01-15 08:30:00 -0800","qty":8432}]}]}}'` returns success with recordsIngested count
    6. Verify `ls data/health/` shows `2024-01-15.json` was created
    7. Re-run the same curl command and verify `recordsSkipped` increases (dedup working)
  </how-to-verify>
  <resume-signal>Type "approved" or describe issues</resume-signal>
</task>

</tasks>

<verification>
1. `npm install` in server completes with `sax` and `multer` added to package.json
2. POST /api/health/ingest accepts and persists Health Auto Export JSON
3. POST /api/health/import/xml accepts XML upload via multipart form
4. Import tab shows Apple Health XML import section with progress bar
5. `cd server && npm test` — all existing tests pass
6. `cd client && npm run build` — client compiles cleanly
</verification>

<success_criteria>
- Apple Health XML files (500MB+) can be imported without OOM via SAX streaming
- Import progress updates in real-time via WebSocket (record count every 10k records)
- XML metric types are normalized to match JSON ingest metric names
- Sleep XML categorical values are correctly converted to duration data
- Import tab has working file picker replacing the disabled placeholder
- All existing tests pass
- Client builds without errors
</success_criteria>

<output>
After completion, create `.planning/phases/03-apple-health-integration/03-02-SUMMARY.md`
</output>
