convert/docs/superpowers/plans/2026-07-31-markdown-conversion.md

# Markdown (.md) Document Conversion Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add Markdown (`.md`) as a document format convertible in both directions against `html`, `txt`, `docx`, and `pdf`.

**Architecture:** All new conversion functions live in the existing `src/converters/document.js` and are registered through the existing `registerDocumentConverters()` function via the existing `register()` call in `src/converters/registry.js` — no registry/API/frontend changes are needed because `GET /api/formats` and `POST /api/jobs` are already format-agnostic (verified by reading `src/app.js`: they call `listTargetFormats`/`resolveConverter` generically). `md->html` and `html->md` are direct library calls; `md->pdf` and `pdf->md` reuse the existing `renderHtmlToPdf` / `extractPdfPageTexts` helpers; `docx->md` reuses the existing `mammoth.convertToHtml` call piped through the new HTML→MD library; `md->docx` is the only genuinely new algorithm (a `markdown-it` token-stream walk into `docx` `Paragraph`/`TextRun` objects); `md<->txt` is a raw file copy.

**Tech Stack:** Node.js, `markdown-it` (Markdown → HTML, v15.0.0), `turndown` (HTML → Markdown, v7.2.4), existing `docx`/`mammoth`/`puppeteer`/`pdfjs-dist` already in the project.

## Global Constraints

- MIME type for `.md` is `text/markdown` (per the spec's MIME plumbing section).
- `.md` files are undetectable by magic bytes — `resolveInputFormat` must trust the declared extension, exactly like the existing `txt`/`html` entries in `UNDETECTABLE_TEXT_FORMATS`.
- No changes to `src/app.js`, `src/jobs/jobRepository.js`, the database schema, or any frontend file — confirmed unnecessary in the spec's Scope section.
- `md -> docx` must never throw on unrecognized Markdown constructs (tables, blockquotes, images) — it degrades to a plain-text paragraph for that block rather than failing the job, per the spec's fallback rule.
- Every new registered conversion pair must have a passing test in the style already used in `test/converters/document.test.js` / `test/converters/documentFromPdf.test.js` (fixture in a `beforeAll` tmp dir, `resolve(source, target)` from the registry, assert on output content/validity).

---

## Task 1: Dependencies and MIME plumbing for `.md`

**Files:**
- Modify: `package.json` (add `markdown-it` and `turndown` to `dependencies`)
- Modify: `src/mime.js`
- Test: `test/mime.test.js`

**Interfaces:**
- Produces: `OUTPUT_MIME_TYPES.md === 'text/markdown'`, `UNDETECTABLE_TEXT_FORMATS.md === 'text/markdown'` in `src/mime.js`, consumed by later tasks' tests and by `src/mime.js`'s existing exported functions (`outputMimeType`, `resolveInputFormat`) unchanged in signature.

- [ ] **Step 1: Install the new dependencies**

Run: `npm install markdown-it turndown`
Expected: `package.json` gains `"markdown-it": "^15.0.0"` and `"turndown": "^7.2.4"` under `"dependencies"`; `package-lock.json` updates accordingly. (If already installed from prior research in this session, this is a no-op — verify with `npm ls markdown-it turndown`.)

- [ ] **Step 2: Write the failing tests**

In `test/mime.test.js`, add inside the existing `describe('outputMimeType', ...)` block, as a new assertion appended to the existing "returns the MIME type for a known target format" test body:

```js
    expect(outputMimeType('md')).toBe('text/markdown');
```

Add a new test inside the existing `describe('resolveInputFormat', ...)` block, modeled directly on the existing "trusts the declared format for undetectable txt files" test right above it:

```js
  it('trusts the declared format for undetectable md files', async () => {
    const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample.md');
    await fs.writeFile(fixturePath, '# plain markdown, no magic bytes');

    const result = await resolveInputFormat(fixturePath, 'md');

    expect(result).toEqual({ mime: 'text/markdown', valid: true });

    await fs.unlink(fixturePath);
  });
```

- [ ] **Step 3: Run tests to verify they fail**

Run: `npx vitest run test/mime.test.js`
Expected: FAIL — `outputMimeType('md')` throws `No known MIME type for target format "md"`, and the new `resolveInputFormat` test gets `{ mime: null, valid: false }` instead of the expected object.

- [ ] **Step 4: Implement the MIME plumbing**

In `src/mime.js`, add `md: 'text/markdown'` to the `OUTPUT_MIME_TYPES` object (after the `docx` entry):

```js
const OUTPUT_MIME_TYPES = {
  jpg: 'image/jpeg',
  jpeg: 'image/jpeg',
  png: 'image/png',
  webp: 'image/webp',
  gif: 'image/gif',
  tiff: 'image/tiff',
  avif: 'image/avif',
  bmp: 'image/bmp',
  pdf: 'application/pdf',
  html: 'text/html',
  txt: 'text/plain',
  docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
  md: 'text/markdown',
};
```

And add `md: 'text/markdown'` to `UNDETECTABLE_TEXT_FORMATS`:

```js
const UNDETECTABLE_TEXT_FORMATS = {
  txt: 'text/plain',
  html: 'text/html',
  md: 'text/markdown',
};
```

- [ ] **Step 5: Run tests to verify they pass**

Run: `npx vitest run test/mime.test.js`
Expected: PASS (all tests in the file, including the two new assertions).

- [ ] **Step 6: Commit**

```bash
git add package.json package-lock.json src/mime.js test/mime.test.js
git commit -m "feat: add MIME plumbing for markdown (.md) format"
```

---

## Task 2: `md -> html` and `html -> md` converters

**Files:**
- Modify: `src/converters/document.js`
- Test: `test/converters/document.test.js`

**Interfaces:**
- Consumes: `register` from `./registry.js` (existing).
- Produces: registry entries `resolve('md', 'html')` and `resolve('html', 'md')`, each `{ family: 'document', convert: async (inputPath, outputPath) => void }`, consumed by Task 3's `md -> pdf` implementation (which reuses the same `MarkdownIt` render call) and by the test suite.

- [ ] **Step 1: Write the failing tests**

In `test/converters/document.test.js`, add two new `it` blocks inside `describe('document converters', ...)`, after the existing "converts DOCX to a valid PDF" test:

```js
  it('converts MD to HTML', async () => {
    const inputPath = path.join(tmpDir, 'fixture.md');
    await fs.writeFile(inputPath, '# Heading\n\nSome **bold** text.');
    const outputPath = path.join(tmpDir, 'from-md.html');
    const entry = resolve('md', 'html');

    await entry.convert(inputPath, outputPath);

    const html = await fs.readFile(outputPath, 'utf8');
    expect(html).toContain('<h1>Heading</h1>');
    expect(html).toContain('<strong>bold</strong>');
  });

  it('converts HTML to MD', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-md.html');
    await fs.writeFile(inputPath, '<h1>Heading</h1><p>Some <strong>bold</strong> text.</p>');
    const outputPath = path.join(tmpDir, 'from-html.md');
    const entry = resolve('html', 'md');

    await entry.convert(inputPath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('# Heading');
    expect(markdown).toContain('**bold**');
  });
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run test/converters/document.test.js`
Expected: FAIL with `TypeError: Cannot read properties of null (reading 'convert')` for both new tests (since `resolve('md', 'html')` and `resolve('html', 'md')` return `null`).

- [ ] **Step 3: Implement the converters**

In `src/converters/document.js`, add the two new imports at the top, alongside the existing ones:

```js
import MarkdownIt from 'markdown-it';
import TurndownService from 'turndown';
```

Add the two new functions after `convertDocxToPdf` (and before `extractPdfPageTexts`):

```js
async function convertMdToHtml(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const html = new MarkdownIt().render(markdown);
  await fs.writeFile(outputPath, html);
}

async function convertHtmlToMd(inputPath, outputPath) {
  const html = await fs.readFile(inputPath, 'utf8');
  const markdown = new TurndownService({ headingStyle: 'atx' }).turndown(html);
  await fs.writeFile(outputPath, markdown);
}
```

Register both in `registerDocumentConverters()`, after the existing `register({ ..., sourceFormat: 'html', targetFormat: 'pdf', ... })` line:

```js
  register({ family: 'document', sourceFormat: 'md', targetFormat: 'html', convert: convertMdToHtml });
  register({ family: 'document', sourceFormat: 'html', targetFormat: 'md', convert: convertHtmlToMd });
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run test/converters/document.test.js`
Expected: PASS (all tests in the file).

- [ ] **Step 5: Commit**

```bash
git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->html document converters"
```

---

## Task 3: `md -> pdf` and `pdf -> md` converters

**Files:**
- Modify: `src/converters/document.js`
- Test: `test/converters/document.test.js`, `test/converters/documentFromPdf.test.js`

**Interfaces:**
- Consumes: `renderHtmlToPdf(html, outputPath)` (existing, defined at the top of `document.js`), `extractPdfPageTexts(inputPath)` (existing), `convertPdfToTxt` (existing, defined via `extractPdfPageTexts`).
- Produces: registry entries `resolve('md', 'pdf')` and `resolve('pdf', 'md')`.

- [ ] **Step 1: Write the failing tests**

In `test/converters/document.test.js`, add after the `md -> html` / `html -> md` tests from Task 2:

```js
  it('converts MD to a valid PDF', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-pdf.md');
    await fs.writeFile(inputPath, '# Heading\n\nSome text.');
    const outputPath = path.join(tmpDir, 'from-md.pdf');
    const entry = resolve('md', 'pdf');

    await entry.convert(inputPath, outputPath);

    const detected = await detectInputMime(outputPath);
    expect(detected.mime).toBe('application/pdf');
  }, 20000);
```

In `test/converters/documentFromPdf.test.js`, add after the existing "extracts text from PDF to HTML" test:

```js
  it('extracts text from PDF to MD', async () => {
    const outputPath = path.join(tmpDir, 'output.md');
    const entry = resolve('pdf', 'md');

    await entry.convert(pdfFixturePath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('Extractable fixture text');
  });
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run test/converters/document.test.js test/converters/documentFromPdf.test.js`
Expected: FAIL with `TypeError: Cannot read properties of null (reading 'convert')` for both new tests.

- [ ] **Step 3: Implement the converters**

In `src/converters/document.js`, add after `convertHtmlToMd` (from Task 2):

```js
async function convertMdToPdf(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const html = new MarkdownIt().render(markdown);
  await renderHtmlToPdf(html, outputPath);
}
```

Register in `registerDocumentConverters()`, after the `md -> html` / `html -> md` lines added in Task 2:

```js
  register({ family: 'document', sourceFormat: 'md', targetFormat: 'pdf', convert: convertMdToPdf });
  register({ family: 'document', sourceFormat: 'pdf', targetFormat: 'md', convert: convertPdfToTxt });
```

Note `pdf -> md` reuses the existing `convertPdfToTxt` function directly (no new function needed) — extracted PDF text is already valid Markdown, and this keeps the two outputs byte-identical by construction.

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run test/converters/document.test.js test/converters/documentFromPdf.test.js`
Expected: PASS (all tests in both files).

- [ ] **Step 5: Commit**

```bash
git add src/converters/document.js test/converters/document.test.js test/converters/documentFromPdf.test.js
git commit -m "feat: add md<->pdf document converters"
```

---

## Task 4: `docx -> md` and `md -> docx` converters

**Files:**
- Modify: `src/converters/document.js`
- Test: `test/converters/document.test.js`

**Interfaces:**
- Consumes: `mammoth.convertToHtml({ path })` (existing import), `TurndownService` (from Task 2's import), `Document`, `Paragraph`, `TextRun`, `Packer` from `docx` (existing import — `HeadingLevel` added in this task), `MarkdownIt` (from Task 2's import).
- Produces: registry entries `resolve('docx', 'md')` and `resolve('md', 'docx')`; a new internal (unexported) helper function `markdownToDocxParagraphs(markdownText)` returning `Paragraph[]`.

- [ ] **Step 1: Write the failing tests**

In `test/converters/document.test.js`, add after the `md -> pdf` test from Task 3:

```js
  it('converts DOCX to MD containing the source text', async () => {
    const outputPath = path.join(tmpDir, 'from-docx.md');
    const entry = resolve('docx', 'md');

    await entry.convert(docxFixturePath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('Hello from the fixture document');
  }, 20000);

  it('converts MD to DOCX preserving headings, bold text, and bullet lists', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-docx.md');
    await fs.writeFile(
      inputPath,
      '# Report Title\n\nSome **bold** finding.\n\n- first point\n- second point\n'
    );
    const outputPath = path.join(tmpDir, 'from-md.docx');
    const entry = resolve('md', 'docx');

    await entry.convert(inputPath, outputPath);

    const mammoth = await import('mammoth');
    const result = await mammoth.default.convertToHtml({ path: outputPath });
    expect(result.value).toContain('Report Title');
    expect(result.value).toContain('<strong>bold</strong>');
    expect(result.value).toContain('first point');
    expect(result.value).toContain('<li>');
  }, 20000);
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run test/converters/document.test.js`
Expected: FAIL with `TypeError: Cannot read properties of null (reading 'convert')` for both new tests.

- [ ] **Step 3: Implement the converters**

In `src/converters/document.js`, change the existing `docx` import line from:

```js
import { Document, Paragraph, TextRun, Packer } from 'docx';
```

to:

```js
import { Document, Paragraph, TextRun, HeadingLevel, Packer } from 'docx';
```

Add, after `convertMdToPdf` (from Task 3):

```js
const MARKDOWN_HEADING_LEVELS = {
  h1: HeadingLevel.HEADING_1,
  h2: HeadingLevel.HEADING_2,
  h3: HeadingLevel.HEADING_3,
  h4: HeadingLevel.HEADING_4,
  h5: HeadingLevel.HEADING_5,
  h6: HeadingLevel.HEADING_6,
};

function markdownInlineTokensToRuns(children) {
  const runs = [];
  let bold = false;
  let italics = false;
  for (const token of children ?? []) {
    if (token.type === 'strong_open') bold = true;
    else if (token.type === 'strong_close') bold = false;
    else if (token.type === 'em_open') italics = true;
    else if (token.type === 'em_close') italics = false;
    else if (token.type === 'text' || token.type === 'code_inline') {
      if (token.content) runs.push(new TextRun({ text: token.content, bold, italics }));
    } else if (token.type === 'softbreak' || token.type === 'hardbreak') {
      runs.push(new TextRun({ text: ' ' }));
    }
  }
  return runs;
}

function markdownToDocxParagraphs(markdownText) {
  const tokens = new MarkdownIt().parse(markdownText, {});
  const paragraphs = [];
  const listStack = [];

  for (let i = 0; i < tokens.length; i += 1) {
    const token = tokens[i];

    if (token.type === 'heading_open') {
      const inline = tokens[i + 1];
      paragraphs.push(
        new Paragraph({
          heading: MARKDOWN_HEADING_LEVELS[token.tag],
          children: markdownInlineTokensToRuns(inline.children),
        })
      );
      i += 2;
      continue;
    }

    if (token.type === 'bullet_list_open') {
      listStack.push({ type: 'bullet' });
      continue;
    }
    if (token.type === 'ordered_list_open') {
      listStack.push({ type: 'ordered', counter: 0 });
      continue;
    }
    if (token.type === 'bullet_list_close' || token.type === 'ordered_list_close') {
      listStack.pop();
      continue;
    }
    if (token.type === 'list_item_open') {
      const current = listStack[listStack.length - 1];
      if (current?.type === 'ordered') current.counter += 1;
      continue;
    }
    if (token.type === 'list_item_close') {
      continue;
    }

    if (token.type === 'paragraph_open') {
      const inline = tokens[i + 1];
      const runs = markdownInlineTokensToRuns(inline.children);
      const current = listStack[listStack.length - 1];
      if (current?.type === 'bullet') {
        paragraphs.push(new Paragraph({ children: runs, bullet: { level: listStack.length - 1 } }));
      } else if (current?.type === 'ordered') {
        paragraphs.push(
          new Paragraph({ children: [new TextRun({ text: `${current.counter}. ` }), ...runs] })
        );
      } else {
        paragraphs.push(new Paragraph({ children: runs }));
      }
      i += 2;
      continue;
    }

    if (token.type === 'fence' || token.type === 'code_block') {
      if (token.content.trim()) paragraphs.push(new Paragraph({ children: [new TextRun({ text: token.content })] }));
      continue;
    }

    if (token.type === 'hr') {
      paragraphs.push(new Paragraph({ children: [new TextRun({ text: '---' })] }));
    }
  }

  return paragraphs;
}

async function convertDocxToMd(inputPath, outputPath) {
  const result = await mammoth.convertToHtml({ path: inputPath });
  const markdown = new TurndownService({ headingStyle: 'atx' }).turndown(result.value);
  await fs.writeFile(outputPath, markdown);
}

async function convertMdToDocx(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const paragraphs = markdownToDocxParagraphs(markdown);
  const doc = new Document({ sections: [{ children: paragraphs }] });
  const buffer = await Packer.toBuffer(doc);
  await fs.writeFile(outputPath, buffer);
}
```

Register in `registerDocumentConverters()`, after the `md -> pdf` / `pdf -> md` lines added in Task 3:

```js
  register({ family: 'document', sourceFormat: 'docx', targetFormat: 'md', convert: convertDocxToMd });
  register({ family: 'document', sourceFormat: 'md', targetFormat: 'docx', convert: convertMdToDocx });
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run test/converters/document.test.js`
Expected: PASS (all tests in the file).

- [ ] **Step 5: Commit**

```bash
git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->docx document converters"
```

---

## Task 5: `md -> txt` and `txt -> md` passthrough converters

**Files:**
- Modify: `src/converters/document.js`
- Test: `test/converters/document.test.js`

**Interfaces:**
- Produces: registry entries `resolve('md', 'txt')` and `resolve('txt', 'md')`.

- [ ] **Step 1: Write the failing tests**

In `test/converters/document.test.js`, add after the `md -> docx` test from Task 4:

```js
  it('copies MD content unchanged to TXT', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-txt.md');
    const content = '# Heading\n\nSome **bold** text.\n';
    await fs.writeFile(inputPath, content);
    const outputPath = path.join(tmpDir, 'from-md.txt');
    const entry = resolve('md', 'txt');

    await entry.convert(inputPath, outputPath);

    const output = await fs.readFile(outputPath, 'utf8');
    expect(output).toBe(content);
  });

  it('copies TXT content unchanged to MD', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-md-from-txt.txt');
    const content = 'Plain text content.\n';
    await fs.writeFile(inputPath, content);
    const outputPath = path.join(tmpDir, 'from-txt.md');
    const entry = resolve('txt', 'md');

    await entry.convert(inputPath, outputPath);

    const output = await fs.readFile(outputPath, 'utf8');
    expect(output).toBe(content);
  });
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `npx vitest run test/converters/document.test.js`
Expected: FAIL with `TypeError: Cannot read properties of null (reading 'convert')` for both new tests.

- [ ] **Step 3: Implement the converters**

In `src/converters/document.js`, add after `convertMdToDocx` (from Task 4):

```js
async function copyFileContents(inputPath, outputPath) {
  await fs.copyFile(inputPath, outputPath);
}
```

Register in `registerDocumentConverters()`, after the `docx -> md` / `md -> docx` lines added in Task 4:

```js
  register({ family: 'document', sourceFormat: 'md', targetFormat: 'txt', convert: copyFileContents });
  register({ family: 'document', sourceFormat: 'txt', targetFormat: 'md', convert: copyFileContents });
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `npx vitest run test/converters/document.test.js`
Expected: PASS (all tests in the file).

- [ ] **Step 5: Run the full test suite to confirm no regressions**

Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run`
Expected: PASS for all files except the two pre-existing, unrelated failures documented in `CLAUDE.md` (`test/cleanup.test.js` and `test/jobs/jobRepository.test.js`, both about expired-job clock/timezone handling). If any other test fails, investigate before proceeding — do not assume it's pre-existing without checking against `main`.

- [ ] **Step 6: Commit**

```bash
git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->txt passthrough document converters"
```

---

## Final Verification

After Task 5, the full Markdown conversion matrix (`md<->html`, `md<->txt`, `md<->docx`, `md<->pdf`) is registered and tested. As a manual sanity check (optional but recommended before considering this done):

1. Run `npm run build` and start the app locally (`npm start` plus `npm run worker` in another terminal, per existing project scripts).
2. Upload a `.md` file through the UI and confirm `html`, `txt`, `docx`, and `pdf` all appear as target format options (served by the now-generic `/api/formats` endpoint).
3. Upload a `.docx`, `.html`, `.txt`, and `.pdf` file and confirm `md` appears as a target format option for each.
4. Convert one file in each direction and download the result to confirm it opens correctly in a real viewer (e.g. Word for `.docx`, a browser for `.html`/`.pdf`).