convert/docs/superpowers/plans/2026-07-31-markdown-conversion.md

Markdown (.md) Document Conversion Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add Markdown (.md) as a document format convertible in both directions against html, txt, docx, and pdf.

Architecture: All new conversion functions live in the existing src/converters/document.js and are registered through the existing registerDocumentConverters() function via the existing register() call in src/converters/registry.js — no registry/API/frontend changes are needed because GET /api/formats and POST /api/jobs are already format-agnostic (verified by reading src/app.js: they call listTargetFormats/resolveConverter generically). md->html and html->md are direct library calls; md->pdf and pdf->md reuse the existing renderHtmlToPdf / extractPdfPageTexts helpers; docx->md reuses the existing mammoth.convertToHtml call piped through the new HTML→MD library; md->docx is the only genuinely new algorithm (a markdown-it token-stream walk into docx Paragraph/TextRun objects); md<->txt is a raw file copy.

Tech Stack: Node.js, markdown-it (Markdown → HTML, v15.0.0), turndown (HTML → Markdown, v7.2.4), existing docx/mammoth/puppeteer/pdfjs-dist already in the project.

Global Constraints

• MIME type for .md is text/markdown (per the spec's MIME plumbing section).
• .md files are undetectable by magic bytes — resolveInputFormat must trust the declared extension, exactly like the existing txt/html entries in UNDETECTABLE_TEXT_FORMATS.
• No changes to src/app.js, src/jobs/jobRepository.js, the database schema, or any frontend file — confirmed unnecessary in the spec's Scope section.
• md -> docx must never throw on unrecognized Markdown constructs (tables, blockquotes, images) — it degrades to a plain-text paragraph for that block rather than failing the job, per the spec's fallback rule.
• Every new registered conversion pair must have a passing test in the style already used in test/converters/document.test.js / test/converters/documentFromPdf.test.js (fixture in a beforeAll tmp dir, resolve(source, target) from the registry, assert on output content/validity).

---

Task 1: Dependencies and MIME plumbing for .md

Files:

• Modify: package.json (add markdown-it and turndown to dependencies)
• Modify: src/mime.js
• Test: test/mime.test.js

Interfaces:

• Produces: OUTPUT_MIME_TYPES.md === 'text/markdown', UNDETECTABLE_TEXT_FORMATS.md === 'text/markdown' in src/mime.js, consumed by later tasks' tests and by src/mime.js's existing exported functions (outputMimeType, resolveInputFormat) unchanged in signature.

• Step 1: Install the new dependencies

Run: npm install markdown-it turndown Expected: package.json gains "markdown-it": "^15.0.0" and "turndown": "^7.2.4" under "dependencies"; package-lock.json updates accordingly. (If already installed from prior research in this session, this is a no-op — verify with npm ls markdown-it turndown.)

• Step 2: Write the failing tests

In test/mime.test.js, add inside the existing describe('outputMimeType', ...) block, as a new assertion appended to the existing "returns the MIME type for a known target format" test body:

    expect(outputMimeType('md')).toBe('text/markdown');

Add a new test inside the existing describe('resolveInputFormat', ...) block, modeled directly on the existing "trusts the declared format for undetectable txt files" test right above it:

  it('trusts the declared format for undetectable md files', async () => {
    const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample.md');
    await fs.writeFile(fixturePath, '# plain markdown, no magic bytes');

    const result = await resolveInputFormat(fixturePath, 'md');

    expect(result).toEqual({ mime: 'text/markdown', valid: true });

    await fs.unlink(fixturePath);
  });

• Step 3: Run tests to verify they fail

Run: npx vitest run test/mime.test.js Expected: FAIL — outputMimeType('md') throws No known MIME type for target format "md", and the new resolveInputFormat test gets { mime: null, valid: false } instead of the expected object.

• Step 4: Implement the MIME plumbing

In src/mime.js, add md: 'text/markdown' to the OUTPUT_MIME_TYPES object (after the docx entry):

const OUTPUT_MIME_TYPES = {
  jpg: 'image/jpeg',
  jpeg: 'image/jpeg',
  png: 'image/png',
  webp: 'image/webp',
  gif: 'image/gif',
  tiff: 'image/tiff',
  avif: 'image/avif',
  bmp: 'image/bmp',
  pdf: 'application/pdf',
  html: 'text/html',
  txt: 'text/plain',
  docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
  md: 'text/markdown',
};

And add md: 'text/markdown' to UNDETECTABLE_TEXT_FORMATS:

const UNDETECTABLE_TEXT_FORMATS = {
  txt: 'text/plain',
  html: 'text/html',
  md: 'text/markdown',
};

• Step 5: Run tests to verify they pass

Run: npx vitest run test/mime.test.js Expected: PASS (all tests in the file, including the two new assertions).

• Step 6: Commit

git add package.json package-lock.json src/mime.js test/mime.test.js
git commit -m "feat: add MIME plumbing for markdown (.md) format"

---

Task 2: md -> html and html -> md converters

Files:

• Modify: src/converters/document.js
• Test: test/converters/document.test.js

Interfaces:

• Consumes: register from ./registry.js (existing).

• Produces: registry entries resolve('md', 'html') and resolve('html', 'md'), each { family: 'document', convert: async (inputPath, outputPath) => void }, consumed by Task 3's md -> pdf implementation (which reuses the same MarkdownIt render call) and by the test suite.

• Step 1: Write the failing tests

In test/converters/document.test.js, add two new it blocks inside describe('document converters', ...), after the existing "converts DOCX to a valid PDF" test:

  it('converts MD to HTML', async () => {
    const inputPath = path.join(tmpDir, 'fixture.md');
    await fs.writeFile(inputPath, '# Heading\n\nSome **bold** text.');
    const outputPath = path.join(tmpDir, 'from-md.html');
    const entry = resolve('md', 'html');

    await entry.convert(inputPath, outputPath);

    const html = await fs.readFile(outputPath, 'utf8');
    expect(html).toContain('<h1>Heading</h1>');
    expect(html).toContain('<strong>bold</strong>');
  });

  it('converts HTML to MD', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-md.html');
    await fs.writeFile(inputPath, '<h1>Heading</h1><p>Some <strong>bold</strong> text.</p>');
    const outputPath = path.join(tmpDir, 'from-html.md');
    const entry = resolve('html', 'md');

    await entry.convert(inputPath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('# Heading');
    expect(markdown).toContain('**bold**');
  });

• Step 2: Run tests to verify they fail

Run: npx vitest run test/converters/document.test.js Expected: FAIL with TypeError: Cannot read properties of null (reading 'convert') for both new tests (since resolve('md', 'html') and resolve('html', 'md') return null).

• Step 3: Implement the converters

In src/converters/document.js, add the two new imports at the top, alongside the existing ones:

import MarkdownIt from 'markdown-it';
import TurndownService from 'turndown';

Add the two new functions after convertDocxToPdf (and before extractPdfPageTexts):

async function convertMdToHtml(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const html = new MarkdownIt().render(markdown);
  await fs.writeFile(outputPath, html);
}

async function convertHtmlToMd(inputPath, outputPath) {
  const html = await fs.readFile(inputPath, 'utf8');
  const markdown = new TurndownService({ headingStyle: 'atx' }).turndown(html);
  await fs.writeFile(outputPath, markdown);
}

Register both in registerDocumentConverters(), after the existing register({ ..., sourceFormat: 'html', targetFormat: 'pdf', ... }) line:

  register({ family: 'document', sourceFormat: 'md', targetFormat: 'html', convert: convertMdToHtml });
  register({ family: 'document', sourceFormat: 'html', targetFormat: 'md', convert: convertHtmlToMd });

• Step 4: Run tests to verify they pass

Run: npx vitest run test/converters/document.test.js Expected: PASS (all tests in the file).

• Step 5: Commit

git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->html document converters"

---

Task 3: md -> pdf and pdf -> md converters

Files:

• Modify: src/converters/document.js
• Test: test/converters/document.test.js, test/converters/documentFromPdf.test.js

Interfaces:

• Consumes: renderHtmlToPdf(html, outputPath) (existing, defined at the top of document.js), extractPdfPageTexts(inputPath) (existing), convertPdfToTxt (existing, defined via extractPdfPageTexts).

• Produces: registry entries resolve('md', 'pdf') and resolve('pdf', 'md').

• Step 1: Write the failing tests

In test/converters/document.test.js, add after the md -> html / html -> md tests from Task 2:

  it('converts MD to a valid PDF', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-pdf.md');
    await fs.writeFile(inputPath, '# Heading\n\nSome text.');
    const outputPath = path.join(tmpDir, 'from-md.pdf');
    const entry = resolve('md', 'pdf');

    await entry.convert(inputPath, outputPath);

    const detected = await detectInputMime(outputPath);
    expect(detected.mime).toBe('application/pdf');
  }, 20000);

In test/converters/documentFromPdf.test.js, add after the existing "extracts text from PDF to HTML" test:

  it('extracts text from PDF to MD', async () => {
    const outputPath = path.join(tmpDir, 'output.md');
    const entry = resolve('pdf', 'md');

    await entry.convert(pdfFixturePath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('Extractable fixture text');
  });

• Step 2: Run tests to verify they fail

Run: npx vitest run test/converters/document.test.js test/converters/documentFromPdf.test.js Expected: FAIL with TypeError: Cannot read properties of null (reading 'convert') for both new tests.

• Step 3: Implement the converters

In src/converters/document.js, add after convertHtmlToMd (from Task 2):

async function convertMdToPdf(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const html = new MarkdownIt().render(markdown);
  await renderHtmlToPdf(html, outputPath);
}

Register in registerDocumentConverters(), after the md -> html / html -> md lines added in Task 2:

  register({ family: 'document', sourceFormat: 'md', targetFormat: 'pdf', convert: convertMdToPdf });
  register({ family: 'document', sourceFormat: 'pdf', targetFormat: 'md', convert: convertPdfToTxt });

Note pdf -> md reuses the existing convertPdfToTxt function directly (no new function needed) — extracted PDF text is already valid Markdown, and this keeps the two outputs byte-identical by construction.

• Step 4: Run tests to verify they pass

Run: npx vitest run test/converters/document.test.js test/converters/documentFromPdf.test.js Expected: PASS (all tests in both files).

• Step 5: Commit

git add src/converters/document.js test/converters/document.test.js test/converters/documentFromPdf.test.js
git commit -m "feat: add md<->pdf document converters"

---

Task 4: docx -> md and md -> docx converters

Files:

• Modify: src/converters/document.js
• Test: test/converters/document.test.js

Interfaces:

• Consumes: mammoth.convertToHtml({ path }) (existing import), TurndownService (from Task 2's import), Document, Paragraph, TextRun, Packer from docx (existing import — HeadingLevel added in this task), MarkdownIt (from Task 2's import).

• Produces: registry entries resolve('docx', 'md') and resolve('md', 'docx'); a new internal (unexported) helper function markdownToDocxParagraphs(markdownText) returning Paragraph[].

• Step 1: Write the failing tests

In test/converters/document.test.js, add after the md -> pdf test from Task 3:

  it('converts DOCX to MD containing the source text', async () => {
    const outputPath = path.join(tmpDir, 'from-docx.md');
    const entry = resolve('docx', 'md');

    await entry.convert(docxFixturePath, outputPath);

    const markdown = await fs.readFile(outputPath, 'utf8');
    expect(markdown).toContain('Hello from the fixture document');
  }, 20000);

  it('converts MD to DOCX preserving headings, bold text, and bullet lists', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-docx.md');
    await fs.writeFile(
      inputPath,
      '# Report Title\n\nSome **bold** finding.\n\n- first point\n- second point\n'
    );
    const outputPath = path.join(tmpDir, 'from-md.docx');
    const entry = resolve('md', 'docx');

    await entry.convert(inputPath, outputPath);

    const mammoth = await import('mammoth');
    const result = await mammoth.default.convertToHtml({ path: outputPath });
    expect(result.value).toContain('Report Title');
    expect(result.value).toContain('<strong>bold</strong>');
    expect(result.value).toContain('first point');
    expect(result.value).toContain('<li>');
  }, 20000);

• Step 2: Run tests to verify they fail

Run: npx vitest run test/converters/document.test.js Expected: FAIL with TypeError: Cannot read properties of null (reading 'convert') for both new tests.

• Step 3: Implement the converters

In src/converters/document.js, change the existing docx import line from:

import { Document, Paragraph, TextRun, Packer } from 'docx';

to:

import { Document, Paragraph, TextRun, HeadingLevel, Packer } from 'docx';

Add, after convertMdToPdf (from Task 3):

const MARKDOWN_HEADING_LEVELS = {
  h1: HeadingLevel.HEADING_1,
  h2: HeadingLevel.HEADING_2,
  h3: HeadingLevel.HEADING_3,
  h4: HeadingLevel.HEADING_4,
  h5: HeadingLevel.HEADING_5,
  h6: HeadingLevel.HEADING_6,
};

function markdownInlineTokensToRuns(children) {
  const runs = [];
  let bold = false;
  let italics = false;
  for (const token of children ?? []) {
    if (token.type === 'strong_open') bold = true;
    else if (token.type === 'strong_close') bold = false;
    else if (token.type === 'em_open') italics = true;
    else if (token.type === 'em_close') italics = false;
    else if (token.type === 'text' || token.type === 'code_inline') {
      if (token.content) runs.push(new TextRun({ text: token.content, bold, italics }));
    } else if (token.type === 'softbreak' || token.type === 'hardbreak') {
      runs.push(new TextRun({ text: ' ' }));
    }
  }
  return runs;
}

function markdownToDocxParagraphs(markdownText) {
  const tokens = new MarkdownIt().parse(markdownText, {});
  const paragraphs = [];
  const listStack = [];

  for (let i = 0; i < tokens.length; i += 1) {
    const token = tokens[i];

    if (token.type === 'heading_open') {
      const inline = tokens[i + 1];
      paragraphs.push(
        new Paragraph({
          heading: MARKDOWN_HEADING_LEVELS[token.tag],
          children: markdownInlineTokensToRuns(inline.children),
        })
      );
      i += 2;
      continue;
    }

    if (token.type === 'bullet_list_open') {
      listStack.push({ type: 'bullet' });
      continue;
    }
    if (token.type === 'ordered_list_open') {
      listStack.push({ type: 'ordered', counter: 0 });
      continue;
    }
    if (token.type === 'bullet_list_close' || token.type === 'ordered_list_close') {
      listStack.pop();
      continue;
    }
    if (token.type === 'list_item_open') {
      const current = listStack[listStack.length - 1];
      if (current?.type === 'ordered') current.counter += 1;
      continue;
    }
    if (token.type === 'list_item_close') {
      continue;
    }

    if (token.type === 'paragraph_open') {
      const inline = tokens[i + 1];
      const runs = markdownInlineTokensToRuns(inline.children);
      const current = listStack[listStack.length - 1];
      if (current?.type === 'bullet') {
        paragraphs.push(new Paragraph({ children: runs, bullet: { level: listStack.length - 1 } }));
      } else if (current?.type === 'ordered') {
        paragraphs.push(
          new Paragraph({ children: [new TextRun({ text: `${current.counter}. ` }), ...runs] })
        );
      } else {
        paragraphs.push(new Paragraph({ children: runs }));
      }
      i += 2;
      continue;
    }

    if (token.type === 'fence' || token.type === 'code_block') {
      if (token.content.trim()) paragraphs.push(new Paragraph({ children: [new TextRun({ text: token.content })] }));
      continue;
    }

    if (token.type === 'hr') {
      paragraphs.push(new Paragraph({ children: [new TextRun({ text: '---' })] }));
    }
  }

  return paragraphs;
}

async function convertDocxToMd(inputPath, outputPath) {
  const result = await mammoth.convertToHtml({ path: inputPath });
  const markdown = new TurndownService({ headingStyle: 'atx' }).turndown(result.value);
  await fs.writeFile(outputPath, markdown);
}

async function convertMdToDocx(inputPath, outputPath) {
  const markdown = await fs.readFile(inputPath, 'utf8');
  const paragraphs = markdownToDocxParagraphs(markdown);
  const doc = new Document({ sections: [{ children: paragraphs }] });
  const buffer = await Packer.toBuffer(doc);
  await fs.writeFile(outputPath, buffer);
}

Register in registerDocumentConverters(), after the md -> pdf / pdf -> md lines added in Task 3:

  register({ family: 'document', sourceFormat: 'docx', targetFormat: 'md', convert: convertDocxToMd });
  register({ family: 'document', sourceFormat: 'md', targetFormat: 'docx', convert: convertMdToDocx });

• Step 4: Run tests to verify they pass

Run: npx vitest run test/converters/document.test.js Expected: PASS (all tests in the file).

• Step 5: Commit

git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->docx document converters"

---

Task 5: md -> txt and txt -> md passthrough converters

Files:

• Modify: src/converters/document.js
• Test: test/converters/document.test.js

Interfaces:

• Produces: registry entries resolve('md', 'txt') and resolve('txt', 'md').

• Step 1: Write the failing tests

In test/converters/document.test.js, add after the md -> docx test from Task 4:

  it('copies MD content unchanged to TXT', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-txt.md');
    const content = '# Heading\n\nSome **bold** text.\n';
    await fs.writeFile(inputPath, content);
    const outputPath = path.join(tmpDir, 'from-md.txt');
    const entry = resolve('md', 'txt');

    await entry.convert(inputPath, outputPath);

    const output = await fs.readFile(outputPath, 'utf8');
    expect(output).toBe(content);
  });

  it('copies TXT content unchanged to MD', async () => {
    const inputPath = path.join(tmpDir, 'fixture-for-md-from-txt.txt');
    const content = 'Plain text content.\n';
    await fs.writeFile(inputPath, content);
    const outputPath = path.join(tmpDir, 'from-txt.md');
    const entry = resolve('txt', 'md');

    await entry.convert(inputPath, outputPath);

    const output = await fs.readFile(outputPath, 'utf8');
    expect(output).toBe(content);
  });

• Step 2: Run tests to verify they fail

Run: npx vitest run test/converters/document.test.js Expected: FAIL with TypeError: Cannot read properties of null (reading 'convert') for both new tests.

• Step 3: Implement the converters

In src/converters/document.js, add after convertMdToDocx (from Task 4):

async function copyFileContents(inputPath, outputPath) {
  await fs.copyFile(inputPath, outputPath);
}

Register in registerDocumentConverters(), after the docx -> md / md -> docx lines added in Task 4:

  register({ family: 'document', sourceFormat: 'md', targetFormat: 'txt', convert: copyFileContents });
  register({ family: 'document', sourceFormat: 'txt', targetFormat: 'md', convert: copyFileContents });

• Step 4: Run tests to verify they pass

Run: npx vitest run test/converters/document.test.js Expected: PASS (all tests in the file).

• Step 5: Run the full test suite to confirm no regressions

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run Expected: PASS for all files except the two pre-existing, unrelated failures documented in CLAUDE.md (test/cleanup.test.js and test/jobs/jobRepository.test.js, both about expired-job clock/timezone handling). If any other test fails, investigate before proceeding — do not assume it's pre-existing without checking against main.

• Step 6: Commit

git add src/converters/document.js test/converters/document.test.js
git commit -m "feat: add md<->txt passthrough document converters"

---

Final Verification

After Task 5, the full Markdown conversion matrix (md<->html, md<->txt, md<->docx, md<->pdf) is registered and tested. As a manual sanity check (optional but recommended before considering this done):

1. Run npm run build and start the app locally (npm start plus npm run worker in another terminal, per existing project scripts).
2. Upload a .md file through the UI and confirm html, txt, docx, and pdf all appear as target format options (served by the now-generic /api/formats endpoint).
3. Upload a .docx, .html, .txt, and .pdf file and confirm md appears as a target format option for each.
4. Convert one file in each direction and download the result to confirm it opens correctly in a real viewer (e.g. Word for .docx, a browser for .html/.pdf).