58 lines
5.7 KiB
Markdown
58 lines
5.7 KiB
Markdown
# Markdown (.md) document conversion
|
|
|
|
Date: 2026-07-31
|
|
|
|
## Goal
|
|
|
|
Add Markdown as a supported document format, convertible in both directions against every other format already handled by `src/converters/document.js`: `html`, `txt`, `docx`, `pdf`. No existing conversion pair changes behavior.
|
|
|
|
## Scope
|
|
|
|
- `src/converters/document.js`: 8 new registry entries (`md<->html`, `md<->txt`, `md<->docx`, `md<->pdf`).
|
|
- `src/mime.js`: MIME plumbing for the new `md` format.
|
|
- Two new dependencies: `markdown-it` (Markdown → HTML) and `turndown` (HTML → Markdown).
|
|
- Out of scope: `src/app.js`, the frontend, and the database. `/api/jobs` and `/api/formats` already work purely off the registry (`resolveConverter`, `listTargetFormats`) and the upload flow has no format allowlist — confirmed by reading `src/app.js`. No frontend file survives a hardcoded format list either (confirmed via `frontend/src/App.jsx` — target formats come from `GET /api/formats`, and the `<input type="file">` has no `accept` restriction). Adding registry entries is sufficient for `.md` to appear end-to-end.
|
|
|
|
## Dependencies
|
|
|
|
Add via `npm install markdown-it turndown` (npm resolves current versions; no version numbers are hand-picked). Both are added to `dependencies` in `package.json`, alongside the existing `mammoth`/`puppeteer`/`docx`/`pdfjs-dist` document stack.
|
|
|
|
## MIME plumbing (`src/mime.js`)
|
|
|
|
- `OUTPUT_MIME_TYPES`: add `md: 'text/markdown'`.
|
|
- `UNDETECTABLE_TEXT_FORMATS`: add `md: 'text/markdown'`. A `.md` file has no distinguishing magic bytes, so `file-type` cannot detect it (same situation as `txt` and `html` today) — `resolveInputFormat` must trust the declared extension.
|
|
|
|
## Conversion functions (`src/converters/document.js`)
|
|
|
|
All functions follow the existing `async (inputPath, outputPath) => {...}` shape and are wired up in `registerDocumentConverters()`.
|
|
|
|
- **`md -> html`**: `new MarkdownIt().render(await fs.readFile(inputPath, 'utf8'))`, write result to `outputPath`.
|
|
- **`html -> md`**: `new TurndownService().turndown(await fs.readFile(inputPath, 'utf8'))`, write result to `outputPath`.
|
|
- **`md -> pdf`**: render the Markdown to HTML via `MarkdownIt` (same call as `md -> html`), then pass the HTML string into the existing `renderHtmlToPdf` helper — no new PDF logic.
|
|
- **`pdf -> md`**: reuse the existing `extractPdfPageTexts` helper (already used by `convertPdfToTxt`); write the page texts joined the same way (`pageTexts.join('\n\n')`) to `outputPath`. Output is content-identical to `pdf -> txt`, just registered under the `md` target key, since extracted plain text is already valid Markdown.
|
|
- **`docx -> md`**: reuse `mammoth.convertToHtml({ path: inputPath })` (as `convertDocxToHtml` already does), then pipe `result.value` through `TurndownService`.
|
|
- **`md -> docx`** (the one pair with no existing reverse path to lean on): parse the Markdown with `MarkdownIt().parse()` and walk the resulting token stream to build a `docx` `Document`:
|
|
- `heading_open` (h1-h6) → a `Paragraph` with `heading: HeadingLevel.HEADING_1` .. `HEADING_6`.
|
|
- `strong_open`/`em_open` inline tokens → `TextRun({ bold: true })` / `TextRun({ italics: true })` for the enclosed text.
|
|
- `bullet_list_open` / `ordered_list_open` + their `list_item_open` children → paragraphs with `bullet: { level: 0 }` (bulleted) or sequential numbering text prefix (ordered), consistent with the level of effort already in this file.
|
|
- Any other/unrecognized block token (tables, images, blockquotes, code fences) falls back to a single plain-text `Paragraph` of that block's inner text, rather than throwing — mirrors this codebase's existing precedent of `convertPdfToDocx` producing flat, unstyled paragraphs rather than failing on content it can't fully model.
|
|
- **`md -> txt`** and **`txt -> md`**: straight passthrough copy of file content — `fs.copyFile(inputPath, outputPath)` or an equivalent read/write. No parsing: a Markdown file is already readable plain text and a plain text file is already valid Markdown. This mirrors the existing low-effort precedent in this file (e.g. `convertTxtToPdf` does no smart processing of its input either).
|
|
|
|
## Testing
|
|
|
|
Following the existing pattern in `test/converters/document.test.js` and `test/converters/documentFromPdf.test.js` (fixture files in a `beforeAll`-created tmp dir, `resolve(source, target)` from the registry, assertions on output content/validity):
|
|
|
|
- `md -> html`: output contains a rendered tag (e.g. `<h1>` for a `# Heading` fixture, `<strong>` for `**bold**`).
|
|
- `html -> md`: output contains Markdown syntax (e.g. `#`, `**`) for equivalent HTML input.
|
|
- `md -> pdf`: output is a valid PDF (`detectInputMime(...).mime === 'application/pdf'`), matching the assertion style already used for `txt -> pdf` / `html -> pdf` / `docx -> pdf`.
|
|
- `pdf -> md`: output file contains the fixture's known text, matching the style already used in `test/converters/documentFromPdf.test.js` for `pdf -> txt`.
|
|
- `docx -> md`: output contains the fixture docx's known text.
|
|
- `md -> docx`: produced docx is valid and its extracted text (via the same fixture-reading approach used elsewhere in the test file) contains the source Markdown's text content, including a heading and a bolded word from the fixture.
|
|
- `md -> txt` / `txt -> md`: output content is byte-identical to input.
|
|
- `test/mime.test.js`: extend `outputMimeType` coverage with `md -> 'text/markdown'`, and add an "undetectable md" case mirroring the existing `sample.txt` case in `resolveInputFormat`.
|
|
|
|
## Non-goals
|
|
|
|
- No fidelity guarantees beyond what's listed above (e.g. tables, images, footnotes, nested blockquotes are not specially modeled in `md -> docx`; they degrade to plain paragraphs).
|
|
- No changes to job validation, storage, the database schema, or the frontend — the format is purely additive at the registry/mime layer.
|