Files

T

anthony dd8557e418 docs: add ebook conversion implementation plan

2026-07-31 13:29:11 +02:00

31 KiB

Raw Permalink Blame History

Ebook Conversion (Calibre wrapper) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a new ebook conversion family (epub, fb2, lrf, mobi, pdb, rb, snb, tcr, azw3, pdf — all pairs, both directions) by wrapping Calibre's ebook-convert CLI as a subprocess.

Architecture: A new src/converters/ebook.js module registers all 90 source→target pairs into the existing converters/registry.js, each delegating to ebook-convert via node:child_process's execFile (never exec). The worker gets a per-family conversion timeout (ebook jobs get a longer one than the existing 60s default) and forwards it to execFile's own timeout option so an orphaned Calibre subprocess is actually killed, not just abandoned. mime.js gains output MIME types for the new formats and two input-validation fixes verified against file-type's actual source: an azw3/mobi detection-collision alias, and a trust-the-declared-format fallback for the formats file-type cannot sniff at all (fb2, lrf, pdb, rb, snb, tcr).

Tech Stack: Node.js (ESM), node:child_process (execFile + util.promisify), Vitest with vi.mock/vi.hoisted for subprocess mocking (Calibre is not installed on the dev machine).

Global Constraints

No system binary may be installed via a compiler/root-requiring step on o2switch (shared hosting, no root, no apt-get, no compilation) — Calibre is deployed there as its self-contained Linux tarball via manual SSH upload, never through npm install. This plan's code changes don't perform that deployment step; they only assume ebook-convert is reachable on PATH or via CALIBRE_PATH.
Subprocess invocation must use execFile with an argv array, never exec/string interpolation (shell-injection safety).
Project is ESM throughout ("type": "module" in package.json) — use import/export, not require.
Calibre is confirmed absent from PATH on this Windows dev machine — no task in this plan runs a real ebook-convert invocation. Every test that exercises src/converters/ebook.js mocks node:child_process.
Local test runs must use .env.local-equivalent values passed as inline env vars (never load real .env), e.g.: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run
Follow the existing registry contract exactly: register({ family, sourceFormat, targetFormat, convert }) where convert has signature (inputPath, outputPath, options = {}) => Promise<void>.
MIME types for lrf, rb, snb, tcr have no established registry entry anywhere (confirmed against the installed mime-db package) — use application/octet-stream for these, don't invent a more specific value.

Task 1: `src/converters/ebook.js` — Calibre subprocess wrapper and registration

Files:

Create: src/converters/ebook.js
Create: test/converters/ebook.test.js

Interfaces:

Consumes: register from src/converters/registry.js (existing: register({ family, sourceFormat, targetFormat, convert })).
Produces: registerEbookConverter() (no args, void) — called by src/worker.js and src/app.js in later tasks. EBOOK_FORMATS (exported array of the 10 format strings) — consumed by this task's own test and available for later tasks/tests.
Step 1: Write the failing test for registration coverage

Create test/converters/ebook.test.js:

import { describe, it, expect, beforeEach, vi } from 'vitest';
import { registerEbookConverter, EBOOK_FORMATS } from '../../src/converters/ebook.js';
import { resolve, _resetForTests } from '../../src/converters/registry.js';

const { execFileMock } = vi.hoisted(() => ({ execFileMock: vi.fn() }));
vi.mock('node:child_process', () => ({ execFile: execFileMock }));

beforeEach(() => {
  _resetForTests();
  execFileMock.mockReset();
  execFileMock.mockImplementation((file, args, options, callback) => callback(null, '', ''));
});

describe('registerEbookConverter — registration', () => {
  it('registers every pair among the 10 ebook formats, and nothing for source === target', () => {
    registerEbookConverter();

    for (const sourceFormat of EBOOK_FORMATS) {
      for (const targetFormat of EBOOK_FORMATS) {
        const entry = resolve(sourceFormat, targetFormat);
        if (sourceFormat === targetFormat) {
          expect(entry).toBeNull();
        } else {
          expect(entry).not.toBeNull();
          expect(entry.family).toBe('ebook');
        }
      }
    }
  });

  it('exposes exactly the 10 requested formats', () => {
    expect(EBOOK_FORMATS.sort()).toEqual(
      ['azw3', 'epub', 'fb2', 'lrf', 'mobi', 'pdb', 'pdf', 'rb', 'snb', 'tcr'].sort()
    );
  });
});

Step 2: Run test to verify it fails

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js Expected: FAIL — Cannot find module '../../src/converters/ebook.js' (or similar resolution error), since the module doesn't exist yet.

Step 3: Write the implementation

Create src/converters/ebook.js:

import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { register } from './registry.js';

const execFileAsync = promisify(execFile);

export const EBOOK_FORMATS = ['epub', 'fb2', 'lrf', 'mobi', 'pdb', 'rb', 'snb', 'tcr', 'azw3', 'pdf'];

export function registerEbookConverter() {
  for (const sourceFormat of EBOOK_FORMATS) {
    for (const targetFormat of EBOOK_FORMATS) {
      if (sourceFormat === targetFormat) continue;
      register({
        family: 'ebook',
        sourceFormat,
        targetFormat,
        convert: async (inputPath, outputPath, options = {}) => {
          const calibrePath = process.env.CALIBRE_PATH || 'ebook-convert';
          await execFileAsync(calibrePath, [inputPath, outputPath], { timeout: options.timeoutMs });
        },
      });
    }
  }
}

Note calibrePath is read from process.env inside the convert closure (call time), not at module load time — this matches how test/config.test.js already tests env-driven defaults (set process.env, then call), and avoids any need for vi.resetModules() gymnastics in this task's tests.

Step 4: Run test to verify it passes

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js Expected: PASS (2 tests).

Step 5: Write the failing test for the subprocess call itself

Add to test/converters/ebook.test.js:

describe('registerEbookConverter — subprocess invocation', () => {
  beforeEach(() => {
    registerEbookConverter();
  });

  it('calls ebook-convert with the input and output paths, and no timeout when none is given', async () => {
    await resolve('epub', 'pdf').convert('/tmp/in.epub', '/tmp/out.pdf');

    expect(execFileMock).toHaveBeenCalledTimes(1);
    const [file, args, options] = execFileMock.mock.calls[0];
    expect(file).toBe('ebook-convert');
    expect(args).toEqual(['/tmp/in.epub', '/tmp/out.pdf']);
    expect(options).toEqual({ timeout: undefined });
  });

  it('forwards options.timeoutMs to execFile as its timeout', async () => {
    await resolve('mobi', 'epub').convert('/tmp/in.mobi', '/tmp/out.epub', { timeoutMs: 180000 });

    const [, , options] = execFileMock.mock.calls[0];
    expect(options).toEqual({ timeout: 180000 });
  });

  it('uses CALIBRE_PATH from the environment when set', async () => {
    const previous = process.env.CALIBRE_PATH;
    process.env.CALIBRE_PATH = '/opt/calibre/ebook-convert';

    await resolve('fb2', 'pdf').convert('/tmp/in.fb2', '/tmp/out.pdf');

    expect(execFileMock.mock.calls[0][0]).toBe('/opt/calibre/ebook-convert');

    if (previous === undefined) delete process.env.CALIBRE_PATH;
    else process.env.CALIBRE_PATH = previous;
  });

  it('propagates a rejection from execFile as a rejected promise', async () => {
    execFileMock.mockImplementation((file, args, options, callback) =>
      callback(new Error('ebook-convert exited with code 1'), '', 'error: unknown format')
    );

    await expect(resolve('epub', 'pdf').convert('/tmp/in.epub', '/tmp/out.pdf')).rejects.toThrow(
      /exited with code 1/
    );
  });
});

Step 6: Run test to verify it fails

Run: same command as Step 2, targeting ebook.test.js. Expected: FAIL — the new describe block should already pass against the Step 3 implementation except verify by running it first; if any assertion fails, it should only be because the implementation is genuinely missing that behavior (it isn't — Step 3's implementation already satisfies all four cases). Confirm all 6 tests pass together.

Step 7: Run full test file to confirm everything passes

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js Expected: PASS (6 tests).

Step 8: Commit

git add src/converters/ebook.js test/converters/ebook.test.js
git commit -m "feat: add ebook converter wrapping Calibre's ebook-convert CLI"

Task 2: `src/config.js` — per-family ebook job timeout

Files:

Modify: src/config.js:20-26
Modify: test/config.test.js

Interfaces:

Consumes: nothing new.
Produces: config.ebookJobTimeoutMs (number, default 180000) — consumed by src/worker.js in Task 3.
Step 1: Write the failing test

Add to test/config.test.js, inside the existing describe('loadConfig', ...) block:

  it('defaults ebookJobTimeoutMs to 180000ms and reads EBOOK_JOB_TIMEOUT_MS when set', () => {
    process.env.STORAGE_DIR = './storage';
    process.env.DB_HOST = 'localhost';
    process.env.DB_USER = 'user';
    process.env.DB_PASSWORD = 'pass';
    process.env.DB_NAME = 'db';
    delete process.env.EBOOK_JOB_TIMEOUT_MS;

    expect(loadConfig().ebookJobTimeoutMs).toBe(180000);

    process.env.EBOOK_JOB_TIMEOUT_MS = '240000';
    expect(loadConfig().ebookJobTimeoutMs).toBe(240000);
  });

Step 2: Run test to verify it fails

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/config.test.js Expected: FAIL — expect(loadConfig().ebookJobTimeoutMs).toBe(180000) receives undefined.

Step 3: Write the implementation

In src/config.js, add one line to the object returned by loadConfig(), right after the existing workerConcurrency line:

    workerConcurrency: Number(process.env.WORKER_CONCURRENCY ?? 3),
    ebookJobTimeoutMs: Number(process.env.EBOOK_JOB_TIMEOUT_MS ?? 180000),

Step 4: Run test to verify it passes

Run: same command as Step 2. Expected: PASS.

Step 5: Commit

git add src/config.js test/config.test.js
git commit -m "feat: add configurable ebookJobTimeoutMs to config"

Task 3: `src/worker.js` — per-family timeout selection + ebook registration

Files:

Modify: src/worker.js:1-17 (imports + JOB_TIMEOUT_MS), src/worker.js:27-60 (processJob), src/worker.js:77-92 (main)
Modify: test/worker.test.js

Interfaces:

Consumes: registerEbookConverter from src/converters/ebook.js (Task 1), config.ebookJobTimeoutMs from src/config.js (Task 2), entry.family from resolveConverter(...) (existing, registry.js).
Produces: no new exports — processPendingJobs/startWorker signatures are unchanged.
Step 1: Write the failing test

Add to test/worker.test.js. First, add the mock and a sync fs import near the top of the file (mocks must be declared before any other code in the file per Vitest hoisting, but since vi.mock calls are auto-hoisted by Vitest to the top of the module regardless of source position, placing this block right after the existing imports is fine):

import { writeFileSync } from 'node:fs';

const { execFileMock } = vi.hoisted(() => ({ execFileMock: vi.fn() }));
vi.mock('node:child_process', () => ({ execFile: execFileMock }));

Add vi to the existing import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; line (becomes import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';).

Add import { registerEbookConverter } from '../src/converters/ebook.js'; alongside the other converter imports, and call registerEbookConverter(); inside the existing beforeAll alongside registerImageConverters();/registerIcoConverter();.

Then add a new test inside describe('processPendingJobs', ...):

  it('converts a pending ebook job to done using config.ebookJobTimeoutMs, not the default 60s timeout', async () => {
    execFileMock.mockReset();
    execFileMock.mockImplementation((file, args, options, callback) => {
      writeFileSync(args[1], 'fake converted ebook output');
      callback(null, '', '');
    });

    const uuid = '11111111-1111-4111-8111-111111111111';
    const inputFilePath = uploadPath(config, uuid, 'epub');
    await fs.writeFile(inputFilePath, 'fake epub content');

    await createJob(prisma, {
      uuid,
      family: 'ebook',
      sourceFormat: 'epub',
      targetFormat: 'pdf',
      originalFilename: 'book.epub',
      inputPath: `${uuid}.epub`,
      inputMimeType: 'application/epub+zip',
      inputSizeBytes: 18,
      expiresAt: new Date(Date.now() + 3600 * 1000),
    });

    await processPendingJobs(prisma, config);

    const job = await getJobByUuid(prisma, uuid);
    expect(job.status).toBe('done');
    expect(execFileMock).toHaveBeenCalledWith(
      'ebook-convert',
      [inputFilePath, outputPath(config, uuid, 'pdf')],
      { timeout: config.ebookJobTimeoutMs }
    );
    expect(config.ebookJobTimeoutMs).not.toBe(60000);
  });

Step 2: Run test to verify it fails

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/worker.test.js Expected: FAIL — either No converter registered for epub -> pdf (registration missing) or the execFileMock assertion shows { timeout: 60000 } instead of config.ebookJobTimeoutMs (per-family selection missing), depending on how much of Task 1/2 is already wired into worker.js.

Step 3: Write the implementation

In src/worker.js, add the import next to the other register* imports:

import { registerEbookConverter } from './converters/ebook.js';

In processJob, replace the single JOB_TIMEOUT_MS use with a per-family choice. Current code:

    const entry = resolveConverter(job.sourceFormat, job.targetFormat);
    if (!entry) {
      throw new Error(`No converter registered for ${job.sourceFormat} -> ${job.targetFormat}`);
    }

    const startedAt = Date.now();
    await withTimeout(
      entry.convert(inputFilePath, outputFilePath, { quality: job.quality, iconSize: job.iconSize }),
      JOB_TIMEOUT_MS
    );

Replace with:

    const entry = resolveConverter(job.sourceFormat, job.targetFormat);
    if (!entry) {
      throw new Error(`No converter registered for ${job.sourceFormat} -> ${job.targetFormat}`);
    }

    const timeoutMs = entry.family === 'ebook' ? config.ebookJobTimeoutMs : JOB_TIMEOUT_MS;
    const startedAt = Date.now();
    await withTimeout(
      entry.convert(inputFilePath, outputFilePath, { quality: job.quality, iconSize: job.iconSize, timeoutMs }),
      timeoutMs
    );

In main(), add the registration call next to the other six:

  registerImageConverters();
  registerImageToPdfConverter();
  registerDocumentConverters();
  registerIcoConverter();
  registerHeicConverter();
  registerFontConverter();
  registerDfontConverter();
  registerEbookConverter();

Step 4: Run test to verify it passes

Run: same command as Step 2. Expected: PASS. Also re-run the full worker suite to confirm no regression in the existing image-family tests (they don't set execFileMock, so they're unaffected by the mock — node:child_process is only ever touched by the ebook converter):

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/worker.test.js Expected: all tests PASS.

Step 5: Commit

git add src/worker.js test/worker.test.js
git commit -m "feat: give ebook conversions their own worker timeout and register the converter"

Task 4: `src/mime.js` — output MIME types, azw3/mobi collision fix, undetectable-format fallback

Files:

Modify: src/mime.js (whole file — small, ~74 lines)
Modify: test/mime.test.js

Interfaces:

Consumes: EBOOK_FORMATS is not needed here (formats are spelled out directly, matching the existing style of OUTPUT_MIME_TYPES/UNDETECTABLE_TEXT_FORMATS).
Produces: outputMimeType('epub'|'fb2'|'lrf'|'mobi'|'pdb'|'rb'|'snb'|'tcr'|'azw3') now return values instead of throwing; resolveInputFormat(path, 'azw3') now accepts real mobi/azw3-family content; resolveInputFormat(path, 'fb2'|'lrf'|'pdb'|'rb'|'snb'|'tcr') now trusts the declared format when content can't be sniffed.

Verified facts this task relies on (confirmed by reading the installed file-type@22 source and the installed mime-db package directly, not assumed):

file-type's zip detector already returns { ext: 'epub', mime: 'application/epub+zip' } for real EPUB content (checks the zip's mimetype entry for the literal string application/epub+zip) — no code change needed for epub detection, it already works and needs no fallback.
file-type detects both real .mobi and real .azw3 files identically as { ext: 'mobi', mime: 'application/x-mobipocket-ebook' }, because its check is only an 8-byte "BOOKMOBI" string match at byte offset 60 (the PDB header's type+creator fields) — a byte pattern both formats share, since AZW3 is itself a MOBI/PDB container. This means a declared azw3 upload is currently rejected (normalizeFormat('azw3') !== normalizeFormat('mobi')) — needs the same kind of alias normalizeFormat already has for jpg/jpeg and heif/heic.
fb2, lrf, pdb, rb, snb, tcr are absent from file-type's supportedExtensions entirely (confirmed) — sniffing real content of these types always returns null, hitting the same code path as today's UNDETECTABLE_TEXT_FORMATS fallback.
MIME values, cross-checked against the installed mime-db package (ground truth, not memory): epub → application/epub+zip (IANA), mobi → application/x-mobipocket-ebook (Apache), pdb → application/vnd.palm (IANA), lrf → no dedicated entry, IANA lists it only under application/octet-stream. azw3/rb/snb/tcr are absent from mime-db entirely; azw3 uses application/vnd.amazon.ebook (the sibling azw extension's Apache-sourced value — the closest established convention, since AZW3 has no separate registration); rb/snb/tcr use application/octet-stream (no established convention found anywhere). fb2 is also absent from mime-db, but application/x-fictionbook+xml is confirmed (via the freedesktop shared-mime-info project and KDE Dolphin) as the de facto standard value used across Linux desktop environments.
Step 1: Write the failing tests

Add to test/mime.test.js:

describe('outputMimeType — ebooks', () => {
  it('returns the correct MIME type for each ebook target format', () => {
    expect(outputMimeType('epub')).toBe('application/epub+zip');
    expect(outputMimeType('fb2')).toBe('application/x-fictionbook+xml');
    expect(outputMimeType('mobi')).toBe('application/x-mobipocket-ebook');
    expect(outputMimeType('azw3')).toBe('application/vnd.amazon.ebook');
    expect(outputMimeType('pdb')).toBe('application/vnd.palm');
    expect(outputMimeType('lrf')).toBe('application/octet-stream');
    expect(outputMimeType('rb')).toBe('application/octet-stream');
    expect(outputMimeType('snb')).toBe('application/octet-stream');
    expect(outputMimeType('tcr')).toBe('application/octet-stream');
  });
});

describe('resolveInputFormat — azw3/mobi collision', () => {
  function buildMobiFamilyBuffer() {
    // PDB header: 8-byte "BOOKMOBI" type+creator magic at offset 60, shared by both
    // real .mobi and real .azw3 files (confirmed in file-type's source).
    const buffer = Buffer.alloc(68);
    buffer.write('BOOKMOBI', 60, 'ascii');
    return buffer;
  }

  it('confirms file-type reports mobi-family content as ext "mobi" regardless of which of the two formats it is (documents the collision this fix works around)', async () => {
    const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
    await fs.writeFile(fixturePath, buildMobiFamilyBuffer());

    const detected = await detectInputMime(fixturePath);
    expect(detected).toEqual({ ext: 'mobi', mime: 'application/x-mobipocket-ebook' });

    await fs.unlink(fixturePath);
  });

  it('accepts a real mobi-family file declared as mobi', async () => {
    const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
    await fs.writeFile(fixturePath, buildMobiFamilyBuffer());

    const result = await resolveInputFormat(fixturePath, 'mobi');
    expect(result).toEqual({ mime: 'application/x-mobipocket-ebook', valid: true });

    await fs.unlink(fixturePath);
  });

  it('accepts a real mobi-family file declared as azw3 despite file-type reporting it as mobi', async () => {
    const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
    await fs.writeFile(fixturePath, buildMobiFamilyBuffer());

    const result = await resolveInputFormat(fixturePath, 'azw3');
    expect(result).toEqual({ mime: 'application/x-mobipocket-ebook', valid: true });

    await fs.unlink(fixturePath);
  });
});

describe('resolveInputFormat — undetectable ebook formats', () => {
  const undetectableFormats = {
    fb2: 'application/x-fictionbook+xml',
    lrf: 'application/octet-stream',
    pdb: 'application/vnd.palm',
    rb: 'application/octet-stream',
    snb: 'application/octet-stream',
    tcr: 'application/octet-stream',
  };

  it.each(Object.entries(undetectableFormats))(
    'trusts the declared format for undetectable %s files',
    async (declaredFormat, expectedMime) => {
      const fixturePath = path.join(import.meta.dirname, 'fixtures', `sample.${declaredFormat}`);
      await fs.writeFile(fixturePath, 'arbitrary bytes with no recognizable magic number');

      const result = await resolveInputFormat(fixturePath, declaredFormat);

      expect(result).toEqual({ mime: expectedMime, valid: true });

      await fs.unlink(fixturePath);
    }
  );
});

Step 2: Run test to verify it fails

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/mime.test.js Expected: FAIL — outputMimeType('epub') throws No known MIME type for target format "epub"; the azw3 test gets { mime: null, valid: false }; the undetectable-format tests get { mime: null, valid: false }.

Step 3: Write the implementation

Replace the full contents of src/mime.js:

import { fileTypeFromFile } from 'file-type';
import fs from 'node:fs/promises';
import { probeDfont } from './converters/dfont.js';

const OUTPUT_MIME_TYPES = {
  jpg: 'image/jpeg',
  jpeg: 'image/jpeg',
  png: 'image/png',
  webp: 'image/webp',
  gif: 'image/gif',
  tiff: 'image/tiff',
  avif: 'image/avif',
  bmp: 'image/bmp',
  ico: 'image/x-icon',
  pdf: 'application/pdf',
  html: 'text/html',
  txt: 'text/plain',
  docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
  md: 'text/markdown',
  csv: 'text/csv',
  xlsx: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
  ttf: 'font/ttf',
  otf: 'font/otf',
  woff: 'font/woff',
  epub: 'application/epub+zip',
  fb2: 'application/x-fictionbook+xml',
  lrf: 'application/octet-stream',
  mobi: 'application/x-mobipocket-ebook',
  pdb: 'application/vnd.palm',
  rb: 'application/octet-stream',
  snb: 'application/octet-stream',
  tcr: 'application/octet-stream',
  azw3: 'application/vnd.amazon.ebook',
};

export async function detectInputMime(filePath) {
  const result = await fileTypeFromFile(filePath);
  return result ?? null;
}

export function outputMimeType(targetFormat) {
  const mime = OUTPUT_MIME_TYPES[targetFormat];
  if (!mime) {
    throw new Error(`No known MIME type for target format "${targetFormat}"`);
  }
  return mime;
}

const UNDETECTABLE_TEXT_FORMATS = {
  txt: 'text/plain',
  html: 'text/html',
  md: 'text/markdown',
  csv: 'text/csv',
};

const UNDETECTABLE_EBOOK_FORMATS = {
  fb2: 'application/x-fictionbook+xml',
  lrf: 'application/octet-stream',
  pdb: 'application/vnd.palm',
  rb: 'application/octet-stream',
  snb: 'application/octet-stream',
  tcr: 'application/octet-stream',
};

function normalizeFormat(format) {
  if (format === 'jpg') return 'jpeg';
  if (format === 'heif') return 'heic';
  if (format === 'azw3') return 'mobi';
  return format;
}

export async function resolveInputFormat(filePath, declaredFormat) {
  if (declaredFormat === 'dfont') {
    const buffer = await fs.readFile(filePath);
    return probeDfont(buffer)
      ? { mime: 'application/x-dfont', valid: true }
      : { mime: null, valid: false };
  }

  const detected = await detectInputMime(filePath);

  if (!detected) {
    const fallbackMime = UNDETECTABLE_TEXT_FORMATS[declaredFormat] ?? UNDETECTABLE_EBOOK_FORMATS[declaredFormat];
    if (fallbackMime) {
      return { mime: fallbackMime, valid: true };
    }
    return { mime: null, valid: false };
  }

  const valid = normalizeFormat(detected.ext) === normalizeFormat(declaredFormat);
  return { mime: detected.mime, valid };
}

The only changes from the current file: nine new OUTPUT_MIME_TYPES entries, the new UNDETECTABLE_EBOOK_FORMATS map, the azw3 → mobi line in normalizeFormat, and resolveInputFormat's fallback line now checking both undetectable-format maps.

Step 4: Run test to verify it passes

Run: same command as Step 2. Expected: PASS. Also re-run the whole file to confirm no regression in the pre-existing dfont/heic/font/image cases:

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/mime.test.js Expected: all tests PASS.

Step 5: Commit

git add src/mime.js test/mime.test.js
git commit -m "feat: add ebook MIME types, fix azw3/mobi detection collision, trust undetectable ebook formats"

Task 5: `src/app.js` — wire the converter into the API + end-to-end job-creation coverage

Files:

Modify: src/app.js:8-14 (imports), src/app.js:38-48 (registerAllConverters)
Modify: test/api/jobs.test.js

Interfaces:

Consumes: registerEbookConverter from src/converters/ebook.js (Task 1); relies on Task 4's mime.js fallback for the fb2 fixture used in the new test.
Produces: nothing new — GET /api/formats and POST /api/jobs now also serve the ebook family, with zero route/handler changes (confirmed: both endpoints are entirely registry-driven already).
Step 1: Write the failing tests

Add to test/api/jobs.test.js:

describe('GET /api/formats — ebooks', () => {
  it('lists the other 9 ebook formats as targets for epub, and never lists epub as its own target', async () => {
    const response = await request(app).get('/api/formats').query({ source: 'epub' });

    expect(response.body.targets).toEqual(
      expect.arrayContaining(['fb2', 'lrf', 'mobi', 'pdb', 'rb', 'snb', 'tcr', 'azw3', 'pdf'])
    );
    expect(response.body.targets).not.toContain('epub');
  });
});

describe('POST /api/jobs — ebooks', () => {
  it('creates a pending job converting an fb2 upload to epub, trusting the declared format since fb2 has no sniffable magic bytes', async () => {
    const fixturePath = path.join(config.storageDir, 'book.fb2');
    await fs.writeFile(
      fixturePath,
      '<?xml version="1.0" encoding="utf-8"?><FictionBook><body><p>hello</p></body></FictionBook>'
    );

    const response = await request(app)
      .post('/api/jobs')
      .field('targetFormats', JSON.stringify(['epub']))
      .attach('files', fixturePath, 'book.fb2');

    expect(response.status).toBe(201);
    expect(response.body.jobs[0].status).toBe('pending');

    const job = await getJobByUuid(prisma, response.body.jobs[0].id);
    expect(job.sourceFormat).toBe('fb2');
    expect(job.targetFormat).toBe('epub');
    expect(job.family).toBe('ebook');
    expect(job.inputMimeType).toBe('application/x-fictionbook+xml');

    await fs.unlink(fixturePath);
  });
});

Step 2: Run test to verify it fails

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/api/jobs.test.js Expected: FAIL — GET /api/formats?source=epub returns an empty targets array (no converter registered yet in app.js's registry instance), and the POST /api/jobs case gets error: 'Unsupported conversion: fb2 to epub' instead of a pending job.

Step 3: Write the implementation

In src/app.js, add the import next to the other register* imports:

import { registerEbookConverter } from './converters/ebook.js';

In registerAllConverters(), add the call next to the other six:

function registerAllConverters() {
  if (convertersRegistered) return;
  registerImageConverters();
  registerImageToPdfConverter();
  registerDocumentConverters();
  registerIcoConverter();
  registerHeicConverter();
  registerFontConverter();
  registerDfontConverter();
  registerEbookConverter();
  convertersRegistered = true;
}

Step 4: Run test to verify it passes

Run: same command as Step 2. Expected: PASS.

Step 5: Run the full test suite to confirm no regressions anywhere

Run: DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run Expected: all tests PASS, except the two pre-existing unrelated failures documented in CLAUDE.md (test/cleanup.test.js and test/jobs/jobRepository.test.js, both known to fail on main independent of this change — confirm by checking they're the only failures, nothing else new).

Step 6: Commit

git add src/app.js test/api/jobs.test.js
git commit -m "feat: register ebook converter in the app so the API serves the new family"

Deliberately out of scope (documented, not silently dropped)

Actually installing Calibre on o2switch. This plan only makes the app able to call ebook-convert; deploying the self-contained Linux Calibre build via SSH and confirming CALIBRE_PATH/PATH on the production host is an infrastructure step outside this codebase change, to be done once before these jobs can succeed in production.
Real end-to-end conversion verification. Every test here mocks the subprocess. The first real Calibre invocation against real files happens either after o2switch deployment or if Calibre for Windows is installed locally later — both explicitly deferred per the approved design.
A separate WORKER_CONCURRENCY limit for the ebook family. Noted as a future consideration in the design if ebook traffic turns out to dominate the shared-hosting resource budget; not implemented now (YAGNI).

31 KiB Raw Permalink Blame History

Ebook Conversion (Calibre wrapper) Implementation Plan

Global Constraints

Task 1: src/converters/ebook.js — Calibre subprocess wrapper and registration

Task 2: src/config.js — per-family ebook job timeout

Task 3: src/worker.js — per-family timeout selection + ebook registration

Task 4: src/mime.js — output MIME types, azw3/mobi collision fix, undetectable-format fallback

Task 5: src/app.js — wire the converter into the API + end-to-end job-creation coverage

Deliberately out of scope (documented, not silently dropped)

31 KiB

Raw Permalink Blame History

Task 1: `src/converters/ebook.js` — Calibre subprocess wrapper and registration

Task 2: `src/config.js` — per-family ebook job timeout

Task 3: `src/worker.js` — per-family timeout selection + ebook registration

Task 4: `src/mime.js` — output MIME types, azw3/mobi collision fix, undetectable-format fallback

Task 5: `src/app.js` — wire the converter into the API + end-to-end job-creation coverage