docs: add ebook conversion implementation plan
This commit is contained in:
@@ -0,0 +1,718 @@
|
|||||||
|
# Ebook Conversion (Calibre wrapper) Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Add a new `ebook` conversion family (epub, fb2, lrf, mobi, pdb, rb, snb, tcr, azw3, pdf — all pairs, both directions) by wrapping Calibre's `ebook-convert` CLI as a subprocess.
|
||||||
|
|
||||||
|
**Architecture:** A new `src/converters/ebook.js` module registers all 90 source→target pairs into the existing `converters/registry.js`, each delegating to `ebook-convert` via `node:child_process`'s `execFile` (never `exec`). The worker gets a per-family conversion timeout (ebook jobs get a longer one than the existing 60s default) and forwards it to `execFile`'s own `timeout` option so an orphaned Calibre subprocess is actually killed, not just abandoned. `mime.js` gains output MIME types for the new formats and two input-validation fixes verified against `file-type`'s actual source: an `azw3`/`mobi` detection-collision alias, and a trust-the-declared-format fallback for the formats `file-type` cannot sniff at all (`fb2`, `lrf`, `pdb`, `rb`, `snb`, `tcr`).
|
||||||
|
|
||||||
|
**Tech Stack:** Node.js (ESM), `node:child_process` (`execFile` + `util.promisify`), Vitest with `vi.mock`/`vi.hoisted` for subprocess mocking (Calibre is not installed on the dev machine).
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- No system binary may be installed via a compiler/root-requiring step on o2switch (shared hosting, no root, no `apt-get`, no compilation) — Calibre is deployed there as its self-contained Linux tarball via manual SSH upload, never through `npm install`. This plan's code changes don't perform that deployment step; they only assume `ebook-convert` is reachable on `PATH` or via `CALIBRE_PATH`.
|
||||||
|
- Subprocess invocation must use `execFile` with an argv array, never `exec`/string interpolation (shell-injection safety).
|
||||||
|
- Project is ESM throughout (`"type": "module"` in `package.json`) — use `import`/`export`, not `require`.
|
||||||
|
- Calibre is confirmed absent from `PATH` on this Windows dev machine — no task in this plan runs a real `ebook-convert` invocation. Every test that exercises `src/converters/ebook.js` mocks `node:child_process`.
|
||||||
|
- Local test runs must use `.env.local`-equivalent values passed as inline env vars (never load real `.env`), e.g.:
|
||||||
|
`DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run`
|
||||||
|
- Follow the existing registry contract exactly: `register({ family, sourceFormat, targetFormat, convert })` where `convert` has signature `(inputPath, outputPath, options = {}) => Promise<void>`.
|
||||||
|
- MIME types for `lrf`, `rb`, `snb`, `tcr` have no established registry entry anywhere (confirmed against the installed `mime-db` package) — use `application/octet-stream` for these, don't invent a more specific value.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: `src/converters/ebook.js` — Calibre subprocess wrapper and registration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/converters/ebook.js`
|
||||||
|
- Create: `test/converters/ebook.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `register` from `src/converters/registry.js` (existing: `register({ family, sourceFormat, targetFormat, convert })`).
|
||||||
|
- Produces: `registerEbookConverter()` (no args, void) — called by `src/worker.js` and `src/app.js` in later tasks. `EBOOK_FORMATS` (exported array of the 10 format strings) — consumed by this task's own test and available for later tasks/tests.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test for registration coverage**
|
||||||
|
|
||||||
|
Create `test/converters/ebook.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { describe, it, expect, beforeEach, vi } from 'vitest';
|
||||||
|
import { registerEbookConverter, EBOOK_FORMATS } from '../../src/converters/ebook.js';
|
||||||
|
import { resolve, _resetForTests } from '../../src/converters/registry.js';
|
||||||
|
|
||||||
|
const { execFileMock } = vi.hoisted(() => ({ execFileMock: vi.fn() }));
|
||||||
|
vi.mock('node:child_process', () => ({ execFile: execFileMock }));
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
_resetForTests();
|
||||||
|
execFileMock.mockReset();
|
||||||
|
execFileMock.mockImplementation((file, args, options, callback) => callback(null, '', ''));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('registerEbookConverter — registration', () => {
|
||||||
|
it('registers every pair among the 10 ebook formats, and nothing for source === target', () => {
|
||||||
|
registerEbookConverter();
|
||||||
|
|
||||||
|
for (const sourceFormat of EBOOK_FORMATS) {
|
||||||
|
for (const targetFormat of EBOOK_FORMATS) {
|
||||||
|
const entry = resolve(sourceFormat, targetFormat);
|
||||||
|
if (sourceFormat === targetFormat) {
|
||||||
|
expect(entry).toBeNull();
|
||||||
|
} else {
|
||||||
|
expect(entry).not.toBeNull();
|
||||||
|
expect(entry.family).toBe('ebook');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('exposes exactly the 10 requested formats', () => {
|
||||||
|
expect(EBOOK_FORMATS.sort()).toEqual(
|
||||||
|
['azw3', 'epub', 'fb2', 'lrf', 'mobi', 'pdb', 'pdf', 'rb', 'snb', 'tcr'].sort()
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js`
|
||||||
|
Expected: FAIL — `Cannot find module '../../src/converters/ebook.js'` (or similar resolution error), since the module doesn't exist yet.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write the implementation**
|
||||||
|
|
||||||
|
Create `src/converters/ebook.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { execFile } from 'node:child_process';
|
||||||
|
import { promisify } from 'node:util';
|
||||||
|
import { register } from './registry.js';
|
||||||
|
|
||||||
|
const execFileAsync = promisify(execFile);
|
||||||
|
|
||||||
|
export const EBOOK_FORMATS = ['epub', 'fb2', 'lrf', 'mobi', 'pdb', 'rb', 'snb', 'tcr', 'azw3', 'pdf'];
|
||||||
|
|
||||||
|
export function registerEbookConverter() {
|
||||||
|
for (const sourceFormat of EBOOK_FORMATS) {
|
||||||
|
for (const targetFormat of EBOOK_FORMATS) {
|
||||||
|
if (sourceFormat === targetFormat) continue;
|
||||||
|
register({
|
||||||
|
family: 'ebook',
|
||||||
|
sourceFormat,
|
||||||
|
targetFormat,
|
||||||
|
convert: async (inputPath, outputPath, options = {}) => {
|
||||||
|
const calibrePath = process.env.CALIBRE_PATH || 'ebook-convert';
|
||||||
|
await execFileAsync(calibrePath, [inputPath, outputPath], { timeout: options.timeoutMs });
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Note `calibrePath` is read from `process.env` **inside** the `convert` closure (call time), not at module load time — this matches how `test/config.test.js` already tests env-driven defaults (set `process.env`, then call), and avoids any need for `vi.resetModules()` gymnastics in this task's tests.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js`
|
||||||
|
Expected: PASS (2 tests).
|
||||||
|
|
||||||
|
- [ ] **Step 5: Write the failing test for the subprocess call itself**
|
||||||
|
|
||||||
|
Add to `test/converters/ebook.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
describe('registerEbookConverter — subprocess invocation', () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
registerEbookConverter();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('calls ebook-convert with the input and output paths, and no timeout when none is given', async () => {
|
||||||
|
await resolve('epub', 'pdf').convert('/tmp/in.epub', '/tmp/out.pdf');
|
||||||
|
|
||||||
|
expect(execFileMock).toHaveBeenCalledTimes(1);
|
||||||
|
const [file, args, options] = execFileMock.mock.calls[0];
|
||||||
|
expect(file).toBe('ebook-convert');
|
||||||
|
expect(args).toEqual(['/tmp/in.epub', '/tmp/out.pdf']);
|
||||||
|
expect(options).toEqual({ timeout: undefined });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('forwards options.timeoutMs to execFile as its timeout', async () => {
|
||||||
|
await resolve('mobi', 'epub').convert('/tmp/in.mobi', '/tmp/out.epub', { timeoutMs: 180000 });
|
||||||
|
|
||||||
|
const [, , options] = execFileMock.mock.calls[0];
|
||||||
|
expect(options).toEqual({ timeout: 180000 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('uses CALIBRE_PATH from the environment when set', async () => {
|
||||||
|
const previous = process.env.CALIBRE_PATH;
|
||||||
|
process.env.CALIBRE_PATH = '/opt/calibre/ebook-convert';
|
||||||
|
|
||||||
|
await resolve('fb2', 'pdf').convert('/tmp/in.fb2', '/tmp/out.pdf');
|
||||||
|
|
||||||
|
expect(execFileMock.mock.calls[0][0]).toBe('/opt/calibre/ebook-convert');
|
||||||
|
|
||||||
|
if (previous === undefined) delete process.env.CALIBRE_PATH;
|
||||||
|
else process.env.CALIBRE_PATH = previous;
|
||||||
|
});
|
||||||
|
|
||||||
|
it('propagates a rejection from execFile as a rejected promise', async () => {
|
||||||
|
execFileMock.mockImplementation((file, args, options, callback) =>
|
||||||
|
callback(new Error('ebook-convert exited with code 1'), '', 'error: unknown format')
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(resolve('epub', 'pdf').convert('/tmp/in.epub', '/tmp/out.pdf')).rejects.toThrow(
|
||||||
|
/exited with code 1/
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: same command as Step 2, targeting `ebook.test.js`.
|
||||||
|
Expected: FAIL — the new `describe` block should already pass against the Step 3 implementation *except* verify by running it first; if any assertion fails, it should only be because the implementation is genuinely missing that behavior (it isn't — Step 3's implementation already satisfies all four cases). Confirm all 6 tests pass together.
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run full test file to confirm everything passes**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/converters/ebook.test.js`
|
||||||
|
Expected: PASS (6 tests).
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/converters/ebook.js test/converters/ebook.test.js
|
||||||
|
git commit -m "feat: add ebook converter wrapping Calibre's ebook-convert CLI"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: `src/config.js` — per-family ebook job timeout
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/config.js:20-26`
|
||||||
|
- Modify: `test/config.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: nothing new.
|
||||||
|
- Produces: `config.ebookJobTimeoutMs` (number, default `180000`) — consumed by `src/worker.js` in Task 3.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Add to `test/config.test.js`, inside the existing `describe('loadConfig', ...)` block:
|
||||||
|
|
||||||
|
```js
|
||||||
|
it('defaults ebookJobTimeoutMs to 180000ms and reads EBOOK_JOB_TIMEOUT_MS when set', () => {
|
||||||
|
process.env.STORAGE_DIR = './storage';
|
||||||
|
process.env.DB_HOST = 'localhost';
|
||||||
|
process.env.DB_USER = 'user';
|
||||||
|
process.env.DB_PASSWORD = 'pass';
|
||||||
|
process.env.DB_NAME = 'db';
|
||||||
|
delete process.env.EBOOK_JOB_TIMEOUT_MS;
|
||||||
|
|
||||||
|
expect(loadConfig().ebookJobTimeoutMs).toBe(180000);
|
||||||
|
|
||||||
|
process.env.EBOOK_JOB_TIMEOUT_MS = '240000';
|
||||||
|
expect(loadConfig().ebookJobTimeoutMs).toBe(240000);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/config.test.js`
|
||||||
|
Expected: FAIL — `expect(loadConfig().ebookJobTimeoutMs).toBe(180000)` receives `undefined`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write the implementation**
|
||||||
|
|
||||||
|
In `src/config.js`, add one line to the object returned by `loadConfig()`, right after the existing `workerConcurrency` line:
|
||||||
|
|
||||||
|
```js
|
||||||
|
workerConcurrency: Number(process.env.WORKER_CONCURRENCY ?? 3),
|
||||||
|
ebookJobTimeoutMs: Number(process.env.EBOOK_JOB_TIMEOUT_MS ?? 180000),
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: same command as Step 2.
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/config.js test/config.test.js
|
||||||
|
git commit -m "feat: add configurable ebookJobTimeoutMs to config"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: `src/worker.js` — per-family timeout selection + ebook registration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/worker.js:1-17` (imports + `JOB_TIMEOUT_MS`), `src/worker.js:27-60` (`processJob`), `src/worker.js:77-92` (`main`)
|
||||||
|
- Modify: `test/worker.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `registerEbookConverter` from `src/converters/ebook.js` (Task 1), `config.ebookJobTimeoutMs` from `src/config.js` (Task 2), `entry.family` from `resolveConverter(...)` (existing, `registry.js`).
|
||||||
|
- Produces: no new exports — `processPendingJobs`/`startWorker` signatures are unchanged.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Add to `test/worker.test.js`. First, add the mock and a sync `fs` import near the top of the file (mocks must be declared before any other code in the file per Vitest hoisting, but since `vi.mock` calls are auto-hoisted by Vitest to the top of the module regardless of source position, placing this block right after the existing imports is fine):
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { writeFileSync } from 'node:fs';
|
||||||
|
```
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { execFileMock } = vi.hoisted(() => ({ execFileMock: vi.fn() }));
|
||||||
|
vi.mock('node:child_process', () => ({ execFile: execFileMock }));
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `vi` to the existing `import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';` line (becomes `import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';`).
|
||||||
|
|
||||||
|
Add `import { registerEbookConverter } from '../src/converters/ebook.js';` alongside the other converter imports, and call `registerEbookConverter();` inside the existing `beforeAll` alongside `registerImageConverters();`/`registerIcoConverter();`.
|
||||||
|
|
||||||
|
Then add a new test inside `describe('processPendingJobs', ...)`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
it('converts a pending ebook job to done using config.ebookJobTimeoutMs, not the default 60s timeout', async () => {
|
||||||
|
execFileMock.mockReset();
|
||||||
|
execFileMock.mockImplementation((file, args, options, callback) => {
|
||||||
|
writeFileSync(args[1], 'fake converted ebook output');
|
||||||
|
callback(null, '', '');
|
||||||
|
});
|
||||||
|
|
||||||
|
const uuid = '11111111-1111-4111-8111-111111111111';
|
||||||
|
const inputFilePath = uploadPath(config, uuid, 'epub');
|
||||||
|
await fs.writeFile(inputFilePath, 'fake epub content');
|
||||||
|
|
||||||
|
await createJob(prisma, {
|
||||||
|
uuid,
|
||||||
|
family: 'ebook',
|
||||||
|
sourceFormat: 'epub',
|
||||||
|
targetFormat: 'pdf',
|
||||||
|
originalFilename: 'book.epub',
|
||||||
|
inputPath: `${uuid}.epub`,
|
||||||
|
inputMimeType: 'application/epub+zip',
|
||||||
|
inputSizeBytes: 18,
|
||||||
|
expiresAt: new Date(Date.now() + 3600 * 1000),
|
||||||
|
});
|
||||||
|
|
||||||
|
await processPendingJobs(prisma, config);
|
||||||
|
|
||||||
|
const job = await getJobByUuid(prisma, uuid);
|
||||||
|
expect(job.status).toBe('done');
|
||||||
|
expect(execFileMock).toHaveBeenCalledWith(
|
||||||
|
'ebook-convert',
|
||||||
|
[inputFilePath, outputPath(config, uuid, 'pdf')],
|
||||||
|
{ timeout: config.ebookJobTimeoutMs }
|
||||||
|
);
|
||||||
|
expect(config.ebookJobTimeoutMs).not.toBe(60000);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/worker.test.js`
|
||||||
|
Expected: FAIL — either `No converter registered for epub -> pdf` (registration missing) or the `execFileMock` assertion shows `{ timeout: 60000 }` instead of `config.ebookJobTimeoutMs` (per-family selection missing), depending on how much of Task 1/2 is already wired into `worker.js`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write the implementation**
|
||||||
|
|
||||||
|
In `src/worker.js`, add the import next to the other `register*` imports:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { registerEbookConverter } from './converters/ebook.js';
|
||||||
|
```
|
||||||
|
|
||||||
|
In `processJob`, replace the single `JOB_TIMEOUT_MS` use with a per-family choice. Current code:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const entry = resolveConverter(job.sourceFormat, job.targetFormat);
|
||||||
|
if (!entry) {
|
||||||
|
throw new Error(`No converter registered for ${job.sourceFormat} -> ${job.targetFormat}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const startedAt = Date.now();
|
||||||
|
await withTimeout(
|
||||||
|
entry.convert(inputFilePath, outputFilePath, { quality: job.quality, iconSize: job.iconSize }),
|
||||||
|
JOB_TIMEOUT_MS
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace with:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const entry = resolveConverter(job.sourceFormat, job.targetFormat);
|
||||||
|
if (!entry) {
|
||||||
|
throw new Error(`No converter registered for ${job.sourceFormat} -> ${job.targetFormat}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const timeoutMs = entry.family === 'ebook' ? config.ebookJobTimeoutMs : JOB_TIMEOUT_MS;
|
||||||
|
const startedAt = Date.now();
|
||||||
|
await withTimeout(
|
||||||
|
entry.convert(inputFilePath, outputFilePath, { quality: job.quality, iconSize: job.iconSize, timeoutMs }),
|
||||||
|
timeoutMs
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
In `main()`, add the registration call next to the other six:
|
||||||
|
|
||||||
|
```js
|
||||||
|
registerImageConverters();
|
||||||
|
registerImageToPdfConverter();
|
||||||
|
registerDocumentConverters();
|
||||||
|
registerIcoConverter();
|
||||||
|
registerHeicConverter();
|
||||||
|
registerFontConverter();
|
||||||
|
registerDfontConverter();
|
||||||
|
registerEbookConverter();
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: same command as Step 2.
|
||||||
|
Expected: PASS. Also re-run the full worker suite to confirm no regression in the existing image-family tests (they don't set `execFileMock`, so they're unaffected by the mock — `node:child_process` is only ever touched by the `ebook` converter):
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/worker.test.js`
|
||||||
|
Expected: all tests PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/worker.js test/worker.test.js
|
||||||
|
git commit -m "feat: give ebook conversions their own worker timeout and register the converter"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: `src/mime.js` — output MIME types, azw3/mobi collision fix, undetectable-format fallback
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/mime.js` (whole file — small, ~74 lines)
|
||||||
|
- Modify: `test/mime.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `EBOOK_FORMATS` is not needed here (formats are spelled out directly, matching the existing style of `OUTPUT_MIME_TYPES`/`UNDETECTABLE_TEXT_FORMATS`).
|
||||||
|
- Produces: `outputMimeType('epub'|'fb2'|'lrf'|'mobi'|'pdb'|'rb'|'snb'|'tcr'|'azw3')` now return values instead of throwing; `resolveInputFormat(path, 'azw3')` now accepts real mobi/azw3-family content; `resolveInputFormat(path, 'fb2'|'lrf'|'pdb'|'rb'|'snb'|'tcr')` now trusts the declared format when content can't be sniffed.
|
||||||
|
|
||||||
|
Verified facts this task relies on (confirmed by reading the installed `file-type@22` source and the installed `mime-db` package directly, not assumed):
|
||||||
|
- `file-type`'s zip detector already returns `{ ext: 'epub', mime: 'application/epub+zip' }` for real EPUB content (checks the zip's mimetype entry for the literal string `application/epub+zip`) — **no code change needed for epub detection**, it already works and needs no fallback.
|
||||||
|
- `file-type` detects **both** real `.mobi` and real `.azw3` files identically as `{ ext: 'mobi', mime: 'application/x-mobipocket-ebook' }`, because its check is only an 8-byte `"BOOKMOBI"` string match at byte offset 60 (the PDB header's type+creator fields) — a byte pattern both formats share, since AZW3 is itself a MOBI/PDB container. This means a declared `azw3` upload is currently rejected (`normalizeFormat('azw3') !== normalizeFormat('mobi')`) — needs the same kind of alias `normalizeFormat` already has for `jpg`/`jpeg` and `heif`/`heic`.
|
||||||
|
- `fb2`, `lrf`, `pdb`, `rb`, `snb`, `tcr` are absent from `file-type`'s `supportedExtensions` entirely (confirmed) — sniffing real content of these types always returns `null`, hitting the same code path as today's `UNDETECTABLE_TEXT_FORMATS` fallback.
|
||||||
|
- MIME values, cross-checked against the installed `mime-db` package (ground truth, not memory): `epub` → `application/epub+zip` (IANA), `mobi` → `application/x-mobipocket-ebook` (Apache), `pdb` → `application/vnd.palm` (IANA), `lrf` → no dedicated entry, IANA lists it only under `application/octet-stream`. `azw3`/`rb`/`snb`/`tcr` are absent from `mime-db` entirely; `azw3` uses `application/vnd.amazon.ebook` (the sibling `azw` extension's Apache-sourced value — the closest established convention, since AZW3 has no separate registration); `rb`/`snb`/`tcr` use `application/octet-stream` (no established convention found anywhere). `fb2` is also absent from `mime-db`, but `application/x-fictionbook+xml` is confirmed (via the freedesktop shared-mime-info project and KDE Dolphin) as the de facto standard value used across Linux desktop environments.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests**
|
||||||
|
|
||||||
|
Add to `test/mime.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
describe('outputMimeType — ebooks', () => {
|
||||||
|
it('returns the correct MIME type for each ebook target format', () => {
|
||||||
|
expect(outputMimeType('epub')).toBe('application/epub+zip');
|
||||||
|
expect(outputMimeType('fb2')).toBe('application/x-fictionbook+xml');
|
||||||
|
expect(outputMimeType('mobi')).toBe('application/x-mobipocket-ebook');
|
||||||
|
expect(outputMimeType('azw3')).toBe('application/vnd.amazon.ebook');
|
||||||
|
expect(outputMimeType('pdb')).toBe('application/vnd.palm');
|
||||||
|
expect(outputMimeType('lrf')).toBe('application/octet-stream');
|
||||||
|
expect(outputMimeType('rb')).toBe('application/octet-stream');
|
||||||
|
expect(outputMimeType('snb')).toBe('application/octet-stream');
|
||||||
|
expect(outputMimeType('tcr')).toBe('application/octet-stream');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('resolveInputFormat — azw3/mobi collision', () => {
|
||||||
|
function buildMobiFamilyBuffer() {
|
||||||
|
// PDB header: 8-byte "BOOKMOBI" type+creator magic at offset 60, shared by both
|
||||||
|
// real .mobi and real .azw3 files (confirmed in file-type's source).
|
||||||
|
const buffer = Buffer.alloc(68);
|
||||||
|
buffer.write('BOOKMOBI', 60, 'ascii');
|
||||||
|
return buffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
it('confirms file-type reports mobi-family content as ext "mobi" regardless of which of the two formats it is (documents the collision this fix works around)', async () => {
|
||||||
|
const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
|
||||||
|
await fs.writeFile(fixturePath, buildMobiFamilyBuffer());
|
||||||
|
|
||||||
|
const detected = await detectInputMime(fixturePath);
|
||||||
|
expect(detected).toEqual({ ext: 'mobi', mime: 'application/x-mobipocket-ebook' });
|
||||||
|
|
||||||
|
await fs.unlink(fixturePath);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('accepts a real mobi-family file declared as mobi', async () => {
|
||||||
|
const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
|
||||||
|
await fs.writeFile(fixturePath, buildMobiFamilyBuffer());
|
||||||
|
|
||||||
|
const result = await resolveInputFormat(fixturePath, 'mobi');
|
||||||
|
expect(result).toEqual({ mime: 'application/x-mobipocket-ebook', valid: true });
|
||||||
|
|
||||||
|
await fs.unlink(fixturePath);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('accepts a real mobi-family file declared as azw3 despite file-type reporting it as mobi', async () => {
|
||||||
|
const fixturePath = path.join(import.meta.dirname, 'fixtures', 'sample-mobi-family.bin');
|
||||||
|
await fs.writeFile(fixturePath, buildMobiFamilyBuffer());
|
||||||
|
|
||||||
|
const result = await resolveInputFormat(fixturePath, 'azw3');
|
||||||
|
expect(result).toEqual({ mime: 'application/x-mobipocket-ebook', valid: true });
|
||||||
|
|
||||||
|
await fs.unlink(fixturePath);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('resolveInputFormat — undetectable ebook formats', () => {
|
||||||
|
const undetectableFormats = {
|
||||||
|
fb2: 'application/x-fictionbook+xml',
|
||||||
|
lrf: 'application/octet-stream',
|
||||||
|
pdb: 'application/vnd.palm',
|
||||||
|
rb: 'application/octet-stream',
|
||||||
|
snb: 'application/octet-stream',
|
||||||
|
tcr: 'application/octet-stream',
|
||||||
|
};
|
||||||
|
|
||||||
|
it.each(Object.entries(undetectableFormats))(
|
||||||
|
'trusts the declared format for undetectable %s files',
|
||||||
|
async (declaredFormat, expectedMime) => {
|
||||||
|
const fixturePath = path.join(import.meta.dirname, 'fixtures', `sample.${declaredFormat}`);
|
||||||
|
await fs.writeFile(fixturePath, 'arbitrary bytes with no recognizable magic number');
|
||||||
|
|
||||||
|
const result = await resolveInputFormat(fixturePath, declaredFormat);
|
||||||
|
|
||||||
|
expect(result).toEqual({ mime: expectedMime, valid: true });
|
||||||
|
|
||||||
|
await fs.unlink(fixturePath);
|
||||||
|
}
|
||||||
|
);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/mime.test.js`
|
||||||
|
Expected: FAIL — `outputMimeType('epub')` throws `No known MIME type for target format "epub"`; the azw3 test gets `{ mime: null, valid: false }`; the undetectable-format tests get `{ mime: null, valid: false }`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write the implementation**
|
||||||
|
|
||||||
|
Replace the full contents of `src/mime.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { fileTypeFromFile } from 'file-type';
|
||||||
|
import fs from 'node:fs/promises';
|
||||||
|
import { probeDfont } from './converters/dfont.js';
|
||||||
|
|
||||||
|
const OUTPUT_MIME_TYPES = {
|
||||||
|
jpg: 'image/jpeg',
|
||||||
|
jpeg: 'image/jpeg',
|
||||||
|
png: 'image/png',
|
||||||
|
webp: 'image/webp',
|
||||||
|
gif: 'image/gif',
|
||||||
|
tiff: 'image/tiff',
|
||||||
|
avif: 'image/avif',
|
||||||
|
bmp: 'image/bmp',
|
||||||
|
ico: 'image/x-icon',
|
||||||
|
pdf: 'application/pdf',
|
||||||
|
html: 'text/html',
|
||||||
|
txt: 'text/plain',
|
||||||
|
docx: 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||||
|
md: 'text/markdown',
|
||||||
|
csv: 'text/csv',
|
||||||
|
xlsx: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||||
|
ttf: 'font/ttf',
|
||||||
|
otf: 'font/otf',
|
||||||
|
woff: 'font/woff',
|
||||||
|
epub: 'application/epub+zip',
|
||||||
|
fb2: 'application/x-fictionbook+xml',
|
||||||
|
lrf: 'application/octet-stream',
|
||||||
|
mobi: 'application/x-mobipocket-ebook',
|
||||||
|
pdb: 'application/vnd.palm',
|
||||||
|
rb: 'application/octet-stream',
|
||||||
|
snb: 'application/octet-stream',
|
||||||
|
tcr: 'application/octet-stream',
|
||||||
|
azw3: 'application/vnd.amazon.ebook',
|
||||||
|
};
|
||||||
|
|
||||||
|
export async function detectInputMime(filePath) {
|
||||||
|
const result = await fileTypeFromFile(filePath);
|
||||||
|
return result ?? null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function outputMimeType(targetFormat) {
|
||||||
|
const mime = OUTPUT_MIME_TYPES[targetFormat];
|
||||||
|
if (!mime) {
|
||||||
|
throw new Error(`No known MIME type for target format "${targetFormat}"`);
|
||||||
|
}
|
||||||
|
return mime;
|
||||||
|
}
|
||||||
|
|
||||||
|
const UNDETECTABLE_TEXT_FORMATS = {
|
||||||
|
txt: 'text/plain',
|
||||||
|
html: 'text/html',
|
||||||
|
md: 'text/markdown',
|
||||||
|
csv: 'text/csv',
|
||||||
|
};
|
||||||
|
|
||||||
|
const UNDETECTABLE_EBOOK_FORMATS = {
|
||||||
|
fb2: 'application/x-fictionbook+xml',
|
||||||
|
lrf: 'application/octet-stream',
|
||||||
|
pdb: 'application/vnd.palm',
|
||||||
|
rb: 'application/octet-stream',
|
||||||
|
snb: 'application/octet-stream',
|
||||||
|
tcr: 'application/octet-stream',
|
||||||
|
};
|
||||||
|
|
||||||
|
function normalizeFormat(format) {
|
||||||
|
if (format === 'jpg') return 'jpeg';
|
||||||
|
if (format === 'heif') return 'heic';
|
||||||
|
if (format === 'azw3') return 'mobi';
|
||||||
|
return format;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function resolveInputFormat(filePath, declaredFormat) {
|
||||||
|
if (declaredFormat === 'dfont') {
|
||||||
|
const buffer = await fs.readFile(filePath);
|
||||||
|
return probeDfont(buffer)
|
||||||
|
? { mime: 'application/x-dfont', valid: true }
|
||||||
|
: { mime: null, valid: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
const detected = await detectInputMime(filePath);
|
||||||
|
|
||||||
|
if (!detected) {
|
||||||
|
const fallbackMime = UNDETECTABLE_TEXT_FORMATS[declaredFormat] ?? UNDETECTABLE_EBOOK_FORMATS[declaredFormat];
|
||||||
|
if (fallbackMime) {
|
||||||
|
return { mime: fallbackMime, valid: true };
|
||||||
|
}
|
||||||
|
return { mime: null, valid: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
const valid = normalizeFormat(detected.ext) === normalizeFormat(declaredFormat);
|
||||||
|
return { mime: detected.mime, valid };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The only changes from the current file: nine new `OUTPUT_MIME_TYPES` entries, the new `UNDETECTABLE_EBOOK_FORMATS` map, the `azw3` → `mobi` line in `normalizeFormat`, and `resolveInputFormat`'s fallback line now checking both undetectable-format maps.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: same command as Step 2.
|
||||||
|
Expected: PASS. Also re-run the whole file to confirm no regression in the pre-existing dfont/heic/font/image cases:
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/mime.test.js`
|
||||||
|
Expected: all tests PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/mime.js test/mime.test.js
|
||||||
|
git commit -m "feat: add ebook MIME types, fix azw3/mobi detection collision, trust undetectable ebook formats"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: `src/app.js` — wire the converter into the API + end-to-end job-creation coverage
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/app.js:8-14` (imports), `src/app.js:38-48` (`registerAllConverters`)
|
||||||
|
- Modify: `test/api/jobs.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `registerEbookConverter` from `src/converters/ebook.js` (Task 1); relies on Task 4's `mime.js` fallback for the `fb2` fixture used in the new test.
|
||||||
|
- Produces: nothing new — `GET /api/formats` and `POST /api/jobs` now also serve the `ebook` family, with zero route/handler changes (confirmed: both endpoints are entirely registry-driven already).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests**
|
||||||
|
|
||||||
|
Add to `test/api/jobs.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
describe('GET /api/formats — ebooks', () => {
|
||||||
|
it('lists the other 9 ebook formats as targets for epub, and never lists epub as its own target', async () => {
|
||||||
|
const response = await request(app).get('/api/formats').query({ source: 'epub' });
|
||||||
|
|
||||||
|
expect(response.body.targets).toEqual(
|
||||||
|
expect.arrayContaining(['fb2', 'lrf', 'mobi', 'pdb', 'rb', 'snb', 'tcr', 'azw3', 'pdf'])
|
||||||
|
);
|
||||||
|
expect(response.body.targets).not.toContain('epub');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('POST /api/jobs — ebooks', () => {
|
||||||
|
it('creates a pending job converting an fb2 upload to epub, trusting the declared format since fb2 has no sniffable magic bytes', async () => {
|
||||||
|
const fixturePath = path.join(config.storageDir, 'book.fb2');
|
||||||
|
await fs.writeFile(
|
||||||
|
fixturePath,
|
||||||
|
'<?xml version="1.0" encoding="utf-8"?><FictionBook><body><p>hello</p></body></FictionBook>'
|
||||||
|
);
|
||||||
|
|
||||||
|
const response = await request(app)
|
||||||
|
.post('/api/jobs')
|
||||||
|
.field('targetFormats', JSON.stringify(['epub']))
|
||||||
|
.attach('files', fixturePath, 'book.fb2');
|
||||||
|
|
||||||
|
expect(response.status).toBe(201);
|
||||||
|
expect(response.body.jobs[0].status).toBe('pending');
|
||||||
|
|
||||||
|
const job = await getJobByUuid(prisma, response.body.jobs[0].id);
|
||||||
|
expect(job.sourceFormat).toBe('fb2');
|
||||||
|
expect(job.targetFormat).toBe('epub');
|
||||||
|
expect(job.family).toBe('ebook');
|
||||||
|
expect(job.inputMimeType).toBe('application/x-fictionbook+xml');
|
||||||
|
|
||||||
|
await fs.unlink(fixturePath);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run test/api/jobs.test.js`
|
||||||
|
Expected: FAIL — `GET /api/formats?source=epub` returns an empty `targets` array (no converter registered yet in `app.js`'s registry instance), and the `POST /api/jobs` case gets `error: 'Unsupported conversion: fb2 to epub'` instead of a `pending` job.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write the implementation**
|
||||||
|
|
||||||
|
In `src/app.js`, add the import next to the other `register*` imports:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { registerEbookConverter } from './converters/ebook.js';
|
||||||
|
```
|
||||||
|
|
||||||
|
In `registerAllConverters()`, add the call next to the other six:
|
||||||
|
|
||||||
|
```js
|
||||||
|
function registerAllConverters() {
|
||||||
|
if (convertersRegistered) return;
|
||||||
|
registerImageConverters();
|
||||||
|
registerImageToPdfConverter();
|
||||||
|
registerDocumentConverters();
|
||||||
|
registerIcoConverter();
|
||||||
|
registerHeicConverter();
|
||||||
|
registerFontConverter();
|
||||||
|
registerDfontConverter();
|
||||||
|
registerEbookConverter();
|
||||||
|
convertersRegistered = true;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: same command as Step 2.
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run the full test suite to confirm no regressions anywhere**
|
||||||
|
|
||||||
|
Run: `DB_HOST=127.0.0.1 DB_USER=convert_user DB_PASSWORD=change_me DB_NAME=file_converter STORAGE_DIR=./storage PORT=3000 npx vitest run`
|
||||||
|
Expected: all tests PASS, except the two pre-existing unrelated failures documented in `CLAUDE.md` (`test/cleanup.test.js` and `test/jobs/jobRepository.test.js`, both known to fail on `main` independent of this change — confirm by checking they're the *only* failures, nothing else new).
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/app.js test/api/jobs.test.js
|
||||||
|
git commit -m "feat: register ebook converter in the app so the API serves the new family"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deliberately out of scope (documented, not silently dropped)
|
||||||
|
|
||||||
|
- **Actually installing Calibre on o2switch.** This plan only makes the app *able* to call `ebook-convert`; deploying the self-contained Linux Calibre build via SSH and confirming `CALIBRE_PATH`/`PATH` on the production host is an infrastructure step outside this codebase change, to be done once before these jobs can succeed in production.
|
||||||
|
- **Real end-to-end conversion verification.** Every test here mocks the subprocess. The first real Calibre invocation against real files happens either after o2switch deployment or if Calibre for Windows is installed locally later — both explicitly deferred per the approved design.
|
||||||
|
- **A separate `WORKER_CONCURRENCY` limit for the `ebook` family.** Noted as a future consideration in the design if `ebook` traffic turns out to dominate the shared-hosting resource budget; not implemented now (YAGNI).
|
||||||
Reference in New Issue
Block a user