specs

2026-08-02 10:57:41 +02:00
parent 3a9836d58b
commit 74c676df9f
4 changed files with 2134 additions and 1 deletions
@@ -0,0 +1,149 @@
+# Archive conversion (zip, tar, tar.gz, tar.bz2, 7z, tar.7z, rar)
+
+Date: 2026-08-01
+
+## Goal
+
+Add a new `archive` family. Requested formats: `zip`, `tar`, `tar.gz`, `7z`, `rar`, `tar.bz2` (user wrote `tar.bz`, meaning bzip2), `tar.7z`. All-pairs, **except**:
+
+- `rar` is source-only. Confirmed with the user: the `unrar` library's license explicitly bars using it to re-implement the RAR compression algorithm, so no npm package (or anything else) can legally create `.rar` files. Every existing RAR-handling package (`node-rar`, `node-unrar`, `node-unrar-js`, `rarfile`) is extraction-only for this reason. `rar` is therefore accepted as an input format and never offered as a target.
+- `tar.7z` (not a standard extension) is, per the user, confirmed to mean: a `.tar` stream compressed with the 7z/LZMA algorithm — the same relationship `tar.gz` has to gzip and `tar.bz2` has to bzip2.
+- Unlike every other family in this codebase (image, document, font, ebook — all reject `sourceFormat === targetFormat`), the user explicitly asked for same-format pairs to be allowed here (`zip -> zip`, `7z -> 7z`, etc.), since recompressing at a different level is a real use case specific to this family.
+
+Net registration: 7 source formats (`zip`, `tar`, `tar.gz`, `tar.bz2`, `7z`, `tar.7z`, `rar`) × 6 target formats (same list minus `rar`) = 42 pairs, family `'archive'`.
+
+## Architecture: extract-then-rebuild
+
+One generic pipeline in `src/converters/archive.js`, `sourceFormat`/`targetFormat` bound via closures at registration time (same pattern `image.js` uses for `sharpFormatName`):
+
+```js
+async function convert(inputPath, outputPath, { quality } = {}, sourceFormat, targetFormat) {
+  const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'archive-convert-'));
+  try {
+    const extractDir = path.join(tmpDir, 'extracted');
+    await fs.mkdir(extractDir, { recursive: true });
+    await EXTRACTORS[sourceFormat](inputPath, extractDir);
+    await assertNoPathEscape(extractDir); // see Security
+    await CREATORS[targetFormat](extractDir, outputPath, quality);
+  } finally {
+    await fs.rm(tmpDir, { recursive: true, force: true });
+  }
+}
+
+export function registerArchiveConverters() {
+  for (const sourceFormat of Object.keys(EXTRACTORS)) {
+    for (const targetFormat of Object.keys(CREATORS)) {
+      register({
+        family: 'archive',
+        sourceFormat,
+        targetFormat,
+        convert: (inputPath, outputPath, options) => convert(inputPath, outputPath, options, sourceFormat, targetFormat),
+      });
+    }
+  }
+}
+```
+
+No `config`/storage-dir threading needed — `os.tmpdir()` + `fs.mkdtemp` is the standard Node scratch-space pattern and keeps the `convert(inputPath, outputPath, options)` signature identical to every other family.
+
+### Libraries (all verified: prebuilt binary or pure JS, nothing requires compilation)
+
+| Format | Extract | Create | Compression level |
+|---|---|---|---|
+| `zip` | `adm-zip` (pure JS) | `archiver` (pure JS, `zlib: { level }`) | 0 (store) – 9 (max deflate) |
+| `tar` | `tar` (pure JS) | `tar` | none (like `gif`/`ico` today) |
+| `tar.gz` | `tar` (`gzip` option decompresses) | `tar` (`{ gzip: { level } }`) | 0–9, zlib gzip level |
+| `tar.bz2` | `7zip-min` (`unpack`) decompresses the bzip2 layer to an intermediate `.tar`, then `tar` extracts that (7za reads bzip2 natively — no separate bzip2 library needed) | `tar` builds a plain `.tar`, then `7zip-min` recompresses it (`cmd(['a', '-tbzip2', '-mx=' + level, outputPath, tarPath])`) | 1–9 (bzip2 has no level 0; a requested `0` is clamped to `1` inside the converter, not at the validation layer — same layering `image.js`'s `buildFormatOptions` already uses for PNG vs. other formats) |
+| `7z` | `7zip-min` (`unpack`) | `7zip-min` (`cmd(['a', '-mx=' + level, outputPath, extractDir + '/*'])`) | 0–9, mapped straight to `-mx` |
+| `tar.7z` | `7zip-min` unpacks the `.7z` to get an intermediate `.tar`, then `tar` extracts that | `tar` builds a plain `.tar`, then `7zip-min` packs it into a `.7z` (`-mx` level) | 0–9, same as `7z` |
+| `rar` (extract only) | `node-unrar-js` (WASM, pure JS — this is *why* it's extraction-only: the same license restriction that blocks every other package blocks this one too) | — | n/a |
+
+`7zip-min` wraps a **precompiled** `7za` binary (via its `7zip-bin` dependency) invoked through `child_process` — no compilation, same category of dependency as `sharp` (bundles libvips) and `puppeteer` (bundles Chromium, and already proves child-process spawning works on the o2switch deployment target). Verified API: `pack(src, dest)`, `unpack(archive, destDir)`, and the low-level `cmd(argsArray)` for the `-mx=N` level flag and the `-tbzip2` format switch, all promise-returning.
+
+New dependencies to add to `package.json` (root only — these are backend-only Node deps with no frontend import, so the o2switch single-`node_modules` mirroring rule for frontend runtime deps doesn't apply): `adm-zip`, `archiver`, `tar`, `7zip-min`, `node-unrar-js`.
+
+## Compression level: reuse `quality`, uniform 0–9 scale
+
+No schema change — `ConversionJob.quality` (`Int?`) already exists and already flows `app.js` → `createJob` → `worker.js` → `entry.convert(..., { quality })` unchanged.
+
+`isValidQuality(targetFormat, quality)` in `src/app.js` gains:
+```js
+if (['zip', 'tar.gz', 'tar.bz2', '7z', 'tar.7z'].includes(targetFormat)) return quality >= 0 && quality <= 9;
+if (targetFormat === 'tar') return false; // no compression knob, same rule as gif/ico today
+```
+placed alongside the existing `gif`/`png` branches (order matters: `tar` must be checked before any fallthrough).
+
+## Double extensions: a real gap in today's extension parsing
+
+`app.js` currently derives `sourceFormat` via `path.extname(...).slice(1)` in two places (the multer `filename` callback, and the route handler reading `file.filename`), and the download route derives the base filename via `path.parse(job.originalFilename).name`. `path.extname`/`path.parse` only ever see the *last* dot segment — for `backup.tar.gz`, `path.extname` returns `.gz`, not `.tar.gz`, and `path.parse(...).name` returns `backup.tar`, not `backup`. This is a pre-existing gap that becomes load-bearing now that `tar.gz`/`tar.bz2`/`tar.7z` are real formats.
+
+Fix: new small module `src/archiveExtensions.js`:
+```js
+const DOUBLE_EXTENSIONS = ['tar.gz', 'tar.bz2', 'tar.7z'];
+
+export function extractExtension(filename) {
+  const lower = filename.toLowerCase();
+  const match = DOUBLE_EXTENSIONS.find((ext) => lower.endsWith(`.${ext}`));
+  return match ?? path.extname(filename).slice(1).toLowerCase();
+}
+
+export function stripExtension(filename) {
+  const ext = extractExtension(filename);
+  return filename.slice(0, filename.length - ext.length - 1);
+}
+```
+Used in place of the raw `path.extname`/`path.parse(...).name` calls at all three call sites (multer `filename`, `sourceFormat` computation, download `downloadFilename` base). This is a mechanical fix, not a product decision — flagging it here because it's easy to miss and the feature silently mis-files every double-extension upload without it.
+
+## `src/mime.js`
+
+`OUTPUT_MIME_TYPES` additions: `zip: 'application/zip'`, `tar: 'application/x-tar'`, `'tar.gz': 'application/gzip'` (IANA-registered per RFC 6713), `'tar.bz2': 'application/x-bzip2'`, `'7z': 'application/x-7z-compressed'`, `'tar.7z': 'application/x-7z-compressed'`.
+
+`resolveInputFormat`/`normalizeFormat`: confirmed by reading the installed `file-type` v22's `source/supported.js` — it detects `zip`, `tar`, `rar`, `gz`, `bz2`, `7z` by magic bytes (including `tar`, via the `ustar` marker at offset 257). It sniffs the *outer compression layer only* — a `.tar.gz` sniffs as `gz`, a `.tar.bz2` as `bz2`, a `.tar.7z` as `7z`, with no way to confirm the decompressed payload is actually a tar stream (same class of limitation the ebook design accepted for `fb2`/`lrf`/etc. — not a security gap, since a mismatched payload fails cleanly inside `convert()` with the existing generic `failed` status, just a weaker input check). `normalizeFormat` gains:
+```js
+if (format === 'tar.gz') return 'gz';
+if (format === 'tar.bz2') return 'bz2';
+if (format === 'tar.7z') return '7z';
+```
+(existing identity fallback already makes the comparison symmetric — `normalizeFormat('gz') === 'gz'`).
+
+## Security: path-escape defense, applied uniformly post-extraction
+
+Rather than trusting each of the five extraction libraries individually to guard against zip-slip/path-traversal (some — `tar`'s default extraction — are documented to strip `..`/absolute paths already; others — `adm-zip`, `node-unrar-js`, `7zip-min` shelling to `7za` — make no such guarantee, and `7za` itself has had real historical path-escape CVEs), one shared check runs after every extraction, before the rebuild step:
+
+```js
+async function assertNoPathEscape(extractDir) {
+  const resolvedRoot = await fs.realpath(extractDir);
+  for (const entry of await fs.readdir(extractDir, { recursive: true, withFileTypes: true })) {
+    const fullPath = path.join(entry.parentPath, entry.name);
+    const real = entry.isSymbolicLink() ? await fs.realpath(fullPath) : fullPath;
+    if (!real.startsWith(resolvedRoot + path.sep) && real !== resolvedRoot) {
+      throw new Error('Archive entry escapes extraction directory');
+    }
+  }
+}
+```
+This throws into the existing generic `catch` in `worker.js`'s `processJob` unchanged (job marked `failed`, generic user-facing message, real error in `errorLog`).
+
+Also: a cap on total decompressed bytes during extraction (zip-bomb defense) — accumulate size as each extractor writes files, abort past `config.maxFileSizeMb * 20`. Since converters don't currently receive `config`, this multiplier is a constant defined in `archive.js` itself (e.g. `MAX_EXTRACTED_BYTES = 2 * 1024 * 1024 * 1024`, 2 GiB flat) rather than threading `config` through — simpler, and consistent with `archive.js` having no other config dependency.
+
+## Registration wiring
+
+`src/app.js` and `src/worker.js`: add `import { registerArchiveConverters } from './converters/archive.js'` and one call inside `registerAllConverters()` / `main()`, alongside the existing seven.
+
+## Frontend
+
+- `frontend/src/data/formats.js`: new entry `{ key: 'archives', formats: ['zip', 'tar', 'tar.gz', 'tar.bz2', '7z', 'tar.7z', 'rar'] }` (rar included here only so upload/icon detection recognizes it as an archive; it never appears as a selectable target, which falls out naturally from `GET /api/formats` reflecting the registry — no special-case frontend code needed).
+- `frontend/src/utils/fileFamily.js`: `FAMILY_ICONS.archives` — `Archive` icon from `@phosphor-icons/react` (already a dependency).
+- `frontend/src/components/FileConfigCard.jsx`: new conditional block, same `RangeField` pattern as the existing PNG control, min 0 max 9, shown when `item.targetFormat` is one of `zip`, `tar.gz`, `tar.bz2`, `7z`, `tar.7z` (not `tar`, matching the `gif`/`ico` precedent of no control at all when there's nothing to tune).
+
+No DB/schema changes anywhere in this feature.
+
+## Testing
+
+`test/converters/archive.test.js`, same convention as `document.test.js`/`image.test.js`: small fixture archives (2–3 files + one subdirectory) for each of the 7 source formats. Not all 42 pairs — one representative test per extractor and per creator (covers every library once), plus explicitly: `zip -> zip` (the same-format recompression case), and a hand-built malicious zip (a `../../evil` entry) asserting `convert` rejects it via `assertNoPathEscape` rather than writing outside the temp dir.
+
+`test/mime.test.js`: cases for the `tar.gz`/`tar.bz2`/`tar.7z` → `gz`/`bz2`/`7z` alias mapping.
+
+`test/archiveExtensions.test.js` (new, small): `extractExtension`/`stripExtension` against plain and double-extension filenames.
+
+Frontend: manual browser check (upload a `.zip`, confirm target chips include the other 5 formats and the compression slider appears/disappears correctly per target).