150 lines
12 KiB
Markdown
150 lines
12 KiB
Markdown
# Archive conversion (zip, tar, tar.gz, tar.bz2, 7z, tar.7z, rar)
|
||
|
||
Date: 2026-08-01
|
||
|
||
## Goal
|
||
|
||
Add a new `archive` family. Requested formats: `zip`, `tar`, `tar.gz`, `7z`, `rar`, `tar.bz2` (user wrote `tar.bz`, meaning bzip2), `tar.7z`. All-pairs, **except**:
|
||
|
||
- `rar` is source-only. Confirmed with the user: the `unrar` library's license explicitly bars using it to re-implement the RAR compression algorithm, so no npm package (or anything else) can legally create `.rar` files. Every existing RAR-handling package (`node-rar`, `node-unrar`, `node-unrar-js`, `rarfile`) is extraction-only for this reason. `rar` is therefore accepted as an input format and never offered as a target.
|
||
- `tar.7z` (not a standard extension) is, per the user, confirmed to mean: a `.tar` stream compressed with the 7z/LZMA algorithm — the same relationship `tar.gz` has to gzip and `tar.bz2` has to bzip2.
|
||
- Unlike every other family in this codebase (image, document, font, ebook — all reject `sourceFormat === targetFormat`), the user explicitly asked for same-format pairs to be allowed here (`zip -> zip`, `7z -> 7z`, etc.), since recompressing at a different level is a real use case specific to this family.
|
||
|
||
Net registration: 7 source formats (`zip`, `tar`, `tar.gz`, `tar.bz2`, `7z`, `tar.7z`, `rar`) × 6 target formats (same list minus `rar`) = 42 pairs, family `'archive'`.
|
||
|
||
## Architecture: extract-then-rebuild
|
||
|
||
One generic pipeline in `src/converters/archive.js`, `sourceFormat`/`targetFormat` bound via closures at registration time (same pattern `image.js` uses for `sharpFormatName`):
|
||
|
||
```js
|
||
async function convert(inputPath, outputPath, { quality } = {}, sourceFormat, targetFormat) {
|
||
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'archive-convert-'));
|
||
try {
|
||
const extractDir = path.join(tmpDir, 'extracted');
|
||
await fs.mkdir(extractDir, { recursive: true });
|
||
await EXTRACTORS[sourceFormat](inputPath, extractDir);
|
||
await assertNoPathEscape(extractDir); // see Security
|
||
await CREATORS[targetFormat](extractDir, outputPath, quality);
|
||
} finally {
|
||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||
}
|
||
}
|
||
|
||
export function registerArchiveConverters() {
|
||
for (const sourceFormat of Object.keys(EXTRACTORS)) {
|
||
for (const targetFormat of Object.keys(CREATORS)) {
|
||
register({
|
||
family: 'archive',
|
||
sourceFormat,
|
||
targetFormat,
|
||
convert: (inputPath, outputPath, options) => convert(inputPath, outputPath, options, sourceFormat, targetFormat),
|
||
});
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
No `config`/storage-dir threading needed — `os.tmpdir()` + `fs.mkdtemp` is the standard Node scratch-space pattern and keeps the `convert(inputPath, outputPath, options)` signature identical to every other family.
|
||
|
||
### Libraries (all verified: prebuilt binary or pure JS, nothing requires compilation)
|
||
|
||
| Format | Extract | Create | Compression level |
|
||
|---|---|---|---|
|
||
| `zip` | `adm-zip` (pure JS) | `archiver` (pure JS, `zlib: { level }`) | 0 (store) – 9 (max deflate) |
|
||
| `tar` | `tar` (pure JS) | `tar` | none (like `gif`/`ico` today) |
|
||
| `tar.gz` | `tar` (`gzip` option decompresses) | `tar` (`{ gzip: { level } }`) | 0–9, zlib gzip level |
|
||
| `tar.bz2` | `7zip-min` (`unpack`) decompresses the bzip2 layer to an intermediate `.tar`, then `tar` extracts that (7za reads bzip2 natively — no separate bzip2 library needed) | `tar` builds a plain `.tar`, then `7zip-min` recompresses it (`cmd(['a', '-tbzip2', '-mx=' + level, outputPath, tarPath])`) | 1–9 (bzip2 has no level 0; a requested `0` is clamped to `1` inside the converter, not at the validation layer — same layering `image.js`'s `buildFormatOptions` already uses for PNG vs. other formats) |
|
||
| `7z` | `7zip-min` (`unpack`) | `7zip-min` (`cmd(['a', '-mx=' + level, outputPath, extractDir + '/*'])`) | 0–9, mapped straight to `-mx` |
|
||
| `tar.7z` | `7zip-min` unpacks the `.7z` to get an intermediate `.tar`, then `tar` extracts that | `tar` builds a plain `.tar`, then `7zip-min` packs it into a `.7z` (`-mx` level) | 0–9, same as `7z` |
|
||
| `rar` (extract only) | `node-unrar-js` (WASM, pure JS — this is *why* it's extraction-only: the same license restriction that blocks every other package blocks this one too) | — | n/a |
|
||
|
||
`7zip-min` wraps a **precompiled** `7za` binary (via its `7zip-bin` dependency) invoked through `child_process` — no compilation, same category of dependency as `sharp` (bundles libvips) and `puppeteer` (bundles Chromium, and already proves child-process spawning works on the o2switch deployment target). Verified API: `pack(src, dest)`, `unpack(archive, destDir)`, and the low-level `cmd(argsArray)` for the `-mx=N` level flag and the `-tbzip2` format switch, all promise-returning.
|
||
|
||
New dependencies to add to `package.json` (root only — these are backend-only Node deps with no frontend import, so the o2switch single-`node_modules` mirroring rule for frontend runtime deps doesn't apply): `adm-zip`, `archiver`, `tar`, `7zip-min`, `node-unrar-js`.
|
||
|
||
## Compression level: reuse `quality`, uniform 0–9 scale
|
||
|
||
No schema change — `ConversionJob.quality` (`Int?`) already exists and already flows `app.js` → `createJob` → `worker.js` → `entry.convert(..., { quality })` unchanged.
|
||
|
||
`isValidQuality(targetFormat, quality)` in `src/app.js` gains:
|
||
```js
|
||
if (['zip', 'tar.gz', 'tar.bz2', '7z', 'tar.7z'].includes(targetFormat)) return quality >= 0 && quality <= 9;
|
||
if (targetFormat === 'tar') return false; // no compression knob, same rule as gif/ico today
|
||
```
|
||
placed alongside the existing `gif`/`png` branches (order matters: `tar` must be checked before any fallthrough).
|
||
|
||
## Double extensions: a real gap in today's extension parsing
|
||
|
||
`app.js` currently derives `sourceFormat` via `path.extname(...).slice(1)` in two places (the multer `filename` callback, and the route handler reading `file.filename`), and the download route derives the base filename via `path.parse(job.originalFilename).name`. `path.extname`/`path.parse` only ever see the *last* dot segment — for `backup.tar.gz`, `path.extname` returns `.gz`, not `.tar.gz`, and `path.parse(...).name` returns `backup.tar`, not `backup`. This is a pre-existing gap that becomes load-bearing now that `tar.gz`/`tar.bz2`/`tar.7z` are real formats.
|
||
|
||
Fix: new small module `src/archiveExtensions.js`:
|
||
```js
|
||
const DOUBLE_EXTENSIONS = ['tar.gz', 'tar.bz2', 'tar.7z'];
|
||
|
||
export function extractExtension(filename) {
|
||
const lower = filename.toLowerCase();
|
||
const match = DOUBLE_EXTENSIONS.find((ext) => lower.endsWith(`.${ext}`));
|
||
return match ?? path.extname(filename).slice(1).toLowerCase();
|
||
}
|
||
|
||
export function stripExtension(filename) {
|
||
const ext = extractExtension(filename);
|
||
return filename.slice(0, filename.length - ext.length - 1);
|
||
}
|
||
```
|
||
Used in place of the raw `path.extname`/`path.parse(...).name` calls at all three call sites (multer `filename`, `sourceFormat` computation, download `downloadFilename` base). This is a mechanical fix, not a product decision — flagging it here because it's easy to miss and the feature silently mis-files every double-extension upload without it.
|
||
|
||
## `src/mime.js`
|
||
|
||
`OUTPUT_MIME_TYPES` additions: `zip: 'application/zip'`, `tar: 'application/x-tar'`, `'tar.gz': 'application/gzip'` (IANA-registered per RFC 6713), `'tar.bz2': 'application/x-bzip2'`, `'7z': 'application/x-7z-compressed'`, `'tar.7z': 'application/x-7z-compressed'`.
|
||
|
||
`resolveInputFormat`/`normalizeFormat`: confirmed by reading the installed `file-type` v22's `source/supported.js` — it detects `zip`, `tar`, `rar`, `gz`, `bz2`, `7z` by magic bytes (including `tar`, via the `ustar` marker at offset 257). It sniffs the *outer compression layer only* — a `.tar.gz` sniffs as `gz`, a `.tar.bz2` as `bz2`, a `.tar.7z` as `7z`, with no way to confirm the decompressed payload is actually a tar stream (same class of limitation the ebook design accepted for `fb2`/`lrf`/etc. — not a security gap, since a mismatched payload fails cleanly inside `convert()` with the existing generic `failed` status, just a weaker input check). `normalizeFormat` gains:
|
||
```js
|
||
if (format === 'tar.gz') return 'gz';
|
||
if (format === 'tar.bz2') return 'bz2';
|
||
if (format === 'tar.7z') return '7z';
|
||
```
|
||
(existing identity fallback already makes the comparison symmetric — `normalizeFormat('gz') === 'gz'`).
|
||
|
||
## Security: path-escape defense, applied uniformly post-extraction
|
||
|
||
Rather than trusting each of the five extraction libraries individually to guard against zip-slip/path-traversal (some — `tar`'s default extraction — are documented to strip `..`/absolute paths already; others — `adm-zip`, `node-unrar-js`, `7zip-min` shelling to `7za` — make no such guarantee, and `7za` itself has had real historical path-escape CVEs), one shared check runs after every extraction, before the rebuild step:
|
||
|
||
```js
|
||
async function assertNoPathEscape(extractDir) {
|
||
const resolvedRoot = await fs.realpath(extractDir);
|
||
for (const entry of await fs.readdir(extractDir, { recursive: true, withFileTypes: true })) {
|
||
const fullPath = path.join(entry.parentPath, entry.name);
|
||
const real = entry.isSymbolicLink() ? await fs.realpath(fullPath) : fullPath;
|
||
if (!real.startsWith(resolvedRoot + path.sep) && real !== resolvedRoot) {
|
||
throw new Error('Archive entry escapes extraction directory');
|
||
}
|
||
}
|
||
}
|
||
```
|
||
This throws into the existing generic `catch` in `worker.js`'s `processJob` unchanged (job marked `failed`, generic user-facing message, real error in `errorLog`).
|
||
|
||
Also: a cap on total decompressed bytes during extraction (zip-bomb defense) — accumulate size as each extractor writes files, abort past `config.maxFileSizeMb * 20`. Since converters don't currently receive `config`, this multiplier is a constant defined in `archive.js` itself (e.g. `MAX_EXTRACTED_BYTES = 2 * 1024 * 1024 * 1024`, 2 GiB flat) rather than threading `config` through — simpler, and consistent with `archive.js` having no other config dependency.
|
||
|
||
## Registration wiring
|
||
|
||
`src/app.js` and `src/worker.js`: add `import { registerArchiveConverters } from './converters/archive.js'` and one call inside `registerAllConverters()` / `main()`, alongside the existing seven.
|
||
|
||
## Frontend
|
||
|
||
- `frontend/src/data/formats.js`: new entry `{ key: 'archives', formats: ['zip', 'tar', 'tar.gz', 'tar.bz2', '7z', 'tar.7z', 'rar'] }` (rar included here only so upload/icon detection recognizes it as an archive; it never appears as a selectable target, which falls out naturally from `GET /api/formats` reflecting the registry — no special-case frontend code needed).
|
||
- `frontend/src/utils/fileFamily.js`: `FAMILY_ICONS.archives` — `Archive` icon from `@phosphor-icons/react` (already a dependency).
|
||
- `frontend/src/components/FileConfigCard.jsx`: new conditional block, same `RangeField` pattern as the existing PNG control, min 0 max 9, shown when `item.targetFormat` is one of `zip`, `tar.gz`, `tar.bz2`, `7z`, `tar.7z` (not `tar`, matching the `gif`/`ico` precedent of no control at all when there's nothing to tune).
|
||
|
||
No DB/schema changes anywhere in this feature.
|
||
|
||
## Testing
|
||
|
||
`test/converters/archive.test.js`, same convention as `document.test.js`/`image.test.js`: small fixture archives (2–3 files + one subdirectory) for each of the 7 source formats. Not all 42 pairs — one representative test per extractor and per creator (covers every library once), plus explicitly: `zip -> zip` (the same-format recompression case), and a hand-built malicious zip (a `../../evil` entry) asserting `convert` rejects it via `assertNoPathEscape` rather than writing outside the temp dir.
|
||
|
||
`test/mime.test.js`: cases for the `tar.gz`/`tar.bz2`/`tar.7z` → `gz`/`bz2`/`7z` alias mapping.
|
||
|
||
`test/archiveExtensions.test.js` (new, small): `extractExtension`/`stripExtension` against plain and double-extension filenames.
|
||
|
||
Frontend: manual browser check (upload a `.zip`, confirm target chips include the other 5 formats and the compression slider appears/disappears correctly per target).
|