1974 lines
52 KiB
Markdown
1974 lines
52 KiB
Markdown
# Ombrora-YTDLP Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Build a public web interface for submitting video URLs to download via yt-dlp, with a MariaDB-backed queue, background Node.js worker, and temporary expiring download links.
|
|
|
|
**Architecture:** Next.js 15 (App Router) handles UI and API routes; a separate Node.js worker script (`worker/index.ts`) runs under Passenger on o2switch, polls a MariaDB queue via Prisma, and spawns concurrent yt-dlp processes. Files are stored locally as `{uuid}.{ext}` and served via expiring token links.
|
|
|
|
**Tech Stack:** Node.js, Next.js 15 (App Router), React, TypeScript, Prisma, MariaDB, Jest, ts-jest, Docker Compose (local dev only), yt-dlp (precompiled binary, not an npm package)
|
|
|
|
## Global Constraints
|
|
|
|
- Single `package.json` / `node_modules` at root — o2switch Passenger constraint, no subfolder installs
|
|
- No row is ever deleted from the DB (`Download` or `DownloadToken`) — full history preserved
|
|
- Files stored as `{STORAGE_PATH}/{uuid}.{extension}` — never by original video title
|
|
- `uuid` is the only identifier exposed on interfaces; `id` (cuid) is internal only
|
|
- yt-dlp always invoked with `--no-playlist`
|
|
- Passenger runs a single Next.js process — rate limiter is a plain in-memory Map (no Redis)
|
|
- All tunable values live in `config/app.config.ts` — no hardcoded values elsewhere
|
|
|
|
---
|
|
|
|
## File Map
|
|
|
|
```
|
|
Ombrora-YTDLP/
|
|
├── config/
|
|
│ └── app.config.ts # All tunable config values
|
|
├── prisma/
|
|
│ └── schema.prisma # Download + DownloadToken models
|
|
├── src/
|
|
│ ├── app/
|
|
│ │ ├── page.tsx # Submission form page (/)
|
|
│ │ ├── status/[uuid]/page.tsx # Status page
|
|
│ │ └── api/
|
|
│ │ ├── downloads/
|
|
│ │ │ ├── route.ts # POST /api/downloads
|
|
│ │ │ └── [uuid]/route.ts # GET /api/downloads/[uuid]
|
|
│ │ └── download/
|
|
│ │ └── [token]/route.ts # GET /api/download/[token] (stream)
|
|
│ ├── components/
|
|
│ │ ├── SubmitForm.tsx # 'use client' form
|
|
│ │ └── StatusView.tsx # 'use client' polling status
|
|
│ └── lib/
|
|
│ ├── prisma.ts # Prisma client singleton
|
|
│ ├── rate-limit.ts # IP rate limiter (in-memory Map)
|
|
│ ├── token.ts # createToken / validateToken
|
|
│ └── ytdlp.ts # yt-dlp argument builder
|
|
├── worker/
|
|
│ ├── index.ts # Main poll loop (exports pollOnce for tests)
|
|
│ ├── processor.ts # processDownload() — runs yt-dlp for one item
|
|
│ ├── cron-cleanup.ts # Deletes expired files, marks FILE_DELETED
|
|
│ └── cron-check.ts # Restarts worker if not running (PID file)
|
|
├── docker-compose.yml
|
|
├── .env.example
|
|
└── package.json
|
|
```
|
|
|
|
---
|
|
|
|
### Task 1: Project scaffold
|
|
|
|
**Files:**
|
|
- Create: `package.json`
|
|
- Create: `tsconfig.json`
|
|
- Create: `next.config.ts`
|
|
- Create: `docker-compose.yml`
|
|
- Create: `.env.example`
|
|
- Create: `.gitignore`
|
|
|
|
**Interfaces:**
|
|
- Produces: working `npm run dev`, `npm run build`, `npm test` commands; MariaDB accessible on `localhost:3306`
|
|
|
|
- [ ] **Step 1: Init npm**
|
|
|
|
```bash
|
|
npm init -y
|
|
```
|
|
|
|
- [ ] **Step 2: Install dependencies**
|
|
|
|
```bash
|
|
npm install next react react-dom
|
|
npm install prisma @prisma/client
|
|
npm install -D typescript @types/node @types/react @types/react-dom
|
|
npm install -D ts-node
|
|
npm install -D jest @types/jest ts-jest jest-environment-node
|
|
```
|
|
|
|
- [ ] **Step 3: Write `package.json` scripts and jest config**
|
|
|
|
Replace the `scripts` section and add `jest` config in `package.json`:
|
|
|
|
```json
|
|
{
|
|
"scripts": {
|
|
"dev": "next dev",
|
|
"build": "next build",
|
|
"start": "next start",
|
|
"test": "jest",
|
|
"worker": "ts-node -r tsconfig-paths/register worker/index.ts",
|
|
"worker:cleanup": "ts-node -r tsconfig-paths/register worker/cron-cleanup.ts",
|
|
"worker:check": "ts-node -r tsconfig-paths/register worker/cron-check.ts",
|
|
"db:migrate": "prisma migrate dev",
|
|
"db:generate": "prisma generate"
|
|
},
|
|
"jest": {
|
|
"testEnvironment": "node",
|
|
"transform": { "^.+\\.tsx?$": "ts-jest" },
|
|
"moduleNameMapper": { "^@/(.*)$": "<rootDir>/src/$1" },
|
|
"testMatch": ["**/__tests__/**/*.test.ts"]
|
|
}
|
|
}
|
|
```
|
|
|
|
Also run:
|
|
```bash
|
|
npm install -D tsconfig-paths
|
|
```
|
|
|
|
- [ ] **Step 4: Write `tsconfig.json`**
|
|
|
|
```json
|
|
{
|
|
"compilerOptions": {
|
|
"target": "ES2020",
|
|
"lib": ["dom", "dom.iterable", "esnext"],
|
|
"allowJs": true,
|
|
"skipLibCheck": true,
|
|
"strict": true,
|
|
"noEmit": true,
|
|
"esModuleInterop": true,
|
|
"module": "esnext",
|
|
"moduleResolution": "bundler",
|
|
"resolveJsonModule": true,
|
|
"isolatedModules": true,
|
|
"jsx": "preserve",
|
|
"incremental": true,
|
|
"paths": {
|
|
"@/*": ["./src/*"]
|
|
}
|
|
},
|
|
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
|
|
"exclude": ["node_modules"]
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 5: Write `next.config.ts`**
|
|
|
|
```typescript
|
|
import type { NextConfig } from 'next'
|
|
|
|
const nextConfig: NextConfig = {}
|
|
|
|
export default nextConfig
|
|
```
|
|
|
|
- [ ] **Step 6: Write `docker-compose.yml`**
|
|
|
|
```yaml
|
|
services:
|
|
db:
|
|
image: mariadb:11
|
|
environment:
|
|
MYSQL_ROOT_PASSWORD: root
|
|
MYSQL_DATABASE: ombrora
|
|
MYSQL_USER: ombrora
|
|
MYSQL_PASSWORD: ombrora
|
|
ports:
|
|
- "3306:3306"
|
|
volumes:
|
|
- db_data:/var/lib/mysql
|
|
|
|
volumes:
|
|
db_data:
|
|
```
|
|
|
|
- [ ] **Step 7: Write `.env.example`**
|
|
|
|
```
|
|
DATABASE_URL="mysql://ombrora:ombrora@localhost:3306/ombrora"
|
|
STORAGE_PATH="/absolute/path/to/local/storage"
|
|
```
|
|
|
|
- [ ] **Step 8: Write `.gitignore`**
|
|
|
|
```
|
|
node_modules/
|
|
.next/
|
|
.env
|
|
*.log
|
|
storage/
|
|
worker/worker.pid
|
|
```
|
|
|
|
- [ ] **Step 9: Start MariaDB and verify**
|
|
|
|
```bash
|
|
docker compose up -d
|
|
docker compose ps
|
|
```
|
|
|
|
Expected: `db` container status `Up`.
|
|
|
|
- [ ] **Step 10: Commit**
|
|
|
|
```bash
|
|
git init
|
|
git add package.json tsconfig.json next.config.ts docker-compose.yml .env.example .gitignore
|
|
git commit -m "chore: scaffold project"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Configuration + Prisma schema + Prisma client
|
|
|
|
**Files:**
|
|
- Create: `config/app.config.ts`
|
|
- Create: `prisma/schema.prisma`
|
|
- Create: `src/lib/prisma.ts`
|
|
- Create: `.env` (not committed)
|
|
|
|
**Interfaces:**
|
|
- Produces:
|
|
- `config` — typed const object from `config/app.config.ts`
|
|
- `prisma` — PrismaClient singleton from `src/lib/prisma.ts`
|
|
- DB tables: `Download`, `DownloadToken` with all fields from spec
|
|
|
|
- [ ] **Step 1: Write `.env`**
|
|
|
|
```
|
|
DATABASE_URL="mysql://ombrora:ombrora@localhost:3306/ombrora"
|
|
STORAGE_PATH="/absolute/path/to/your/local/storage"
|
|
```
|
|
|
|
Replace `STORAGE_PATH` with an actual absolute path that exists on your machine (create the folder if needed).
|
|
|
|
- [ ] **Step 2: Write `config/app.config.ts`**
|
|
|
|
```typescript
|
|
export const config = {
|
|
DOWNLOAD_LINK_TTL_HOURS: 24,
|
|
WORKER_CONCURRENCY: 3,
|
|
WORKER_POLL_INTERVAL_MS: 10_000,
|
|
STORAGE_PATH: process.env.STORAGE_PATH ?? '',
|
|
RATE_LIMIT_MAX: 5,
|
|
RATE_LIMIT_WINDOW_MS: 3_600_000,
|
|
} as const
|
|
```
|
|
|
|
- [ ] **Step 3: Write `prisma/schema.prisma`**
|
|
|
|
```prisma
|
|
generator client {
|
|
provider = "prisma-client-js"
|
|
}
|
|
|
|
datasource db {
|
|
provider = "mysql"
|
|
url = env("DATABASE_URL")
|
|
}
|
|
|
|
model Download {
|
|
id String @id @default(cuid())
|
|
uuid String @unique @default(uuid())
|
|
url String @db.Text
|
|
status Status @default(PENDING)
|
|
format String
|
|
quality String
|
|
subtitles Boolean @default(false)
|
|
extraArgs String? @db.Text
|
|
|
|
filePath String? @db.Text
|
|
fileName String?
|
|
fileSize BigInt?
|
|
errorMsg String? @db.Text
|
|
|
|
ipAddress String
|
|
submittedAt DateTime @default(now())
|
|
startedAt DateTime?
|
|
completedAt DateTime?
|
|
deletedAt DateTime?
|
|
|
|
tokens DownloadToken[]
|
|
}
|
|
|
|
model DownloadToken {
|
|
id String @id @default(cuid())
|
|
download Download @relation(fields: [downloadId], references: [id])
|
|
downloadId String
|
|
token String @unique @default(uuid())
|
|
expiresAt DateTime
|
|
usedAt DateTime?
|
|
createdAt DateTime @default(now())
|
|
}
|
|
|
|
enum Status {
|
|
PENDING
|
|
PROCESSING
|
|
DONE
|
|
FAILED
|
|
FILE_DELETED
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Write `src/lib/prisma.ts`**
|
|
|
|
```typescript
|
|
import { PrismaClient } from '@prisma/client'
|
|
|
|
const globalForPrisma = globalThis as unknown as { prisma: PrismaClient }
|
|
|
|
export const prisma =
|
|
globalForPrisma.prisma ?? new PrismaClient()
|
|
|
|
if (process.env.NODE_ENV !== 'production') globalForPrisma.prisma = prisma
|
|
```
|
|
|
|
- [ ] **Step 5: Run Prisma migration**
|
|
|
|
```bash
|
|
npx prisma migrate dev --name init
|
|
```
|
|
|
|
Expected: `Your database is now in sync with your schema.`
|
|
|
|
- [ ] **Step 6: Verify tables**
|
|
|
|
```bash
|
|
docker exec -it $(docker compose ps -q db) mariadb -u ombrora -pombrora ombrora -e "SHOW TABLES;"
|
|
```
|
|
|
|
Expected output includes `Download`, `DownloadToken`, `_prisma_migrations`.
|
|
|
|
- [ ] **Step 7: Commit**
|
|
|
|
```bash
|
|
git add config/ prisma/ src/lib/prisma.ts
|
|
git commit -m "chore: add config, prisma schema and client"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Rate limiter
|
|
|
|
**Files:**
|
|
- Create: `src/lib/rate-limit.ts`
|
|
- Create: `src/lib/__tests__/rate-limit.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `config.RATE_LIMIT_MAX`, `config.RATE_LIMIT_WINDOW_MS`
|
|
- Produces: `isRateLimited(ip: string): boolean` — returns `true` if the IP has exceeded the limit, increments counter otherwise
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/lib/__tests__/rate-limit.test.ts`:
|
|
|
|
```typescript
|
|
import { isRateLimited } from '../rate-limit'
|
|
|
|
beforeEach(() => {
|
|
jest.useFakeTimers()
|
|
jest.resetModules()
|
|
})
|
|
|
|
afterEach(() => {
|
|
jest.useRealTimers()
|
|
})
|
|
|
|
describe('isRateLimited', () => {
|
|
it('allows requests under the limit', () => {
|
|
const ip = '1.2.3.4'
|
|
for (let i = 0; i < 5; i++) {
|
|
expect(isRateLimited(ip)).toBe(false)
|
|
}
|
|
})
|
|
|
|
it('blocks the 6th request within the window', () => {
|
|
const ip = '10.0.0.1'
|
|
for (let i = 0; i < 5; i++) isRateLimited(ip)
|
|
expect(isRateLimited(ip)).toBe(true)
|
|
})
|
|
|
|
it('resets after the window expires', () => {
|
|
const ip = '10.0.0.2'
|
|
for (let i = 0; i < 5; i++) isRateLimited(ip)
|
|
expect(isRateLimited(ip)).toBe(true)
|
|
jest.advanceTimersByTime(3_600_001)
|
|
expect(isRateLimited(ip)).toBe(false)
|
|
})
|
|
|
|
it('tracks different IPs independently', () => {
|
|
const ipA = '192.168.1.1'
|
|
const ipB = '192.168.1.2'
|
|
for (let i = 0; i < 5; i++) isRateLimited(ipA)
|
|
expect(isRateLimited(ipA)).toBe(true)
|
|
expect(isRateLimited(ipB)).toBe(false)
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/rate-limit.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../rate-limit'`
|
|
|
|
- [ ] **Step 3: Write `src/lib/rate-limit.ts`**
|
|
|
|
```typescript
|
|
import { config } from '../../config/app.config'
|
|
|
|
type Entry = { count: number; resetAt: number }
|
|
const store = new Map<string, Entry>()
|
|
|
|
export function isRateLimited(ip: string): boolean {
|
|
const now = Date.now()
|
|
const entry = store.get(ip)
|
|
|
|
if (!entry || now > entry.resetAt) {
|
|
store.set(ip, { count: 1, resetAt: now + config.RATE_LIMIT_WINDOW_MS })
|
|
return false
|
|
}
|
|
|
|
if (entry.count >= config.RATE_LIMIT_MAX) return true
|
|
|
|
entry.count++
|
|
return false
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/rate-limit.test.ts
|
|
```
|
|
|
|
Expected: PASS (4 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/rate-limit.ts src/lib/__tests__/rate-limit.test.ts
|
|
git commit -m "feat: add IP-based rate limiter"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Token utilities
|
|
|
|
**Files:**
|
|
- Create: `src/lib/token.ts`
|
|
- Create: `src/lib/__tests__/token.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `prisma`, `config.DOWNLOAD_LINK_TTL_HOURS`
|
|
- Produces:
|
|
- `createToken(downloadId: string): Promise<string>` — inserts a `DownloadToken` row, returns the `token` string
|
|
- `validateToken(token: string): Promise<{ downloadId: string; filePath: string } | null>` — returns null if expired, not found, or filePath is null
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/lib/__tests__/token.test.ts`:
|
|
|
|
```typescript
|
|
import { createToken, validateToken } from '../token'
|
|
import { prisma } from '../prisma'
|
|
|
|
jest.mock('../prisma', () => ({
|
|
prisma: {
|
|
downloadToken: {
|
|
create: jest.fn(),
|
|
findUnique: jest.fn(),
|
|
},
|
|
},
|
|
}))
|
|
|
|
const mockCreate = (prisma.downloadToken.create as jest.Mock)
|
|
const mockFindUnique = (prisma.downloadToken.findUnique as jest.Mock)
|
|
|
|
describe('createToken', () => {
|
|
it('inserts a DownloadToken and returns the token string', async () => {
|
|
mockCreate.mockResolvedValue({ token: 'generated-token' })
|
|
|
|
const result = await createToken('dl-id-1')
|
|
|
|
expect(mockCreate).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
data: expect.objectContaining({ downloadId: 'dl-id-1' }),
|
|
})
|
|
)
|
|
expect(result).toBe('generated-token')
|
|
})
|
|
})
|
|
|
|
describe('validateToken', () => {
|
|
it('returns null when token not found', async () => {
|
|
mockFindUnique.mockResolvedValue(null)
|
|
expect(await validateToken('bad')).toBeNull()
|
|
})
|
|
|
|
it('returns null when token is expired', async () => {
|
|
mockFindUnique.mockResolvedValue({
|
|
expiresAt: new Date(Date.now() - 1000),
|
|
download: { id: 'dl-1', filePath: '/storage/abc.mp4' },
|
|
})
|
|
expect(await validateToken('expired-tok')).toBeNull()
|
|
})
|
|
|
|
it('returns null when filePath is null', async () => {
|
|
mockFindUnique.mockResolvedValue({
|
|
expiresAt: new Date(Date.now() + 3_600_000),
|
|
download: { id: 'dl-1', filePath: null },
|
|
})
|
|
expect(await validateToken('tok')).toBeNull()
|
|
})
|
|
|
|
it('returns downloadId and filePath for a valid token', async () => {
|
|
mockFindUnique.mockResolvedValue({
|
|
expiresAt: new Date(Date.now() + 3_600_000),
|
|
download: { id: 'dl-1', filePath: '/storage/abc.mp4' },
|
|
})
|
|
expect(await validateToken('tok')).toEqual({
|
|
downloadId: 'dl-1',
|
|
filePath: '/storage/abc.mp4',
|
|
})
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/token.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../token'`
|
|
|
|
- [ ] **Step 3: Write `src/lib/token.ts`**
|
|
|
|
```typescript
|
|
import { prisma } from './prisma'
|
|
import { config } from '../../config/app.config'
|
|
|
|
export async function createToken(downloadId: string): Promise<string> {
|
|
const expiresAt = new Date(
|
|
Date.now() + config.DOWNLOAD_LINK_TTL_HOURS * 60 * 60 * 1000
|
|
)
|
|
const record = await prisma.downloadToken.create({
|
|
data: { downloadId, expiresAt },
|
|
})
|
|
return record.token
|
|
}
|
|
|
|
export async function validateToken(
|
|
token: string
|
|
): Promise<{ downloadId: string; filePath: string } | null> {
|
|
const record = await prisma.downloadToken.findUnique({
|
|
where: { token },
|
|
include: { download: { select: { id: true, filePath: true } } },
|
|
})
|
|
if (!record) return null
|
|
if (record.expiresAt < new Date()) return null
|
|
if (!record.download.filePath) return null
|
|
return { downloadId: record.download.id, filePath: record.download.filePath }
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/token.test.ts
|
|
```
|
|
|
|
Expected: PASS (5 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/token.ts src/lib/__tests__/token.test.ts
|
|
git commit -m "feat: add token create/validate utilities"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: yt-dlp argument builder
|
|
|
|
**Files:**
|
|
- Create: `src/lib/ytdlp.ts`
|
|
- Create: `src/lib/__tests__/ytdlp.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `config.STORAGE_PATH`
|
|
- Produces:
|
|
- `YtdlpParams` type: `{ url: string; uuid: string; format: string; quality: string; subtitles: boolean; extraArgs: string | null }`
|
|
- `buildYtdlpArgs(params: YtdlpParams): string[]` — full argv array to pass to `spawn('yt-dlp', args)`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/lib/__tests__/ytdlp.test.ts`:
|
|
|
|
```typescript
|
|
import { buildYtdlpArgs } from '../ytdlp'
|
|
|
|
const base = {
|
|
url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
|
|
uuid: 'abc-123',
|
|
format: 'mp4',
|
|
quality: 'best',
|
|
subtitles: false,
|
|
extraArgs: null,
|
|
}
|
|
|
|
describe('buildYtdlpArgs', () => {
|
|
it('always includes --no-playlist', () => {
|
|
expect(buildYtdlpArgs(base)).toContain('--no-playlist')
|
|
})
|
|
|
|
it('sets output path with uuid and %(ext)s template', () => {
|
|
const args = buildYtdlpArgs(base)
|
|
const idx = args.indexOf('-o')
|
|
expect(idx).toBeGreaterThan(-1)
|
|
expect(args[idx + 1]).toContain('abc-123')
|
|
expect(args[idx + 1]).toContain('%(ext)s')
|
|
})
|
|
|
|
it('adds format filter when quality is not "best"', () => {
|
|
const args = buildYtdlpArgs({ ...base, quality: '1080p' })
|
|
expect(args).toContain('-f')
|
|
})
|
|
|
|
it('does not add -f flag when quality is "best"', () => {
|
|
expect(buildYtdlpArgs(base)).not.toContain('-f')
|
|
})
|
|
|
|
it('adds subtitle flags when subtitles is true', () => {
|
|
const args = buildYtdlpArgs({ ...base, subtitles: true })
|
|
expect(args).toContain('--write-sub')
|
|
expect(args).toContain('--sub-lang')
|
|
})
|
|
|
|
it('does not add subtitle flags when subtitles is false', () => {
|
|
expect(buildYtdlpArgs(base)).not.toContain('--write-sub')
|
|
})
|
|
|
|
it('appends extra args from a JSON array string', () => {
|
|
const args = buildYtdlpArgs({
|
|
...base,
|
|
extraArgs: '["--sponsorblock-remove","all"]',
|
|
})
|
|
expect(args).toContain('--sponsorblock-remove')
|
|
expect(args).toContain('all')
|
|
})
|
|
|
|
it('ignores malformed extraArgs JSON', () => {
|
|
expect(() =>
|
|
buildYtdlpArgs({ ...base, extraArgs: 'not-json' })
|
|
).not.toThrow()
|
|
})
|
|
|
|
it('url is the last argument', () => {
|
|
const args = buildYtdlpArgs(base)
|
|
expect(args[args.length - 1]).toBe(base.url)
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/ytdlp.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../ytdlp'`
|
|
|
|
- [ ] **Step 3: Write `src/lib/ytdlp.ts`**
|
|
|
|
```typescript
|
|
import path from 'path'
|
|
import { config } from '../../config/app.config'
|
|
|
|
export type YtdlpParams = {
|
|
url: string
|
|
uuid: string
|
|
format: string
|
|
quality: string
|
|
subtitles: boolean
|
|
extraArgs: string | null
|
|
}
|
|
|
|
export function buildYtdlpArgs(params: YtdlpParams): string[] {
|
|
const { url, uuid, format, quality, subtitles, extraArgs } = params
|
|
const outputTemplate = path.join(config.STORAGE_PATH, `${uuid}.%(ext)s`)
|
|
|
|
const args: string[] = ['--no-playlist', '-o', outputTemplate]
|
|
|
|
if (quality !== 'best') {
|
|
const height = quality.replace('p', '')
|
|
args.push('-f', `${format}[height<=?${height}]+bestaudio/best[height<=?${height}]`)
|
|
}
|
|
|
|
if (subtitles) {
|
|
args.push('--write-sub', '--sub-lang', 'fr,en')
|
|
}
|
|
|
|
if (extraArgs) {
|
|
try {
|
|
const extra = JSON.parse(extraArgs) as string[]
|
|
args.push(...extra)
|
|
} catch {
|
|
// malformed extraArgs — skip silently
|
|
}
|
|
}
|
|
|
|
args.push(url)
|
|
return args
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- src/lib/__tests__/ytdlp.test.ts
|
|
```
|
|
|
|
Expected: PASS (9 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/lib/ytdlp.ts src/lib/__tests__/ytdlp.test.ts
|
|
git commit -m "feat: add yt-dlp argument builder"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 6: POST /api/downloads — Submission endpoint
|
|
|
|
**Files:**
|
|
- Create: `src/app/api/downloads/route.ts`
|
|
- Create: `src/app/api/downloads/__tests__/post.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `prisma.download.create`, `isRateLimited(ip: string): boolean`
|
|
- Produces: `POST /api/downloads`
|
|
- Request body: `{ url: string; format: string; quality: string; subtitles: boolean; extraArgs?: string | null }`
|
|
- Response 201: `{ uuid: string }`
|
|
- Response 400: `{ error: string }` — missing or invalid fields
|
|
- Response 429: `{ error: 'Rate limit exceeded' }`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/app/api/downloads/__tests__/post.test.ts`:
|
|
|
|
```typescript
|
|
import { POST } from '../route'
|
|
import { prisma } from '@/lib/prisma'
|
|
import * as rateLimit from '@/lib/rate-limit'
|
|
|
|
jest.mock('@/lib/prisma', () => ({
|
|
prisma: { download: { create: jest.fn() } },
|
|
}))
|
|
jest.mock('@/lib/rate-limit')
|
|
|
|
const mockCreate = prisma.download.create as jest.Mock
|
|
const mockIsRateLimited = rateLimit.isRateLimited as jest.Mock
|
|
|
|
function req(body: object, ip = '1.2.3.4') {
|
|
return new Request('http://localhost/api/downloads', {
|
|
method: 'POST',
|
|
headers: { 'content-type': 'application/json', 'x-forwarded-for': ip },
|
|
body: JSON.stringify(body),
|
|
})
|
|
}
|
|
|
|
describe('POST /api/downloads', () => {
|
|
beforeEach(() => {
|
|
mockIsRateLimited.mockReturnValue(false)
|
|
mockCreate.mockResolvedValue({ uuid: 'test-uuid' })
|
|
})
|
|
|
|
it('returns 429 when rate limited', async () => {
|
|
mockIsRateLimited.mockReturnValue(true)
|
|
const res = await POST(req({ url: 'https://y.com', format: 'mp4', quality: 'best', subtitles: false }))
|
|
expect(res.status).toBe(429)
|
|
})
|
|
|
|
it('returns 400 when url is missing', async () => {
|
|
const res = await POST(req({ format: 'mp4', quality: 'best', subtitles: false }))
|
|
expect(res.status).toBe(400)
|
|
})
|
|
|
|
it('returns 400 when format is missing', async () => {
|
|
const res = await POST(req({ url: 'https://y.com', quality: 'best', subtitles: false }))
|
|
expect(res.status).toBe(400)
|
|
})
|
|
|
|
it('returns 201 with uuid on success', async () => {
|
|
const res = await POST(req({ url: 'https://youtube.com/watch?v=abc', format: 'mp4', quality: 'best', subtitles: false }))
|
|
expect(res.status).toBe(201)
|
|
expect(await res.json()).toEqual({ uuid: 'test-uuid' })
|
|
})
|
|
|
|
it('passes correct fields to prisma', async () => {
|
|
await POST(req({ url: 'https://y.com/watch?v=x', format: 'mp3', quality: '720p', subtitles: true }, '9.9.9.9'))
|
|
expect(mockCreate).toHaveBeenCalledWith({
|
|
data: expect.objectContaining({
|
|
url: 'https://y.com/watch?v=x',
|
|
format: 'mp3',
|
|
quality: '720p',
|
|
subtitles: true,
|
|
ipAddress: '9.9.9.9',
|
|
}),
|
|
})
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- src/app/api/downloads/__tests__/post.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../route'`
|
|
|
|
- [ ] **Step 3: Write `src/app/api/downloads/route.ts`**
|
|
|
|
```typescript
|
|
import { NextRequest, NextResponse } from 'next/server'
|
|
import { prisma } from '@/lib/prisma'
|
|
import { isRateLimited } from '@/lib/rate-limit'
|
|
|
|
export async function POST(req: NextRequest) {
|
|
const ip =
|
|
req.headers.get('x-forwarded-for')?.split(',')[0].trim() ?? '0.0.0.0'
|
|
|
|
if (isRateLimited(ip)) {
|
|
return NextResponse.json({ error: 'Rate limit exceeded' }, { status: 429 })
|
|
}
|
|
|
|
const body = await req.json().catch(() => null)
|
|
|
|
if (
|
|
!body?.url ||
|
|
!body?.format ||
|
|
!body?.quality ||
|
|
body?.subtitles === undefined
|
|
) {
|
|
return NextResponse.json(
|
|
{ error: 'Missing required fields: url, format, quality, subtitles' },
|
|
{ status: 400 }
|
|
)
|
|
}
|
|
|
|
const download = await prisma.download.create({
|
|
data: {
|
|
url: String(body.url),
|
|
format: String(body.format),
|
|
quality: String(body.quality),
|
|
subtitles: Boolean(body.subtitles),
|
|
extraArgs: body.extraArgs ?? null,
|
|
ipAddress: ip,
|
|
},
|
|
})
|
|
|
|
return NextResponse.json({ uuid: download.uuid }, { status: 201 })
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- src/app/api/downloads/__tests__/post.test.ts
|
|
```
|
|
|
|
Expected: PASS (5 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/app/api/downloads/
|
|
git commit -m "feat: add POST /api/downloads submission endpoint"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: GET /api/downloads/[uuid] — Status endpoint
|
|
|
|
**Files:**
|
|
- Create: `src/app/api/downloads/[uuid]/route.ts`
|
|
- Create: `src/app/api/downloads/[uuid]/__tests__/get.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `prisma.download.findUnique` with `tokens` relation
|
|
- Produces: `GET /api/downloads/[uuid]`
|
|
- Response 200: `{ uuid, status, format, quality, subtitles, fileName, fileSize, errorMsg, submittedAt, completedAt, downloadToken, tokenExpiresAt }`
|
|
- `fileSize` serialized as string (BigInt not JSON-serializable)
|
|
- `downloadToken` — token string, only present when `status === 'DONE'` and a non-expired token exists; otherwise `null`
|
|
- Response 404: `{ error: 'Not found' }`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/app/api/downloads/[uuid]/__tests__/get.test.ts`:
|
|
|
|
```typescript
|
|
import { GET } from '../route'
|
|
import { prisma } from '@/lib/prisma'
|
|
|
|
jest.mock('@/lib/prisma', () => ({
|
|
prisma: { download: { findUnique: jest.fn() } },
|
|
}))
|
|
|
|
const mockFindUnique = prisma.download.findUnique as jest.Mock
|
|
|
|
function req(uuid: string) {
|
|
return new Request(`http://localhost/api/downloads/${uuid}`)
|
|
}
|
|
|
|
const baseDownload = {
|
|
uuid: 'abc',
|
|
status: 'DONE',
|
|
format: 'mp4',
|
|
quality: 'best',
|
|
subtitles: false,
|
|
fileName: 'video.mp4',
|
|
fileSize: BigInt(1_048_576),
|
|
errorMsg: null,
|
|
submittedAt: new Date('2026-01-01'),
|
|
completedAt: new Date('2026-01-01'),
|
|
}
|
|
|
|
describe('GET /api/downloads/[uuid]', () => {
|
|
it('returns 404 when not found', async () => {
|
|
mockFindUnique.mockResolvedValue(null)
|
|
const res = await GET(req('missing'), { params: Promise.resolve({ uuid: 'missing' }) })
|
|
expect(res.status).toBe(404)
|
|
})
|
|
|
|
it('returns 200 with download fields', async () => {
|
|
mockFindUnique.mockResolvedValue({ ...baseDownload, tokens: [] })
|
|
const res = await GET(req('abc'), { params: Promise.resolve({ uuid: 'abc' }) })
|
|
expect(res.status).toBe(200)
|
|
const body = await res.json()
|
|
expect(body.uuid).toBe('abc')
|
|
expect(body.fileSize).toBe('1048576')
|
|
})
|
|
|
|
it('returns downloadToken when status is DONE and token is valid', async () => {
|
|
mockFindUnique.mockResolvedValue({
|
|
...baseDownload,
|
|
tokens: [{ token: 'valid-tok', expiresAt: new Date(Date.now() + 3_600_000) }],
|
|
})
|
|
const res = await GET(req('abc'), { params: Promise.resolve({ uuid: 'abc' }) })
|
|
const body = await res.json()
|
|
expect(body.downloadToken).toBe('valid-tok')
|
|
})
|
|
|
|
it('returns null downloadToken when token is expired', async () => {
|
|
mockFindUnique.mockResolvedValue({
|
|
...baseDownload,
|
|
tokens: [{ token: 'expired-tok', expiresAt: new Date(Date.now() - 1000) }],
|
|
})
|
|
const res = await GET(req('abc'), { params: Promise.resolve({ uuid: 'abc' }) })
|
|
const body = await res.json()
|
|
expect(body.downloadToken).toBeNull()
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- "src/app/api/downloads/\[uuid\]/__tests__/get.test.ts"
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../route'`
|
|
|
|
- [ ] **Step 3: Write `src/app/api/downloads/[uuid]/route.ts`**
|
|
|
|
```typescript
|
|
import { NextRequest, NextResponse } from 'next/server'
|
|
import { prisma } from '@/lib/prisma'
|
|
|
|
export async function GET(
|
|
_req: NextRequest,
|
|
{ params }: { params: Promise<{ uuid: string }> }
|
|
) {
|
|
const { uuid } = await params
|
|
|
|
const download = await prisma.download.findUnique({
|
|
where: { uuid },
|
|
include: {
|
|
tokens: { orderBy: { createdAt: 'desc' }, take: 1 },
|
|
},
|
|
})
|
|
|
|
if (!download) {
|
|
return NextResponse.json({ error: 'Not found' }, { status: 404 })
|
|
}
|
|
|
|
const latestToken = download.tokens[0]
|
|
const validToken =
|
|
latestToken && latestToken.expiresAt > new Date() ? latestToken : null
|
|
|
|
return NextResponse.json({
|
|
uuid: download.uuid,
|
|
status: download.status,
|
|
format: download.format,
|
|
quality: download.quality,
|
|
subtitles: download.subtitles,
|
|
fileName: download.fileName,
|
|
fileSize: download.fileSize?.toString() ?? null,
|
|
errorMsg: download.errorMsg,
|
|
submittedAt: download.submittedAt,
|
|
completedAt: download.completedAt,
|
|
downloadToken: download.status === 'DONE' ? (validToken?.token ?? null) : null,
|
|
tokenExpiresAt: download.status === 'DONE' ? (validToken?.expiresAt ?? null) : null,
|
|
})
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- "src/app/api/downloads/\[uuid\]/__tests__/get.test.ts"
|
|
```
|
|
|
|
Expected: PASS (4 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add "src/app/api/downloads/[uuid]/"
|
|
git commit -m "feat: add GET /api/downloads/[uuid] status endpoint"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 8: GET /api/download/[token] — File streaming endpoint
|
|
|
|
**Files:**
|
|
- Create: `src/app/api/download/[token]/route.ts`
|
|
- Create: `src/app/api/download/[token]/__tests__/get.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `validateToken(token): Promise<{ downloadId: string; filePath: string } | null>`, `prisma.downloadToken.update`
|
|
- Produces: `GET /api/download/[token]`
|
|
- Response 200: file stream, headers `Content-Disposition: attachment; filename="..."`, `Content-Type: application/octet-stream`
|
|
- Response 404: `{ error: 'Invalid or expired token' }` or `{ error: 'File not found on server' }`
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `src/app/api/download/[token]/__tests__/get.test.ts`:
|
|
|
|
```typescript
|
|
import { GET } from '../route'
|
|
import * as tokenLib from '@/lib/token'
|
|
import { prisma } from '@/lib/prisma'
|
|
|
|
jest.mock('@/lib/token')
|
|
jest.mock('@/lib/prisma', () => ({
|
|
prisma: { downloadToken: { update: jest.fn() } },
|
|
}))
|
|
|
|
const mockValidate = tokenLib.validateToken as jest.Mock
|
|
const mockUpdate = prisma.downloadToken.update as jest.Mock
|
|
|
|
function req(token: string) {
|
|
return new Request(`http://localhost/api/download/${token}`)
|
|
}
|
|
|
|
describe('GET /api/download/[token]', () => {
|
|
it('returns 404 for invalid or expired token', async () => {
|
|
mockValidate.mockResolvedValue(null)
|
|
const res = await GET(req('bad'), { params: Promise.resolve({ token: 'bad' }) })
|
|
expect(res.status).toBe(404)
|
|
const body = await res.json()
|
|
expect(body.error).toBe('Invalid or expired token')
|
|
})
|
|
|
|
it('returns 404 when file does not exist on disk', async () => {
|
|
mockValidate.mockResolvedValue({ downloadId: 'dl-1', filePath: '/nonexistent/path/abc.mp4' })
|
|
mockUpdate.mockResolvedValue({})
|
|
const res = await GET(req('ok-tok'), { params: Promise.resolve({ token: 'ok-tok' }) })
|
|
expect(res.status).toBe(404)
|
|
const body = await res.json()
|
|
expect(body.error).toBe('File not found on server')
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- "src/app/api/download/\[token\]/__tests__/get.test.ts"
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../route'`
|
|
|
|
- [ ] **Step 3: Write `src/app/api/download/[token]/route.ts`**
|
|
|
|
```typescript
|
|
import { NextRequest, NextResponse } from 'next/server'
|
|
import { createReadStream, statSync } from 'fs'
|
|
import path from 'path'
|
|
import { validateToken } from '@/lib/token'
|
|
import { prisma } from '@/lib/prisma'
|
|
|
|
export async function GET(
|
|
_req: NextRequest,
|
|
{ params }: { params: Promise<{ token: string }> }
|
|
) {
|
|
const { token } = await params
|
|
const result = await validateToken(token)
|
|
|
|
if (!result) {
|
|
return NextResponse.json({ error: 'Invalid or expired token' }, { status: 404 })
|
|
}
|
|
|
|
const { filePath } = result
|
|
|
|
let stat: ReturnType<typeof statSync>
|
|
try {
|
|
stat = statSync(filePath)
|
|
} catch {
|
|
return NextResponse.json({ error: 'File not found on server' }, { status: 404 })
|
|
}
|
|
|
|
await prisma.downloadToken.update({
|
|
where: { token },
|
|
data: { usedAt: new Date() },
|
|
})
|
|
|
|
const fileName = path.basename(filePath)
|
|
const stream = createReadStream(filePath)
|
|
|
|
return new NextResponse(stream as unknown as ReadableStream, {
|
|
headers: {
|
|
'Content-Disposition': `attachment; filename="${fileName}"`,
|
|
'Content-Length': String(stat.size),
|
|
'Content-Type': 'application/octet-stream',
|
|
},
|
|
})
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- "src/app/api/download/\[token\]/__tests__/get.test.ts"
|
|
```
|
|
|
|
Expected: PASS (2 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add "src/app/api/download/[token]/"
|
|
git commit -m "feat: add GET /api/download/[token] file streaming endpoint"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 9: Worker processor
|
|
|
|
**Files:**
|
|
- Create: `worker/processor.ts`
|
|
- Create: `worker/__tests__/processor.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `buildYtdlpArgs(params: YtdlpParams)`, `prisma.download.findUnique`, `prisma.download.update`, `createToken(downloadId: string): Promise<string>`
|
|
- Produces: `processDownload(downloadId: string): Promise<void>` — never throws; updates DB on success or failure
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `worker/__tests__/processor.test.ts`:
|
|
|
|
```typescript
|
|
import { EventEmitter } from 'events'
|
|
import { processDownload } from '../processor'
|
|
import { prisma } from '@/lib/prisma'
|
|
import * as tokenLib from '@/lib/token'
|
|
import * as cp from 'child_process'
|
|
|
|
jest.mock('@/lib/prisma', () => ({
|
|
prisma: {
|
|
download: { findUnique: jest.fn(), update: jest.fn() },
|
|
},
|
|
}))
|
|
jest.mock('@/lib/token')
|
|
jest.mock('child_process')
|
|
jest.mock('fs', () => ({
|
|
readdirSync: jest.fn(() => ['uuid-abc.mp4']),
|
|
statSync: jest.fn(() => ({ size: 1024 })),
|
|
}))
|
|
|
|
const mockFindUnique = prisma.download.findUnique as jest.Mock
|
|
const mockUpdate = prisma.download.update as jest.Mock
|
|
const mockCreateToken = tokenLib.createToken as jest.Mock
|
|
const mockSpawn = cp.spawn as jest.Mock
|
|
|
|
const baseDownload = {
|
|
id: 'dl-1',
|
|
uuid: 'uuid-abc',
|
|
url: 'https://youtube.com/watch?v=abc',
|
|
format: 'mp4',
|
|
quality: 'best',
|
|
subtitles: false,
|
|
extraArgs: null,
|
|
}
|
|
|
|
function makeChildProcess(exitCode: number, stderrMsg = '') {
|
|
const proc = new EventEmitter() as any
|
|
proc.stdout = new EventEmitter()
|
|
proc.stderr = new EventEmitter()
|
|
setImmediate(() => {
|
|
if (stderrMsg) proc.stderr.emit('data', Buffer.from(stderrMsg))
|
|
proc.emit('close', exitCode)
|
|
})
|
|
return proc
|
|
}
|
|
|
|
describe('processDownload', () => {
|
|
beforeEach(() => {
|
|
mockFindUnique.mockResolvedValue(baseDownload)
|
|
mockUpdate.mockResolvedValue({})
|
|
mockCreateToken.mockResolvedValue('new-token')
|
|
})
|
|
|
|
it('marks download as DONE on yt-dlp exit code 0', async () => {
|
|
mockSpawn.mockReturnValue(makeChildProcess(0))
|
|
await processDownload('dl-1')
|
|
expect(mockUpdate).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
where: { id: 'dl-1' },
|
|
data: expect.objectContaining({ status: 'DONE' }),
|
|
})
|
|
)
|
|
})
|
|
|
|
it('creates a token on success', async () => {
|
|
mockSpawn.mockReturnValue(makeChildProcess(0))
|
|
await processDownload('dl-1')
|
|
expect(mockCreateToken).toHaveBeenCalledWith('dl-1')
|
|
})
|
|
|
|
it('marks download as FAILED on non-zero exit code', async () => {
|
|
mockSpawn.mockReturnValue(makeChildProcess(1, 'unsupported URL'))
|
|
await processDownload('dl-1')
|
|
expect(mockUpdate).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
data: expect.objectContaining({
|
|
status: 'FAILED',
|
|
errorMsg: expect.stringContaining('unsupported URL'),
|
|
}),
|
|
})
|
|
)
|
|
})
|
|
|
|
it('does nothing when download not found', async () => {
|
|
mockFindUnique.mockResolvedValue(null)
|
|
await expect(processDownload('nonexistent')).resolves.toBeUndefined()
|
|
expect(mockSpawn).not.toHaveBeenCalled()
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- worker/__tests__/processor.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `Cannot find module '../processor'`
|
|
|
|
- [ ] **Step 3: Write `worker/processor.ts`**
|
|
|
|
```typescript
|
|
import { spawn } from 'child_process'
|
|
import { readdirSync, statSync } from 'fs'
|
|
import path from 'path'
|
|
import { prisma } from '@/lib/prisma'
|
|
import { buildYtdlpArgs } from '@/lib/ytdlp'
|
|
import { createToken } from '@/lib/token'
|
|
import { config } from '../config/app.config'
|
|
|
|
export async function processDownload(downloadId: string): Promise<void> {
|
|
const download = await prisma.download.findUnique({ where: { id: downloadId } })
|
|
if (!download) return
|
|
|
|
const args = buildYtdlpArgs({
|
|
url: download.url,
|
|
uuid: download.uuid,
|
|
format: download.format,
|
|
quality: download.quality,
|
|
subtitles: download.subtitles,
|
|
extraArgs: download.extraArgs,
|
|
})
|
|
|
|
let stderr = ''
|
|
|
|
await new Promise<void>((resolve) => {
|
|
const proc = spawn('yt-dlp', args)
|
|
proc.stderr.on('data', (chunk: Buffer) => { stderr += chunk.toString() })
|
|
|
|
proc.on('close', async (code) => {
|
|
if (code === 0) {
|
|
const files = readdirSync(config.STORAGE_PATH).filter((f) =>
|
|
f.startsWith(download.uuid)
|
|
)
|
|
const relFile = files[0] ?? null
|
|
const filePath = relFile ? path.join(config.STORAGE_PATH, relFile) : null
|
|
const fileSize = filePath ? BigInt(statSync(filePath).size) : null
|
|
|
|
await prisma.download.update({
|
|
where: { id: downloadId },
|
|
data: {
|
|
status: 'DONE',
|
|
filePath,
|
|
fileName: relFile,
|
|
fileSize,
|
|
completedAt: new Date(),
|
|
},
|
|
})
|
|
await createToken(downloadId)
|
|
} else {
|
|
await prisma.download.update({
|
|
where: { id: downloadId },
|
|
data: {
|
|
status: 'FAILED',
|
|
errorMsg: stderr || `yt-dlp exited with code ${code}`,
|
|
completedAt: new Date(),
|
|
},
|
|
})
|
|
}
|
|
resolve()
|
|
})
|
|
})
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- worker/__tests__/processor.test.ts
|
|
```
|
|
|
|
Expected: PASS (4 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add worker/processor.ts worker/__tests__/processor.test.ts
|
|
git commit -m "feat: add yt-dlp download processor"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 10: Worker main loop
|
|
|
|
**Files:**
|
|
- Create: `worker/index.ts`
|
|
- Create: `worker/__tests__/index.test.ts`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `prisma.download.findMany`, `prisma.download.updateMany`, `processDownload(id: string): Promise<void>`, `config.WORKER_CONCURRENCY`, `config.WORKER_POLL_INTERVAL_MS`
|
|
- Produces:
|
|
- `pollOnce(): Promise<void>` — exported for testing; one poll cycle
|
|
- `runWorker(): Promise<never>` — infinite loop; entry point when run directly
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
Create `worker/__tests__/index.test.ts`:
|
|
|
|
```typescript
|
|
import { prisma } from '@/lib/prisma'
|
|
import * as processor from '../processor'
|
|
|
|
jest.mock('@/lib/prisma', () => ({
|
|
prisma: {
|
|
download: {
|
|
findMany: jest.fn(),
|
|
updateMany: jest.fn(),
|
|
},
|
|
},
|
|
}))
|
|
jest.mock('../processor')
|
|
|
|
const mockFindMany = prisma.download.findMany as jest.Mock
|
|
const mockUpdateMany = prisma.download.updateMany as jest.Mock
|
|
const mockProcess = processor.processDownload as jest.Mock
|
|
|
|
describe('pollOnce', () => {
|
|
beforeEach(() => {
|
|
jest.resetModules()
|
|
mockProcess.mockClear()
|
|
})
|
|
|
|
it('processes all PENDING downloads', async () => {
|
|
mockFindMany.mockResolvedValue([{ id: 'dl-1' }, { id: 'dl-2' }])
|
|
mockUpdateMany.mockResolvedValue({ count: 2 })
|
|
mockProcess.mockResolvedValue(undefined)
|
|
|
|
const { pollOnce } = await import('../index')
|
|
await pollOnce()
|
|
|
|
expect(mockUpdateMany).toHaveBeenCalledWith(
|
|
expect.objectContaining({
|
|
where: expect.objectContaining({ status: 'PENDING' }),
|
|
data: expect.objectContaining({ status: 'PROCESSING' }),
|
|
})
|
|
)
|
|
expect(mockProcess).toHaveBeenCalledTimes(2)
|
|
expect(mockProcess).toHaveBeenCalledWith('dl-1')
|
|
expect(mockProcess).toHaveBeenCalledWith('dl-2')
|
|
})
|
|
|
|
it('does nothing when queue is empty', async () => {
|
|
mockFindMany.mockResolvedValue([])
|
|
|
|
const { pollOnce } = await import('../index')
|
|
await pollOnce()
|
|
|
|
expect(mockUpdateMany).not.toHaveBeenCalled()
|
|
expect(mockProcess).not.toHaveBeenCalled()
|
|
})
|
|
})
|
|
```
|
|
|
|
- [ ] **Step 2: Run test to verify it fails**
|
|
|
|
```bash
|
|
npm test -- worker/__tests__/index.test.ts
|
|
```
|
|
|
|
Expected: FAIL — `pollOnce is not exported` or module not found
|
|
|
|
- [ ] **Step 3: Write `worker/index.ts`**
|
|
|
|
```typescript
|
|
import { prisma } from '@/lib/prisma'
|
|
import { config } from '../config/app.config'
|
|
import { processDownload } from './processor'
|
|
|
|
export async function pollOnce(): Promise<void> {
|
|
const pending = await prisma.download.findMany({
|
|
where: { status: 'PENDING' },
|
|
orderBy: { submittedAt: 'asc' },
|
|
take: config.WORKER_CONCURRENCY,
|
|
select: { id: true },
|
|
})
|
|
|
|
if (pending.length === 0) return
|
|
|
|
const ids = pending.map((d) => d.id)
|
|
|
|
await prisma.download.updateMany({
|
|
where: { id: { in: ids }, status: 'PENDING' },
|
|
data: { status: 'PROCESSING', startedAt: new Date() },
|
|
})
|
|
|
|
await Promise.all(ids.map((id) => processDownload(id)))
|
|
}
|
|
|
|
export async function runWorker(): Promise<never> {
|
|
console.log('[worker] started')
|
|
while (true) {
|
|
try {
|
|
await pollOnce()
|
|
} catch (err) {
|
|
console.error('[worker] poll error:', err)
|
|
}
|
|
await new Promise((resolve) =>
|
|
setTimeout(resolve, config.WORKER_POLL_INTERVAL_MS)
|
|
)
|
|
}
|
|
}
|
|
|
|
if (require.main === module) {
|
|
runWorker().catch(console.error)
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Run test to verify it passes**
|
|
|
|
```bash
|
|
npm test -- worker/__tests__/index.test.ts
|
|
```
|
|
|
|
Expected: PASS (2 tests)
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add worker/index.ts worker/__tests__/index.test.ts
|
|
git commit -m "feat: add worker main poll loop"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 11: Cron scripts
|
|
|
|
**Files:**
|
|
- Create: `worker/cron-cleanup.ts`
|
|
- Create: `worker/cron-check.ts`
|
|
|
|
**Interfaces:**
|
|
- `cron-cleanup.ts`: reads DONE downloads where `completedAt + TTL < now`, deletes files, sets `status=FILE_DELETED`, sets `deletedAt`
|
|
- `cron-check.ts`: reads `worker/worker.pid`; if PID is alive, exits 0; otherwise spawns `worker/index.ts` detached and writes new PID
|
|
|
|
- [ ] **Step 1: Write `worker/cron-cleanup.ts`**
|
|
|
|
```typescript
|
|
import { prisma } from '@/lib/prisma'
|
|
import { config } from '../config/app.config'
|
|
import { unlinkSync, existsSync } from 'fs'
|
|
|
|
async function cleanup() {
|
|
const cutoff = new Date(
|
|
Date.now() - config.DOWNLOAD_LINK_TTL_HOURS * 60 * 60 * 1000
|
|
)
|
|
|
|
const expired = await prisma.download.findMany({
|
|
where: { status: 'DONE', completedAt: { lt: cutoff } },
|
|
select: { id: true, filePath: true },
|
|
})
|
|
|
|
for (const dl of expired) {
|
|
if (dl.filePath && existsSync(dl.filePath)) {
|
|
unlinkSync(dl.filePath)
|
|
}
|
|
await prisma.download.update({
|
|
where: { id: dl.id },
|
|
data: { status: 'FILE_DELETED', deletedAt: new Date() },
|
|
})
|
|
}
|
|
|
|
console.log(`[cron-cleanup] removed ${expired.length} expired file(s)`)
|
|
}
|
|
|
|
cleanup().catch(console.error).finally(() => process.exit(0))
|
|
```
|
|
|
|
- [ ] **Step 2: Write `worker/cron-check.ts`**
|
|
|
|
```typescript
|
|
import { existsSync, readFileSync, writeFileSync } from 'fs'
|
|
import { spawn } from 'child_process'
|
|
import path from 'path'
|
|
|
|
const PID_FILE = path.join(__dirname, 'worker.pid')
|
|
|
|
function isRunning(pid: number): boolean {
|
|
try {
|
|
process.kill(pid, 0)
|
|
return true
|
|
} catch {
|
|
return false
|
|
}
|
|
}
|
|
|
|
if (existsSync(PID_FILE)) {
|
|
const pid = parseInt(readFileSync(PID_FILE, 'utf-8').trim(), 10)
|
|
if (!isNaN(pid) && isRunning(pid)) {
|
|
console.log(`[cron-check] worker already running (PID ${pid})`)
|
|
process.exit(0)
|
|
}
|
|
}
|
|
|
|
const child = spawn(
|
|
'node',
|
|
['-r', 'ts-node/register', '-r', 'tsconfig-paths/register', path.join(__dirname, 'index.ts')],
|
|
{ detached: true, stdio: 'ignore' }
|
|
)
|
|
child.unref()
|
|
|
|
if (child.pid) {
|
|
writeFileSync(PID_FILE, String(child.pid))
|
|
console.log(`[cron-check] worker started (PID ${child.pid})`)
|
|
}
|
|
|
|
process.exit(0)
|
|
```
|
|
|
|
- [ ] **Step 3: Verify TypeScript compiles**
|
|
|
|
```bash
|
|
npx tsc --noEmit
|
|
```
|
|
|
|
Expected: no errors.
|
|
|
|
- [ ] **Step 4: Commit**
|
|
|
|
```bash
|
|
git add worker/cron-cleanup.ts worker/cron-check.ts
|
|
git commit -m "feat: add cron cleanup and worker-check scripts"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 12: Frontend — Submission form
|
|
|
|
**Files:**
|
|
- Create: `src/app/layout.tsx`
|
|
- Create: `src/app/page.tsx`
|
|
- Create: `src/components/SubmitForm.tsx`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `POST /api/downloads` returns `{ uuid: string }`
|
|
- Produces: form at `/`; on success redirects to `/status/{uuid}`
|
|
|
|
- [ ] **Step 1: Write `src/app/layout.tsx`**
|
|
|
|
```tsx
|
|
import type { Metadata } from 'next'
|
|
|
|
export const metadata: Metadata = {
|
|
title: 'Ombrora — Telechargeur de videos',
|
|
}
|
|
|
|
export default function RootLayout({ children }: { children: React.ReactNode }) {
|
|
return (
|
|
<html lang="fr">
|
|
<body>{children}</body>
|
|
</html>
|
|
)
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Write `src/components/SubmitForm.tsx`**
|
|
|
|
```tsx
|
|
'use client'
|
|
|
|
import { useState, FormEvent } from 'react'
|
|
import { useRouter } from 'next/navigation'
|
|
|
|
const FORMATS = ['mp4', 'mp3', 'webm', 'mkv']
|
|
const QUALITIES = ['best', '1080p', '720p', '480p', '360p']
|
|
|
|
export function SubmitForm() {
|
|
const router = useRouter()
|
|
const [url, setUrl] = useState('')
|
|
const [format, setFormat] = useState('mp4')
|
|
const [quality, setQuality] = useState('best')
|
|
const [subtitles, setSubtitles] = useState(false)
|
|
const [extraArgs, setExtraArgs] = useState('')
|
|
const [error, setError] = useState<string | null>(null)
|
|
const [loading, setLoading] = useState(false)
|
|
|
|
async function handleSubmit(e: FormEvent) {
|
|
e.preventDefault()
|
|
setError(null)
|
|
setLoading(true)
|
|
|
|
const res = await fetch('/api/downloads', {
|
|
method: 'POST',
|
|
headers: { 'content-type': 'application/json' },
|
|
body: JSON.stringify({
|
|
url,
|
|
format,
|
|
quality,
|
|
subtitles,
|
|
extraArgs: extraArgs.trim() || null,
|
|
}),
|
|
})
|
|
|
|
setLoading(false)
|
|
|
|
if (res.status === 429) {
|
|
setError('Trop de soumissions. Reessayez dans une heure.')
|
|
return
|
|
}
|
|
if (!res.ok) {
|
|
const body = await res.json().catch(() => ({}))
|
|
setError((body as { error?: string }).error ?? 'Erreur lors de la soumission.')
|
|
return
|
|
}
|
|
|
|
const { uuid } = (await res.json()) as { uuid: string }
|
|
router.push(`/status/${uuid}`)
|
|
}
|
|
|
|
return (
|
|
<form onSubmit={handleSubmit} style={{ display: 'flex', flexDirection: 'column', gap: '1rem', maxWidth: 480 }}>
|
|
<label>
|
|
URL de la video
|
|
<input
|
|
type="url"
|
|
value={url}
|
|
onChange={(e) => setUrl(e.target.value)}
|
|
required
|
|
placeholder="https://www.youtube.com/watch?v=..."
|
|
style={{ display: 'block', width: '100%' }}
|
|
/>
|
|
</label>
|
|
|
|
<label>
|
|
Format
|
|
<select value={format} onChange={(e) => setFormat(e.target.value)} style={{ display: 'block' }}>
|
|
{FORMATS.map((f) => <option key={f}>{f}</option>)}
|
|
</select>
|
|
</label>
|
|
|
|
<label>
|
|
Qualite
|
|
<select value={quality} onChange={(e) => setQuality(e.target.value)} style={{ display: 'block' }}>
|
|
{QUALITIES.map((q) => <option key={q}>{q}</option>)}
|
|
</select>
|
|
</label>
|
|
|
|
<label>
|
|
<input
|
|
type="checkbox"
|
|
checked={subtitles}
|
|
onChange={(e) => setSubtitles(e.target.checked)}
|
|
/>{' '}
|
|
Telecharger les sous-titres (fr, en)
|
|
</label>
|
|
|
|
<label>
|
|
Options avancees (optionnel, JSON)
|
|
<input
|
|
type="text"
|
|
value={extraArgs}
|
|
onChange={(e) => setExtraArgs(e.target.value)}
|
|
placeholder='["--sponsorblock-remove","all"]'
|
|
style={{ display: 'block', width: '100%' }}
|
|
/>
|
|
</label>
|
|
|
|
{error && <p style={{ color: 'red' }}>{error}</p>}
|
|
|
|
<button type="submit" disabled={loading}>
|
|
{loading ? 'Envoi...' : 'Telecharger'}
|
|
</button>
|
|
</form>
|
|
)
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 3: Write `src/app/page.tsx`**
|
|
|
|
```tsx
|
|
import { SubmitForm } from '@/components/SubmitForm'
|
|
|
|
export default function Home() {
|
|
return (
|
|
<main style={{ padding: '2rem' }}>
|
|
<h1>Ombrora</h1>
|
|
<SubmitForm />
|
|
</main>
|
|
)
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 4: Start dev server and verify**
|
|
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
Open `http://localhost:3000`. Verify the form renders with all fields. Submit an invalid URL — expect a browser validation error. Correct it with a valid URL, submit, and verify a redirect to `/status/{uuid}` (the status page will show an error since it does not exist yet).
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/app/ src/components/SubmitForm.tsx
|
|
git commit -m "feat: add video submission form"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 13: Frontend — Status page
|
|
|
|
**Files:**
|
|
- Create: `src/app/status/[uuid]/page.tsx`
|
|
- Create: `src/components/StatusView.tsx`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `GET /api/downloads/[uuid]` polled every 5 seconds
|
|
- Produces: status page at `/status/[uuid]`
|
|
- `PENDING` / `PROCESSING` — spinner text, auto-refreshing
|
|
- `DONE` — download link (`/api/download/{token}`), file name, size, expiry date
|
|
- `FAILED` — error message from yt-dlp
|
|
- `FILE_DELETED` — expiry notice
|
|
|
|
- [ ] **Step 1: Write `src/components/StatusView.tsx`**
|
|
|
|
```tsx
|
|
'use client'
|
|
|
|
import { useEffect, useState } from 'react'
|
|
|
|
type DownloadData = {
|
|
uuid: string
|
|
status: 'PENDING' | 'PROCESSING' | 'DONE' | 'FAILED' | 'FILE_DELETED'
|
|
format: string
|
|
quality: string
|
|
fileName: string | null
|
|
fileSize: string | null
|
|
errorMsg: string | null
|
|
downloadToken: string | null
|
|
tokenExpiresAt: string | null
|
|
}
|
|
|
|
export function StatusView({ uuid }: { uuid: string }) {
|
|
const [data, setData] = useState<DownloadData | null>(null)
|
|
const [notFound, setNotFound] = useState(false)
|
|
|
|
useEffect(() => {
|
|
let active = true
|
|
|
|
async function poll() {
|
|
const res = await fetch(`/api/downloads/${uuid}`)
|
|
if (!active) return
|
|
|
|
if (res.status === 404) {
|
|
setNotFound(true)
|
|
return
|
|
}
|
|
|
|
const body = (await res.json()) as DownloadData
|
|
setData(body)
|
|
|
|
if (body.status === 'PENDING' || body.status === 'PROCESSING') {
|
|
setTimeout(poll, 5_000)
|
|
}
|
|
}
|
|
|
|
poll()
|
|
return () => { active = false }
|
|
}, [uuid])
|
|
|
|
if (notFound) return <p>Telechargement introuvable.</p>
|
|
if (!data) return <p>Chargement...</p>
|
|
|
|
if (data.status === 'PENDING' || data.status === 'PROCESSING') {
|
|
return (
|
|
<div>
|
|
<p>Statut : {data.status === 'PENDING' ? 'En attente' : 'En cours...'}</p>
|
|
<p>Cette page se rafraichit automatiquement toutes les 5 secondes.</p>
|
|
</div>
|
|
)
|
|
}
|
|
|
|
if (data.status === 'DONE' && data.downloadToken) {
|
|
const sizeMb = data.fileSize
|
|
? `(${(Number(data.fileSize) / 1_048_576).toFixed(1)} Mo)`
|
|
: ''
|
|
return (
|
|
<div>
|
|
<p>Telechargement pret {sizeMb}</p>
|
|
<a
|
|
href={`/api/download/${data.downloadToken}`}
|
|
download={data.fileName ?? undefined}
|
|
>
|
|
Telecharger {data.fileName}
|
|
</a>
|
|
{data.tokenExpiresAt && (
|
|
<p style={{ fontSize: '0.85rem', color: '#666' }}>
|
|
Lien valable jusqu'au{' '}
|
|
{new Date(data.tokenExpiresAt).toLocaleString('fr-FR')}
|
|
</p>
|
|
)}
|
|
</div>
|
|
)
|
|
}
|
|
|
|
if (data.status === 'FAILED') {
|
|
return (
|
|
<div>
|
|
<p>Echec du telechargement.</p>
|
|
<pre style={{ background: '#fee', padding: '0.5rem', overflowX: 'auto' }}>
|
|
{data.errorMsg}
|
|
</pre>
|
|
</div>
|
|
)
|
|
}
|
|
|
|
if (data.status === 'FILE_DELETED') {
|
|
return (
|
|
<p>
|
|
Le fichier a expire et a ete supprime. Le telechargement n'est plus
|
|
disponible.
|
|
</p>
|
|
)
|
|
}
|
|
|
|
return <p>Statut inconnu : {data.status}</p>
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Write `src/app/status/[uuid]/page.tsx`**
|
|
|
|
```tsx
|
|
import { StatusView } from '@/components/StatusView'
|
|
|
|
export default async function StatusPage({
|
|
params,
|
|
}: {
|
|
params: Promise<{ uuid: string }>
|
|
}) {
|
|
const { uuid } = await params
|
|
return (
|
|
<main style={{ padding: '2rem' }}>
|
|
<h1>Statut du telechargement</h1>
|
|
<StatusView uuid={uuid} />
|
|
</main>
|
|
)
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 3: End-to-end golden path test**
|
|
|
|
With MariaDB running (`docker compose up -d`) and dev server running (`npm run dev`):
|
|
|
|
1. Open `http://localhost:3000`.
|
|
2. Paste a valid YouTube URL, keep format `mp4`, quality `best`, click Telecharger.
|
|
3. Verify redirect to `/status/{uuid}` showing "En attente".
|
|
4. In a second terminal, verify yt-dlp is installed: `yt-dlp --version`
|
|
5. Start the worker: `npm run worker`
|
|
6. Wait — the status page should transition from "En attente" to "En cours..." to "Telechargement pret".
|
|
7. Click the download link and verify the file downloads.
|
|
8. Run cleanup manually and verify FILE_DELETED:
|
|
```bash
|
|
# Temporarily set TTL to 0 to force expiry — edit config, then:
|
|
npm run worker:cleanup
|
|
# Reload /status/{uuid} — should show expiry message
|
|
```
|
|
|
|
- [ ] **Step 4: Run full test suite**
|
|
|
|
```bash
|
|
npm test
|
|
```
|
|
|
|
Expected: all tests pass.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/app/status/ src/components/StatusView.tsx
|
|
git commit -m "feat: add status page with polling and download link"
|
|
```
|
|
|
|
---
|
|
|
|
## Self-Review
|
|
|
|
### Spec coverage
|
|
|
|
| Requirement | Task |
|
|
|------------|------|
|
|
| Public web interface | 12 |
|
|
| URL + format + quality + subtitles + extraArgs | 5, 6, 12 |
|
|
| DB-backed queue, no deletions ever | 2, 10 |
|
|
| Worker with concurrency (WORKER_CONCURRENCY) | 2, 10 |
|
|
| yt-dlp with --no-playlist | 5, 9 |
|
|
| Files stored as {uuid}.{ext} | 5, 9 |
|
|
| Rate limiting by IP | 3, 6 |
|
|
| Status page with polling | 13 |
|
|
| Temporary download link (token) | 4, 8 |
|
|
| Token TTL configurable, default 24h | 2, 4 |
|
|
| FILE_DELETED status + cron cleanup | 2, 11 |
|
|
| cron-check to restart worker | 11 |
|
|
| uuid on interfaces, id internal only | 2, 7 |
|
|
| Docker Compose MariaDB for local dev | 1 |
|
|
| Single package.json (o2switch) | 1 |
|