BlueGit

timfa filemanager

Merge fix/subtitle-buffer-stall-diagnostics

timfa Tim Falken · pushed by timfa

Authored 2026-05-25 · 5 months ago

Hash
fe80f2a9d5fa90f896d21c8c318525f1e9634ea1
5 files changed +214 −20
.github/technical-reference.md +6 −5
@@ -11,13 +11,13 @@
11 11
12 12 ## server/image-subtitles.js
13 13 - parseRenderOptions(options) - Normalises subtitle-frame request options and validates filePath/streamIndex/time/canvas dimensions; stream index `0` is valid and must not be treated as missing; canvas is scaled down within a 1280x720 ceiling, then multiplied by `IMAGE_SUBTITLE_RENDER_SCALE` (default `0.75`) while preserving requested aspect ratio.
14 - renderImageSubtitleFrame(options) - Extracts bitmap subtitle streams to cached `.sup`, parses PGS display sets, and returns {frame: PNG buffer, left, top, width, height, canvasWidth, canvasHeight} with metadata for proper client-side upscaling; the server preserves positioning info instead of throwing it away, allowing the client to calculate percentage-based sizing for downscaled frames.
14 - renderImageSubtitleFrame(options) - Extracts bitmap subtitle streams to cached `.sup`, parses PGS display sets, and returns {frame: PNG buffer, left, top, width, height, canvasWidth, canvasHeight} with metadata for proper client-side upscaling; the server preserves positioning info instead of throwing it away, allowing the client to calculate percentage-based sizing for downscaled frames; request/build/render timing counters are tracked for debug diagnostics.
15 15 - clearCachesForFile(filePath) - Removes all event and frame cache entries for a given file path using path-prefixed cache keys, also deletes the corresponding `.sup` file from disk for each evicted event entry; called when video stream sessions end to prevent memory and disk accumulation.
16 16 - pruneOldImageSubtitleCaches(filePath, slowestViewerTimeSeconds) - Drops subtitle frame caches for events that have passed their retention threshold of end time + event duration + 5 seconds (e.g., event ending at 30s with 10s duration is dropped after slowest viewer reaches 45s); used during segment pruning to keep memory usage bounded on long videos, especially on resource-limited hardware like Raspberry Pi.
17 17 - purgeSubtitleCacheDir() - Removes the entire subtitle `.sup` file cache directory from disk; called once at server startup to clean up any files left over from a previous run.
18 18 - touchCachedEntry(cache, key, ttlMs) - Refreshes the TTL of an existing cache entry without changing its value; used to keep actively-watched subtitle tracks alive past their initial 5-minute TTL and prevent expensive full re-parses during long playback sessions.
19 - getImageSubtitleDebugSnapshot() - Returns event/frame cache totals and per-video cache usage ({event entries/bytes/count, frame entries/bytes}) so admin diagnostics can list cached subtitle data even when no viewer is connected.
20 - IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS / IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES / IMAGE_SUBTITLE_FRAME_CACHE_MAX_BYTES / IMAGE_SUBTITLE_RENDER_SCALE / IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS - Environment knobs for subtitle cache retention, item caps, byte caps, render resolution, and prune grace window.
19 - getImageSubtitleDebugSnapshot() - Returns event/frame cache totals, timing counters (cache hits/misses, event build timing, frame render/request timing), and per-video cache usage ({event entries/bytes/count, frame entries/bytes}) so admin diagnostics can distinguish cache-thrash or cold-parse stalls from steady-state rendering.
20 - IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS / IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES / IMAGE_SUBTITLE_FRAME_CACHE_MAX_BYTES / IMAGE_SUBTITLE_RENDER_SCALE / IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS - Environment knobs for subtitle cache retention, optional item caps (`0`/`unbounded` disables the cap so byte limits govern), byte caps, render resolution, and prune grace window.
21 21
22 22 ## server/thumbnail-cache.js
23 23 - syncThumbnailCacheOnce() - Mirrors configured roots and userPath into public/thumbs and generates 99x99-max WebP thumbnails for image files, video files (ffprobe/ffmpeg frame at ~10%), and 3D model files (STL/OBJ, isometric SVG projection via Sharp); removes stale thumbnails and updates manifest hashes.
@@ -132,17 +132,18 @@
132 132 - sendVideoStreamAsset(sessionId, assetPath, req, res) - Resolves viewer leases to a shared stream session, serves HLS assets, tracks lease activity, and prunes segments using the slowest active viewer position.
133 133 - releaseVideoStreamSession(sessionId) - Releases one viewer lease; only destroys the underlying transcode session when no active leases remain; cleanup only clears image subtitle frame/event caches once no other active stream session for the same source file remains.
134 134 - parseSegmentIndexFromAssetPath(assetPath) - Extracts the numeric segment index from an HLS segment filename; matches both numeric variant identifiers (stream_0_segment_000000.ts) and named variants (stream_video_segment_000000.ts, stream_Audio-1-jpn_segment_000000.ts) — ffmpeg substitutes the var_stream_map name: value for %v when named streams are used.
135 - runSegmentPrune(session) - Removes HLS segment files that fall behind the slowest active viewer and calls pruneImageSubtitleCachesForSession() using the slowest active viewer across all live sessions for the same source file, so file-wide subtitle buffers are not pruned ahead of a newly started second session; segment filename regex also uses named-variant pattern to match production filenames correctly.
135 - runSegmentPrune(session) - Removes HLS segment files that fall behind the slowest active viewer and calls pruneImageSubtitleCachesForSession() using the slowest active viewer across all live sessions for the same source file, so file-wide subtitle buffers are not pruned ahead of a newly started second session; segment deletion now runs in bounded batches and records last prune duration/deleted-segment counts for diagnostics; segment filename regex also uses named-variant pattern to match production filenames correctly.
136 136 - hasInitialHlsAssets(outputDir) - Returns true once master.m3u8 or at least one variant playlist/segment exists in an HLS output directory; accepts named variant filenames (stream_video.m3u8) in addition to numeric ones.
137 137 - purgeStreamRootDir() - Best-effort recursive delete of the temporary HLS stream root; used at startup and by idle sweeps to remove orphaned stream session directories from previous runs.
138 138 - cleanupIdleStreamArtifacts() - Periodic idle-time cleaner that purges stream-root artifacts when there are no active sessions or leases, preventing stale segment directories from accumulating after failures.
139 139 - getVideoStreamDebugSnapshot() - Returns active stream sessions with viewer counts and output-dir stats (segment counts/bytes), plus orphaned output directories left under the stream root for diagnostics.
140 - getVideoStreamDebugSnapshot() - Includes segment retention and prune telemetry per session (segmentRetentionCount, slowestViewerSegmentIndex, segmentPrunedUntil) to explain growth in segment counts.
140 - getVideoStreamDebugSnapshot() - Includes segment retention and prune telemetry per session (segmentRetentionCount, slowestViewerSegmentIndex, segmentPrunedUntil, lastSegmentPrune, streamPruneDeleteBatchSize) to explain growth or cleanup stalls in segment counts.
141 141 - STREAM_ROOT_DIR - Temporary root directory for active ffmpeg-backed stream sessions.
142 142 - STREAM_SESSION_TTL_MS - Idle lifetime for a stream session before cleanup.
143 143 - STREAM_IDLE_STOP_MS - Maximum client inactivity window before an active stream session is terminated (5 minutes).
144 144 - STREAM_SEGMENT_RETENTION_COUNT - Number of most-recent HLS segment indices kept on disk while older segments are pruned; when not overridden with VIDEO_STREAM_SEGMENT_RETENTION_COUNT it is derived from hardware capacity (RAM + CPU), current memory/CPU pressure, and Raspberry Pi 5 profile bounds.
145 145 - MAX_STREAM_SEGMENT_RETENTION_COUNT - Upper safety cap for retention (360) to avoid runaway in-session segment growth from oversized overrides.
146 - VIDEO_STREAM_PRUNE_DELETE_BATCH_SIZE - Optional environment knob that limits how many old HLS segment files are unlinked in parallel during one prune batch; smaller values reduce I/O spikes at the cost of slower cleanup.
146 147
147 148 ## server/db.js
148 149 - initDb() - Initializes SQLite schema for users, file metadata, auth_sessions, and share_tokens (expires_at index).
@@ -11,13 +11,13 @@
11
11
12## server/image-subtitles.js
12## server/image-subtitles.js
13- parseRenderOptions(options) - Normalises subtitle-frame request options and validates filePath/streamIndex/time/canvas dimensions; stream index `0` is valid and must not be treated as missing; canvas is scaled down within a 1280x720 ceiling, then multiplied by `IMAGE_SUBTITLE_RENDER_SCALE` (default `0.75`) while preserving requested aspect ratio.
13- parseRenderOptions(options) - Normalises subtitle-frame request options and validates filePath/streamIndex/time/canvas dimensions; stream index `0` is valid and must not be treated as missing; canvas is scaled down within a 1280x720 ceiling, then multiplied by `IMAGE_SUBTITLE_RENDER_SCALE` (default `0.75`) while preserving requested aspect ratio.
14- renderImageSubtitleFrame(options) - Extracts bitmap subtitle streams to cached `.sup`, parses PGS display sets, and returns {frame: PNG buffer, left, top, width, height, canvasWidth, canvasHeight} with metadata for proper client-side upscaling; the server preserves positioning info instead of throwing it away, allowing the client to calculate percentage-based sizing for downscaled frames.
14- renderImageSubtitleFrame(options) - Extracts bitmap subtitle streams to cached `.sup`, parses PGS display sets, and returns {frame: PNG buffer, left, top, width, height, canvasWidth, canvasHeight} with metadata for proper client-side upscaling; the server preserves positioning info instead of throwing it away, allowing the client to calculate percentage-based sizing for downscaled frames; request/build/render timing counters are tracked for debug diagnostics.
15- clearCachesForFile(filePath) - Removes all event and frame cache entries for a given file path using path-prefixed cache keys, also deletes the corresponding `.sup` file from disk for each evicted event entry; called when video stream sessions end to prevent memory and disk accumulation.
15- clearCachesForFile(filePath) - Removes all event and frame cache entries for a given file path using path-prefixed cache keys, also deletes the corresponding `.sup` file from disk for each evicted event entry; called when video stream sessions end to prevent memory and disk accumulation.
16- pruneOldImageSubtitleCaches(filePath, slowestViewerTimeSeconds) - Drops subtitle frame caches for events that have passed their retention threshold of end time + event duration + 5 seconds (e.g., event ending at 30s with 10s duration is dropped after slowest viewer reaches 45s); used during segment pruning to keep memory usage bounded on long videos, especially on resource-limited hardware like Raspberry Pi.
16- pruneOldImageSubtitleCaches(filePath, slowestViewerTimeSeconds) - Drops subtitle frame caches for events that have passed their retention threshold of end time + event duration + 5 seconds (e.g., event ending at 30s with 10s duration is dropped after slowest viewer reaches 45s); used during segment pruning to keep memory usage bounded on long videos, especially on resource-limited hardware like Raspberry Pi.
17- purgeSubtitleCacheDir() - Removes the entire subtitle `.sup` file cache directory from disk; called once at server startup to clean up any files left over from a previous run.
17- purgeSubtitleCacheDir() - Removes the entire subtitle `.sup` file cache directory from disk; called once at server startup to clean up any files left over from a previous run.
18- touchCachedEntry(cache, key, ttlMs) - Refreshes the TTL of an existing cache entry without changing its value; used to keep actively-watched subtitle tracks alive past their initial 5-minute TTL and prevent expensive full re-parses during long playback sessions.
18- touchCachedEntry(cache, key, ttlMs) - Refreshes the TTL of an existing cache entry without changing its value; used to keep actively-watched subtitle tracks alive past their initial 5-minute TTL and prevent expensive full re-parses during long playback sessions.
19- getImageSubtitleDebugSnapshot() - Returns event/frame cache totals and per-video cache usage ({event entries/bytes/count, frame entries/bytes}) so admin diagnostics can list cached subtitle data even when no viewer is connected.
19- getImageSubtitleDebugSnapshot() - Returns event/frame cache totals, timing counters (cache hits/misses, event build timing, frame render/request timing), and per-video cache usage ({event entries/bytes/count, frame entries/bytes}) so admin diagnostics can distinguish cache-thrash or cold-parse stalls from steady-state rendering.
20- IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS / IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES / IMAGE_SUBTITLE_FRAME_CACHE_MAX_BYTES / IMAGE_SUBTITLE_RENDER_SCALE / IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS - Environment knobs for subtitle cache retention, item caps, byte caps, render resolution, and prune grace window.
20- IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS / IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS / IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES / IMAGE_SUBTITLE_FRAME_CACHE_MAX_BYTES / IMAGE_SUBTITLE_RENDER_SCALE / IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS - Environment knobs for subtitle cache retention, optional item caps (`0`/`unbounded` disables the cap so byte limits govern), byte caps, render resolution, and prune grace window.
21
21
22## server/thumbnail-cache.js
22## server/thumbnail-cache.js
23- syncThumbnailCacheOnce() - Mirrors configured roots and userPath into public/thumbs and generates 99x99-max WebP thumbnails for image files, video files (ffprobe/ffmpeg frame at ~10%), and 3D model files (STL/OBJ, isometric SVG projection via Sharp); removes stale thumbnails and updates manifest hashes.
23- syncThumbnailCacheOnce() - Mirrors configured roots and userPath into public/thumbs and generates 99x99-max WebP thumbnails for image files, video files (ffprobe/ffmpeg frame at ~10%), and 3D model files (STL/OBJ, isometric SVG projection via Sharp); removes stale thumbnails and updates manifest hashes.
@@ -132,17 +132,18 @@
132- sendVideoStreamAsset(sessionId, assetPath, req, res) - Resolves viewer leases to a shared stream session, serves HLS assets, tracks lease activity, and prunes segments using the slowest active viewer position.
132- sendVideoStreamAsset(sessionId, assetPath, req, res) - Resolves viewer leases to a shared stream session, serves HLS assets, tracks lease activity, and prunes segments using the slowest active viewer position.
133- releaseVideoStreamSession(sessionId) - Releases one viewer lease; only destroys the underlying transcode session when no active leases remain; cleanup only clears image subtitle frame/event caches once no other active stream session for the same source file remains.
133- releaseVideoStreamSession(sessionId) - Releases one viewer lease; only destroys the underlying transcode session when no active leases remain; cleanup only clears image subtitle frame/event caches once no other active stream session for the same source file remains.
134- parseSegmentIndexFromAssetPath(assetPath) - Extracts the numeric segment index from an HLS segment filename; matches both numeric variant identifiers (stream_0_segment_000000.ts) and named variants (stream_video_segment_000000.ts, stream_Audio-1-jpn_segment_000000.ts) — ffmpeg substitutes the var_stream_map name: value for %v when named streams are used.
134- parseSegmentIndexFromAssetPath(assetPath) - Extracts the numeric segment index from an HLS segment filename; matches both numeric variant identifiers (stream_0_segment_000000.ts) and named variants (stream_video_segment_000000.ts, stream_Audio-1-jpn_segment_000000.ts) — ffmpeg substitutes the var_stream_map name: value for %v when named streams are used.
135- runSegmentPrune(session) - Removes HLS segment files that fall behind the slowest active viewer and calls pruneImageSubtitleCachesForSession() using the slowest active viewer across all live sessions for the same source file, so file-wide subtitle buffers are not pruned ahead of a newly started second session; segment filename regex also uses named-variant pattern to match production filenames correctly.
135- runSegmentPrune(session) - Removes HLS segment files that fall behind the slowest active viewer and calls pruneImageSubtitleCachesForSession() using the slowest active viewer across all live sessions for the same source file, so file-wide subtitle buffers are not pruned ahead of a newly started second session; segment deletion now runs in bounded batches and records last prune duration/deleted-segment counts for diagnostics; segment filename regex also uses named-variant pattern to match production filenames correctly.
136- hasInitialHlsAssets(outputDir) - Returns true once master.m3u8 or at least one variant playlist/segment exists in an HLS output directory; accepts named variant filenames (stream_video.m3u8) in addition to numeric ones.
136- hasInitialHlsAssets(outputDir) - Returns true once master.m3u8 or at least one variant playlist/segment exists in an HLS output directory; accepts named variant filenames (stream_video.m3u8) in addition to numeric ones.
137- purgeStreamRootDir() - Best-effort recursive delete of the temporary HLS stream root; used at startup and by idle sweeps to remove orphaned stream session directories from previous runs.
137- purgeStreamRootDir() - Best-effort recursive delete of the temporary HLS stream root; used at startup and by idle sweeps to remove orphaned stream session directories from previous runs.
138- cleanupIdleStreamArtifacts() - Periodic idle-time cleaner that purges stream-root artifacts when there are no active sessions or leases, preventing stale segment directories from accumulating after failures.
138- cleanupIdleStreamArtifacts() - Periodic idle-time cleaner that purges stream-root artifacts when there are no active sessions or leases, preventing stale segment directories from accumulating after failures.
139- getVideoStreamDebugSnapshot() - Returns active stream sessions with viewer counts and output-dir stats (segment counts/bytes), plus orphaned output directories left under the stream root for diagnostics.
139- getVideoStreamDebugSnapshot() - Returns active stream sessions with viewer counts and output-dir stats (segment counts/bytes), plus orphaned output directories left under the stream root for diagnostics.
140- getVideoStreamDebugSnapshot() - Includes segment retention and prune telemetry per session (segmentRetentionCount, slowestViewerSegmentIndex, segmentPrunedUntil) to explain growth in segment counts.
140- getVideoStreamDebugSnapshot() - Includes segment retention and prune telemetry per session (segmentRetentionCount, slowestViewerSegmentIndex, segmentPrunedUntil, lastSegmentPrune, streamPruneDeleteBatchSize) to explain growth or cleanup stalls in segment counts.
141- STREAM_ROOT_DIR - Temporary root directory for active ffmpeg-backed stream sessions.
141- STREAM_ROOT_DIR - Temporary root directory for active ffmpeg-backed stream sessions.
142- STREAM_SESSION_TTL_MS - Idle lifetime for a stream session before cleanup.
142- STREAM_SESSION_TTL_MS - Idle lifetime for a stream session before cleanup.
143- STREAM_IDLE_STOP_MS - Maximum client inactivity window before an active stream session is terminated (5 minutes).
143- STREAM_IDLE_STOP_MS - Maximum client inactivity window before an active stream session is terminated (5 minutes).
144- STREAM_SEGMENT_RETENTION_COUNT - Number of most-recent HLS segment indices kept on disk while older segments are pruned; when not overridden with VIDEO_STREAM_SEGMENT_RETENTION_COUNT it is derived from hardware capacity (RAM + CPU), current memory/CPU pressure, and Raspberry Pi 5 profile bounds.
144- STREAM_SEGMENT_RETENTION_COUNT - Number of most-recent HLS segment indices kept on disk while older segments are pruned; when not overridden with VIDEO_STREAM_SEGMENT_RETENTION_COUNT it is derived from hardware capacity (RAM + CPU), current memory/CPU pressure, and Raspberry Pi 5 profile bounds.
145- MAX_STREAM_SEGMENT_RETENTION_COUNT - Upper safety cap for retention (360) to avoid runaway in-session segment growth from oversized overrides.
145- MAX_STREAM_SEGMENT_RETENTION_COUNT - Upper safety cap for retention (360) to avoid runaway in-session segment growth from oversized overrides.
146- VIDEO_STREAM_PRUNE_DELETE_BATCH_SIZE - Optional environment knob that limits how many old HLS segment files are unlinked in parallel during one prune batch; smaller values reduce I/O spikes at the cost of slower cleanup.
146
147
147## server/db.js
148## server/db.js
148- initDb() - Initializes SQLite schema for users, file metadata, auth_sessions, and share_tokens (expires_at index).
149- initDb() - Initializes SQLite schema for users, file metadata, auth_sessions, and share_tokens (expires_at index).
server/image-subtitles.js +122 −6
@@ -11,8 +11,8 @@ const sharp = require('sharp');
11 11 const SUBTITLE_CACHE_DIR = path.join(os.tmpdir(), 'filemanager-video-subtitles');
12 12 const EVENT_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS, 5 * 60 * 1000);
13 13 const FRAME_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS, 2 * 60 * 1000);
14 const MAX_EVENT_CACHE_ITEMS = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS, 3);
15 const MAX_FRAME_CACHE_ITEMS = toPositiveInt(process.env.IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS, 50);
14 const MAX_EVENT_CACHE_ITEMS = toCacheItemLimit(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS, Infinity);
15 const MAX_FRAME_CACHE_ITEMS = toCacheItemLimit(process.env.IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS, Infinity);
16 16 // Removed byte size cap for event cache to prevent thrashing
17 17 const MAX_EVENT_CACHE_BYTES = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES, 50 * 1024 * 1024);
18 18 const MIN_EVENT_WINDOW_SECONDS = 30; // Always keep at least this much for seeking
@@ -23,9 +23,60 @@ const IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS = toPositiveInt(process.env.IMAGE_SUBTI
23 23
24 24 const eventCache = createCacheState(MAX_EVENT_CACHE_ITEMS, MAX_EVENT_CACHE_BYTES);
25 25 const frameCache = createCacheState(MAX_FRAME_CACHE_ITEMS, MAX_FRAME_CACHE_BYTES);
26 const subtitleDebugStats = createSubtitleDebugStats();
26 27
27 28 let parseDisplaySets = null;
28 29
30 function createDurationStats ()
31 {
32 return {
33 count: 0,
34 totalMs: 0,
35 maxMs: 0,
36 lastMs: 0,
37 };
38 }
39
40 function createSubtitleDebugStats ()
41 {
42 return {
43 eventCacheHits: 0,
44 eventCacheMisses: 0,
45 eventCachePendingHits: 0,
46 eventCacheBuilds: 0,
47 eventCacheBuildErrors: 0,
48 frameCacheHits: 0,
49 frameCacheMisses: 0,
50 frameRenders: 0,
51 eventBuildDurationMs: createDurationStats(),
52 frameRenderDurationMs: createDurationStats(),
53 frameRequestDurationMs: createDurationStats(),
54 };
55 }
56
57 function recordDuration (stats, startedAt)
58 {
59 if (!stats || !Number.isFinite(startedAt)) return 0;
60
61 const elapsedMs = Math.max(0, Date.now() - startedAt);
62 stats.count += 1;
63 stats.totalMs += elapsedMs;
64 stats.lastMs = elapsedMs;
65 if (elapsedMs > stats.maxMs) stats.maxMs = elapsedMs;
66 return elapsedMs;
67 }
68
69 function averageDurationMs (stats)
70 {
71 if (!stats || !stats.count) return 0;
72 return Number((stats.totalMs / stats.count).toFixed(1));
73 }
74
75 function debugLimitValue (value)
76 {
77 return Number.isFinite(value) ? value : null;
78 }
79
29 80 function ensureImageDataGlobal ()
30 81 {
31 82 if (typeof globalThis.ImageData === 'function') return;
@@ -68,6 +119,20 @@ function toPositiveInt (value, fallback)
68 119 return fallback;
69 120 }
70 121
122 function toCacheItemLimit (value, fallback)
123 {
124 const raw = String(value || '').trim().toLowerCase();
125 if (!raw) return fallback;
126 if (raw === '0' || raw === 'none' || raw === 'infinity' || raw === 'inf' || raw === 'unbounded' || raw === 'unlimited')
127 {
128 return Infinity;
129 }
130
131 const number = Number.parseInt(raw, 10);
132 if (Number.isInteger(number) && number > 0) return number;
133 return fallback;
134 }
135
71 136 function toBoundedNumber (value, fallback, min, max)
72 137 {
73 138 const numeric = Number.parseFloat(String(value || ''));
@@ -385,20 +450,27 @@ async function getEventCacheEntry (filePath, streamIndex)
385 450 const cached = getCachedValue(eventCache, identityKey);
386 451 if (cached)
387 452 {
453 if (typeof cached.then === 'function') subtitleDebugStats.eventCachePendingHits += 1;
454 else subtitleDebugStats.eventCacheHits += 1;
388 455 touchCachedEntry(eventCache, identityKey, EVENT_CACHE_TTL_MS);
389 456 return cached;
390 457 }
391 458
459 subtitleDebugStats.eventCacheMisses += 1;
460 subtitleDebugStats.eventCacheBuilds += 1;
461 const buildStartedAt = Date.now();
392 462 const pending = buildEventCacheEntry(normalisedPath, streamIndex, identityKey);
393 463 setCachedValue(eventCache, identityKey, pending, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
394 464
395 465 try
396 466 {
397 467 const resolved = await pending;
468 recordDuration(subtitleDebugStats.eventBuildDurationMs, buildStartedAt);
398 469 setCachedValue(eventCache, identityKey, resolved, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
399 470 return resolved;
400 471 } catch (error)
401 472 {
473 subtitleDebugStats.eventCacheBuildErrors += 1;
402 474 deleteCachedValue(eventCache, identityKey);
403 475 throw error;
404 476 }
@@ -511,32 +583,50 @@ function parseRenderOptions (options)
511 583
512 584 async function renderImageSubtitleFrame (options)
513 585 {
586 const requestStartedAt = Date.now();
514 587 const parsed = parseRenderOptions(options);
515 588 const { filePath, streamIndex, timeSeconds, canvasWidth, canvasHeight } = parsed;
516 589
517 590 if (!parsed.isValid)
518 591 {
592 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
519 593 return null;
520 594 }
521 595
522 596 const entry = await getEventCacheEntry(filePath, streamIndex);
523 if (!entry || !Array.isArray(entry.events) || entry.events.length === 0) return null;
597 if (!entry || !Array.isArray(entry.events) || entry.events.length === 0)
598 {
599 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
600 return null;
601 }
524 602
525 603 const eventIndex = findActiveEventIndex(entry.events, timeSeconds);
526 if (eventIndex < 0) return null;
604 if (eventIndex < 0)
605 {
606 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
607 return null;
608 }
527 609
528 610 const frameKey = `${entry.cacheKey}|${eventIndex}|${canvasWidth}x${canvasHeight}`;
529 611 const cachedFrame = getCachedValue(frameCache, frameKey);
530 612 if (cachedFrame)
531 613 {
614 subtitleDebugStats.frameCacheHits += 1;
615 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
532 616 return cachedFrame;
533 617 }
534 618
619 subtitleDebugStats.frameCacheMisses += 1;
620 subtitleDebugStats.frameRenders += 1;
621 const renderStartedAt = Date.now();
535 622 const frameData = await renderEventFrame(entry.events[eventIndex], canvasWidth, canvasHeight);
536 623 if (!frameData)
537 624 {
625 recordDuration(subtitleDebugStats.frameRenderDurationMs, renderStartedAt);
626 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
538 627 return null;
539 628 }
629 recordDuration(subtitleDebugStats.frameRenderDurationMs, renderStartedAt);
540 630
541 631 const event = entry.events[eventIndex];
542 632 const cacheValue = {
@@ -557,6 +647,7 @@ async function renderImageSubtitleFrame (options)
557 647 FRAME_CACHE_TTL_MS,
558 648 estimateFrameCacheBytes,
559 649 );
650 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
560 651 return cacheValue;
561 652 }
562 653
@@ -750,15 +841,40 @@ function getImageSubtitleDebugSnapshot ()
750 841 eventCache: {
751 842 entries: eventCache.map.size,
752 843 bytes: eventCache.totalBytes,
753 maxItems: eventCache.maxItems,
844 maxItems: debugLimitValue(eventCache.maxItems),
754 845 maxBytes: eventCache.maxBytes,
755 846 },
756 847 frameCache: {
757 848 entries: frameCache.map.size,
758 849 bytes: frameCache.totalBytes,
759 maxItems: frameCache.maxItems,
850 maxItems: debugLimitValue(frameCache.maxItems),
760 851 maxBytes: frameCache.maxBytes,
761 852 },
853 metrics: {
854 eventCacheHits: subtitleDebugStats.eventCacheHits,
855 eventCacheMisses: subtitleDebugStats.eventCacheMisses,
856 eventCachePendingHits: subtitleDebugStats.eventCachePendingHits,
857 eventCacheBuilds: subtitleDebugStats.eventCacheBuilds,
858 eventCacheBuildErrors: subtitleDebugStats.eventCacheBuildErrors,
859 frameCacheHits: subtitleDebugStats.frameCacheHits,
860 frameCacheMisses: subtitleDebugStats.frameCacheMisses,
861 frameRenders: subtitleDebugStats.frameRenders,
862 eventBuildDurationMs: {
863 last: subtitleDebugStats.eventBuildDurationMs.lastMs,
864 max: subtitleDebugStats.eventBuildDurationMs.maxMs,
865 average: averageDurationMs(subtitleDebugStats.eventBuildDurationMs),
866 },
867 frameRenderDurationMs: {
868 last: subtitleDebugStats.frameRenderDurationMs.lastMs,
869 max: subtitleDebugStats.frameRenderDurationMs.maxMs,
870 average: averageDurationMs(subtitleDebugStats.frameRenderDurationMs),
871 },
872 frameRequestDurationMs: {
873 last: subtitleDebugStats.frameRequestDurationMs.lastMs,
874 max: subtitleDebugStats.frameRequestDurationMs.maxMs,
875 average: averageDurationMs(subtitleDebugStats.frameRequestDurationMs),
876 },
877 },
762 878 byFile,
763 879 };
764 880 }
@@ -11,8 +11,8 @@ const sharp = require('sharp');
11const SUBTITLE_CACHE_DIR = path.join(os.tmpdir(), 'filemanager-video-subtitles');
11const SUBTITLE_CACHE_DIR = path.join(os.tmpdir(), 'filemanager-video-subtitles');
12const EVENT_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS, 5 * 60 * 1000);
12const EVENT_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_TTL_MS, 5 * 60 * 1000);
13const FRAME_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS, 2 * 60 * 1000);
13const FRAME_CACHE_TTL_MS = toPositiveInt(process.env.IMAGE_SUBTITLE_FRAME_CACHE_TTL_MS, 2 * 60 * 1000);
14const MAX_EVENT_CACHE_ITEMS = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS, 3);
14const MAX_EVENT_CACHE_ITEMS = toCacheItemLimit(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_ITEMS, Infinity);
15const MAX_FRAME_CACHE_ITEMS = toPositiveInt(process.env.IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS, 50);
15const MAX_FRAME_CACHE_ITEMS = toCacheItemLimit(process.env.IMAGE_SUBTITLE_FRAME_CACHE_MAX_ITEMS, Infinity);
16// Removed byte size cap for event cache to prevent thrashing
16// Removed byte size cap for event cache to prevent thrashing
17const MAX_EVENT_CACHE_BYTES = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES, 50 * 1024 * 1024);
17const MAX_EVENT_CACHE_BYTES = toPositiveInt(process.env.IMAGE_SUBTITLE_EVENT_CACHE_MAX_BYTES, 50 * 1024 * 1024);
18const MIN_EVENT_WINDOW_SECONDS = 30; // Always keep at least this much for seeking
18const MIN_EVENT_WINDOW_SECONDS = 30; // Always keep at least this much for seeking
@@ -23,9 +23,60 @@ const IMAGE_SUBTITLE_PRUNE_GRACE_SECONDS = toPositiveInt(process.env.IMAGE_SUBTI
23
23
24const eventCache = createCacheState(MAX_EVENT_CACHE_ITEMS, MAX_EVENT_CACHE_BYTES);
24const eventCache = createCacheState(MAX_EVENT_CACHE_ITEMS, MAX_EVENT_CACHE_BYTES);
25const frameCache = createCacheState(MAX_FRAME_CACHE_ITEMS, MAX_FRAME_CACHE_BYTES);
25const frameCache = createCacheState(MAX_FRAME_CACHE_ITEMS, MAX_FRAME_CACHE_BYTES);
26const subtitleDebugStats = createSubtitleDebugStats();
26
27
27let parseDisplaySets = null;
28let parseDisplaySets = null;
28
29
30function createDurationStats ()
31{
32 return {
33 count: 0,
34 totalMs: 0,
35 maxMs: 0,
36 lastMs: 0,
37 };
38}
39
40function createSubtitleDebugStats ()
41{
42 return {
43 eventCacheHits: 0,
44 eventCacheMisses: 0,
45 eventCachePendingHits: 0,
46 eventCacheBuilds: 0,
47 eventCacheBuildErrors: 0,
48 frameCacheHits: 0,
49 frameCacheMisses: 0,
50 frameRenders: 0,
51 eventBuildDurationMs: createDurationStats(),
52 frameRenderDurationMs: createDurationStats(),
53 frameRequestDurationMs: createDurationStats(),
54 };
55}
56
57function recordDuration (stats, startedAt)
58{
59 if (!stats || !Number.isFinite(startedAt)) return 0;
60
61 const elapsedMs = Math.max(0, Date.now() - startedAt);
62 stats.count += 1;
63 stats.totalMs += elapsedMs;
64 stats.lastMs = elapsedMs;
65 if (elapsedMs > stats.maxMs) stats.maxMs = elapsedMs;
66 return elapsedMs;
67}
68
69function averageDurationMs (stats)
70{
71 if (!stats || !stats.count) return 0;
72 return Number((stats.totalMs / stats.count).toFixed(1));
73}
74
75function debugLimitValue (value)
76{
77 return Number.isFinite(value) ? value : null;
78}
79
29function ensureImageDataGlobal ()
80function ensureImageDataGlobal ()
30{
81{
31 if (typeof globalThis.ImageData === 'function') return;
82 if (typeof globalThis.ImageData === 'function') return;
@@ -68,6 +119,20 @@ function toPositiveInt (value, fallback)
68 return fallback;
119 return fallback;
69}
120}
70
121
122function toCacheItemLimit (value, fallback)
123{
124 const raw = String(value || '').trim().toLowerCase();
125 if (!raw) return fallback;
126 if (raw === '0' || raw === 'none' || raw === 'infinity' || raw === 'inf' || raw === 'unbounded' || raw === 'unlimited')
127 {
128 return Infinity;
129 }
130
131 const number = Number.parseInt(raw, 10);
132 if (Number.isInteger(number) && number > 0) return number;
133 return fallback;
134}
135
71function toBoundedNumber (value, fallback, min, max)
136function toBoundedNumber (value, fallback, min, max)
72{
137{
73 const numeric = Number.parseFloat(String(value || ''));
138 const numeric = Number.parseFloat(String(value || ''));
@@ -385,20 +450,27 @@ async function getEventCacheEntry (filePath, streamIndex)
385 const cached = getCachedValue(eventCache, identityKey);
450 const cached = getCachedValue(eventCache, identityKey);
386 if (cached)
451 if (cached)
387 {
452 {
453 if (typeof cached.then === 'function') subtitleDebugStats.eventCachePendingHits += 1;
454 else subtitleDebugStats.eventCacheHits += 1;
388 touchCachedEntry(eventCache, identityKey, EVENT_CACHE_TTL_MS);
455 touchCachedEntry(eventCache, identityKey, EVENT_CACHE_TTL_MS);
389 return cached;
456 return cached;
390 }
457 }
391
458
459 subtitleDebugStats.eventCacheMisses += 1;
460 subtitleDebugStats.eventCacheBuilds += 1;
461 const buildStartedAt = Date.now();
392 const pending = buildEventCacheEntry(normalisedPath, streamIndex, identityKey);
462 const pending = buildEventCacheEntry(normalisedPath, streamIndex, identityKey);
393 setCachedValue(eventCache, identityKey, pending, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
463 setCachedValue(eventCache, identityKey, pending, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
394
464
395 try
465 try
396 {
466 {
397 const resolved = await pending;
467 const resolved = await pending;
468 recordDuration(subtitleDebugStats.eventBuildDurationMs, buildStartedAt);
398 setCachedValue(eventCache, identityKey, resolved, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
469 setCachedValue(eventCache, identityKey, resolved, EVENT_CACHE_TTL_MS, estimateEventCacheBytes);
399 return resolved;
470 return resolved;
400 } catch (error)
471 } catch (error)
401 {
472 {
473 subtitleDebugStats.eventCacheBuildErrors += 1;
402 deleteCachedValue(eventCache, identityKey);
474 deleteCachedValue(eventCache, identityKey);
403 throw error;
475 throw error;
404 }
476 }
@@ -511,32 +583,50 @@ function parseRenderOptions (options)
511
583
512async function renderImageSubtitleFrame (options)
584async function renderImageSubtitleFrame (options)
513{
585{
586 const requestStartedAt = Date.now();
514 const parsed = parseRenderOptions(options);
587 const parsed = parseRenderOptions(options);
515 const { filePath, streamIndex, timeSeconds, canvasWidth, canvasHeight } = parsed;
588 const { filePath, streamIndex, timeSeconds, canvasWidth, canvasHeight } = parsed;
516
589
517 if (!parsed.isValid)
590 if (!parsed.isValid)
518 {
591 {
592 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
519 return null;
593 return null;
520 }
594 }
521
595
522 const entry = await getEventCacheEntry(filePath, streamIndex);
596 const entry = await getEventCacheEntry(filePath, streamIndex);
523 if (!entry || !Array.isArray(entry.events) || entry.events.length === 0) return null;
597 if (!entry || !Array.isArray(entry.events) || entry.events.length === 0)
598 {
599 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
600 return null;
601 }
524
602
525 const eventIndex = findActiveEventIndex(entry.events, timeSeconds);
603 const eventIndex = findActiveEventIndex(entry.events, timeSeconds);
526 if (eventIndex < 0) return null;
604 if (eventIndex < 0)
605 {
606 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
607 return null;
608 }
527
609
528 const frameKey = `${entry.cacheKey}|${eventIndex}|${canvasWidth}x${canvasHeight}`;
610 const frameKey = `${entry.cacheKey}|${eventIndex}|${canvasWidth}x${canvasHeight}`;
529 const cachedFrame = getCachedValue(frameCache, frameKey);
611 const cachedFrame = getCachedValue(frameCache, frameKey);
530 if (cachedFrame)
612 if (cachedFrame)
531 {
613 {
614 subtitleDebugStats.frameCacheHits += 1;
615 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
532 return cachedFrame;
616 return cachedFrame;
533 }
617 }
534
618
619 subtitleDebugStats.frameCacheMisses += 1;
620 subtitleDebugStats.frameRenders += 1;
621 const renderStartedAt = Date.now();
535 const frameData = await renderEventFrame(entry.events[eventIndex], canvasWidth, canvasHeight);
622 const frameData = await renderEventFrame(entry.events[eventIndex], canvasWidth, canvasHeight);
536 if (!frameData)
623 if (!frameData)
537 {
624 {
625 recordDuration(subtitleDebugStats.frameRenderDurationMs, renderStartedAt);
626 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
538 return null;
627 return null;
539 }
628 }
629 recordDuration(subtitleDebugStats.frameRenderDurationMs, renderStartedAt);
540
630
541 const event = entry.events[eventIndex];
631 const event = entry.events[eventIndex];
542 const cacheValue = {
632 const cacheValue = {
@@ -557,6 +647,7 @@ async function renderImageSubtitleFrame (options)
557 FRAME_CACHE_TTL_MS,
647 FRAME_CACHE_TTL_MS,
558 estimateFrameCacheBytes,
648 estimateFrameCacheBytes,
559 );
649 );
650 recordDuration(subtitleDebugStats.frameRequestDurationMs, requestStartedAt);
560 return cacheValue;
651 return cacheValue;
561}
652}
562
653
@@ -750,15 +841,40 @@ function getImageSubtitleDebugSnapshot ()
750 eventCache: {
841 eventCache: {
751 entries: eventCache.map.size,
842 entries: eventCache.map.size,
752 bytes: eventCache.totalBytes,
843 bytes: eventCache.totalBytes,
753 maxItems: eventCache.maxItems,
844 maxItems: debugLimitValue(eventCache.maxItems),
754 maxBytes: eventCache.maxBytes,
845 maxBytes: eventCache.maxBytes,
755 },
846 },
756 frameCache: {
847 frameCache: {
757 entries: frameCache.map.size,
848 entries: frameCache.map.size,
758 bytes: frameCache.totalBytes,
849 bytes: frameCache.totalBytes,
759 maxItems: frameCache.maxItems,
850 maxItems: debugLimitValue(frameCache.maxItems),
760 maxBytes: frameCache.maxBytes,
851 maxBytes: frameCache.maxBytes,
761 },
852 },
853 metrics: {
854 eventCacheHits: subtitleDebugStats.eventCacheHits,
855 eventCacheMisses: subtitleDebugStats.eventCacheMisses,
856 eventCachePendingHits: subtitleDebugStats.eventCachePendingHits,
857 eventCacheBuilds: subtitleDebugStats.eventCacheBuilds,
858 eventCacheBuildErrors: subtitleDebugStats.eventCacheBuildErrors,
859 frameCacheHits: subtitleDebugStats.frameCacheHits,
860 frameCacheMisses: subtitleDebugStats.frameCacheMisses,
861 frameRenders: subtitleDebugStats.frameRenders,
862 eventBuildDurationMs: {
863 last: subtitleDebugStats.eventBuildDurationMs.lastMs,
864 max: subtitleDebugStats.eventBuildDurationMs.maxMs,
865 average: averageDurationMs(subtitleDebugStats.eventBuildDurationMs),
866 },
867 frameRenderDurationMs: {
868 last: subtitleDebugStats.frameRenderDurationMs.lastMs,
869 max: subtitleDebugStats.frameRenderDurationMs.maxMs,
870 average: averageDurationMs(subtitleDebugStats.frameRenderDurationMs),
871 },
872 frameRequestDurationMs: {
873 last: subtitleDebugStats.frameRequestDurationMs.lastMs,
874 max: subtitleDebugStats.frameRequestDurationMs.maxMs,
875 average: averageDurationMs(subtitleDebugStats.frameRequestDurationMs),
876 },
877 },
762 byFile,
878 byFile,
763 };
879 };
764}
880}
server/video-stream.js +51 −8
@@ -13,6 +13,7 @@ const STREAM_READY_TIMEOUT_MS = 20 * 1000;
13 13 const STREAM_IDLE_STOP_MS = 5 * 60 * 1000;
14 14 const ESTIMATED_SEGMENT_BYTES = 1024 * 1024;
15 15 const ESTIMATED_SEGMENT_DURATION_SECONDS = 6;
16 const STREAM_PRUNE_DELETE_BATCH_SIZE = parsePruneDeleteBatchSize(process.env.VIDEO_STREAM_PRUNE_DELETE_BATCH_SIZE);
16 17 const MAX_STREAM_SEGMENT_RETENTION_COUNT = 360;
17 18 const STREAM_SEGMENT_RETENTION_COUNT = parseSegmentRetentionCount(process.env.VIDEO_STREAM_SEGMENT_RETENTION_COUNT);
18 19 const IMAGE_BASED_SUBTITLE_CODECS = new Set([
@@ -68,6 +69,16 @@ function parseSegmentRetentionCount (value)
68 69 return defaultSegmentRetentionCount();
69 70 }
70 71
72 function parsePruneDeleteBatchSize (value)
73 {
74 const parsed = Number.parseInt(String(value || ''), 10);
75 if (Number.isInteger(parsed) && parsed >= 1)
76 {
77 return clampNumber(parsed, 1, 128);
78 }
79 return 16;
80 }
81
71 82 function readRaspberryPiModel ()
72 83 {
73 84 try
@@ -400,6 +411,7 @@ function runSegmentPrune (session)
400 411 if (session.segmentPrunePendingCutoff <= session.segmentPrunedUntil) return;
401 412
402 413 const cutoff = session.segmentPrunePendingCutoff;
414 const pruneStartedAt = Date.now();
403 415 session.segmentPruneRunning = true;
404 416
405 417 fs.promises.readdir(session.outputDir)
@@ -414,18 +426,20 @@ function runSegmentPrune (session)
414 426 if (!Number.isInteger(segmentIndex)) continue;
415 427 if (segmentIndex > cutoff) continue;
416 428
417 removals.push(
418 fs.promises.unlink(path.join(session.outputDir, entryName)).catch(() =>
419 {
420 // Ignore races while ffmpeg rotates temp files and request handlers read segments.
421 })
422 );
429 removals.push(path.join(session.outputDir, entryName));
423 430 }
424 return Promise.all(removals);
431 return deleteFilesInBatches(removals, STREAM_PRUNE_DELETE_BATCH_SIZE);
425 432 })
426 .then(() =>
433 .then((deletedCount) =>
427 434 {
435 const completedAt = Date.now();
428 436 session.segmentPrunedUntil = Math.max(session.segmentPrunedUntil, cutoff);
437 session.lastSegmentPrune = {
438 cutoff,
439 deletedSegments: deletedCount,
440 durationMs: Math.max(0, completedAt - pruneStartedAt),
441 completedAt,
442 };
429 443 pruneImageSubtitleCachesForSession(session);
430 444 if (session.segmentPrunedUntil >= 0 && reusableSessionIdBySource.get(session.sourcePath) === session.id)
431 445 {
@@ -446,6 +460,31 @@ function runSegmentPrune (session)
446 460 });
447 461 }
448 462
463 async function deleteFilesInBatches (pathsToDelete, batchSize)
464 {
465 const paths = Array.isArray(pathsToDelete) ? pathsToDelete : [];
466 if (!paths.length) return 0;
467
468 let deletedCount = 0;
469 for (let index = 0; index < paths.length; index += batchSize)
470 {
471 const batch = paths.slice(index, index + batchSize);
472 await Promise.all(batch.map((targetPath) =>
473 fs.promises.unlink(targetPath)
474 .then(() =>
475 {
476 deletedCount += 1;
477 })
478 .catch(() =>
479 {
480 // Ignore races while ffmpeg rotates temp files and request handlers read segments.
481 })
482 ));
483 }
484
485 return deletedCount;
486 }
487
449 488 function slowestLeaseSegmentIndex (session)
450 489 {
451 490 if (!session.leaseIds.size) return null;
@@ -993,6 +1032,7 @@ async function createVideoStreamSession (sourcePath)
993 1032 segmentPruneRunning: false,
994 1033 segmentPrunePendingCutoff: -1,
995 1034 segmentPrunedUntil: -1,
1035 lastSegmentPrune: null,
996 1036 leaseIds: new Set(),
997 1037 };
998 1038
@@ -1209,9 +1249,11 @@ function getVideoStreamDebugSnapshot ()
1209 1249 activeRequests: Number(session.activeRequests || 0),
1210 1250 expiresAt: Number(session.expiresAt || 0),
1211 1251 segmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1252 segmentPruneDeleteBatchSize: STREAM_PRUNE_DELETE_BATCH_SIZE,
1212 1253 segmentPrunePendingCutoff: Number(session.segmentPrunePendingCutoff || 0),
1213 1254 segmentPrunedUntil: Number(session.segmentPrunedUntil || 0),
1214 1255 slowestViewerSegmentIndex: slowestLeaseSegmentIndex(session) ?? -1,
1256 lastSegmentPrune: session.lastSegmentPrune || null,
1215 1257 output: collectOutputDirStats(session.outputDir),
1216 1258 });
1217 1259 }
@@ -1240,6 +1282,7 @@ function getVideoStreamDebugSnapshot ()
1240 1282 streamRootDir: STREAM_ROOT_DIR,
1241 1283 streamSegmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1242 1284 streamSegmentRetentionMax: MAX_STREAM_SEGMENT_RETENTION_COUNT,
1285 streamPruneDeleteBatchSize: STREAM_PRUNE_DELETE_BATCH_SIZE,
1243 1286 activeSessionCount: sessions.size,
1244 1287 activeLeaseCount: leases.size,
1245 1288 activeStreams,
@@ -13,6 +13,7 @@ const STREAM_READY_TIMEOUT_MS = 20 * 1000;
13const STREAM_IDLE_STOP_MS = 5 * 60 * 1000;
13const STREAM_IDLE_STOP_MS = 5 * 60 * 1000;
14const ESTIMATED_SEGMENT_BYTES = 1024 * 1024;
14const ESTIMATED_SEGMENT_BYTES = 1024 * 1024;
15const ESTIMATED_SEGMENT_DURATION_SECONDS = 6;
15const ESTIMATED_SEGMENT_DURATION_SECONDS = 6;
16const STREAM_PRUNE_DELETE_BATCH_SIZE = parsePruneDeleteBatchSize(process.env.VIDEO_STREAM_PRUNE_DELETE_BATCH_SIZE);
16const MAX_STREAM_SEGMENT_RETENTION_COUNT = 360;
17const MAX_STREAM_SEGMENT_RETENTION_COUNT = 360;
17const STREAM_SEGMENT_RETENTION_COUNT = parseSegmentRetentionCount(process.env.VIDEO_STREAM_SEGMENT_RETENTION_COUNT);
18const STREAM_SEGMENT_RETENTION_COUNT = parseSegmentRetentionCount(process.env.VIDEO_STREAM_SEGMENT_RETENTION_COUNT);
18const IMAGE_BASED_SUBTITLE_CODECS = new Set([
19const IMAGE_BASED_SUBTITLE_CODECS = new Set([
@@ -68,6 +69,16 @@ function parseSegmentRetentionCount (value)
68 return defaultSegmentRetentionCount();
69 return defaultSegmentRetentionCount();
69}
70}
70
71
72function parsePruneDeleteBatchSize (value)
73{
74 const parsed = Number.parseInt(String(value || ''), 10);
75 if (Number.isInteger(parsed) && parsed >= 1)
76 {
77 return clampNumber(parsed, 1, 128);
78 }
79 return 16;
80}
81
71function readRaspberryPiModel ()
82function readRaspberryPiModel ()
72{
83{
73 try
84 try
@@ -400,6 +411,7 @@ function runSegmentPrune (session)
400 if (session.segmentPrunePendingCutoff <= session.segmentPrunedUntil) return;
411 if (session.segmentPrunePendingCutoff <= session.segmentPrunedUntil) return;
401
412
402 const cutoff = session.segmentPrunePendingCutoff;
413 const cutoff = session.segmentPrunePendingCutoff;
414 const pruneStartedAt = Date.now();
403 session.segmentPruneRunning = true;
415 session.segmentPruneRunning = true;
404
416
405 fs.promises.readdir(session.outputDir)
417 fs.promises.readdir(session.outputDir)
@@ -414,18 +426,20 @@ function runSegmentPrune (session)
414 if (!Number.isInteger(segmentIndex)) continue;
426 if (!Number.isInteger(segmentIndex)) continue;
415 if (segmentIndex > cutoff) continue;
427 if (segmentIndex > cutoff) continue;
416
428
417 removals.push(
429 removals.push(path.join(session.outputDir, entryName));
418 fs.promises.unlink(path.join(session.outputDir, entryName)).catch(() =>
419 {
420 // Ignore races while ffmpeg rotates temp files and request handlers read segments.
421 })
422 );
423 }
430 }
424 return Promise.all(removals);
431 return deleteFilesInBatches(removals, STREAM_PRUNE_DELETE_BATCH_SIZE);
425 })
432 })
426 .then(() =>
433 .then((deletedCount) =>
427 {
434 {
435 const completedAt = Date.now();
428 session.segmentPrunedUntil = Math.max(session.segmentPrunedUntil, cutoff);
436 session.segmentPrunedUntil = Math.max(session.segmentPrunedUntil, cutoff);
437 session.lastSegmentPrune = {
438 cutoff,
439 deletedSegments: deletedCount,
440 durationMs: Math.max(0, completedAt - pruneStartedAt),
441 completedAt,
442 };
429 pruneImageSubtitleCachesForSession(session);
443 pruneImageSubtitleCachesForSession(session);
430 if (session.segmentPrunedUntil >= 0 && reusableSessionIdBySource.get(session.sourcePath) === session.id)
444 if (session.segmentPrunedUntil >= 0 && reusableSessionIdBySource.get(session.sourcePath) === session.id)
431 {
445 {
@@ -446,6 +460,31 @@ function runSegmentPrune (session)
446 });
460 });
447}
461}
448
462
463async function deleteFilesInBatches (pathsToDelete, batchSize)
464{
465 const paths = Array.isArray(pathsToDelete) ? pathsToDelete : [];
466 if (!paths.length) return 0;
467
468 let deletedCount = 0;
469 for (let index = 0; index < paths.length; index += batchSize)
470 {
471 const batch = paths.slice(index, index + batchSize);
472 await Promise.all(batch.map((targetPath) =>
473 fs.promises.unlink(targetPath)
474 .then(() =>
475 {
476 deletedCount += 1;
477 })
478 .catch(() =>
479 {
480 // Ignore races while ffmpeg rotates temp files and request handlers read segments.
481 })
482 ));
483 }
484
485 return deletedCount;
486}
487
449function slowestLeaseSegmentIndex (session)
488function slowestLeaseSegmentIndex (session)
450{
489{
451 if (!session.leaseIds.size) return null;
490 if (!session.leaseIds.size) return null;
@@ -993,6 +1032,7 @@ async function createVideoStreamSession (sourcePath)
993 segmentPruneRunning: false,
1032 segmentPruneRunning: false,
994 segmentPrunePendingCutoff: -1,
1033 segmentPrunePendingCutoff: -1,
995 segmentPrunedUntil: -1,
1034 segmentPrunedUntil: -1,
1035 lastSegmentPrune: null,
996 leaseIds: new Set(),
1036 leaseIds: new Set(),
997 };
1037 };
998
1038
@@ -1209,9 +1249,11 @@ function getVideoStreamDebugSnapshot ()
1209 activeRequests: Number(session.activeRequests || 0),
1249 activeRequests: Number(session.activeRequests || 0),
1210 expiresAt: Number(session.expiresAt || 0),
1250 expiresAt: Number(session.expiresAt || 0),
1211 segmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1251 segmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1252 segmentPruneDeleteBatchSize: STREAM_PRUNE_DELETE_BATCH_SIZE,
1212 segmentPrunePendingCutoff: Number(session.segmentPrunePendingCutoff || 0),
1253 segmentPrunePendingCutoff: Number(session.segmentPrunePendingCutoff || 0),
1213 segmentPrunedUntil: Number(session.segmentPrunedUntil || 0),
1254 segmentPrunedUntil: Number(session.segmentPrunedUntil || 0),
1214 slowestViewerSegmentIndex: slowestLeaseSegmentIndex(session) ?? -1,
1255 slowestViewerSegmentIndex: slowestLeaseSegmentIndex(session) ?? -1,
1256 lastSegmentPrune: session.lastSegmentPrune || null,
1215 output: collectOutputDirStats(session.outputDir),
1257 output: collectOutputDirStats(session.outputDir),
1216 });
1258 });
1217 }
1259 }
@@ -1240,6 +1282,7 @@ function getVideoStreamDebugSnapshot ()
1240 streamRootDir: STREAM_ROOT_DIR,
1282 streamRootDir: STREAM_ROOT_DIR,
1241 streamSegmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1283 streamSegmentRetentionCount: STREAM_SEGMENT_RETENTION_COUNT,
1242 streamSegmentRetentionMax: MAX_STREAM_SEGMENT_RETENTION_COUNT,
1284 streamSegmentRetentionMax: MAX_STREAM_SEGMENT_RETENTION_COUNT,
1285 streamPruneDeleteBatchSize: STREAM_PRUNE_DELETE_BATCH_SIZE,
1243 activeSessionCount: sessions.size,
1286 activeSessionCount: sessions.size,
1244 activeLeaseCount: leases.size,
1287 activeLeaseCount: leases.size,
1245 activeStreams,
1288 activeStreams,
tests/image-subtitles.test.js +22 −1
@@ -3,7 +3,7 @@
3 3 const fs = require('fs');
4 4 const os = require('os');
5 5 const path = require('path');
6 const { parseRenderOptions, pruneOldImageSubtitleCaches, purgeSubtitleCacheDir } = require('../server/image-subtitles');
6 const { parseRenderOptions, getImageSubtitleDebugSnapshot, purgeSubtitleCacheDir } = require('../server/image-subtitles');
7 7
8 8 describe('image subtitle option parsing', () =>
9 9 {
@@ -90,6 +90,27 @@ describe('image subtitle option parsing', () =>
90 90
91 91 describe('subtitle cache directory management', () =>
92 92 {
93 test('debug snapshot reports unbounded item caps and timing metrics', () =>
94 {
95 const snapshot = getImageSubtitleDebugSnapshot();
96
97 expect(snapshot.eventCache.maxItems).toBeNull();
98 expect(snapshot.frameCache.maxItems).toBeNull();
99 expect(snapshot.metrics).toEqual(expect.objectContaining({
100 eventCacheHits: expect.any(Number),
101 eventCacheMisses: expect.any(Number),
102 eventCacheBuilds: expect.any(Number),
103 frameCacheHits: expect.any(Number),
104 frameCacheMisses: expect.any(Number),
105 frameRenders: expect.any(Number),
106 }));
107 expect(snapshot.metrics.eventBuildDurationMs).toEqual(expect.objectContaining({
108 last: expect.any(Number),
109 max: expect.any(Number),
110 average: expect.any(Number),
111 }));
112 });
113
93 114 test('purgeSubtitleCacheDir removes the subtitle cache directory and its contents', () =>
94 115 {
95 116 const testDir = path.join(os.tmpdir(), 'filemanager-video-subtitles');
@@ -3,7 +3,7 @@
3const fs = require('fs');
3const fs = require('fs');
4const os = require('os');
4const os = require('os');
5const path = require('path');
5const path = require('path');
6const { parseRenderOptions, pruneOldImageSubtitleCaches, purgeSubtitleCacheDir } = require('../server/image-subtitles');
6const { parseRenderOptions, getImageSubtitleDebugSnapshot, purgeSubtitleCacheDir } = require('../server/image-subtitles');
7
7
8describe('image subtitle option parsing', () =>
8describe('image subtitle option parsing', () =>
9{
9{
@@ -90,6 +90,27 @@ describe('image subtitle option parsing', () =>
90
90
91describe('subtitle cache directory management', () =>
91describe('subtitle cache directory management', () =>
92{
92{
93 test('debug snapshot reports unbounded item caps and timing metrics', () =>
94 {
95 const snapshot = getImageSubtitleDebugSnapshot();
96
97 expect(snapshot.eventCache.maxItems).toBeNull();
98 expect(snapshot.frameCache.maxItems).toBeNull();
99 expect(snapshot.metrics).toEqual(expect.objectContaining({
100 eventCacheHits: expect.any(Number),
101 eventCacheMisses: expect.any(Number),
102 eventCacheBuilds: expect.any(Number),
103 frameCacheHits: expect.any(Number),
104 frameCacheMisses: expect.any(Number),
105 frameRenders: expect.any(Number),
106 }));
107 expect(snapshot.metrics.eventBuildDurationMs).toEqual(expect.objectContaining({
108 last: expect.any(Number),
109 max: expect.any(Number),
110 average: expect.any(Number),
111 }));
112 });
113
93 test('purgeSubtitleCacheDir removes the subtitle cache directory and its contents', () =>
114 test('purgeSubtitleCacheDir removes the subtitle cache directory and its contents', () =>
94 {
115 {
95 const testDir = path.join(os.tmpdir(), 'filemanager-video-subtitles');
116 const testDir = path.join(os.tmpdir(), 'filemanager-video-subtitles');
tests/video-stream.test.js +13 −0
@@ -517,6 +517,19 @@ describe('video stream bootstrap', () =>
517 517 expect(fs.existsSync(path.join(sessionDir, 'stream_Audio-1_segment_000041.ts'))).toBe(true);
518 518 expect(pruneOldImageSubtitleCaches).toHaveBeenCalled();
519 519
520 const debugSnapshot = streamModule.getVideoStreamDebugSnapshot();
521 const activeStream = debugSnapshot.activeStreams.find((stream) => stream.sourcePath === sourceFile);
522 expect(debugSnapshot.streamPruneDeleteBatchSize).toBe(16);
523 expect(activeStream).toEqual(expect.objectContaining({
524 lastSegmentPrune: expect.objectContaining({
525 cutoff: 40,
526 deletedSegments: expect.any(Number),
527 durationMs: expect.any(Number),
528 completedAt: expect.any(Number),
529 }),
530 }));
531 expect(activeStream.lastSegmentPrune.deletedSegments).toBeGreaterThan(0);
532
520 533 streamModule.releaseVideoStreamSession(session.sessionId);
521 534 });
522 535
@@ -517,6 +517,19 @@ describe('video stream bootstrap', () =>
517 expect(fs.existsSync(path.join(sessionDir, 'stream_Audio-1_segment_000041.ts'))).toBe(true);
517 expect(fs.existsSync(path.join(sessionDir, 'stream_Audio-1_segment_000041.ts'))).toBe(true);
518 expect(pruneOldImageSubtitleCaches).toHaveBeenCalled();
518 expect(pruneOldImageSubtitleCaches).toHaveBeenCalled();
519
519
520 const debugSnapshot = streamModule.getVideoStreamDebugSnapshot();
521 const activeStream = debugSnapshot.activeStreams.find((stream) => stream.sourcePath === sourceFile);
522 expect(debugSnapshot.streamPruneDeleteBatchSize).toBe(16);
523 expect(activeStream).toEqual(expect.objectContaining({
524 lastSegmentPrune: expect.objectContaining({
525 cutoff: 40,
526 deletedSegments: expect.any(Number),
527 durationMs: expect.any(Number),
528 completedAt: expect.any(Number),
529 }),
530 }));
531 expect(activeStream.lastSegmentPrune.deletedSegments).toBeGreaterThan(0);
532
520 streamModule.releaseVideoStreamSession(session.sessionId);
533 streamModule.releaseVideoStreamSession(session.sessionId);
521 });
534 });
522
535