hl-valdemar/tubular
1
0
Fork 0
Code Issues Pull Requests Actions Packages Releases Wiki Activity

feat: integrate Miruro playback and episode progress

Browse Source
This commit is contained in:
Valdemar Høgsvig Lorenzen
2026-10-12 22:23:15 +02:00
parent 0fe24ff302
commit 0e909001d0
9 changed files with 1676 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+16
View File
@@ -1380,6 +1380,12 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "tinyvec"
version = "1.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]]
name = "toml_datetime"
version = "1.1.2+spec-1.1.0"
@@ -1430,6 +1436,7 @@ dependencies = [
"serde_json",
"time",
"toml_edit",
"unicode-normalization",
"ureq",
]
@@ -1439,6 +1446,15 @@ version = "1.0.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
[[package]]
name = "unicode-normalization"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8"
dependencies = [
"tinyvec",
]
[[package]]
name = "unicode-segmentation"
version = "1.13.3"
Cargo.toml
+2 -1
View File
@@ -2,7 +2,7 @@
name = "tubular"
version = "0.1.0"
edition = "2024"
description = "A terminal YouTube browser that opens selected videos in Terebi"
description = "A terminal YouTube and Miruro browser that opens selected videos in Terebi"
[dependencies]
clap = { version = "4.6.7", features = ["derive"] }
@@ -13,6 +13,7 @@ serde = { version = "1.0.229", features = ["derive"] }
serde_json = { version = "1.0.151", features = ["raw_value"] }
toml_edit = "0.25.17"
time = "0.3.55"
unicode-normalization = "0.1.25"
ureq = { version = "3.4.2", default-features = false, features = ["rustls"] }
[target."cfg(unix)".dependencies]
README.md
+22 -12
View File
@@ -1,13 +1,13 @@
# tubular
A modal terminal YouTube browser built with Rust and [Ratatui](https://ratatui.rs/). Search for videos, organize subscribed channels with overlapping tags, browse their feeds, and open a selected video in Terebi.
A modal terminal YouTube and Miruro browser built with Rust and [Ratatui](https://ratatui.rs/). Search for videos or anime episodes, organize subscribed YouTube channels with overlapping tags, and open a selection in Terebi.
## Requirements
- A recent stable Rust toolchain, including Cargo, to build and install.
- [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) on `PATH` for searches, feeds, and full descriptions.
- [`yt-dlp`](https://github.com/yt-dlp/yt-dlp) on `PATH` for YouTube searches, feeds, and full descriptions; Miruro uses the native API client.
- `curl` on `PATH` for thumbnails.
- The **Terebi CLI** installed on `PATH`, together with its app bundle and streaming dependencies, for playback. It must support `--new-instance --url … --progress-file PATH`.
- The **Terebi CLI** installed on `PATH`, together with its app bundle and streaming dependencies, for playback. YouTube requires `--new-instance --url … --progress-file PATH`; Miruro additionally requires `--source-file PATH` and version-2 episode progress support.
The default playback setup requires **macOS 15+**, since Terebi is a native macOS player. Install Terebi separately and follow its dependency/setup instructions; installing Tubular does not install the player.
@@ -34,6 +34,7 @@ cargo run --locked -- "rust programming"
```sh
tubular # Start without a query
tubular "rust programming" # Search immediately
tubular --miruro "apothecary" # Search Miruro instead
tubular --player /path/to/terebi "japanese podcast"
tubular --help
```
@@ -49,6 +50,15 @@ terebi --new-instance --url 'https://www.youtube.com/watch?v=VIDEO_ID' \
Each selection opens an independent player instance without replacing existing playback. Tubular creates a private reporting session and consumes Terebi's progress records independently of the launcher. `--player` overrides the Terebi CLI executable; it is not a generic mpv/IINA adapter. Tubular reports launcher failures, while later streaming/playback errors appear in Terebi. Quitting Tubular leaves launched players running.
### Miruro episodes
Use `--miruro` at startup, or reach the permanent **Miruro** tab with Tab / Shift-Tab. Press `i` to edit its search and Enter to submit. Autocomplete returns at most eight catalog titles, without pagination. Enter on a title loads its returned episodes; Enter on an episode resolves a fresh direct **ssub / sun** HLS source and launches Terebi with a private source document. `/` still opens YouTube Search.
Episodes support watch later, saved videos, currently watching, and watched status. Only episodes—not catalog titles—can be saved. Reopening a saved episode resolves new URLs; identity is the opaque Miruro catalog ID plus its returned episode number. Miruro entries appear in combined collections, not YouTube channel categories. Channel/tag actions and YouTube thumbnails/extraction do not apply.
Stream and external VTT subtitle headers are supplied explicitly; Terebi selects the source's declared default subtitle. Sources requiring media redirects, cross-origin HLS, proxies, iframe playback, or provider-specific playlist rewriting fail explicitly. No provider/language fallback or automatic expired-source refresh is performed.
## Controls
These are the default bindings. Overrides in `[keys]` replace an action's bindings; the live hints and `?` reference show the configured keys.
@@ -58,9 +68,9 @@ These are the default bindings. Overrides in `[keys]` replace an action's bindin
| Key | Action |
| --- | --- |
| `j` / `k`, ↓ / ↑ | Select the next / previous video |
| Enter | Open the selected video in Terebi |
| Enter | Open a Miruro catalog title's episodes, or launch the selected video/episode in Terebi |
| `g` / Home, `G` / End | Select the first / last loaded video |
| `i` | Edit the current category/combined feed's search, or general Search from other views |
| `i` | Edit the current category/combined feed or Miruro search; other views use general YouTube Search |
| `/` | Go directly to general YouTube Search |
| `F` (Shift-F) | Go directly to the subscription Feed |
| `d` | Open video info; press again for the full description, again to return to info |
@@ -205,7 +215,7 @@ Category entries accept `@handles`, HTTPS YouTube channel URLs, or canonical `UC
Video tables color channel names **cyan** when the creator belongs to at least one category. Untagged registrations keep the default text color; selected rows retain their black-on-cyan highlight. Membership is matched by channel ID or handle, not the displayed name, and updates after saving tags or reloading config.
- **Search** and **Feed** are permanent tabs. Feed combines all `[channels]` registrations, including untagged channels, plus channels entered directly in categories, without duplicates.
- **Search**, **Feed**, and **Miruro** are permanent tabs. Feed combines all YouTube `[channels]` registrations, including untagged channels, plus channels entered directly in categories, without duplicates.
- Browsing an unpinned category, combined-category feed, channel, watch-later list, or saved-video list uses one shared temporary tab; choosing another replaces it. Pinned categories keep their own tabs, marked with a dimmed `p`.
- Removing a channel from a category does **not** unsubscribe it from Feed. To unsubscribe, remove its `[channels]` mappings and category references, then press `r`.
- Temporary tabs do not survive restarting; saved categories and pins do. Hidden views retain loaded rows, selection, and pagination during the session.
@@ -214,7 +224,7 @@ Bulk editing refuses to clear all tags when a target has no `[channels]` registr
## Watch later
Watch later is **one collection**, deduplicated by YouTube video ID. Category lists are derived from the saved videos' channel identities and the **current** subscription categories: adding or removing a channel from a category changes its watch-later list after saving tags or reloading config with `r`. No category memberships are stored on individual videos.
Watch later is **one collection**, deduplicated by YouTube video ID or stable Miruro episode identity. Category lists are derived from saved YouTube videos' channel identities and the **current** subscription categories: adding or removing a channel from a category changes its watch-later list after saving tags or reloading config with `r`. Miruro episodes appear only in the combined list. No category memberships are stored on individual videos.
`W` opens a searchable collection selector from any video view or preview. The All categories (combined) entry comes first, followed by configured categories; it remains available even when there are no categories. Navigate with `j` / `k` or arrows in normal mode, or press `i` to filter by name and use arrows / `Ctrl-j` / `Ctrl-k` while typing. `Enter` browses the selected match; with no matches it does nothing. `Esc` returns to normal mode, then closes the selector without navigating. Reopening clears the search. `Alt-w` still opens the current category's watch later directly.
@@ -234,15 +244,15 @@ Files are siblings of `config.toml`:
- `saved-videos.toml`: independent versioned saved-video metadata.
- `currently-watching.toml`: versioned metadata for started, unfinished videos.
- `watch-state.toml`: versioned global watched video IDs.
- `playback-sessions/`: private per-launch progress streams, expected video identities, video snapshots, and durable admission markers.
- `playback-sessions/`: private per-launch progress streams, expected identities, metadata snapshots, durable admission markers, and Miruro source documents.
State writes are atomic and owner-only. Invalid state files or detected external edits are reported rather than overwritten; repair the file or restart before retrying. Each state file is limited to 8 MiB.
Progress sessions survive Tubular exit so later player reports can be recovered on the next launch. They are retained because the GUI may still be writing; up to 4,096 sessions are monitored. If that limit is reached, close the corresponding players, remove their inactive session directories, and restart Tubular. Never remove an active player's reporting directory.
Progress sessions survive Tubular exit so later player reports can be recovered on the next launch. Miruro source documents are `0600` files in `0700` session directories and remain unchanged, even after launcher exit or reported end/close; they contain private, expiring resource URLs and headers. Sessions are retained because CLI exit does not prove the independent GUI has finished; up to 4,096 are monitored. Close the corresponding players before removing their inactive session directories, then restart Tubular. Never remove an active player's directory.
## Saved videos
Saved videos is a separate collection from watch later, deduplicated by YouTube video ID. It uses the same offline title, creator, and publication metadata and follows **current channel category memberships**, not categories stored on individual videos.
Saved videos is a separate collection from watch later, deduplicated by YouTube video ID or stable Miruro episode identity. It uses the same offline title, creator, and publication metadata; YouTube entries follow **current channel category memberships**, not categories stored on individual videos. Miruro episodes appear in All categories (combined).
`V` opens the same searchable selector for saved videos, with **All categories (combined)** first and configured categories beneath it. `Alt-v` opens the current category's saved videos directly. `Alt-w` / `Alt-v` retain the explicit category when switching between category-scoped collection lists; without a current category, they leave the view unchanged. All four shortcuts work in normal mode and previews, not text-entry fields or modal selectors. Both collections share the existing temporary-tab and back/forward history behavior.
@@ -264,7 +274,7 @@ Use `x` for individual removal or `X` for bulk confirmation. Removal neither mar
## Collection removal
Removal controls apply **only in watch-later, saved-video, and currently-watching views**, including previews opened from those lists. Search, Feed, ordinary channel/category views, and their previews do not expose removal actions or hints. Text fields and selectors retain their normal controls.
Removal controls apply **only in watch-later, saved-video, and currently-watching views**, including previews opened from those lists. Search, Feed, Miruro catalog/episode browsing, ordinary channel/category views, and their previews do not expose removal actions or hints. Text fields and selectors retain their normal controls.
`x` immediately removes only the selected video from the active collection. `X` snapshots the current list and opens bulk confirmation: watch later uses the current category and All/Unwatched/Watched filter; saved videos uses the current category's entire saved list; Currently watching uses its entire combined list. An empty list does not open confirmation.
@@ -322,7 +332,7 @@ Video info shows a thumbnail, title, author/publication details, and a clipped d
## Development
[Miruro API notes](docs/miruro-api.md) document the native `tubular::miruro` wrapper and verified transport behavior. Miruro is not yet integrated into the TUI or Terebi playback.
[Miruro API notes](docs/miruro-api.md) document the native `tubular::miruro` wrapper, source-document projection, stable episode/progress contracts, and verification limits.
```sh
cargo fmt --check
docs/miruro-api.md
+68 -7
View File
@@ -1,6 +1,6 @@
# Miruro API research
Observed **2026-10-12**, scoped to **https://barelystarted.miruro.tv**. These are reverse-engineering notes, not an official API specification. Tubular now has a native API module and runnable smoke example, but no Miruro TUI or Terebi integration.
Observed **2026-10-12**, scoped to **https://barelystarted.miruro.tv**. These are reverse-engineering notes, not an official API specification. Tubular includes a native API module, Miruro catalog/episode browsing, and direct Sun playback through Terebi's implemented source/progress contracts.
Evidence labels:
@@ -263,7 +263,7 @@ Observed `providerOrder`: `sun, bee, pewe, hop, ally, kiwi, moo, bonk, knob, nun
- The browser loads [strm.cx/client.js][strm-client], renders `<strmcx-embed>`, and calls `setSession` with local player state/source/subtitle/skip data over `strmcx/v1` messaging. This was **not required for the tested source or direct-resource GETs**. It is not evidence of an authenticated API session prerequisite.
- strm client contains a best-effort `POST /v1/embeds`; it was source-inspected only, not invoked. Iframe providers and unrelated comment/telemetry endpoints were not exercised.
For a future native player: preserve fresh query tokens, per-server headers and relevant provider config; correctly resolve nested/relative HLS URLs. Do not apply one provider's headers indiscriminately to another provider's subtitles/thumbnails. Terebi currently receives only `--new-instance --url … --progress-file PATH` in Tubular ([existing usage](../README.md#usage)); custom headers, external subtitles, HLS overrides, native seeking and full Miruro playback remain unverified.
Tubular's direct Sun integration preserves fresh URLs and explicitly supplies the selected server's headers to both its stream and verified VTT subtitle resources. Terebi owns native HLS resolution and subtitle loading; other providers' proxies, headers and playlist transforms remain unverified and unsupported. See [the adopted integration contract](#tubular--terebi-integration).
## SvelteKit route data
@@ -399,8 +399,10 @@ Library interface, exposed as `tubular::miruro`:
| `client.search(query)` | `io::Result<Vec<serde_json::Value>>`; trimmed, nonempty autocomplete query; no cursor pagination |
| `client.detail(id, slug)` | `io::Result<Detail>`; decoded `media` JSON and `episodes: Vec<Value>`; validates returned media ID |
| `client.sources(id, episode)` | `io::Result<Value>`; fresh full response preserving headers/config/subtitles; validates positive episode and returned episodeNumber/tracks |
| `client.source_document(id, episode, title)` | `io::Result<SourceDocument>`; fresh validated direct `ssub/sun` HLS/VTT projection for Terebi |
| `Identity::new(id, episode)`, `Identity::from_key(key)` | Validated catalog namespace/episode identity; `key()` is `miruro:<opaque ID>:<episode>` |
The wrapper does not choose providers/languages or implement a player. It preserves JSON metadata rather than freezing every deployment-specific field into a second schema. IDs/slugs are validated as nonempty ASCII alphanumeric/hyphen/underscore path segments. HTTP failures remain errors (`PermissionDenied` for 401/403); HTML and unexpected Content-Types are rejected, not converted to empty results. No tokenized URLs appear in its error messages. Successful bodies are limited to 2 MiB.
The raw methods preserve JSON metadata rather than freezing every deployment-specific field into a second schema. `source_document` deliberately selects only direct Sun soft-subtitle playback; it does not guess another provider or implement a player. IDs/slugs are validated as nonempty ASCII alphanumeric/hyphen/underscore path segments. HTTP failures remain errors (`PermissionDenied` for 401/403); HTML and unexpected Content-Types are rejected, not converted to empty results. No tokenized URLs appear in its error messages. Successful bodies are limited to 2 MiB.
Its private route decoder resolves only the needed detail value and episode promise, using separate value tables and promise IDs. Unrelated sidebar promises are not returned. It supports the JSON-shaped tables verified on this path, **not arbitrary devalue**: tagged special types and negative sentinels are explicit errors, never silently coerced to null. Out-of-range references, rejected/missing/duplicate episode chunks, cycles/excessive depth and decoded-size amplification are rejected. Depth is bounded to 64 and expanded values share a 2 MiB size budget. These are wrapper safety limits, not Miruro server limits.
@@ -425,16 +427,75 @@ The preceding scratch probe also reported HTTP 200 for search (`application/json
The [example](../examples/miruro_probe.rs) deliberately uses the documented title and `ssub/sun` provider, and limits resource requests to HTTPS `*.keeply.top` without redirects. It fails explicitly if that sample disappears or changes instead of silently choosing a different provider. It fetches no nested playlist, media segment, key, sprite or download page, and saves no tokens/bodies. This is an explicit network smoke command, not a network-dependent unit test.
Verification: `cargo test --locked` passed all 164 tests, including the new episode-promise regression; `cargo clippy --locked --all-targets -- -D warnings` passed; `rustfmt --edition 2024 --check src/lib.rs src/miruro.rs examples/miruro_probe.rs` passed. The decoder regression covers independent tables, out-of-order unrelated chunks, preserved episode/absolute numbers and nulls, unresolved/rejected promises, bad references/cycles, unsupported tags and route redirects.
Wrapper-extraction verification, before TUI integration: `cargo test --locked` passed all 164 tests, including the episode-promise regression; `cargo clippy --locked --all-targets -- -D warnings` passed; `rustfmt --edition 2024 --check src/lib.rs src/miruro.rs examples/miruro_probe.rs` passed. The decoder regression covers independent tables, out-of-order unrelated chunks, preserved episode/absolute numbers and nulls, unresolved/rejected promises, bad references/cycles, unsupported tags and route redirects.
## Open questions before TUI/player integration
## Tubular → Terebi integration
`tubular --miruro "apothecary"` starts the permanent Miruro search tab. Catalog Enter opens actual returned episodes; episode Enter resolves fresh sources off-thread. Autocomplete is limited to eight titles, without invented pagination. One worker handles bounded network work; changing views/queries invalidates stale responses rather than launching obsolete selections. Canonical title/slug rules follow the [deployed helper][link-helpers], including Unicode NFD normalization.
Titles normalizing to an empty slug fail explicitly; the wrapper does not guess an alternative detail route. This does not reject accented Latin titles, which use NFD decomposition.
### Source document and launch
`Client::source_document` chooses the first nonempty server/first direct HLS stream from the returned `ssub` track's `sun` provider. It requires native playback, `proxy:false`, and no enabled HLS host/query rewriting. Missing/unsupported sources are errors; no alternate language/provider, embed, proxy or refresh fallback is added.
| Terebi source-v1 field | Projection |
| --- | --- |
| `url` | Selected stream URL, unchanged, including its query tokens |
| `headers` | Selected server map, without trimming or rewriting |
| `identity` | `{provider:"miruro", media_id:<opaque catalog ID>, episode:<returned episode_number>}`; not slug/external ID/absolute number/array index/playback provider |
| `title` | Episode display title |
| `subtitles` | Selected Sun provider's VTT tracks; other external formats excluded |
| Track `url`, `language`, `label`, `default` | Returned `file` and subtitle metadata |
| Track `headers` | Explicit selected Sun server map, verified for Sun resources only; no general inheritance rule |
Projection validates HTTP(S) resource URLs, header syntax/values, case-insensitive duplicate headers, identities, control characters, 32-track/64-header limits and at most one default subtitle. Session creation also enforces the 1 MiB serialized source-document limit. Miruro media IDs allow at most 512 ASCII letters/digits/`-`/`_`; episodes range from 1 through 2147483647.
`State::miruro_session` binds the source identity to the selected episode, creates a unique private session directory (`0700`), writes immutable `source.json` (`0600`) and metadata/identity snapshots, and returns a **nonexistent** progress destination. `PlayerLaunch::start_source` passes absolute filesystem paths, not resource URLs/header values:
```sh
terebi --new-instance --source-file '/private/session/source.json' \
--progress-file '/private/session/progress.jsonl'
```
Terebi exclusively creates the report (`0600`). CLI exit acknowledges only launch acceptance, not successful media opening or completion. Tubular retains source/report files across launcher exit, final events, and Tubular shutdown because the independent GUI may still need them. The existing 4,096-session cap and explicit inactive-session cleanup apply; close the corresponding players before removing their session directories. Private source documents contain expiring URLs/headers; watched/collection state contains only stable identity and display metadata.
Terebi's implemented transport preserves headers on master/variant/segment/key/init requests and probes Sun's image-suffixed media/subtitle resources by content/declared VTT format. All media redirects and foreign-origin HLS resources are rejected; subtitle redirects must stay on their initial origin. Terebi does not refresh expired sources. These restrictions remain visible errors, not silent provider switches.
### Progress and collections
Miruro JSONL reports use version 2:
```json
{"version":2,"event":"progress","media_identity":{"provider":"miruro","media_id":"opaque_ID","episode":1},"position_seconds":95.0,"duration_seconds":100.0,"completion_eligible":true}
```
Tubular validates the version, namespace, media ID, episode, event and numeric fields, then compares the complete identity against that launch's private session marker. Cross-episode/provider/media records cannot admit or complete another episode. YouTube's canonical `media_url` and version-1 validation remain supported, including its existing unknown-field tolerance.
Matching acknowledged positive positions admit currently-watching entries. Watched status retains `completion_eligible && duration_seconds > 0 && position_seconds / duration_seconds >= 0.95`; null duration is ineligible. Seeking near the end can satisfy it. Launch/window closure alone cannot. Duplicate records, partial trailing lines and later reports recovered after Tubular restart retain the existing session-consumption behavior.
Watch later, saved videos, currently watching and watched state accept stable episode keys `miruro:<opaque ID>:<canonical episode>`. Catalog rows (`miruro-series:<ID>`) are not saveable/playable episode identities. Saved episode playback resolves fresh sources; CDN addresses/tokens never become collection keys. Miruro has no YouTube channel identity, so category/tag/channel actions and YouTube extraction/thumbnails are excluded; its episodes use combined collections.
### Integration verification
The actual Tubular terminal workflow was exercised with isolated `$XDG_CONFIG_HOME` state and the updated Terebi developer bundle selected by `--player`: autocomplete returned eight titles; the selected title returned two episodes; episode 1 was saved to both collections, freshly resolved, and launched through `State::miruro_session` / `PlayerLaunch::start_source`.
The private source document retained Origin/Referer headers and nine VTT tracks with English declared default. Session mode `0700` and source/report modes `0600` were checked without logging resource URLs or header values. Actual Terebi v2 records matched the selected opaque ID/episode and acknowledged position advanced from about 7 to 141 seconds, with positive duration and completion eligibility. Tubular admitted that identity to its visible Currently watching list and persisted source-free collection metadata.
After Tubular exited successfully, its independent GUI continued appending matching v2 progress and the private source file remained byte-for-byte unchanged. That player/session was intentionally retained for the user's visible subtitle/seek check; no source cleanup was attempted while playback remained active.
Final integrated checks passed: `cargo fmt --check`, all **175 tests**, `cargo clippy --locked --all-targets -- -D warnings`, `cargo build --locked`, and the built executable's `--help`. Regressions cover v1 compatibility, v2 identity/95%/null-duration boundaries, restart/partial-report recovery, source/session permissions and retention, collection routing, stale responses and canonical Unicode slugging. The installed Terebi CLI was older and lacked `--source-file`; the explicit developer-bundle override avoided changing that installation.
This run proves Tubular-to-Terebi launch, acknowledged playback/progress consumption and independent lifetime—not visible subtitle rendering, perceived A/V synchronization, or seeks through the user's installed release. English subtitle appearance/synchronization before and after seeking still requires a visible-app check.
## Remaining research questions
- Will this same native client remain accessible on other networks/deployments? Its sampled path passed; local curl previously failed, and no workaround is implemented.
- Is there a stable plain metadata/episode API? The wrapper currently owns the bounded JSON-shaped SvelteKit decoder for the observed deployment.
- What are server-side filter/enum constraints, multi-value semantics, cursor lifetime, page-size-cookie limits, numeric-ID lookups and slug behavior?
- What happens for unavailable episodes, empty tracks, provider failures, bad cursors, rejected promise chunks and syntactically valid missing IDs?
- Which headers/config transforms are required per provider, and can Terebi propagate them to HLS playlists/segments/keys and external subtitles?
- Are direct streams usable end-to-end, including rendition/segment access and seeking? Only master/subtitle/thumbnail retrieval was verified.
- Which headers/config transforms are required by providers other than the verified direct Sun path?
- Will other streams require redirects or cross-origin HLS resources that Terebi deliberately rejects? The Terebi team's handoff exercised direct Sun native decode, subtitles and acknowledged seeks, not every provider or comprehensive visible synchronization.
- What are seeded discovery authentication requirements, rate limits, token expiry/IP binding and content-use permissions? Reachable internal endpoints do not imply a supported third-party integration or permission to redistribute content.
Hashed JavaScript URLs below identify the inspected deployment, not stable versions. For later research, re-read the homepage's entry-script imports and [route map][routes] to locate the current bundles; do not hardcode these hashes in the wrapper.
src/main.rs
+381 -38
View File
File diff suppressed because one or more lines are too long
src/miruro.rs
+349 -4
View File
@@ -1,10 +1,15 @@
//! Blocking access to the deployed Miruro catalog and source API.
//!
//! See `docs/miruro-api.md`. Source URLs are ephemeral; retain server headers and
//! provider configuration for playback. This module does not select a provider,
//! bypass protection, decode iframe embeds, or integrate with a player.
use std::{io, mem::size_of, time::Duration};
//! See `docs/miruro-api.md`. Source URLs are ephemeral. Playback projection is
//! limited to direct Sun soft-subtitle sources; no proxy, iframe or retry fallback.
use std::{
collections::{BTreeMap, HashSet},
io,
mem::size_of,
time::Duration,
};
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
const BASE: &str = "https://barelystarted.miruro.tv";
@@ -20,6 +25,262 @@ pub struct Detail {
pub episodes: Vec<Value>,
}
/// Stable catalog identity, never a CDN address or playback-provider name.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub struct Identity {
pub provider: String,
pub media_id: String,
pub episode: u32,
}
impl Identity {
pub fn new(media_id: String, episode: u32) -> io::Result<Self> {
let identity = Self {
provider: "miruro".into(),
media_id,
episode,
};
identity.validate()?;
Ok(identity)
}
pub fn validate(&self) -> io::Result<()> {
if self.provider != "miruro"
|| self.media_id.len() > 512
|| self.episode == 0
|| self.episode > i32::MAX as u32
{
return Err(invalid("Invalid Miruro episode identity"));
}
validate_segment(&self.media_id)
}
pub fn key(&self) -> String {
format!("miruro:{}:{}", self.media_id, self.episode)
}
pub fn from_key(key: &str) -> io::Result<Self> {
let (media_id, episode) = key
.strip_prefix("miruro:")
.and_then(|rest| rest.split_once(':'))
.ok_or_else(|| invalid("Invalid Miruro episode key"))?;
if episode.starts_with('0') || !episode.bytes().all(|byte| byte.is_ascii_digit()) {
return Err(invalid("Noncanonical Miruro episode key"));
}
Self::new(
media_id.into(),
episode
.parse()
.map_err(|_| invalid("Invalid Miruro episode number"))?,
)
}
}
#[derive(Clone, Debug, Serialize)]
pub struct Subtitle {
pub url: String,
pub format: String,
pub language: String,
pub label: String,
pub default: bool,
pub headers: BTreeMap<String, String>,
}
#[derive(Clone, Debug, Serialize)]
pub struct SourceDocument {
pub version: u32,
pub url: String,
pub headers: BTreeMap<String, String>,
pub identity: Identity,
#[serde(skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
pub subtitles: Vec<Subtitle>,
}
impl SourceDocument {
pub fn validate(&self) -> io::Result<()> {
if self.version != 1
|| self.subtitles.len() > 32
|| self.subtitles.iter().filter(|track| track.default).count() > 1
{
return Err(invalid(
"Invalid source document version or subtitle limits",
));
}
if self
.title
.as_ref()
.is_some_and(|title| title.chars().any(char::is_control))
{
return Err(invalid("Invalid source title"));
}
self.identity.validate()?;
validate_url(&self.url)?;
validate_headers(&self.headers)?;
for track in &self.subtitles {
if track.format != "vtt" {
return Err(invalid("Only external VTT subtitles are supported"));
}
if track.label.chars().any(char::is_control)
|| track.language.chars().any(char::is_control)
{
return Err(invalid("Invalid subtitle label or language"));
}
validate_url(&track.url)?;
validate_headers(&track.headers)?;
}
Ok(())
}
/// No provider/language fallback: take the first direct ssub/Sun server.
pub fn from_sources(sources: &Value, identity: Identity, title: &str) -> io::Result<Self> {
identity.validate()?;
if sources["episodeNumber"].as_u64() != Some(u64::from(identity.episode)) {
return Err(invalid(
"Resolved source episode does not match its identity",
));
}
let config = &sources["config"]["sun"];
if config["player"] != "native"
|| config["proxy"] != false
|| config["hls"]["segments"]["enabled"] == true
|| config["hls"]["query"]["enabled"] == true
{
return Err(invalid("Sun requires unsupported proxy or HLS rewriting"));
}
let provider = sources["tracks"]
.as_array()
.and_then(|tracks| tracks.iter().find(|track| track["track"] == "ssub"))
.and_then(|track| track["providers"].as_array())
.and_then(|providers| {
providers
.iter()
.find(|provider| provider["provider"] == "sun")
})
.ok_or_else(|| invalid("Direct Sun soft-subtitle source is unavailable"))?;
let server = provider["servers"]
.as_array()
.and_then(|servers| {
servers.iter().find(|server| {
server["streams"]
.as_array()
.is_some_and(|streams| !streams.is_empty())
})
})
.ok_or_else(|| invalid("Direct Sun stream is unavailable"))?;
let stream = &server["streams"][0];
if !matches!(
stream["format"].as_str(),
Some("hls" | "m3u8" | "application/vnd.apple.mpegurl" | "application/x-mpegurl")
) || !stream["embed"].is_null()
{
return Err(invalid("Selected Sun stream is not direct HLS"));
}
let headers = source_headers(&server["headers"])?;
let mut subtitles = Vec::new();
for subtitle in provider["subtitles"]
.as_array()
.ok_or_else(|| invalid("Missing Sun subtitle array"))?
{
if subtitle["format"] != "vtt" {
continue;
}
if subtitles.len() == 32 {
return Err(invalid("Source subtitle limit exceeded"));
}
subtitles.push(Subtitle {
url: source_string(subtitle, "file")?,
format: "vtt".into(),
language: source_string(subtitle, "language")?,
label: source_string(subtitle, "label")?,
default: subtitle["default"]
.as_bool()
.ok_or_else(|| invalid("Invalid subtitle default flag"))?,
// Verified for direct Sun only. These are explicit resource headers,
// not an inheritance rule for other providers.
headers: headers.clone(),
});
}
let document = Self {
version: 1,
url: source_string(stream, "url")?,
headers,
identity,
title: Some(title.into()),
subtitles,
};
document.validate()?;
Ok(document)
}
}
fn source_string(value: &Value, field: &str) -> io::Result<String> {
value
.get(field)
.and_then(Value::as_str)
.map(str::to_owned)
.ok_or_else(|| invalid("Missing or invalid resolved source field"))
}
fn source_headers(value: &Value) -> io::Result<BTreeMap<String, String>> {
if value.is_null() {
return Ok(BTreeMap::new());
}
let headers = value
.as_object()
.ok_or_else(|| invalid("Invalid source header map"))?;
if headers.len() > 64 {
return Err(invalid("Source header limit exceeded"));
}
headers
.iter()
.map(|(key, value)| {
Ok((
key.clone(),
value
.as_str()
.ok_or_else(|| invalid("Invalid source header value"))?
.into(),
))
})
.collect()
}
fn validate_url(url: &str) -> io::Result<()> {
let uri: ureq::http::Uri = url.parse().map_err(|_| invalid("Invalid source URL"))?;
if !matches!(uri.scheme_str(), Some("http" | "https"))
|| uri.host().is_none_or(str::is_empty)
|| uri
.authority()
.is_some_and(|authority| authority.as_str().contains('@'))
|| url.chars().any(char::is_control)
{
return Err(invalid(
"Source URLs must be absolute HTTP(S) without credentials",
));
}
Ok(())
}
fn validate_headers(headers: &BTreeMap<String, String>) -> io::Result<()> {
if headers.len() > 64 {
return Err(invalid("Source header limit exceeded"));
}
let mut names = HashSet::with_capacity(headers.len());
for (name, value) in headers {
let name = ureq::http::HeaderName::from_bytes(name.as_bytes())
.map_err(|_| invalid("Invalid source header name"))?;
if !names.insert(name)
|| value.chars().any(char::is_control)
|| ureq::http::HeaderValue::from_str(value).is_err()
{
return Err(invalid("Invalid or duplicate source header"));
}
}
Ok(())
}
impl Default for Client {
fn default() -> Self {
Self {
@@ -98,6 +359,16 @@ impl Client {
Ok(value)
}
pub fn source_document(
&self,
id: &str,
episode: u32,
title: &str,
) -> io::Result<SourceDocument> {
let identity = Identity::new(id.into(), episode)?;
SourceDocument::from_sources(&self.sources(id, episode)?, identity, title)
}
fn get(&self, path: &str, query: &[(&str, &str)], types: &[&str]) -> io::Result<String> {
let mut response = self
.agent
@@ -277,6 +548,80 @@ fn expand(table: &[Value], index: usize, depth: usize, budget: &mut usize) -> io
mod tests {
use super::*;
#[test]
fn source_projection_rejects_unsafe_or_unsupported_playback_contracts() {
let sources = serde_json::json!({
"episodeNumber": 1,
"config": { "sun": { "player": "native", "proxy": false } },
"tracks": [{ "track": "ssub", "providers": [{
"provider": "sun",
"servers": [{
"headers": { "Origin": "https://ultracloud.cc", "Referer": "https://ultracloud.cc/" },
"streams": [{ "url": "https://vault.keeply.top/master.m3u8?token=secret", "format": "hls", "embed": null }]
}],
"subtitles": [{
"file": "https://vault.keeply.top/english.jpeg", "format": "vtt",
"language": "en", "label": "English", "default": true
}]
}] }]
});
let identity = Identity::new("catalog".into(), 1).unwrap();
let source = SourceDocument::from_sources(&sources, identity.clone(), "Episode 1").unwrap();
assert_eq!(Identity::from_key(&identity.key()).unwrap(), identity);
assert_ne!(
Identity::new("catalog".into(), 2).unwrap().key(),
source.identity.key()
);
for key in [
"miruro:catalog:01",
"miruro:catalog:0",
"miruro:catalog:2147483648",
"miruro:catalog/:1",
] {
assert!(Identity::from_key(key).is_err());
}
for (pointer, value) in [
("/episodeNumber", serde_json::json!(49)),
("/config/sun/proxy", serde_json::json!({"segments": true})),
(
"/tracks/0/providers/0/servers/0/streams/0/url",
serde_json::json!("file:///secret"),
),
(
"/tracks/0/providers/0/servers/0/streams/0/embed",
serde_json::json!({"url": "https://embed.example"}),
),
(
"/tracks/0/providers/0/servers/0/headers/origin",
serde_json::json!("secret"),
),
(
"/tracks/0/providers/0/servers/0/headers/Referer",
serde_json::json!("secret\r\nX-Injected: true"),
),
] {
let mut invalid_sources = sources.clone();
if let Some(field) = invalid_sources.pointer_mut(pointer) {
*field = value;
} else {
invalid_sources["tracks"][0]["providers"][0]["servers"][0]["headers"]["origin"] =
value;
}
let error =
SourceDocument::from_sources(&invalid_sources, identity.clone(), "Episode 1")
.unwrap_err();
assert!(!error.to_string().contains("secret"));
}
let mut invalid_source = source;
invalid_source
.subtitles
.push(invalid_source.subtitles[0].clone());
assert!(invalid_source.validate().is_err());
invalid_source.subtitles[1].default = false;
invalid_source.subtitles[1].format = "srt".into();
assert!(invalid_source.validate().is_err());
}
#[test]
fn episodes_resolve_by_promise_id_with_independent_tables_or_fail_explicitly() {
let initial = r#"{"type":"data","nodes":[null,{"type":"data","data":[{"detail":1,"episodes":4},{"id":2,"episode_count":3},"series",12,["Promise",5],9]}]}"#;
src/miruro_browser.rs
+276
View File
@@ -0,0 +1,276 @@
use crate::youtube::Video;
use serde_json::Value;
use std::{
io,
sync::mpsc::{self, Receiver, SyncSender},
thread,
};
use tubular::miruro::{Client, Detail, Identity, SourceDocument};
use unicode_normalization::UnicodeNormalization;
pub enum Request {
Search(String),
Episodes { media_id: String, slug: String },
Source(Video),
}
pub enum Response {
Rows(Vec<(Video, String)>),
Source(Video, Box<SourceDocument>),
}
// One running request, no queued network requests. The UI retains only its newest intent.
pub struct Worker {
sender: SyncSender<(u64, Request)>,
receiver: Receiver<(u64, io::Result<Response>)>,
pub busy: bool,
generation: u64,
}
impl Worker {
pub fn new() -> Self {
let (sender, requests) = mpsc::sync_channel::<(u64, Request)>(1);
let (results, receiver) = mpsc::sync_channel(1);
thread::spawn(move || {
let client = Client::default();
while let Ok((generation, request)) = requests.recv() {
let result = match request {
Request::Search(query) => client
.search(&query)
.and_then(catalog_rows)
.map(Response::Rows),
Request::Episodes { media_id, slug } => client
.detail(&media_id, &slug)
.and_then(|detail| episode_rows(&media_id, detail))
.map(Response::Rows),
Request::Source(video) => Identity::from_key(&video.id)
.and_then(|id| {
client.source_document(&id.media_id, id.episode, &video.title)
})
.map(|source| Response::Source(video, Box::new(source))),
};
if results.send((generation, result)).is_err() {
break;
}
}
});
Self {
sender,
receiver,
busy: false,
generation: 0,
}
}
pub fn submit(&mut self, generation: u64, request: Request) -> io::Result<()> {
if self.busy {
return Err(io::Error::other("Miruro worker is busy"));
}
self.sender
.try_send((generation, request))
.map_err(|_| io::Error::other("Miruro worker unavailable"))?;
self.busy = true;
self.generation = generation;
Ok(())
}
pub fn poll(&mut self) -> Option<(u64, io::Result<Response>)> {
match self.receiver.try_recv() {
Ok(result) => {
self.busy = false;
Some(result)
}
Err(mpsc::TryRecvError::Empty) => None,
Err(mpsc::TryRecvError::Disconnected) if self.busy => {
self.busy = false;
Some((
self.generation,
Err(io::Error::other("Miruro worker stopped")),
))
}
Err(_) => None,
}
}
}
pub fn is_miruro(video: &Video) -> bool {
video.id.starts_with("miruro:") || video.id.starts_with("miruro-series:")
}
fn invalid(message: &str) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, message)
}
fn title(media: &Value) -> io::Result<String> {
let get = |language| {
media
.get("title")
.and_then(|v| v.get(language))
.and_then(Value::as_str)
.map(|text| text.trim_matches(js_whitespace))
.filter(|s| !s.is_empty())
};
let selected = match (get("english"), get("romaji")) {
(Some(a), Some(b)) => {
if a.encode_utf16().count() <= b.encode_utf16().count() {
a
} else {
b
}
}
(Some(a), None) | (None, Some(a)) => a,
_ => media
.get("title")
.and_then(|value| value.get("native"))
.and_then(Value::as_str)
.or_else(|| media.get("id").and_then(Value::as_str))
.ok_or_else(|| invalid("Miruro title is missing"))?,
};
Ok(selected.to_owned())
}
// ECMAScript \s/trim, not Rust's Unicode whitespace (which includes U+0085).
fn js_whitespace(ch: char) -> bool {
matches!(ch, '\u{0009}'..='\u{000d}' | ' ' | '\u{00a0}' | '\u{1680}'
| '\u{2000}'..='\u{200a}' | '\u{2028}' | '\u{2029}' | '\u{202f}'
| '\u{205f}' | '\u{3000}' | '\u{feff}')
}
pub fn slug(title: &str) -> io::Result<String> {
let cleaned: String = title
.nfd()
.filter(|ch| !('\u{0300}'..='\u{036f}').contains(ch))
.flat_map(char::to_lowercase)
.filter(|ch| {
ch.is_ascii_lowercase() || ch.is_ascii_digit() || js_whitespace(*ch) || *ch == '-'
})
.collect();
let mut slug = String::new();
for ch in cleaned.trim_matches(js_whitespace).chars() {
if ch.is_ascii_alphanumeric() {
slug.push(ch);
} else if (js_whitespace(ch) || ch == '-') && !slug.ends_with('-') {
slug.push('-');
}
}
if slug.is_empty() {
Err(invalid("Miruro title has no representable route slug"))
} else {
Ok(slug)
}
}
fn video(id: String, title: String, date: Option<String>) -> Video {
Video {
id,
title,
channel: Some("Miruro".into()),
channel_id: None,
uploader_id: None,
timestamp: None,
publication_date: date,
}
}
pub fn catalog_rows(items: Vec<Value>) -> io::Result<Vec<(Video, String)>> {
items
.into_iter()
.map(|media| {
let id = media
.get("id")
.and_then(Value::as_str)
.ok_or_else(|| invalid("Miruro catalog ID is missing"))?;
Identity::new(id.to_owned(), 1)?;
let title = title(&media)?;
let description = media
.get("description")
.and_then(Value::as_str)
.unwrap_or(
"Catalog title: Enter opens returned episodes; only episodes can be saved.",
)
.to_owned();
Ok((
video(format!("miruro-series:{id}"), title, None),
description,
))
})
.collect()
}
pub fn episode_rows(media_id: &str, detail: Detail) -> io::Result<Vec<(Video, String)>> {
let series = title(&detail.media)?;
let mut seen = std::collections::HashSet::new();
detail
.episodes
.into_iter()
.map(|episode| {
let number = episode
.get("episode_number")
.and_then(Value::as_u64)
.and_then(|n| u32::try_from(n).ok())
.ok_or_else(|| invalid("Miruro episode number is invalid"))?;
let identity = Identity::new(media_id.to_owned(), number)?;
if !seen.insert(number) {
return Err(invalid("Miruro returned duplicate episode numbers"));
}
let label = episode.get("title").and_then(Value::as_str).unwrap_or("");
let title = format!(
"{series} · Episode {number}{}",
if label.is_empty() {
String::new()
} else {
format!(" · {label}")
}
);
let description = episode
.get("synopsis")
.and_then(Value::as_str)
.unwrap_or("Episode metadata unavailable.")
.to_owned();
let date = episode
.get("aired_on")
.and_then(Value::as_str)
.map(str::to_owned);
Ok((video(identity.key(), title, date), description))
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn catalog_and_actual_episode_numbers_remain_distinct() {
let media = serde_json::json!({"id":"opaque_ID", "title":{"english":"The Apothecary Diaries", "romaji":"Kusuriya no Hitorigoto"}});
let rows = catalog_rows(vec![media.clone()]).unwrap();
assert_eq!(rows[0].0.id, "miruro-series:opaque_ID");
assert!(Identity::from_key(&rows[0].0.id).is_err());
let episodes = episode_rows(
"opaque_ID",
Detail {
media,
episodes: vec![
serde_json::json!({"episode_number":3,"absolute_number":49,"title":"Returned"}),
],
},
)
.unwrap();
let identity = Identity::from_key(&episodes[0].0.id).unwrap();
assert_eq!(identity.episode, 3);
assert_eq!(identity.media_id, "opaque_ID");
assert!(episodes[0].0.channel_id.is_none());
assert!(slug("薬屋").is_err());
assert_eq!(slug(" A Test!! -- Title ").unwrap(), "a-test-title");
assert_eq!(slug(" --Title-- ").unwrap(), "-title-");
assert_eq!(slug("École déjà-vu").unwrap(), "ecole-deja-vu");
assert_eq!(slug("Cafe\u{0301}").unwrap(), "cafe");
assert_eq!(slug("Ａ薬屋 KK İ").unwrap(), "kk-i");
assert_eq!(
slug("\u{feff}A\u{00a0}B\u{2003}C\u{feff}").unwrap(),
"a-b-c"
);
assert_eq!(slug("A\u{0085}B").unwrap(), "ab");
assert!(slug("!!薬屋!!").is_err());
assert_eq!(
title(&serde_json::json!({"id":"id","title":{"english":"AB","romaji":"CD"}})).unwrap(),
"AB"
);
assert_eq!(
title(&serde_json::json!({"id":"id","title":{"english":"𐀀A","romaji":"BC"}})).unwrap(),
"BC"
);
assert_eq!(
title(&serde_json::json!({"id":"id","title":{"native":""}})).unwrap(),
""
);
}
}
src/video_state.rs
+507 -28
View File
@@ -10,11 +10,14 @@ use serde::Deserialize;
use toml_edit::{Array, ArrayOfTables, DocumentMut, Table, value};
use crate::youtube::{Video, valid_video_id, validate_video};
use tubular::miruro::{Identity, SourceDocument};
const MAX_STATE_BYTES: u64 = 8 * 1024 * 1024;
const MAX_LINE: usize = 16 * 1024;
const STREAM_BYTES: usize = 16 * 1024;
const MAX_STREAMS: usize = 4096;
const MAX_SOURCE_BYTES: usize = 1024 * 1024;
const MAX_IDENTITY_BYTES: u64 = 7 + 512 + 1 + 10;
static NEXT_NAME: AtomicU64 = AtomicU64::new(0);
fn invalid(message: impl Into<String>) -> io::Error {
@@ -239,6 +242,26 @@ impl State {
Ok(path)
}
pub fn miruro_session(
&mut self,
video: &Video,
source: &SourceDocument,
) -> io::Result<(PathBuf, PathBuf)> {
source.validate()?;
if Identity::from_key(&video.id)? != source.identity {
return Err(invalid("Source identity must match the playback episode"));
}
let contents = serde_json::to_vec(source).map_err(|e| invalid(e.to_string()))?;
if contents.len() > MAX_SOURCE_BYTES {
return Err(invalid("Playback source exceeds the 1 MiB limit"));
}
let progress_file = self.progress_file(video)?;
let source_file = progress_file.with_file_name("source.json");
atomic_write(&source_file, None, &contents)?;
register_identity(&progress_file, &video.id)?;
Ok((source_file, progress_file))
}
pub fn poll_progress(&mut self) -> io::Result<bool> {
let mut error = self.discover().err();
let count = self.streams.len().min(16);
@@ -372,6 +395,23 @@ fn validate_snapshot(video: &mut Video, require_channel: bool) -> io::Result<()>
));
}
}
if video.id.starts_with("miruro:") {
Identity::from_key(&video.id)?;
if video.title.trim().is_empty() || video.title.chars().any(char::is_control) {
return Err(invalid(
"Miruro episode title must be nonempty and contain no controls",
));
}
if video.channel.as_deref() != Some("Miruro")
|| video.channel_id.is_some()
|| video.uploader_id.is_some()
{
return Err(invalid(
"Miruro episodes must not contain YouTube channel identities",
));
}
return Ok(());
}
validate_video(video).map_err(invalid)?;
video.publication_date = date;
if let Some(handle) = &video.uploader_id
@@ -448,13 +488,13 @@ fn load_watched(document: &DocumentMut) -> io::Result<BTreeSet<String>> {
return Ok(BTreeSet::new());
};
item.as_array()
.ok_or_else(|| invalid("watched must be an array of YouTube IDs"))?
.ok_or_else(|| invalid("watched must be an array of playback identities"))?
.iter()
.map(|v| {
let id = v
.as_str()
.filter(|id| valid_video_id(id))
.ok_or_else(|| invalid("Invalid watched YouTube ID"))?;
.filter(|id| valid_identity(id))
.ok_or_else(|| invalid("Invalid watched playback identity"))?;
Ok(id.to_owned())
})
.collect()
@@ -641,8 +681,12 @@ fn private_directory(path: &Path, create: bool) -> io::Result<()> {
Ok(())
}
fn valid_identity(id: &str) -> bool {
valid_video_id(id) || Identity::from_key(id).is_ok()
}
pub(crate) fn register_identity(progress_file: &Path, id: &str) -> io::Result<()> {
if !valid_video_id(id)
if !valid_identity(id)
|| progress_file
.file_name()
.is_none_or(|name| name != "progress.jsonl")
@@ -674,7 +718,14 @@ pub(crate) fn register_identity(progress_file: &Path, id: &str) -> io::Result<()
}
#[derive(Deserialize)]
struct ProgressRecord {
#[serde(untagged)]
enum ProgressRecord {
YouTube(YouTubeProgress),
Miruro(MiruroProgress),
}
#[derive(Deserialize)]
struct YouTubeProgress {
version: u32,
event: String,
media_url: String,
@@ -683,6 +734,17 @@ struct ProgressRecord {
completion_eligible: bool,
}
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct MiruroProgress {
version: u32,
event: String,
media_identity: Identity,
position_seconds: f64,
duration_seconds: Option<f64>,
completion_eligible: bool,
}
struct ProgressStream {
path: PathBuf,
id: Option<String>,
@@ -722,11 +784,11 @@ impl ProgressStream {
let marker = self.path.with_file_name("video-id");
let mut bytes = Vec::new();
regular_file(&marker, true)?
.take(65)
.take(MAX_IDENTITY_BYTES + 1)
.read_to_end(&mut bytes)?;
let id = std::str::from_utf8(&bytes)
.ok()
.filter(|id| valid_video_id(id))
.filter(|id| id.len() as u64 <= MAX_IDENTITY_BYTES && valid_identity(id))
.ok_or_else(|| invalid("Invalid video-id session marker"))?;
let snapshot = self.path.with_file_name("video.toml");
let video = match regular_file(&snapshot, true) {
@@ -739,7 +801,12 @@ impl ProgressStream {
}
collection.videos.pop()
}
Err(e) if e.kind() == io::ErrorKind::NotFound => None,
Err(e) if e.kind() == io::ErrorKind::NotFound && valid_video_id(id) => None,
Err(e) if e.kind() == io::ErrorKind::NotFound => {
return Err(invalid(
"Miruro sessions require their bound video metadata",
));
}
Err(e) => return Err(e),
};
let seen_path = self.path.with_file_name("watching-seen.toml");
@@ -856,22 +923,48 @@ impl ProgressStream {
) -> io::Result<()> {
let record: ProgressRecord =
serde_json::from_slice(&self.partial).map_err(|e| self.record_error(e))?;
if record.version != 1 || !matches!(record.event.as_str(), "progress" | "ended" | "closed")
{
return Err(
self.record_error("expected version 1 and progress, ended, or closed event")
);
let miruro_key;
let (id, event, position, duration, eligible) = match &record {
ProgressRecord::YouTube(record) if record.version == 1 => {
let id = record
.media_url
.strip_prefix("https://www.youtube.com/watch?v=")
.filter(|id| valid_video_id(id))
.ok_or_else(|| self.record_error("expected canonical YouTube video URL"))?;
(
id,
record.event.as_str(),
record.position_seconds,
record.duration_seconds,
record.completion_eligible,
)
}
ProgressRecord::Miruro(record) if record.version == 2 => {
record
.media_identity
.validate()
.map_err(|e| self.record_error(e))?;
miruro_key = record.media_identity.key();
(
miruro_key.as_str(),
record.event.as_str(),
record.position_seconds,
record.duration_seconds,
record.completion_eligible,
)
}
_ => {
return Err(self.record_error(
"expected version 1 with media_url or version 2 with media_identity",
));
}
};
if !matches!(event, "progress" | "ended" | "closed") {
return Err(self.record_error("expected progress, ended, or closed event"));
}
let id = record
.media_url
.strip_prefix("https://www.youtube.com/watch?v=")
.filter(|id| valid_video_id(id))
.ok_or_else(|| self.record_error("expected canonical YouTube video URL"))?;
if !record.position_seconds.is_finite()
|| record.position_seconds < 0.0
|| record
.duration_seconds
.is_some_and(|d| !d.is_finite() || d <= 0.0)
if !position.is_finite()
|| position < 0.0
|| duration.is_some_and(|d| !d.is_finite() || d <= 0.0)
{
return Err(self.record_error(
"position must be finite/nonnegative and duration positive/finite or null",
@@ -880,13 +973,11 @@ impl ProgressStream {
if self.id.as_deref() != Some(id) {
return Ok(());
}
if record.position_seconds > 0.0 && !self.seen && !watched.contains(id) {
if position > 0.0 && !self.seen && !watched.contains(id) {
self.started = true;
}
if record.completion_eligible
&& record
.duration_seconds
.is_some_and(|d| record.position_seconds / d >= 0.95)
if eligible
&& duration.is_some_and(|d| position / d >= 0.95)
&& !watched.contains(id)
&& !pending.contains(id)
{
@@ -955,6 +1046,50 @@ mod tests {
)
}
fn miruro_video(episode: u32) -> Video {
Video {
id: Identity::new("opaque_Show-1".into(), episode)
.unwrap()
.key(),
title: format!("Example show · Episode {episode}"),
channel: Some("Miruro".into()),
channel_id: None,
uploader_id: None,
timestamp: Some(1_114_300_000),
publication_date: Some("2005-04-23".into()),
}
}
fn miruro_source(video: &Video) -> SourceDocument {
SourceDocument {
version: 1,
url: "https://cdn.example/video.m3u8?token=private-token".into(),
headers: [("Referer".into(), "https://player.example/".into())].into(),
identity: Identity::from_key(&video.id).unwrap(),
title: Some(video.title.clone()),
subtitles: Vec::new(),
}
}
fn miruro_record(
video: &Video,
event: &str,
eligible: bool,
position: f64,
duration: Option<f64>,
) -> String {
serde_json::json!({
"version": 2,
"event": event,
"media_identity": Identity::from_key(&video.id).unwrap(),
"position_seconds": position,
"duration_seconds": duration,
"completion_eligible": eligible,
})
.to_string()
+ "\n"
}
#[test]
fn collections_are_independent_and_preserve_metadata_and_watched_on_restart() {
let directory = TestDirectory::new();
@@ -1565,4 +1700,348 @@ mod tests {
.unwrap();
assert!(State::open(&directory.config()).is_err());
}
#[test]
fn miruro_sessions_recover_collections_and_watching_without_persisting_sources_in_state() {
let directory = TestDirectory::new();
let video = miruro_video(1);
let mut state = State::open(&directory.config()).unwrap();
for kind in [CollectionKind::WatchLater, CollectionKind::Saved] {
state.save(kind, &video).unwrap();
}
let source = miruro_source(&video);
let (source_file, report) = state.miruro_session(&video, &source).unwrap();
let source_bytes = fs::read(&source_file).unwrap();
assert_eq!(source_bytes, serde_json::to_vec(&source).unwrap());
assert!(!report.exists());
assert_eq!(
fs::read_to_string(report.with_file_name("video-id")).unwrap(),
video.id
);
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
for path in [
&source_file,
&report.with_file_name("video-id"),
&report.with_file_name("video.toml"),
] {
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
0o600
);
}
for path in [
report.parent().unwrap(),
&directory.0.join("playback-sessions"),
] {
assert_eq!(
fs::metadata(path).unwrap().permissions().mode() & 0o777,
0o700
);
}
}
assert!(!state.poll_progress().unwrap());
assert!(state.videos(CollectionKind::Watching).is_empty());
drop(state);
// A GUI can acknowledge its final snapshot after Tubular has quit.
append(
&report,
miruro_record(&video, "closed", false, 10.0, None).as_bytes(),
);
let mut state = State::open(&directory.config()).unwrap();
assert!(state.poll_progress().unwrap());
let stored = &state.videos(CollectionKind::Watching)[0];
assert_eq!(stored.id, video.id);
assert_eq!(stored.title, video.title);
assert_eq!(stored.channel, video.channel);
assert_eq!(stored.timestamp, video.timestamp);
assert_eq!(stored.publication_date, video.publication_date);
drop(state);
let mut state = State::open(&directory.config()).unwrap();
assert!(state.contains(CollectionKind::Watching, &video.id));
assert!(!state.poll_progress().unwrap());
append(
&report,
miruro_record(&video, "ended", true, 95.0, Some(100.0)).as_bytes(),
);
assert!(state.poll_progress().unwrap());
assert!(state.is_watched(&video.id));
assert!(state.videos(CollectionKind::Watching).is_empty());
drop(state);
let mut state = State::open(&directory.config()).unwrap();
assert!(state.is_watched(&video.id));
assert!(!state.poll_progress().unwrap());
for kind in [CollectionKind::WatchLater, CollectionKind::Saved] {
let stored = &state.videos(kind)[0];
assert_eq!(stored.id, video.id);
assert_eq!(stored.title, video.title);
assert_eq!(stored.channel, video.channel);
assert_eq!(stored.channel_id, None);
assert_eq!(stored.uploader_id, None);
assert_eq!(stored.timestamp, video.timestamp);
assert_eq!(stored.publication_date, video.publication_date);
}
for path in [
directory.0.join("watch-later.toml"),
directory.0.join("saved-videos.toml"),
directory.0.join("currently-watching.toml"),
directory.0.join("watch-state.toml"),
report.with_file_name("video.toml"),
] {
let contents = fs::read_to_string(path).unwrap();
assert!(!contents.contains("private-token"));
assert!(!contents.contains("cdn.example"));
}
assert_eq!(fs::read(source_file).unwrap(), source_bytes);
assert!(report.exists());
}
#[test]
fn miruro_v2_requires_matching_episode_eligible_known_duration_and_threshold() {
for event in ["progress", "ended", "closed"] {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let video = miruro_video(1);
let (_, report) = state
.miruro_session(&video, &miruro_source(&video))
.unwrap();
append(
&report,
miruro_record(&miruro_video(2), event, true, 100.0, Some(100.0)).as_bytes(),
);
assert!(!state.poll_progress().unwrap());
assert!(state.videos(CollectionKind::Watching).is_empty());
assert!(!state.is_watched(&miruro_video(2).id));
for (eligible, position, duration) in [
(true, 0.0, Some(100.0)),
(true, 94.9, Some(100.0)),
(false, 100.0, Some(100.0)),
(true, 100.0, None),
] {
append(
&report,
miruro_record(&video, event, eligible, position, duration).as_bytes(),
);
state.poll_progress().unwrap();
assert!(!state.is_watched(&video.id));
}
assert!(state.contains(CollectionKind::Watching, &video.id));
let completion = miruro_record(&video, event, true, 95.0, Some(100.0));
append(&report, completion.as_bytes());
append(&report, completion.as_bytes());
assert!(state.poll_progress().unwrap());
assert!(state.is_watched(&video.id));
assert!(state.videos(CollectionKind::Watching).is_empty());
assert!(!state.poll_progress().unwrap());
}
}
#[test]
fn miruro_v2_rejects_invalid_wire_schemas_numbers_and_namespaces() {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let video = miruro_video(1);
let (_, report) = state
.miruro_session(&video, &miruro_source(&video))
.unwrap();
let valid = miruro_record(&video, "progress", true, 100.0, Some(100.0));
for line in [
valid.replace("\"version\":2", "\"version\":1"),
valid.replace("\"version\":2", "\"version\":3"),
valid.replace("\"provider\":\"miruro\"", "\"provider\":\"youtube\""),
valid.replace("\"episode\":1", "\"episode\":0"),
valid.replace("\"episode\":1", "\"episode\":2147483648"),
valid.replace("\"position_seconds\":100.0", "\"position_seconds\":-1"),
valid.replace("\"position_seconds\":100.0", "\"position_seconds\":1e999"),
valid.replace("\"duration_seconds\":100.0", "\"duration_seconds\":0"),
valid.replace("\"duration_seconds\":100.0", "\"duration_seconds\":-1"),
valid.replace("\"duration_seconds\":100.0", "\"duration_seconds\":1e999"),
valid.replace("\"event\":\"progress\"", "\"event\":\"other\""),
valid.replacen('{', "{\"media_url\":null,", 1),
record(&self::video().id, "ended", true, 100.0)
.replace("\"version\":1", "\"version\":2"),
] {
append(&report, line.as_bytes());
assert!(state.poll_progress().is_err(), "{line}");
assert!(state.videos(CollectionKind::Watching).is_empty());
assert!(!state.is_watched(&video.id));
assert!(!state.poll_progress().unwrap());
}
}
#[test]
fn miruro_partial_reports_and_bad_siblings_recover_after_restart() {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let video = miruro_video(1);
let (source, report) = state
.miruro_session(&video, &miruro_source(&video))
.unwrap();
let completion = miruro_record(&video, "closed", true, 95.0, Some(100.0));
append(&report, &completion.as_bytes()[..completion.len() - 1]);
assert!(!state.poll_progress().unwrap());
drop(state);
let mut state = State::open(&directory.config()).unwrap();
assert!(!state.poll_progress().unwrap());
let other = miruro_video(2);
let (_, bad) = state
.miruro_session(&other, &miruro_source(&other))
.unwrap();
append(&bad, b"not json\n");
append(&report, b"\n");
assert!(state.poll_progress().is_err());
assert!(state.is_watched(&video.id));
assert!(!state.is_watched(&other.id));
assert!(state.poll_progress().unwrap());
assert!(!state.poll_progress().unwrap());
drop(state);
assert!(source.exists());
assert!(report.exists());
let state = State::open(&directory.config()).unwrap();
assert!(state.is_watched(&video.id));
}
#[test]
fn miruro_state_rejects_catalog_rows_bad_titles_and_youtube_metadata() {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
for field in [
"series",
"noncanonical",
"empty-title",
"control-title",
"channel",
"channel-id",
"uploader",
] {
let mut video = miruro_video(1);
match field {
"series" => video.id = "miruro-series:opaque_Show-1".into(),
"noncanonical" => video.id = "miruro:opaque_Show-1:01".into(),
"empty-title" => video.title = " ".into(),
"control-title" => video.title.push('\n'),
"channel" => video.channel = Some("YouTube".into()),
"channel-id" => video.channel_id = self::video().channel_id,
"uploader" => video.uploader_id = Some("@jawed".into()),
_ => unreachable!(),
}
for kind in [
CollectionKind::WatchLater,
CollectionKind::Saved,
CollectionKind::Watching,
] {
assert!(state.save(kind, &video).is_err(), "{field}");
}
assert!(state.progress_file(&video).is_err(), "{field}");
}
for id in [
"miruro-series:opaque_Show-1",
"miruro:opaque_Show-1:01",
"other:opaque_Show-1:1",
] {
let mut watched = DocumentMut::new();
watched["version"] = value(1);
let mut ids = Array::new();
ids.push(id);
watched["watched"] = value(ids);
assert!(load_watched(&watched).is_err());
}
}
#[test]
fn miruro_source_sessions_validate_binding_size_and_maximum_identity_marker() {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let mut video = miruro_video(i32::MAX as u32);
video.id = Identity::new("a".repeat(512), i32::MAX as u32)
.unwrap()
.key();
let mut source = miruro_source(&video);
source.identity.episode -= 1;
assert!(state.miruro_session(&video, &source).is_err());
assert!(state.streams.is_empty());
source.identity = Identity::from_key(&video.id).unwrap();
source.title = Some("x".repeat(MAX_SOURCE_BYTES));
assert!(state.miruro_session(&video, &source).is_err());
assert!(state.streams.is_empty());
source.title = None;
let (_, report) = state.miruro_session(&video, &source).unwrap();
append(
&report,
miruro_record(&video, "ended", true, 95.0, Some(100.0)).as_bytes(),
);
drop(state);
let mut state = State::open(&directory.config()).unwrap();
assert!(state.poll_progress().unwrap());
assert!(state.is_watched(&video.id));
drop(state);
let state = State::open(&directory.config()).unwrap();
assert!(state.is_watched(&video.id));
}
#[test]
fn miruro_recovery_rejects_missing_snapshots_and_overlong_identity_markers() {
for missing_snapshot in [true, false] {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let video = miruro_video(1);
let (_, report) = state
.miruro_session(&video, &miruro_source(&video))
.unwrap();
if missing_snapshot {
fs::remove_file(report.with_file_name("video.toml")).unwrap();
} else {
append(
&report.with_file_name("video-id"),
&vec![b'x'; MAX_IDENTITY_BYTES as usize],
);
}
append(
&report,
miruro_record(&video, "ended", true, 100.0, Some(100.0)).as_bytes(),
);
drop(state);
let mut state = State::open(&directory.config()).unwrap();
assert!(state.poll_progress().is_err());
assert!(!state.is_watched(&video.id));
assert!(state.videos(CollectionKind::Watching).is_empty());
}
}
#[test]
fn youtube_v1_reports_keep_canonical_urls_and_unknown_field_tolerance() {
let directory = TestDirectory::new();
let mut state = State::open(&directory.config()).unwrap();
let video = video();
let report = state.progress_file(&video).unwrap();
register_identity(&report, &video.id).unwrap();
let valid = record(&video.id, "ended", true, 95.0);
for url in [
format!("https://youtu.be/{}", video.id),
format!("https://www.youtube.com/watch?v={}&t=1", video.id),
format!("http://www.youtube.com/watch?v={}", video.id),
] {
append(
&report,
valid
.replace(
&format!("https://www.youtube.com/watch?v={}", video.id),
&url,
)
.as_bytes(),
);
assert!(state.poll_progress().is_err());
assert!(!state.is_watched(&video.id));
assert!(state.videos(CollectionKind::Watching).is_empty());
}
append(
&report,
valid
.replacen('{', "{\"legacy_extension\":true,", 1)
.as_bytes(),
);
assert!(state.poll_progress().unwrap());
assert!(state.is_watched(&video.id));
}
}
src/youtube.rs
+55 -2
View File
@@ -819,14 +819,67 @@ impl PlayerLaunch {
));
}
crate::video_state::register_identity(progress_file, id)?;
let mut player = isolated_command(player)
let mut command = isolated_command(player);
command
.args([
"--new-instance",
"--url",
&format!("https://www.youtube.com/watch?v={id}"),
])
.arg("--progress-file")
.arg(progress_file)
.arg(progress_file);
Self::launch(command)
}
/// Session files are owned and retained by video_state, not by this launcher.
pub fn start_source(
source_file: &Path,
player: &OsStr,
progress_file: &Path,
) -> io::Result<Self> {
if !std::fs::symlink_metadata(source_file)?.is_file() {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"Source document must be a regular file",
));
}
let source_file = std::fs::canonicalize(source_file)?;
let parent = std::fs::canonicalize(progress_file.parent().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidInput, "Missing progress directory")
})?)?;
if source_file.parent() != Some(parent.as_path())
|| progress_file
.file_name()
.is_none_or(|name| name != "progress.jsonl")
{
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"Source and progress must share their private session directory",
));
}
let progress_file = parent.join("progress.jsonl");
match std::fs::symlink_metadata(&progress_file) {
Err(error) if error.kind() == io::ErrorKind::NotFound => {}
Err(error) => return Err(error),
Ok(_) => {
return Err(io::Error::new(
io::ErrorKind::AlreadyExists,
"Progress destination must not exist",
));
}
}
let mut command = isolated_command(player);
command
.arg("--new-instance")
.arg("--source-file")
.arg(source_file)
.arg("--progress-file")
.arg(progress_file);
Self::launch(command)
}
fn launch(mut command: Command) -> io::Result<Self> {
let mut player = command
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::piped())