refactor: a tiny bit of everything

[luxass]

🔗 Linked issue

resolves #152
resolves #149
resolves #151
resolves #150
resolves #340
resolves #331
resolves #427
resolves #428
resolves #429
resolves #430
resolves #432
resolves #335

closes #419
closes #401
closes #380

📚 Description

This PR implements a "tiny" bit of everything.

The main features is the comparison, file-explorer and much much more website related.

TODO:

• Make lockfile hashing work better
• Implement proper fix for the hash header strip (f6962e4)
• Make File Explorer reusable as component
• Implement Diffing for File Explorer
• Implement proper use of "UCDStore" on website
• Update Unicode Utils to support the stripping of header strip (Not inside here, but quite related).
• Add a small label to old unicode versions in file explorer (It should tell the mapped version). So 4.0-Update1 is 4.0.1
• Add Playgrounds for ucd-store-v2 (create better playground examples for ucd-store-v2 #338)
• Make sure Comparison returns following structure (See Compare Structure Response)
• Setup E2E Tests for Store (This can maybe be removed, we'll see.) (add e2e test suite for ucd-store-v2 #337) (Add end-to-end test suite for ucd-store #322)
• Move the UCD Store V2 into UCD Store (merge ucd-store-v2 into ucd-store #427)
• Implement Lockfile Command (add lockfile command #428)
• Implement Lockfile Subcommands (add lockfile hash subcommand (cli) #429) (add lockfile info subcommand (cli) #430)

Note

There is probably a lot more, that should be implemented! But let's see where this ends.

Compare Structure Response

{
  from: "15.1.0",
  to: "16.0.0",
  added: ["NewFile.txt"],
  removed: ["OldFile.txt"], 
  modified: ["UnicodeData.txt"],
  unchanged: 95,
  changes: [
    { 
      file: "UnicodeData.txt",
      from: { size: 1258291, modified: "2023-09-15" },
      to: { size: 1364982, modified: "2024-09-13" }
    }
  ]
}

Summary by CodeRabbit

• New Features

  • Added comprehensive file explorer with directory navigation, search filtering, pattern matching, and sort options.
  • Introduced version-specific feature pages: normalization preview, grapheme visualizer, font/glyph view, bidi & line break, and search within versions.
  • Added new API endpoints for blocks, characters, and properties data retrieval.
  • Enhanced sidebar with dynamic version list and tools section.

• Improvements

  • Replaced deprecated search endpoint with integrated file explorer search.
  • Added large file preview warnings and non-renderable file handling.
  • Enhanced UI with breadcrumb navigation and improved file viewer layout.

• Chores

  • Updated package manager to v10.26.2.
  • Refreshed dependencies across framework packages.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2767ef5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

This PR is a comprehensive refactor that updates the pnpm package manager across the monorepo, restructures API routes with enhanced file listing query support, removes the search endpoint, completely refactors the web app with TanStack Start integration, adds new API endpoints for Unicode blocks/characters/properties, introduces new schemas, and implements version comparison functionality in the UCD Store.

Changes

Cohort / File(s)	Summary
Package Manager Version Bumps package.json, apps/*/package.json, packages/*/package.json, tooling/*/package.json, pnpm-workspace.yaml, vscode/package.json	Updated pnpm from 10.26.1 to 10.26.2 across all package.json manifests; updated dependency catalogs in pnpm-workspace.yaml with bumped versions for TanStack packages, MSW, OpenAI/AI, ESLint, and other tools.
API File Listing Route Refactor apps/api/src/routes/v1_files/$wildcard.ts, apps/api/test/routes/v1_files/$wildcard.test.ts	Added comprehensive query parameter support (query, type, sort, order) to directory listings; implemented filtering by prefix, filtering by type (files/directories), and sorting logic with directories prioritized first; updated response headers with new stat header constants; extended test coverage with prefix-search scenarios and adjusted expected ordering.
Search Endpoint Removal apps/api/src/routes/v1_files/router.ts, apps/api/src/routes/v1_files/search.ts, apps/api/test/routes/v1_files/search.test.ts	Removed registerSearchRoute invocation from router; deleted entire search route implementation including HTML parsing, prefix filtering, and external unicode.org fetch logic; removed corresponding test suite.
Header Constants Rename packages/env/src/constants.ts, packages/env/src/index.ts, apps/api/test/routes/v1_files/$wildcard.test.ts, packages/cli/src/cmd/files/info.ts, packages/cli/test/cmd/files/info.test.ts	Replaced single UCD_FILE_STAT_TYPE_HEADER with set of new headers: UCD_STAT_TYPE_HEADER, UCD_STAT_SIZE_HEADER, UCD_STAT_CHILDREN_HEADER, UCD_STAT_CHILDREN_FILES_HEADER, UCD_STAT_CHILDREN_DIRS_HEADER; updated all imports and usages.
API Route Constants & Tags apps/api/src/constants.ts, apps/api/src/openapi.ts, apps/api/src/worker.ts	Added three new router base path constants (V1_CHARACTERS, V1_PROPERTIES, V1_BLOCKS); added corresponding OpenAPI tags; registered new routers in worker.
New Blocks API Route apps/api/src/routes/v1_blocks/$block.ts, apps/api/src/routes/v1_blocks/list.ts, apps/api/src/routes/v1_blocks/router.ts	Introduced new GET endpoints for listing blocks (/list) and fetching individual block details (/$block) with OpenAPI integration, caching, schema validation; placeholder implementations indicate pending completion.
New Characters API Route apps/api/src/routes/v1_characters/$codepoint.ts, apps/api/src/routes/v1_characters/router.ts	Added new GET /{codepoint} endpoint with support for multiple codepoint formats (U+XXXX, 0xXXXX, decimal, character); includes caching, validation, and parseCodepoint helper; placeholder implementation.
New Properties API Route apps/api/src/routes/v1_properties/$property.ts, apps/api/src/routes/v1_properties/router.ts	Introduced new GET /{property} endpoint with query parameters for filtering (value), formatting (json/ranges/list), and pagination (limit, offset); caching and schema validation included; placeholder implementation.
File Explorer Components Rewrite apps/web/src/components/file-explorer/entry-list.tsx, apps/web/src/components/file-explorer/explorer-entry.tsx, apps/web/src/components/file-explorer/explorer-toolbar.tsx, apps/web/src/components/file-explorer/file-explorer.tsx	Replaced component-based API with internal state management; removed DirectoryItem/FileItem components; introduced EntryList for fetching and rendering files; refactored ExplorerToolbar to manage filtering internally via URL parameters; removed FileExplorer wrapper.
File Viewer & Non-Renderable Updates apps/web/src/components/file-explorer/file-viewer.tsx, apps/web/src/components/file-explorer/non-renderable-file.tsx, apps/web/src/components/file-explorer/large-file-warning.tsx, apps/web/src/components/file-explorer/file.tsx	Removed back-navigation UI from headers; introduced FileViewerSkeleton component; added LargeFileWarning component for files exceeding 1 MB; removed FileItem component entirely.
File Explorer Index & Routes apps/web/src/components/file-explorer/index.ts, apps/web/src/routes/explorer/files.$.tsx, apps/web/src/routes/explorer/index.tsx	Removed all barrel exports; deleted old explorer routes; introduced new file-explorer routes at /file-explorer/$ and /file-explorer/v/$ with advanced query handling, suspense boundaries, and redirects.
File Explorer Route Implementation apps/web/src/routes/file-explorer/route.tsx, apps/web/src/routes/file-explorer/$.tsx, apps/web/src/routes/file-explorer/v.$.tsx	Added layout route with breadcrumb support; implemented directory listing route with filtering/sorting/pagination; implemented file viewer route with size validation and renderability checking; all routes include beforeLoad hooks, loaders, and notFoundComponent boundaries.
Server-Side File Functions apps/web/src/functions/files.ts, apps/web/src/functions/versions.ts	Replaced client-side API consumption with server functions using createServerFn; fetchFiles now handles large files (>1 MB) by returning metadata instead of content; getFileHeadInfo performs HEAD requests for file metadata; filesQueryOptions provides React Query integration.
Web App Dependencies & Config apps/web/package.json, apps/web/tsconfig.json, apps/web/vite.config.ts, apps/web/components.json, apps/web/wrangler.jsonc	Added new dependencies (@tanstack/zod-adapter, @ucdjs-internal/shared, @ucdjs/env); extended tsconfig with base configuration; configured Vite with server entry and disabled prerendering; updated Tailwind CSS config path; added UCDJS_API_BASE_URL environment variable.
Web App Global Styling apps/web/src/globals.css	Updated color tokens (primary, charts, sidebar) to new palette with adjusted luminance and hues; increased border radius from 0.45rem to 0.625rem; added sidebar-ring token in light mode.
Sidebar Component Refactor apps/web/src/components/layout/sidebar/app-sidebar.tsx, apps/web/src/components/layout/sidebar/versions-list.tsx, apps/web/src/components/layout/sidebar/versions-list.tsx, apps/web/src/components/app-sidebar.tsx	Moved AppSidebar to layout subdirectory; introduced internal state management with Tools and Version sections; added VersionsList and VersionsCardList components with status badges (Latest, Mature, etc.); replaced direct data fetching with useSuspenseQuery.
App & Root Route Updates apps/web/src/routes/__root.tsx, apps/web/src/routes/index.tsx, apps/web/src/routes/v/$version/route.tsx	Added AppRouterContext with latestUnicodeVersion and apiBaseUrl; implemented async loader for versions prefetching; added notFoundComponent to root route; updated version routes with notFound handling and VersionNotFoundBoundary.
Version Browsing Routes apps/web/src/routes/v/index.tsx, apps/web/src/routes/v/$version/index.tsx	Added redirect from /v to latest version; added Search Card to version overview linking to /v/$version/search.
Version-Specific Feature Routes apps/web/src/routes/v/$version/bidi-linebreak.tsx, apps/web/src/routes/v/$version/font-glyph-view.tsx, apps/web/src/routes/v/$version/grapheme-visualizer.tsx, apps/web/src/routes/v/$version/normalization-preview.tsx, apps/web/src/routes/v/$version/search.tsx	Introduced five new feature routes under version context with breadcrumb navigation, descriptive UI, and back-links; all are placeholder implementations with pending feature development.
Global Search & Tools Routes apps/web/src/routes/search.tsx, apps/web/src/routes/compare.tsx, apps/web/src/routes/codepoint-inspector.tsx	Added standalone routes for global search, version comparison, and codepoint inspection; all are placeholder implementations with mock data and UI scaffolding.
Documentation Routes apps/web/src/routes/(docs)/api/search.ts, apps/web/src/routes/(docs)/docs/index.tsx, apps/web/src/routes/(docs)/docs/$.tsx	Reorganized documentation routes under (docs) route group; added notFoundComponent boundaries with DocsNotFoundBoundary.
Router & Server Infrastructure apps/web/src/routeTree.gen.ts, apps/web/src/router.tsx, apps/web/src/server.ts	Updated route tree with new route definitions and file route mappings; added zod schema registry patch for HMR; introduced ServerRequestContext and apiBaseUrl context propagation; added server entry with request context handling.
UI Component Updates apps/web/src/components/ui/*, apps/web/src/components/nav.tsx, apps/web/src/components/not-found.tsx	Removed NavItem component; replaced icon imports from lucide-react (ChevronRightIcon → ChevronRight, etc.); updated data attribute selectors (data-[inset] → data-inset); added NotFoundLayout and four page-specific 404 components (AppNotFound, DocsNotFound, ExplorerNotFound, VersionNotFound); removed "use client" directives from several UI components.
New UI Components apps/web/src/components/ucd-logo.tsx, apps/web/src/components/u-version.tsx, apps/web/src/components/home/versions-list.tsx	Added UcdLogo SVG component; introduced UVersion badge component for rendering compact version labels; created VersionsCardList and VersionsCardListSkeleton for home page version grid display.
Type Definitions apps/web/src/types/file-explorer.ts	Added ViewMode type alias ("list" | "cards").
UCD Store v2 Comparison packages/ucd-store-v2/src/operations/compare.ts, packages/ucd-store-v2/src/store.ts, packages/ucd-store-v2/src/types.ts, packages/ucd-store-v2/test/operations/compare.test.ts	Introduced new compare operation for version comparison with file hash support; added CompareOptions and VersionComparison types; exposed compare method on UCDStore; added VersionConflictStrategy type; included comprehensive test suite.
Schema Additions packages/schemas/src/unicode.ts, packages/schemas/src/index.ts	Added four new Unicode data schemas: UnicodeCharacterSchema, UnicodePropertyResponseSchema, UnicodeBlockSchema, UnicodeBlockListSchema with corresponding TypeScript types; all include metadata and descriptive fields.
Security Tests packages/fs-bridge/test/security/http/encoded-attacks.test.ts, packages/fs-bridge/test/security/http/excessive-encoding.test.ts, packages/fs-bridge/test/security/node/encoded-attacks.test.ts, packages/fs-bridge/test/security/node/excessive-encoding.test.ts	Tightened error assertions to expect specific error messages ("Path traversal", "Failed to decode path"); refactored HTTP tests with basePath constants; normalized test title casing.
Miscellaneous Updates packages/fs-bridge/src/bridges/node.ts, packages/fs-bridge/package.json, packages/test-utils/src/mock-store/handlers/well-known.ts, apps/api/src/routes/v1_schemas/router.ts	Added refactor TODO comment in Node bridge; updated package manager version; added non-null assertion for file version access in mock handler; reformatted imports in schemas router.

Sequence Diagram(s)

sequenceDiagram
    actor Client
    participant FileExplorerRoute as File Explorer<br/>Route
    participant EntryList as EntryList<br/>Component
    participant Query as React Query
    participant Server as Server<br/>Function
    participant API as API<br/>Endpoint

    Client->>FileExplorerRoute: Navigate to /file-explorer/$path
    FileExplorerRoute->>FileExplorerRoute: validateSearch, beforeLoad,<br/>resolve path metadata
    FileExplorerRoute->>Server: getFileHeadInfo(path)
    Server->>API: HEAD /api/v1/files/$path
    API-->>Server: headers: type, size, counts
    Server-->>FileExplorerRoute: statType, size, counts
    
    alt Directory
        FileExplorerRoute->>EntryList: Render with currentPath
        EntryList->>Query: useSuspenseQuery(filesQueryOptions)
        Query->>Server: fetchFiles(path, query, sort, order, type)
        Server->>API: GET /api/v1/files/$path?query=...&sort=...
        API-->>Server: JSON array of FileEntry[]
        Server-->>Query: FilesResponse {type: "directory", files}
        Query-->>EntryList: files data
        EntryList->>Client: Render file list with filters/sort
    else File
        FileExplorerRoute->>FileExplorerRoute: Check size & renderability
        alt Too Large (>1MB)
            FileExplorerRoute->>Client: Render LargeFileWarning
        else Non-Renderable
            FileExplorerRoute->>Client: Render NonRenderableFile
        else Renderable
            FileExplorerRoute->>Query: useSuspenseQuery(filesQueryOptions)
            Query->>Server: fetchFiles(path)
            Server->>API: GET /api/v1/files/$path
            API-->>Server: FilesResponse {type: "file", content}
            Server-->>Query: file content
            Query->>Client: Render FileViewer with content
        end
    end

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• feat(env): add environment package #94: Introduces the @ucdjs/env package that this PR now imports and consumes for header constants and environment configuration.
• feat(shared): migrate utilities to @ucdjs-internal/shared #311: Migrates internal utility dependencies from @ucdjs/shared to @ucdjs-internal/shared, which this PR adopts across the codebase.
• refactor: switch to tanstack start for frontend #400: Overlaps significantly on the web app refactoring with TanStack Start integration, route tree changes, and major component restructuring.

Poem

🐰 A warren of routes now flourish and bloom,
Files filter and sort without gloom,
Search endpoints retire, new features take flight,
The web app springs forward with TanStack's might,
Schemas and stores unite in the fold,
A monorepo refactored, magnificent and bold! ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.37% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.
Title check	❓ Inconclusive	The PR title 'refactor: a tiny bit of everything' is vague and does not clearly summarize the main changes. The changeset includes significant additions across multiple areas: new API routes for Unicode blocks/characters/properties, file explorer refactoring, version management UI, server-side routing infrastructure, and numerous dependency updates. The title uses a generic phrase ('a tiny bit of everything') that conveys minimal meaningful information about what was actually changed.	Replace with a more specific title that highlights the primary changes, such as 'refactor: add Unicode API routes and file explorer UI' or 'feat: refactor file explorer and add Unicode blocks/characters/properties API routes'. This would better communicate the scope to reviewers scanning PR history.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🌏 Preview Deployments

Application	Status	Preview URL
API	⏳ In Progress	N/A
Website	⏳ In Progress	N/A

Built from commit: 2767ef5ee6600e9b7f4b56895ed5b04d3793553d

---

_{🤖 This comment will be updated automatically when you push new commits to this PR.}

[github-actions]

📋 OpenAPI Schema Analysis

Summary

API Endpoints

Change Type	Count	Details
🟢 Added	4	New API endpoints
🔴 Removed	1	⚠️ Breaking
🟡 Modified	0	Endpoints with method changes
📊 Method Changes	+4 / -1	Added/Removed HTTP methods

Schema Components

Change Type	Count	Details
🟢 Added	6	New schema components
🔴 Removed	2	⚠️ Breaking
🟡 Modified	2	Changed schema definitions

Overall Status

Status	Result
⚠️ Breaking Changes	Yes - This PR contains breaking changes

Detailed Changes

🟢 Added Endpoints (4)

Path	Methods	Description
/api/v1/characters/{version}/{codepoint}	GET	No description
/api/v1/properties/{version}/{property}	GET	No description
/api/v1/blocks/{version}	GET	No description
/api/v1/blocks/{version}/{block}	GET	No description

🔴 Removed Endpoints (1) ⚠️ Breaking

Path	Methods	Description
/api/v1/files/search	GET	No description

📋 Schema Components Changes

Added Schemas (6):

• UnicodeFileTreeNode
• UnicodeFileTree
• UnicodeCharacter
• UnicodePropertyResponse
• UnicodeBlock
• UnicodeBlockList

Removed Schemas (2): ⚠️

• UnicodeTreeNode
• UnicodeTree

Modified Schemas (2):

• UnicodeVersionDetails
• FileEntryList

Important

⚠️ Breaking Changes Detected

This PR contains breaking changes that require maintainer approval.
Maintainers: Add the approve-breaking-openapi label to approve these changes.

---

_{🤖 This comment is automatically updated when you push new commits.}