Fix multiple race conditions in TMModel #109

[udlose]

Summary

Fixes #109

This PR hardens TMModel's lifecycle and threading behavior under its intended usage (UI-thread document edits + background tokenization), addressing races, deadlock vectors, and resource cleanup issues without changing the public API surface

Fixes / changes included

Lifecycle + disposal correctness

• Added atomic dispose guards using Interlocked.CompareExchange to prevent double-dispose / double-stop behavior under concurrent calls
• Made tokenizer thread cleanup non-blocking by deferring disposal until the worker task completes (via continuation), avoiding UI-thread stalls during shutdown
• Ensured the tokenizer worker's resources (CancellationTokenSource, AutoResetEvent, and worker Task) are cancelled/disposed safely and exactly once

Tokenizer thread state machine hardening

• Introduced a dedicated lifecycle lock inside TokenizerThread to guard start/stop/signal/dispose transitions and eliminate check-then-act races
• Worker loop exits promptly if the active thread instance has been replaced by SetGrammar/Stop (thread identity check under the model lock)
• Capture the current grammar epoch and tokenizer thread when building a token-change event, and re-validate them before delivering callbacks; if SetGrammar advanced the epoch or swapped the thread, drop the event to avoid emitting token-change notifications under a different grammar context (stale publish)

Model state synchronization (race fixes)

• Synchronized reads of ModelLine.State used for tokenization by capturing the starting state under _lock before calling into the tokenizer
• Published tokenization results (ModelLine.Tokens, ModelLine.IsInvalid, and end-state propagation) under _lock with epoch/thread/tokenizer identity checks so stale work is discarded and re-queued safely

Lock ordering + deadlock prevention

• Standardized lock ordering as _lock → listeners and documented it in code comments
• Avoided invoking listeners under lock(listeners) by snapshotting the listener list under the lock and invoking callbacks outside the lock
• Adjusted listener add/remove paths to avoid lock-order inversion with Stop() and to prevent adding listeners to a disposed model

ForceTokenization / invalidation robustness

• ForceTokenization(start,end) clamps indices under _lock and ensures grammar/thread checks are coherent with the captured thread instance (avoids TOCTOU races)
• InvalidateLineRange clamps indices under _lock and signals the worker thread safely after enqueueing invalid work

Notes / non-goals

• This PR intentionally avoids behavioral changes to tokenization semantics and avoids allocations on hot paths (no defensive token copies)

---

• ITMModel.cs: Updates interface to inherit IDisposable, clarifying resource management contract.
• TMModelTests.cs: Adds tests for Dispose behavior, listener removal, and grammar flip deadlock prevention.

[danipen]

Bunch of changes! The PR is well-engineered and addresses real race conditions without changing tokenization semantics. LGTM

[danipen]

Tested with AvaloniaEdit's current master (AvaloniaUI/AvaloniaEdit@7fa5d88). No issues found.

[danipen]

Merged! Thanks for your contribution.