perf(tokenizer): classify char once per token and use ident-start table bit by bartveneman · Pull Request #302 · projectwallace/css-parser

bartveneman · 2026-10-14T11:57:53Z

Summary

Adds a CHAR_IDENT_START bit (ASCII letters and _) to char_types; is_ident_start is now a single table lookup, which benefits all of its callers. The now-unused is_alpha is removed.
next_token_fast reads char_types once per token into ch_type and reuses it for the whitespace, ident-start and digit checks instead of looking the character up again.

No behaviour change intended. The hyphen/dot/plus lookahead blocks and the consume_* functions are untouched.

Tests proving nothing broke

New tests in src/tokenize.test.ts → describe('first-character classification'):

is_ident_start matches letters, underscore and non-ASCII for every code unit: checks all 65,536 code units against an independent oracle.
a lone character becomes IDENT, WHITESPACE or NUMBER exactly when the oracle says so: tokenizes every ASCII/Latin-1 code unit plus a spread of BMP and surrogate code units and checks the token type and span against the oracle.
skip_whitespace=true lands on the same token as an explicit whitespace token, identifiers may start with underscore, a letter or a non-ASCII character, digits start numbers and are not identifiers.

Verification:

The new tests pass on the pre-change code (main behaviour) and on this branch, so they pin existing behaviour rather than the new implementation.
Mutation checks on this branch: dropping _ from CHAR_IDENT_START fails 5 tests, removing the ch >= 128 ident case in next_token_fast fails 2, and disabling the digit branch fails 10.
The existing tokenizer and parser suites (1433 tests overall) also pass, and tsc --noEmit, lint and knip are clean.

Not benchmarked yet; meant to be benchmarked independently of the other open PRs.

🤖 Generated with Claude Code

…le bit Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

github-actions · 2026-10-14T11:58:36Z

⚠️ Package Size Increase

📦 Package	📏 Base Size	📏 Source Size	📈 Size Change
@projectwallace/css-parser	46.1 kB	46.1 kB	+8 B

pkg-pr-new · 2026-10-14T11:58:46Z

Open in StackBlitz

npm i https://pkg.pr.new/@projectwallace/css-parser@a5c743f

commit: a5c743f

…pendent oracle Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

bartveneman · 2026-10-15T20:32:27Z

before:

┌─────────┬─────────────────────────────────────┬────────────────┬─────────────┬────────────────┐
│ (index) │ Benchmark                           │ mode (ops/sec) │ mode (µs)   │ mean (ops/sec) │
├─────────┼─────────────────────────────────────┼────────────────┼─────────────┼────────────────┤
│ 0       │ 'tokenize|Nerdy.dev'                │ '681'          │ '1467.736'  │ '676'          │
│ 1       │ 'parse|Nerdy.dev'                   │ '543'          │ '1840.345'  │ '536'          │
│ 2       │ 'parse (no sub-parsing)|Nerdy.dev'  │ '1198'         │ '834.407'   │ '1181'         │
│ 3       │ 'tokenize|CSS Tricks'               │ '75'           │ '13324.259' │ '75'           │
│ 4       │ 'parse|CSS Tricks'                  │ '48'           │ '21016.999' │ '47'           │
│ 5       │ 'parse (no sub-parsing)|CSS Tricks' │ '161'          │ '6218.423'  │ '160'          │
│ 6       │ 'tokenize|Bootstrap'                │ '432'          │ '2315.883'  │ '429'          │
│ 7       │ 'parse|Bootstrap'                   │ '302'          │ '3315.934'  │ '299'          │
│ 8       │ 'parse (no sub-parsing)|Bootstrap'  │ '595'          │ '1680.778'  │ '591'          │
│ 9       │ 'tokenize|Tailwind'                 │ '35'           │ '28525.508' │ '35'           │
│ 10      │ 'parse|Tailwind'                    │ '24'           │ '42072.361' │ '24'           │
│ 11      │ 'parse (no sub-parsing)|Tailwind'   │ '53'           │ '18843.379' │ '52'           │
│ 12      │ 'parse_selector (simple)'           │ '2455313'      │ '0.407'     │ '1988567'      │
│ 13      │ 'parse_selector (complex)'          │ '822450'       │ '1.216'     │ '799445'       │
│ 14      │ 'parse_selector_list (simple)'      │ '1082263'      │ '0.924'     │ '1029310'      │
│ 15      │ 'parse_selector_list (complex)'     │ '345859'       │ '2.891'     │ '318000'       │
│ 16      │ 'parse_atrule_prelude (media)'      │ '776699'       │ '1.287'     │ '751812'       │
│ 17      │ 'parse_atrule_prelude (supports)'   │ '1047669'      │ '0.954'     │ '993282'       │
│ 18      │ 'parse_declaration'                 │ '776699'       │ '1.287'     │ '751667'       │
│ 19      │ 'parse_value (simple)'              │ '3014318'      │ '0.332'     │ '2384241'      │
│ 20      │ 'parse_value (complex)'             │ '1148594'      │ '0.871'     │ '1113214'      │
│ 21      │ 'parse_dimension'                   │ '12025011'     │ '0.083'     │ '17186763'     │
│ 22      │ 'walk (node.type)|Nerdy.dev'        │ '2414'         │ '414.293'   │ '2409'         │
│ 23      │ 'walk (node.type_name)|Nerdy.dev'   │ '2277'         │ '439.134'   │ '2244'         │
│ 24      │ 'traverse|Nerdy.dev'                │ '3380'         │ '295.875'   │ '3347'         │
│ 25      │ 'walk (node.type)|CSS Tricks'       │ '270'          │ '3701.495'  │ '270'          │
│ 26      │ 'walk (node.type_name)|CSS Tricks'  │ '254'          │ '3940.183'  │ '250'          │
│ 27      │ 'traverse|CSS Tricks'               │ '377'          │ '2653.554'  │ '372'          │
│ 28      │ 'walk (node.type)|Bootstrap'        │ '1274'         │ '784.834'   │ '1265'         │
│ 29      │ 'walk (node.type_name)|Bootstrap'   │ '1167'         │ '856.989'   │ '1164'         │
│ 30      │ 'traverse|Bootstrap'                │ '1781'         │ '561.452'   │ '1761'         │
│ 31      │ 'walk (node.type)|Tailwind'         │ '89'           │ '11176.400' │ '90'           │
│ 32      │ 'walk (node.type_name)|Tailwind'    │ '84'           │ '11949.126' │ '83'           │
│ 33      │ 'traverse|Tailwind'                 │ '123'          │ '8146.113'  │ '125'          │
└─────────┴─────────────────────────────────────┴────────────────┴─────────────┴────────────────┘

── Parse+Walk memory ──

┌─────────┬──────────────┬───────────┬───────────┐
│ (index) │ File         │ Size      │ Memory    │
├─────────┼──────────────┼───────────┼───────────┤
│ 0       │ 'Nerdy.dev'  │ '111 KB'  │ '3.5 MB'  │
│ 1       │ 'CSS Tricks' │ '1308 KB' │ '37.9 MB' │
│ 2       │ 'Bootstrap'  │ '227 KB'  │ '6.9 MB'  │
│ 3       │ 'Tailwind'   │ '2865 KB' │ '86.2 MB' │
└─────────┴──────────────┴───────────┴───────────┘

bartveneman · 2026-10-15T20:34:17Z

after

┌─────────┬─────────────────────────────────────┬────────────────┬─────────────┬────────────────┐
│ (index) │ Benchmark                           │ mode (ops/sec) │ mode (µs)   │ mean (ops/sec) │
├─────────┼─────────────────────────────────────┼────────────────┼─────────────┼────────────────┤
│ 0       │ 'tokenize|Nerdy.dev'                │ '688'          │ '1453.588'  │ '682'          │
│ 1       │ 'parse|Nerdy.dev'                   │ '547'          │ '1827.428'  │ '542'          │
│ 2       │ 'parse (no sub-parsing)|Nerdy.dev'  │ '1194'         │ '837.237'   │ '1190'         │
│ 3       │ 'tokenize|CSS Tricks'               │ '77'           │ '13038.045' │ '76'           │
│ 4       │ 'parse|CSS Tricks'                  │ '49'           │ '20604.146' │ '48'           │
│ 5       │ 'parse (no sub-parsing)|CSS Tricks' │ '161'          │ '6211.887'  │ '160'          │
│ 6       │ 'tokenize|Bootstrap'                │ '448'          │ '2233.774'  │ '443'          │
│ 7       │ 'parse|Bootstrap'                   │ '308'          │ '3245.613'  │ '304'          │
│ 8       │ 'parse (no sub-parsing)|Bootstrap'  │ '599'          │ '1668.365'  │ '595'          │
│ 9       │ 'tokenize|Tailwind'                 │ '36'           │ '27859.530' │ '36'           │
│ 10      │ 'parse|Tailwind'                    │ '24'           │ '42238.324' │ '24'           │
│ 11      │ 'parse (no sub-parsing)|Tailwind'   │ '53'           │ '18719.518' │ '52'           │
│ 12      │ 'parse_selector (simple)'           │ '2454771'      │ '0.407'     │ '2015070'      │
│ 13      │ 'parse_selector (complex)'          │ '852304'       │ '1.173'     │ '822019'       │
│ 14      │ 'parse_selector_list (simple)'      │ '1089372'      │ '0.918'     │ '1030030'      │
│ 15      │ 'parse_selector_list (complex)'     │ '340347'       │ '2.938'     │ '321198'       │
│ 16      │ 'parse_atrule_prelude (media)'      │ '776699'       │ '1.287'     │ '754475'       │
│ 17      │ 'parse_atrule_prelude (supports)'   │ '1041851'      │ '0.960'     │ '989856'       │
│ 18      │ 'parse_declaration'                 │ '822375'       │ '1.216'     │ '787472'       │
│ 19      │ 'parse_value (simple)'              │ '3014318'      │ '0.332'     │ '2371377'      │
│ 20      │ 'parse_value (complex)'             │ '1244385'      │ '0.804'     │ '1181063'      │
│ 21      │ 'parse_dimension'                   │ '12025011'     │ '0.083'     │ '17217040'     │
│ 22      │ 'walk (node.type)|Nerdy.dev'        │ '2473'         │ '404.437'   │ '2425'         │
│ 23      │ 'walk (node.type_name)|Nerdy.dev'   │ '2267'         │ '441.030'   │ '2237'         │
│ 24      │ 'traverse|Nerdy.dev'                │ '3308'         │ '302.322'   │ '3313'         │
│ 25      │ 'walk (node.type)|CSS Tricks'       │ '271'          │ '3695.390'  │ '266'          │
│ 26      │ 'walk (node.type_name)|CSS Tricks'  │ '250'          │ '4007.539'  │ '247'          │
│ 27      │ 'traverse|CSS Tricks'               │ '387'          │ '2585.499'  │ '378'          │
│ 28      │ 'walk (node.type)|Bootstrap'        │ '1280'         │ '781.481'   │ '1273'         │
│ 29      │ 'walk (node.type_name)|Bootstrap'   │ '1170'         │ '854.944'   │ '1161'         │
│ 30      │ 'traverse|Bootstrap'                │ '1812'         │ '551.763'   │ '1761'         │
│ 31      │ 'walk (node.type)|Tailwind'         │ '91'           │ '10938.163' │ '89'           │
│ 32      │ 'walk (node.type_name)|Tailwind'    │ '84'           │ '11938.762' │ '82'           │
│ 33      │ 'traverse|Tailwind'                 │ '129'          │ '7774.011'  │ '126'          │
└─────────┴─────────────────────────────────────┴────────────────┴─────────────┴────────────────┘

── Parse+Walk memory ──

┌─────────┬──────────────┬───────────┬───────────┐
│ (index) │ File         │ Size      │ Memory    │
├─────────┼──────────────┼───────────┼───────────┤
│ 0       │ 'Nerdy.dev'  │ '111 KB'  │ '3.6 MB'  │
│ 1       │ 'CSS Tricks' │ '1308 KB' │ '37.9 MB' │
│ 2       │ 'Bootstrap'  │ '227 KB'  │ '6.9 MB'  │
│ 3       │ 'Tailwind'   │ '2865 KB' │ '86.2 MB' │
└─────────┴──────────────┴───────────┴───────────┘

perf(tokenizer): classify char once per token and use ident-start tab…

edd8a5d

…le bit Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

claude and others added 3 commits October 14, 2026 13:51

test(tokenizer): cover first-character classification against an inde…

ff8c70b

…pendent oracle Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test(tokenizer): fix lint (expect takes one argument)

b2d98e4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge branch 'main' into claude/tokenizer-char-dispatch

a5c743f

bartveneman merged commit 22f35f9 into main Oct 15, 2026
16 checks passed

bartveneman deleted the claude/tokenizer-char-dispatch branch October 15, 2026 20:37

Navigation Menu

Repository navigation

perf(tokenizer): classify char once per token and use ident-start table bit - #302

perf(tokenizer): classify char once per token and use ident-start table bit#302
bartveneman merged 4 commits into
mainfrom
claude/tokenizer-char-dispatch

bartveneman commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

bartveneman commented Oct 15, 2026

Uh oh!

bartveneman commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bartveneman commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests proving nothing broke

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Package Size Increase

Uh oh!

pkg-pr-new Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartveneman commented Oct 15, 2026

Uh oh!

bartveneman commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bartveneman commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

pkg-pr-new Bot commented Oct 14, 2026 •

edited

Loading