Islands Performance Benchmarks by Framework #

Framework comparisons are the most-cited and least-trustworthy artefacts in front-end performance. A screenshot of one framework beating another travels widely, but the workload behind it is almost never controlled: different content, different numbers of islands, different throttling, different metrics. The result is that “framework X is faster” means nothing without the harness that produced it. This guide is not a leaderboard. It is a methodology for building a benchmark you can defend — one where the only variable between measurements is the framework itself — applied to Astro, Next.js, Qwik, SvelteKit, Marko, and SolidStart across the metrics that actually matter: Time to Interactive, Total Blocking Time, Interaction to Next Paint, and transferred JavaScript. It is part of islands performance optimization, and it pairs with the framework-specific islands and streaming SSR guides that explain why each framework produces the numbers it does.

Concept Definition & Scope #

A framework benchmark is only meaningful as a difference under controlled conditions. The absolute numbers are almost worthless — they depend on the test machine, the day, and the throttle — but the ratio between two frameworks measured on the same harness, on the same page, in the same session, is stable and defensible. So the entire discipline reduces to one principle: hold everything constant except the framework. That means an identical workload (same content, same three interactive components, same data payload, same image treatment) and an identical environment (same CPU throttle, same network profile, same viewport, same server), differing only in which framework compiled the page.

The metrics fall into two families. Load-time metrics — Time to Interactive and Total Blocking Time — measure the cost of becoming interactive, dominated by how much JavaScript hydrates and when. Interaction-time metrics — Interaction to Next Paint — measure responsiveness after the page settles. A framework that ships little load JavaScript can still have poor INP if its event handlers are expensive, so both families are mandatory. Transferred JavaScript bytes sits underneath all of them as the causal variable that the payload techniques in reducing JavaScript payload in islands attack directly. Defining the target metrics precisely — what counts as “interactive”, how INP is sampled — is prerequisite work covered in measuring hydration and interactivity metrics.

Technical Mechanics #

The workload must be specified so tightly that each framework’s implementation is a translation, not a redesign. Pin it as a document: one route, a static article of fixed length, and exactly three islands — a search box (client:load equivalent, above the fold), a filter panel (client:idle), and an image carousel (client:visible). Every framework implements these same three, with the same props and the same data. Here is the harness driver that runs identically against each built site:

// bench/run.mjs — one driver, run per framework against its production build.
// The ONLY thing that changes between invocations is TARGET_URL.
import { spawnSync } from 'node:child_process';
import { writeFileSync } from 'node:fs';

const RUNS = 9;
const TARGET_URL = process.env.TARGET_URL; // e.g. http://localhost:4321/bench

const results = [];
for (let i = 0; i < RUNS; i++) {
  // Fixed throttling model so CPU and network are identical across frameworks.
  // --throttling-method=devtools keeps the CPU model consistent with the
  // Performance panel, avoiding Lighthouse's simulated-throttle divergence.
  const out = spawnSync('npx', [
    'lighthouse', TARGET_URL,
    '--only-categories=performance',
    '--throttling-method=devtools',
    '--throttling.cpuSlowdownMultiplier=4',
    '--output=json', '--quiet',
    '--chrome-flags=--headless=new',
  ], { encoding: 'utf8', maxBuffer: 32 * 1024 * 1024 });

  const r = JSON.parse(out.stdout);
  const a = r.audits;
  results.push({
    tti: a['interactive'].numericValue,
    tbt: a['total-blocking-time'].numericValue,
    // Transferred script bytes: the causal variable behind TBT.
    jsBytes: a['network-requests'].details.items
      .filter((it) => it.resourceType === 'Script')
      .reduce((sum, it) => sum + (it.transferSize || 0), 0),
  });
}

// Median is robust to the thermal / JIT-warmup outliers a mean would absorb.
const median = (xs) => xs.sort((a, b) => a - b)[Math.floor(xs.length / 2)];
writeFileSync(`bench/out/${process.env.FRAMEWORK}.json`, JSON.stringify({
  tti: median(results.map((r) => r.tti)),
  tbt: median(results.map((r) => r.tbt)),
  jsBytes: median(results.map((r) => r.jsBytes)),
}, null, 2));

INP cannot be read from a cold Lighthouse load because it requires real interaction. Drive it with a scripted probe that dispatches a click on the search island and measures the interaction latency via the Event Timing API:

// bench/inp-probe.mjs (excerpt) — runs in the page via a driver like Playwright.
// Measures the time from a real input event to the next paint on one island.
await page.evaluate(() => new Promise((resolve) => {
  new PerformanceObserver((list) => {
    for (const entry of list.getEntries()) {
      // 'event' entries carry the full input-to-next-paint duration.
      if (entry.name === 'pointerdown') resolve(entry.duration);
    }
  }).observe({ type: 'event', durationThreshold: 16, buffered: true });
  document.querySelector('[data-island="search"] input').focus();
}));

Representative Results #

The numbers below are representative — the shape a well-controlled harness produces on a fixed three-island page under CPU 4×/4G, not universal truths. Reproduce them on your own workload before quoting them; the point of the table is the pattern, not the digits.

Two patterns survive re-measurement across workloads. First, the ordering on TBT tracks transferred JavaScript almost monotonically — the load-time bill is fundamentally a bytes-executed bill. Second, INP compresses the field: frameworks separated by 3× on TBT sit within a much narrower band on INP, because after load a single island’s handler cost dominates regardless of the framework’s hydration model. Qwik’s resumability wins load-time decisively but its INP lead is modest, since deserialising state on first interaction claws back some of the advantage. The framework-by-framework causes are unpacked in astro islands and client directives and qwik resumable architecture.

Step-by-Step Harness #

1. Freeze the workload spec. Write down the page: content length, the three islands, their props, the data payload, and the image dimensions. This document is the contract every implementation must satisfy.

2. Implement the page in each framework. Translate — do not redesign. If Astro uses client:visible for the carousel, the SvelteKit and SolidStart versions must defer the equivalent island; a framework must not get a free win from a different hydration choice unless that choice is the framework’s genuine default.

3. Build to production and serve identically. Compile every implementation with production flags and serve the static output from the same local static server on the same port pattern, so server behaviour is not a variable.

4. Run the driver nine times per framework. Clear the browser cache between runs, keep the machine otherwise idle, and let the CPU return to a stable thermal state between frameworks.

5. Run the INP probe separately. Load-time and interaction-time measurements do not mix in one session cleanly; collect INP in its own scripted pass.

6. Aggregate to medians with an interquartile range. Report the spread, not just the centre. If the IQR for a framework exceeds ~15% of its median, the environment is noisy and the run is suspect.

The two most-scrutinised head-to-heads get their own end-to-end procedures: Astro vs Next.js: a TTI benchmark workflow and benchmarking Qwik vs React total blocking time.

Measurement & Validation #

A benchmark you cannot validate is an opinion with decimals. Apply three checks before publishing any comparison.

First, variance gating: reject any framework’s result whose interquartile range exceeds 15% of its median. High variance means the environment, not the framework, is the dominant signal. Second, a null control: run the same framework build twice as if it were two competitors. The two “different” results should be statistically indistinguishable; if they are not, your harness is too noisy to detect real differences smaller than that gap. Third, byte-to-blocking sanity: TBT should rise with transferred JavaScript within a framework family. A framework that reports low TBT but high JS bytes is either deferring execution (good — verify it) or the measurement missed a long task (bad — re-check with the Performance panel workflow from measuring hydration and interactivity metrics).

// bench/validate.mjs — the null control: same build labelled as two entrants.
import a from './out/astro-run-a.json' assert { type: 'json' };
import b from './out/astro-run-b.json' assert { type: 'json' };

// If two runs of the SAME framework differ by more than the resolution you
// claim between frameworks, your harness cannot support that claim.
const delta = Math.abs(a.tbt - b.tbt);
const resolution = 20; // ms — the smallest between-framework gap you report
if (delta > resolution) {
  throw new Error(`Harness noise ${delta}ms exceeds claimed resolution ${resolution}ms`);
}
console.log(`✓ Null control passed: run-to-run TBT delta ${delta}ms`);

Report the harness, not just the result. A published comparison needs five pieces of metadata beside every number: the harness version, the date, the machine profile, the throttling model, and a link to the workload specification. Without them the reader cannot tell whether your Astro number and their Astro number differ because of the framework, the page, or the laptop. Publishing the workload spec matters most of all — it is the only part a reader can re-implement to check you.

Re-measure on a schedule, not on a whim. Framework performance moves between minor releases: a compiler change trims a wrapper, a runtime adds a scheduler, a default flips. A benchmark published once decays into folklore within two release cycles, and folklore is what produces the contradictory comparisons this guide exists to replace. Pin the harness in CI, run it monthly against current releases, and keep the previous results so the trend is visible rather than a single snapshot.

Failure Modes #

1. Workload drift (the biggest source of bias). One framework’s page quietly does less work — fewer islands, smaller images, a stubbed data call. The comparison then measures the workload difference, not the framework. Symptom: the “faster” framework also has implausibly lower transferred bytes for content that should be identical. Fix: diff the rendered HTML and the network waterfalls across frameworks; the static content and image bytes should match to within a few percent.

2. Throttling mismatch. Lighthouse’s simulated throttling and DevTools CPU throttling apply the CPU slowdown by different models, so a benchmark that switches methods between frameworks compares apples to oranges. Symptom: TBT ordering flips when you change --throttling-method. Fix: pin one method (devtools) for every run and record it alongside the results.

3. Cold-versus-warm confound. Measuring one framework with a warm CPU and browser cache and the next cold inflates the difference. Symptom: the first framework tested consistently looks worst regardless of which one it is. Fix: clear cache between every run, discard the first run per framework as warm-up, and randomise framework order across sessions.

Frequently Asked Questions #

Related #

• Astro vs Next.js: A TTI Benchmark Workflow — the full end-to-end procedure for the most common head-to-head.
• Benchmarking Qwik vs React Total Blocking Time — isolating TBT differences between a resumable and a hydrating app.
• Framework-Specific Islands & Streaming SSR — the architectural reasons each framework produces the numbers it does.
• Measuring Hydration and Interactivity Metrics — precise definitions of the TTI, TBT, and INP the harness collects.
• Reducing JavaScript Payload in Islands Apps — the causal variable behind most of the between-framework gaps.

← Back to Islands Performance Optimization