Who gets to keep score

Five posts ago I said the browser had started grading sites for agents and nobody was grading the industry. This is the end of that argument, and it is the part I have a stake in, so I am going to say what the stake is up front: I sell the work that fixes a bad score. Read what follows knowing that.

Here is what I keep coming back to. Every measurement layer the web has adopted arrived in the same order, and it is not the order anybody plans. Somebody publishes a comparison. The comparison is partial and people argue with it. The arguing produces the definition. The definition becomes the standard. Accessibility went that way. Page speed went that way, badly, twice. Security went that way when browsers began labeling sites insecure and the industry discovered it had opinions about certificates.

Four conditions, and why each one exists

The method is published before any result is. Not a methodology page written after the fact to justify a ranking. The full rubric, in advance, so a firm can score itself and get the same answer. I published mine in the second post of this series for exactly that reason.

Every check records its evidence. Not “fails structured data” but the URL and the literal thing found or not found there. The first time somebody calls to say the score is wrong, the evidence line is the only thing standing between a correction and a fight. Twice now the caller has been right.

Rescoring is free. A benchmark that charges to acknowledge a fix is a protection racket with a spreadsheet. Fix it, ask, get rescored, and the new number replaces the old one with both dates visible.

The conflicts are stated. Anyone publishing a score is selling something. Say what. A score is not disqualified by its author having a business, it is disqualified by hiding one.

The vendor scoreboard problem

Watch for the benchmark whose criteria happen to be that vendor's feature list. It scores you low, the remedy is their product, and the score rises when you buy it. That is not measurement, it is a quote with a chart on top.

The tell is checkable. Ask whether a firm could score full marks using nothing but open standards and its own engineering team. If the answer is no, the benchmark is a sales tool. My rubric passes that test on purpose: every point on it can be earned with published standards, a text editor and someone who knows what they are doing.

What a first round honestly looks like

Partial. One industry, a hundred or so firms, conformance measured from the public web, published with the date and the rubric version stamped on it. Not a market study, not a prediction, and no claim about anybody's business quality.

It will be argued with. Some of the arguments will be right and the results will get corrected in public, because the alternative is pretending a first attempt was perfect, which is how a benchmark loses the only thing it has.

And one round proves nothing on its own. A single score is a snapshot. Four rounds is a trend, and the trend is the part nobody can reconstruct after the fact, which is why the only version of this worth doing is the one that keeps a schedule.

Where this leaves you

Score yourself before somebody else does. The rubric is published, the checks are mechanical, and the exercise costs an afternoon. Then fix the failures rather than the number, since the failures are specific and the number is only a summary of them.

And keep the two measurements apart in your head. Conformance is not citation. One says software can read you. The other says a model recommends you. A scoreboard can settle the first honestly. Nobody should claim to settle the second with a single number, and when someone tries, that is the moment to ask what they sell.

I am building the first one. When it publishes, it will carry the method, the evidence and this disclosure attached to it.

Common questions