Do property-based tests for parsers find generated-code bugs?

A generated parser can look finished long before anyone has established that it preserves the language you meant to accept. The code has types, branches, error messages, and a dozen tidy example tests. Then a customer sends an empty string inside a nested field, a number one digit past the limit, or two equivalent spellings of the same value. The parser does something nobody asked it to do.

Property-based tests are how you turn those unstated rules into executable pressure. They do not replace examples. They force the parser and validator through combinations that neither the original author nor an assistant thought to write down. That matters most when AI produced the first version of the code, because an assistant is good at filling visible gaps and weak at discovering the invisible contract around them.

The useful unit is not "generate random strings and hope." The useful unit is a property with a generator designed to attack the boundary where meaning changes: text becomes a value, a value becomes canonical text, a field becomes required, an integer becomes out of range, or an accepted request becomes an expensive failure later.

Generated code needs a contract before it needs more examples

A parser has at least three contracts, and teams regularly collapse them into one.

The first contract is syntax: which byte sequences or characters form a legal document. The second is structure: what tree or object the parser returns. The third is policy: which structurally valid objects your application permits. A generated parser often blends all three into a single function because that makes a compact implementation. It also makes defects hard to see.

Take a configuration format with a port field. The syntax may permit "8080" and 8080. The parsed structure might represent both as a JavaScript number. Policy may permit only integers from 1 through 65535. If you test only parse("{\"port\":8080}"), you have not decided whether strings are legal, whether decimals round, whether 0 is rejected, or whether a huge integer loses precision before the validator runs.

Write the contract in a form a test can consume. It does not need formal notation, but it must make choices visible:

• A valid document produces a value or a documented normalization of that value.
• Invalid syntax produces a parse failure, never a partial object presented as valid.
• A structurally valid but disallowed value produces a validation failure with a stable path.
• Parsing and validation do not throw for ordinary hostile input.
• Limits apply before resource use becomes unreasonable.

The distinction between syntax and policy has a practical consequence. If your test generator creates only policy-valid objects and serializes them, it will never exercise the parser's rejection paths. If it emits arbitrary text for every property, almost every case will fail before the meaningful code runs. You need both domains, and you need to name which one each property targets.

The original QuickCheck paper by Koen Claessen and John Hughes made the important move here: describe a general property as code, then generate many inputs against it. The hard work is not pressing the button that runs a thousand cases. The hard work is choosing a statement that remains true across all legitimate inputs.

Round trips prove different things in each direction

parse(print(value)) and print(parse(text)) sound interchangeable. They are not. Treating them as the same property produces false failures in one direction and missed bugs in the other.

For a serializer and parser pair, the strongest ordinary property is usually:

const reparsed = parse(print(value));
expect(reparsed).toEqual(normalize(value));

This starts with a value your system says is legal. You print it, parse the result, and compare the outcome with the normalized value. It catches serializers that omit a field, escape text incorrectly, lose a sign, change a time zone, or produce text that the parser itself cannot read.

The reverse property starts with text:

const result = parse(text);
if (result.ok) {
  expect(print(result.value)).toBe(canonicalize(text));
}

This asks whether accepted text has one stable representation. It is only valid when your language has a defined canonical form. JSON objects can reorder fields. Whitespace can disappear. A decimal such as 1.0 may print as 1. A date might convert from an offset form into UTC. Comparing output to the original text would accuse correct normalization of being a defect.

Use three separate labels in the test suite:

• Lossless round trip: print(parse(text)) === text. Reserve this for formats that promise byte preservation.
• Semantic round trip: parse(print(value)) === normalize(value). This is the workhorse for typed values.
• Canonicalization: print(parse(text)) === canonicalize(text). Use it only after you define canonical text precisely.

Generated code commonly gets the first two confused. An assistant sees a parser and printer, writes a pretty round-trip test, and accidentally compares objects with properties whose order, default values, or absent fields have changed. The test starts failing, someone weakens it with a partial assertion, and the suite loses its teeth.

Build a normalizer that reflects the contract instead. If an omitted enabled field means true, normalize both {} and { enabled: true } to the same internal value. If input allows a string ID with leading zeros but internal values use a number, make that conversion explicit. Do not let deep equality decide the product behavior for you.

Bounds tests must attack the decision, not the number

Most validation defects live one unit from a boundary. Generated code often handles the obvious examples and mishandles the comparison operator, type coercion, or conversion sequence around them.

If a field permits an integer from 1 through 100, random generation from a large number range wastes nearly every case. The generator should spend much of its time on -1, 0, 1, 2, 99, 100, and 101. It should also generate values that look numeric but are not valid numbers for your contract: "1", "01", "1.0", " 1", "1e2", NaN, Infinity, and values beyond the safe integer range if the input path can represent them.

A compact boundary generator in TypeScript might look like this:

import * as fc from "fast-check";

const portCandidates = fc.constantFrom(
  -1, 0, 1, 2, 65534, 65535, 65536,
  Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER + 1
);

const portLike = fc.oneof(
  portCandidates,
  fc.integer({ min: -100, max: 100000 }),
  fc.constantFrom("0", "1", "65535", "65536", "01", "1.0", " 80", "80 ")
);

fc.assert(
  fc.property(portLike, (candidate) => {
    const result = validatePort(candidate);
    const allowed = typeof candidate === "number"
      && Number.isSafeInteger(candidate)
      && candidate >= 1
      && candidate <= 65535;

    expect(result.ok).toBe(allowed);
  })
);

This artifact has a limitation worth stating plainly. It assumes validatePort accepts only numeric JavaScript values. If your API intentionally coerces numeric strings, change allowed to express that policy and add a normalized output assertion. Do not silently allow the implementation to choose coercion rules because JavaScript happened to make them convenient.

Bounds also apply to length, nesting, collection size, token count, and numeric magnitude. A maximum string length needs tests at max - 1, max, and max + 1, but that is not enough. Test a string with the same visible length made from multi-byte characters if the implementation measures bytes. Test combining characters if the product means user-perceived characters. Test a long prefix plus an invalid final token if the parser accumulates data before it rejects.

The question is always the same: where does the code decide to accept, reject, allocate, or recurse? Generate cases around that decision.

Idempotency catches normalization that drifts

Idempotency means that after the first normalization, applying the same operation again does not change the result. For validators, that property exposes a category of defects example tests rarely catch.

Imagine a request validator that trims strings, fills defaults, sorts tags, and converts an email to lowercase. A caller sends a request, gets back the sanitized object, stores it, then validates it again on a later update. The second pass should return the same sanitized object. If it does not, you have a moving contract.

The property is simple:

fc.assert(
  fc.property(rawRequestArb, (raw) => {
    const first = validateAndNormalize(raw);
    if (!first.ok) return true;

    const second = validateAndNormalize(first.value);
    expect(second).toEqual(first);
  })
);

This catches subtle behavior. A common generated implementation adds a default timestamp each time validation runs. Another converts an absent array to [] on the first pass, then rejects that empty array on the second because a different branch sees it. Another trims a field after measuring its length, so input at the maximum boundary passes once and fails after normalization.

Do not apply idempotency blindly. Some operations are intentionally non-idempotent. A validator that allocates a new server-side ID or attaches the current time should not claim otherwise. Separate pure normalization from side effects, then test the pure portion. If your team cannot separate them, the property has already shown you a design problem.

Idempotency also clarifies a recurring argument about defaults. A schema default is not automatically a runtime default. The JSON Schema documentation separates annotations from validation assertions, and default is often metadata rather than an instruction for a validator to mutate data. If your application fills defaults, test that behavior as your own normalization rule. Do not pretend the schema language promised it.

Invalid input needs its own generator and outcome model

Teams often write one generator for valid documents, then mutate a character and call that invalid-input coverage. It is better than nothing, but it misses the ways invalidity actually enters production systems.

Create invalid generators by violation class. A syntax generator should target broken delimiters, truncated strings, malformed escapes, unexpected tokens, and invalid encodings when your transport exposes them. A structural generator should use an otherwise parseable document with a wrong type, an unexpected field, a duplicate field, an absent required field, or an invalid discriminator. A policy generator should preserve the model shape while breaking cross-field rules and bounds.

Each class expects a different result. This model keeps the assertions honest:

type Outcome<T> =
  | { kind: "ok"; value: T }
  | { kind: "parse_error"; offset: number; message: string }
  | { kind: "validation_error"; issues: Issue[] };

A crash is not an Outcome. A timeout is not an Outcome. Returning {} after a parse error is not an Outcome. Make the test harness fail separately for thrown exceptions, unhandled promise rejections, and duration beyond a chosen limit.

The useful property for hostile input is usually not "all malformed inputs return the same message." Error prose changes and can make tests brittle. Assert the stable parts instead: no exception, a recognized error category, an offset inside the input for parse errors, and issue paths that point to an actual location for validation errors.

Here is a failure pattern I have seen repeatedly in generated validators. The code checks typeof value === "object", then reads value.profile.email. null passes the first check in JavaScript and the next line throws. The assistant may have generated tests for a valid object, an object missing profile, and an object whose email is invalid. It often does not generate null, an array, a function, a proxy, or a getter that throws.

Do not force every validator to defend against exotic objects if the boundary has already converted input to plain JSON. Do make the boundary explicit. If the validator accepts arbitrary runtime values, generate arbitrary runtime values. If it accepts decoded JSON only, test the decoder and validator together at the point where that promise becomes true.

Cross-field rules are where assistants guess

Single-field constraints are easy to prompt. The expensive bugs sit in relationships between fields because relationships rely on business meaning that rarely appears in a type definition.

Suppose a deployment request has mode, replicas, minReplicas, maxReplicas, and regions. The field types tell you little. Your contract may require minReplicas <= replicas <= maxReplicas, a single region for one mode, at least two regions for another, and a limit that changes when a feature flag is present.

The wrong testing approach is to generate arbitrary objects and put a large predicate before the assertion:

fc.property(deploymentArb, (x) =>
  fc.pre(isValidDeployment(x)),
  () => expect(validate(x).ok).toBe(true)
);

If valid deployments form a small part of the input space, the framework discards most cases. The test run can look busy while barely testing acceptance. The research on coverage-guided property testing makes the same practical point: sparse preconditions waste generated inputs. You can sometimes use coverage guidance or mutation, but a direct generator is usually the faster first move.

Build a valid object from a valid core. Generate minReplicas first. Generate maxReplicas at or above it. Generate replicas inside that interval. Then mutate exactly one relationship at a time for rejection properties.

const validScaleArb = fc.integer({ min: 0, max: 20 }).chain((min) =>
  fc.integer({ min, max: 30 }).chain((max) =>
    fc.integer({ min, max }).map((replicas) => ({ min, max, replicas }))
  )
);

const invertedScaleArb = validScaleArb.map(({ min, max, replicas }) => ({
  min: max + 1,
  max,
  replicas
}));

Now your assertions can say something sharper. Every value from validScaleArb must pass. Every value from invertedScaleArb must fail with an issue attached to the scale relationship, not an unrelated field. If a future refactor changes the message, the test survives. If it accepts min > max, the test gives you a compact counterexample.

This is where generated code needs a human decision most. An assistant can express min <= max after you state it. It cannot reliably infer whether zero replicas mean "paused," "invalid," or "autoscale disabled" from names alone.

Shrinking is part of the debugging interface

A property test that reports a 40,000-character input, a random seed, and a generic assertion failure will earn a reputation for noise. Engineers will lower the run count or disable the suite. That is not a problem with property testing. It means the generator and failure report were treated as an afterthought.

Shrinking reduces a failing generated value toward a smaller value that still fails. With parsers and validators, the smallest case often tells you which assumption broke: an empty property name, a one-item array, a limit exceeded by one, or a nested object with a missing discriminator.

Design values so they can shrink meaningfully. Use structured arbitraries instead of generating a JSON string by concatenating fragments. Represent a document as a model, then serialize it. A framework can shrink the model's strings, numbers, arrays, and fields while preserving enough structure to reach the code you want.

For invalid cases, decide whether you want to preserve invalidity during shrinking. A malformed JSON generator that removes arbitrary characters may shrink into valid JSON, and then the property can fail for the wrong reason. Use a tagged case when necessary:

type InvalidCase =
  | { kind: "unterminated_string"; text: string }
  | { kind: "duplicate_name"; text: string }
  | { kind: "too_deep"; text: string };

The tag gives the property an expected category. It also gives the eventual failure report context that raw text cannot supply.

Keep the seed, the shrunk input, and the generator version in CI logs. Add the shrunk input to your ordinary regression suite after you fix the defect. Do not delete the property because it found one example. The saved example proves you fixed yesterday's bug. The property keeps looking for tomorrow's variation.

Metamorphic tests work when there is no perfect oracle

A parser often has no independent implementation that can tell you the correct output for every input. That does not leave you helpless. Metamorphic properties compare related inputs whose relationship should preserve or predict behavior.

For an order-insensitive object format, permuting fields should preserve the parsed semantic value. For a parser that ignores surrounding whitespace, adding permitted whitespace should not change the value. For a validator where adding an unknown field is forbidden, adding one should change acceptance from true to false while leaving reported issues for existing fields untouched.

These tests are especially good at finding generated code that accidentally depends on object order or branch order. An assistant may emit a parser that records the first occurrence of a field, while the product contract says the last occurrence wins or duplicates must fail. A field permutation generator makes that ambiguity visible immediately.

Be careful with the word "equivalent." Equivalent inputs must be equivalent under your language, not merely in a developer's intuition. Unicode normalization is the classic trap. Two strings may look identical and still contain different code points. If your identity rules normalize Unicode, generate normalization variants and assert the intended result. If they do not, do not smuggle normalization into tests because it feels user friendly.

A practical metamorphic property for a case-insensitive identifier parser looks like this:

fc.assert(
  fc.property(identifierArb, (id) => {
    const lower = parseIdentifier(id.toLowerCase());
    const upper = parseIdentifier(id.toUpperCase());

    expect(upper).toEqual(lower);
  })
);

That property is wrong if identifiers are case-sensitive, if locale affects casing, or if identifierArb includes characters outside the contract. The code is easy. Establishing the contract is the work.

Put generated parser tests at the trust boundary

The highest return comes from testing the parser or validator where untrusted input becomes a typed business object. That might be an HTTP request, a configuration file, an import job, a webhook, or a message pulled from a queue. Testing every internal helper with arbitrary values can be useful, but it does not excuse a weak boundary test.

Use a layered suite. Keep a few hand-written examples for product rules people need to read quickly. Put property tests around pure parsing, normalization, and validation functions. Add integration tests at the transport boundary for encoding, body-size limits, and error serialization. Each layer should own a distinct failure class.

Do not start by asking an assistant to "add property-based tests." Give it the invariants, generators, and outcome types you have decided on. Ask it to implement one property at a time, then inspect the generated cases and the discarded-case count. The assistant will happily create a test that passes while generating almost no relevant input if you let it choose the domain.

For a team introducing AI coding tools across a mature codebase, a Team & AI Audit should inspect these trust boundaries early. The savings do not come from producing more test files. They come from preventing generated changes from quietly widening accepted input, weakening validation, or turning malformed requests into runtime incidents.

Start with the parser that can hurt you most, write one semantic round-trip property, one boundary property, and one hostile-input outcome property. Run them until they produce a counterexample you did not expect. If they never surprise you, inspect the generator before congratulating the implementation.