Checking
Prop.check #
Prop.check(gen: Gen[T], config: Config = Config(), law: Fn[T, Result[Void]]) -> Result[Void]Runs law on config.runs values of gen. It returns ok() when the law holds for all of them, or an error with the shrunk counterexample, the number of runs and the seed.
import Prop from "pkg/prop/prop"
def main() -> Void
puts(Prop.check(Prop.int(0, 100)) do |n| Prop.eq(n + 0, n) end)
small = Prop.check(Prop.int(0, 100), Prop.Config(seed: some(7))) do |n|
Prop.eq(n < 50, true)
end
puts(small.message)
endok()
falsified after 3 runs, shrunk to 50, seed 7: expected true, got falseProp.Config #
Prop.Config(runs: Int = 100, seed: Optional[Int] = none, discards: Int = 1000, shrinks: Int = 500)How a check runs: how many cases, a fixed seed (32 bits; a bigger number is masked to them), how many discarded cases are too many, and how many laws to run when shrinking.
import Prop from "pkg/prop/prop"
def main() -> Void
config = Prop.Config(runs: 3, seed: some(1))
puts(Prop.check(Prop.int(0, 9), config) do |n| Prop.eq(n * 2, n + n) end)
endok()Prop.eq #
Prop.eq(actual: T, expected: T) -> Result[Void]ok() when the two values encode to the same JSON, else an error that shows both. It is the equality for a law: its failure is a plain Result, so check can shrink it.
import Prop from "pkg/prop/prop"
def main() -> Void
puts(Prop.eq([1, 2], [1, 2]))
puts(Prop.eq([1, 2], [2, 1]))
endok()
error("expected [2,1], got [1,2]")Prop.assume #
Prop.assume(condition: Bool) -> Result[Void]ok() when condition is true. When it is false, the case is discarded: it is neither a pass nor a failure. Use it first in a law, with try.
import Prop from "pkg/prop/prop"
def main() -> Void
result = Prop.check(Prop.int(-50, 50)) do |n|
try Prop.assume(n != 0)
Prop.eq(n * 100 / n, 100)
end
puts(result)
endok()Seeds and replay #
check draws its seed from the Random of the test, so agentica --seed N test replays the whole run. A case depends on its seed alone. The seed of a failure is in its message, and Prop.Config(seed: some(S)) runs that check again, and nothing else. Prop.sample (on the generators page) shows the values that a seed makes.
Discarded cases #
check fails when more than config.discards cases are discarded, so a law cannot pass without being tried:
import Prop from "pkg/prop/prop"
def main() -> Void
config = Prop.Config(discards: 5, seed: some(1))
result = Prop.check(Prop.int(0, 9), config) do |n|
try Prop.assume(n > 100)
ok()
end
puts(result.message)
endgave up: 6 cases discarded and 0 of 100 passed, seed 1Shrinking #
When a case fails, check tries simpler values of the same generator and keeps the simplest one that still fails. Each generator shrinks in its own way (see generators). config.shrinks bounds the laws run to shrink one case; when it runs out, the message says (shrinking stopped after N steps).
Rules for a law #
- Fail with
Prop.eq,Prop.assumeorerror(...). A failedTest.eqorTest.assertinside a law fails the test at the first failing case, beforecheckcan shrink it. - Do not narrow a generator, or weaken the law, to make it pass. Put the counterexample in a plain
test_function and fix the code. Prop.equsesJSON.encode, so what it compares must be JSON-encodable (comparebytes.to_aforBytes). Two maps with the same pairs in another order differ. The counterexample is written by the generator, so a generated type that JSON cannot encode (Prop.bytes, or a struct with bytes throughProp.map_shown) is reported too.- A test has a budget of about 2.5 million steps (and 30 seconds). A run that goes over it ends with
RUNTIME-STEP-LIMIT, whichcheckcannot catch, so the counterexample and the seed are lost. Making a case costs about 200 steps per item (a string is a list of characters, so a list of strings is a list of lists): 100 cases ofProp.list(Prop.string())take about 0.5 million steps before the law runs, ofProp.list(Prop.int(0, 1000), 100)about 0.7 million, and ofProp.list(Prop.string(20), 20)about 1.5 million. A case of aTree(a value and a list of trees) fromProp.oftakes about 4000 steps at depth 2 and 7000 at depth 3. Raisemax_sizeorrunsonly as far as that allows, or split the law into several tests.