pick.tools

Methodology

methodology_version: 1. These definitions produce every number on Rankings, the server pages and the data API.

Call outcomes

Each relayed tools/call gets one outcome:

  • ok: upstream returned a JSON-RPC result without isError: true.
  • tool_error: upstream returned a result with isError: true (the tool ran and reported a tool-level error, e.g. bad arguments). The server answered correctly; this is not a reliability failure.
  • upstream_error: upstream HTTP 5xx/4xx other than 401/403, or a JSON-RPC error response.
  • timeout, unreachable: transport failures.
  • credential_error: upstream 401/403, or the caller omitted a required X-Pick-Key-<slug>. A caller mistake; excluded from the public denominator.
  • rejected: pick.tools refused the call (not proxyable, upstream busy, invalid slug). Never reached upstream; excluded from public stats, kept for operations.

Requests refused by the per-IP rate limit are not recorded at all.

Daily numbers

Per server and UTC day:

  • calls = ok + tool_error + error_calls, where error_calls counts upstream_error, timeout and unreachable. Credential errors and rejections are excluded.
  • answered = ok + tool_error
  • success_rate = answered / calls
  • ranked_calls = Σ over ip_hash of min(answered_by_that_ip, 1000). The per-IP daily cap of 1,000 limits what any single source can add to a server's rank. Events without an IP hash count as one source.
  • p50_ms, p95_ms: latency percentiles over calls.
  • distinct_ip_hashes, distinct_client_names: distinct counts over calls. Indicative: hosted platforms share a few egress IPs. They never order the rankings; distinct_ip_hashes serves the trending threshold and nothing else.

Public aggregates exclude private rows (X-Pick-Private: 1). private_calls counts all non-rejected private events and is shown as a site-wide total, never per server.

Windows and rankings

A window is 1, 7 or 30 UTC days ending as_of, the current UTC day, partial day included. Today's rows refresh every 10 minutes.

  • Rankings sort by ranked_calls in the window, then answered.
  • p50_ms over more than one day is approximated: daily p50s weighted by calls, Σ(p50_ms × calls) / Σ calls.
  • Trend compares ranked_calls with the preceding window of equal length, in percent. Empty when the previous window has 0. The current UTC day is partial, so every trend compares a window ending mid-day with a full one; this shows most in 1d, early in the day.
  • Trending needs at least 100 ranked_calls and at least 3 distinct IP hashes in the window. The hashes are summed over days, an upper bound: one IP on two days counts twice.

Versions

Changing any definition bumps methodology_version and triggers a recompute from raw events where retention allows (90 days of raw events).