Method
How it works
Serendib reads primary sources every day, proposes pairs from different fields, and keeps a pair only when every clause of the contract holds. Most candidates are discarded. The ones that survive reach the editor, then the digest.
Four steps
-
Ingest
Connectors fetch new items from arXiv, OpenAlex, Hacker News, GitHub releases, SEC EDGAR, the Federal Register, and a short list of curated feeds. Each raw payload is kept so an item can be rebuilt. Items are scored for relevance to the beat before they count.
-
Connect
Candidate pairs come from three methods: a moderate-distance nearest-neighbour search across domains, rare shared entities, and Swanson-style paths where two items share a bridge but no direct citation or link. Pairs that a keyword search would already find are removed before any language model sees them.
-
Verify
A verifier reads both items and the bridge, then returns a judgement, the evidence quotes, and a one-sentence analyst action. Every quote must match the stored source text character for character after whitespace and Unicode normalisation. A pair that fails that check is rejected, whatever the verifier said.
-
Deliver
An editor approves each connection before anything is sent. The approved issue goes to subscribers. The public archive shows the same issue one week after it was sent, and never earlier.
The serendipity contract
A connection is delivered only if all four clauses hold. Each clause is enforced by an automated gate, a human review, or both.
-
It spans two sources and two fields.
The two items come from different sources, and their domains differ.
-
A keyword search on either side would not have found the pair.
Title and keyphrase overlap must stay below a threshold, and the pair must be absent from a top-20 lexical search seeded by either item's title.
-
Both sources contain quoted evidence.
The verifier returns at least one quote per item, and each quote must string-match the stored text. A quote that does not match rejects the candidate.
-
Nobody has publicly linked them before.
The index is checked for an existing citation, shared URL, or co-mention. The final shortlist also gets a web search for a prior public link, and the editor confirms the pair.
Scoring
Only pairs that pass the gates are scored. Each connection carries two public scores, non-obvious and confidence, plus a combined score. The formula is shown in full:
non_obvious = w1·d_lex + w2·d_topic + w3·σ(z_pair) + w4·(1 − prior) + w5·u_profile + w6·llm_non_obvious
score = non_obvious × confidence × freshness × credibility- d_lex
- One minus the Jaccard overlap of title and keyphrase tokens.
- d_topic
- The distance between the two primary topics in the OpenAlex hierarchy, divided by four.
- z_pair
- How unusual the co-occurrence of the two source or topic pair is, against a shuffled 90-day window, squashed with a logistic.
- prior
- One if any prior link was found, otherwise a weak-link estimate between zero and one.
- u_profile
- One minus the closest similarity to previously delivered connections and the beat's seed keywords. Zero until a profile exists.
- llm_non_obvious
- The verifier's own non-obvious estimate.
- freshness
- An exponential decay on the older item's age, with a floor, so older material is not dropped but counts for less.
- credibility
- A fixed table by item type: filings and rules at the top, preprints slightly below peer-reviewed work, and community posts lower still.
The weights start equal. They change only through the evaluation harness, and the weights and formula version are stored on every connection.
What Serendib will not do
- Rank by popularity, summarise news, or forecast outcomes. Serendib claims relevance, not prediction.
- Show a connection it cannot back with quotes from both sides, or present an invented example as real. Examples outside the archive are labelled ILLUSTRATIVE.
- Republish full text. Public pages show short quotes with links to the source.
- Use Reddit, X, live patent APIs, or PDFs of arXiv papers as sources in v0.
Where the method comes from
In 1986 Don Swanson showed that dietary fish oil and Raynaud's syndrome were linked through a chain of published papers, even though the two bodies of literature rarely cited each other. Each link was public. The connection was not. Serendib works the same way at larger scale, and the citations are Swanson, Perspectives in Biology and Medicine 30(1), and Swanson, Library Quarterly 56(2).