Retrieval That Crosses Languages on Purpose

Most retrieval datasets treat language as a property of the document. The passage is French, so the queries are French. Train on that data and you get an embedding model that is excellent inside a language box — and unreliable at the exact moment clinical work needs it most, when a Vietnamese question has to find its answer inside a Korean guideline or an English protocol.

Meddies Embedding Data is a query/passage dataset built on the opposite premise: cross-lingual by design, not by accident. For each generated query, the target language is drawn independently of the passage language. A Korean passage yields Vietnamese, English, or Chinese queries as often as Korean ones. Roughly two thirds of the rows are cross-lingual. document_language and query_language are stored separately, and requested_query_language records what the planner asked for before validation confirmed what actually came out.

The planner owns the axes, not the model

The queries are synthetic, generated by an LLM reading a passage split into numbered evidence blocks. What keeps them useful is who decides what. A program-owned plan fixes the information need, the query shape, the register, the lexical relation, and the target language for every slot. The model writes the query; it never chooses those axes.

This is the rule I use across every Meddies synthetic dataset: an LLM is good at producing text inside constraints and unreliable at deciding what the constraints should be. Let the model pick its own difficulty and you get a dataset that is easy to generate and boring to train on.

Grounding and the validation gauntlet

Every query is generated against a passage carrying at least ten evidence blocks. That floor is deliberate: a passage that supports ten near-identical questions produces ten near-identical rows, which is how datasets quietly fill up with duplicates.

Before a row is promoted, it passes deterministic validation:

• fastText language identification against the requested language — the planner's intent is not taken on faith;
• a token-length policy calibrated per language, because ten tokens in Vietnamese and ten in German are different budgets;
• a retrievability gate rejecting vague subjects and meta-references to the passage ("this text discusses…");
• exact-match duplicate rejection against both the current run and previously published data.

Three domains: general (Wikipedia across 17 languages), medical (clinical practice guidelines and reference text), legal (statutes and codified law across 7 languages). The synthetic bundle comes in configurations — synthetic_monolingual, synthetic_cross_lingual (the default), and synthetic_language_unverified, which is quarantine data: opt-in, and not for training without review.

Provenance as a first-class field

Every row carries the machinery of its own creation: the source dataset and pinned upstream revision, the source admission record that authorized generation, the prompt contract version, the tokenizer identity, the transform digest, and the execution identity of the run that produced it. A cohort field names the operating-day run.

This is not bureaucracy. In clinical-adjacent work, a dataset without provenance is a liability: when something looks wrong six months from now, you need to trace which runs produced it, under which contract, from which source revision. If that reconstruction requires archaeology, the dataset is not reproducible.

Privacy before volume

Source corpora are screened before admission, and again by an independent audit that does not share the admission screen's patterns — two chances to catch what one screen misses. Judicial corpora that name private individuals are excluded outright rather than filtered, because rule-based redaction proved unreliable on exactly that kind of text. Rows identified as carrying an identifiable private person are removed from the reservoir, not from the output — so the removal cannot be reverse-engineered from what shipped.

The limits, stated

• Duplicate rejection is exact-match on a normalized key. Cosmetic rewrites of the same information need survive it, so the near-duplicate rate is materially higher than the exact rate.
• A share of rows carry a null query_language — language identification was confident enough to promote the row but not to label it. Filtering on language drops those rows.
• No hard negatives are included. Train with in-batch negatives or mine them separately; this dataset gives you the pairs, not the adversarial distractors.

The top-level license is CC BY-SA 4.0 because the general subset derives from Wikipedia under those terms. The medical and legal subsets derive from sources with varying terms, some non-commercial — the per-row source_admission_license and source_admission_attribution fields are the real license record, because a mixed corpus cannot be summarized by one field.

Dataset: Meddies/meddies-embedding-data — 2.31 GB, viewer-configurable by subset. Part of the Meddies Embeddings collection.