Horyzont
Building and studying a Polish private/public corpus for model training, validation and data research.
Developed by Oris Team. The project is deliberately practical: collect, preserve, measure, train small models, inspect what they learn, then feed the result back into the data pipeline.
State · last three months
| Month | Tokens | Status |
|---|---|---|
| June 2026 | — | no snapshot |
| July 2026 | — | no snapshot |
| August 2026 | ≈7B | active corpus work |
Horyzont v1 → v1.5
The first phase was intentionally simple. It existed to make the pipeline measurable before making it clever. Documents were routed into KEEP, CLEAN, SPLIT and DROP. That routing remains useful as a historical diagnostic, but Horyzont is now moving from v1 to v1.5: decisions are no longer treated as final labels produced by one teacher or one scoring pass.
≈870k classified records
≈490k classified records
≈120k classified records
≈850k classified records
These percentages describe the early ≈2.33M-record routing experiment, not the token share of the full corpus.
What v1.5 is doing
v1.5 is a feedback loop rather than a single filter. A larger teacher is useful for bootstrapping judgments, but its scores are not treated as ground truth. The same applies to later pipeline models: a confident score is still only one measurement of a document.
Source text, provenance and compact routing decisions are kept before destructive processing. Even material currently marked DROP can be useful later when the scoring model, tokenizer or target model changes.
It reviews noisy/web-derived material, proposes cleaning or routing decisions and produces candidate examples. It is a teacher and a tool, not an oracle.
A smaller manually reviewed set is used to measure where teacher judgments are reliable and where they fail. This is also the reference point for later small filtering models.
Koliber 1.0, APT3 and Liquid 230M-class models are used as practical probes. The question is not only “does this document look good?”, but “does keeping or removing this type of sequence help a model at this scale?”
The accepted/rejected boundary is iterated instead of frozen. The goal is a higher and better-understood accept/reject signal, not a one-pass quality score.
After enough gold and model feedback exist, the compact encoder can be used for a much broader pass, including a new evaluation of data already processed by earlier versions.
Preservation does not mean duplicating every intermediate artifact forever. A 40 GB source corpus should not quietly become 60 GB because each stage emits another metadata-heavy copy. v1.5 therefore keeps the original data and the information needed to reproduce or audit a decision, while avoiding gratuitous per-stage metadata expansion.
Do not over-trust the score
A teacher can reject useful data. A small classifier can learn the teacher's bias. A later target model can value a sequence that looked unimportant to both. For that reason, Horyzont treats KEEP/CLEAN/SPLIT/DROP as routing decisions with provenance, not as permanent truths.
Data logic is mostly mathematics
The most useful first filters are often the least glamorous ones: counts, ratios, entropy, repetition statistics, length distributions, token frequencies, near-duplicate similarity and distribution shifts. Mathematics is the best friend of anyone working on language-model data because human-readable text and model-useful text are not the same thing.
A paragraph can look perfectly reasonable to us while being highly repetitive under the tokenizer, nearly duplicated elsewhere, badly segmented, statistically out of distribution, or disproportionately expensive for the information it adds.
Koliber 1.1 · data, tokenizer and saturation
Data, data and data again.
A model is, to a large extent, what you feed it. For Koliber 1.1 the data problem is therefore being split into several connected questions: how much repetition a small model can tolerate, what mixture of domains is useful, when filtering removes diversity, how vocabulary size changes the effective corpus, and where the boundary lies between data the model should see and data that only consumes limited parameter capacity.
Research reference points
These papers are reference points, not a specification for Horyzont. They frame the questions being tested locally; the pipeline is not presented as a reproduction of any one of them.
What is being built
The goal is not to “collect the internet”. It is to preserve enough raw material to keep future options open, then learn which parts are actually worth spending model capacity and compute on.
Community context
Horyzont is developed alongside work already happening in the Polish data community. SpeakLeash and SlayerLab / Polish DynaWord are useful reference points and signals that more open, documented Polish data work is worth doing.
Extended Wikipedia
Polish Wikipedia is valuable but limited in size and domain coverage. One direction is to preserve more of the structured information that conventional text extraction tends to discard, and to cautiously extend coverage with larger documented sources.
FILENAME, FS, RS, NF, NR, OFS, ORS, OFMT, SUBSEP), the operator-precedence table and an infobox containing basic facts about the language.This is simply an example of the difference between extraction optimized for clean prose and extraction optimized for preserving knowledge.