Architecture¶

Brief, current, precise. A PR that changes the structure described here updates this file in the same PR. The language is docs/SPEC.md; plans and refusals are docs/ROADMAP.md; measured results are docs/benchmarks.md, produced by the harness in bench/ — which is also how a claim here gets falsified. The ceiling below is falsified the same way, by docs/ports.md.

python examples/walkthrough.py executes the pipeline below stage by stage and prints what each one produces — the same public calls fk.solve makes, so the demonstration cannot drift from the code. Its output is committed as examples/walkthrough.out and asserted line for line (tests/test_walkthrough.py), so reading it is the same as running it — and a stage that starts telling a different story shows up as a diff in that file.

Thesis¶

A YAML math spec is a closed AST known before any data is touched. That one property makes everything else legal: the whole model can be compiled — to eager xarray/linopy calls, or to a logical plan executed relationally under a fixed memory budget — with both paths provably meaning the same thing. Every rule below protects it.

flowchart TB
    Y[YAML file] -->|"parse + validate<br/>(schema.py, validation.py)"| MS[MathSchema]
    MS -->|"expand macros: / expressions: (expansion.py)<br/>expand piecewise: blocks (piecewise.py)<br/>resolve names to typed nodes (resolution.py)<br/>check dim sets (dimensions.py)<br/>— backends never see any of them"| AST["core AST<br/>= the only contract between layers<br/>fully typed: names resolved, dims checked"]
    AST -->|"api.py: check / build / solve / write"| LOWER
    AST -.->|"farkas.linopy<br/>(opt-in shim: build / extend)"| BUILD
    LOWER -->|"outside the language:<br/>LanguageError naming the construct"| ERR["load error<br/>(no fallback)"]

    subgraph REL["Relational lane — sparse · batched at every sink · linopy-free"]
        direction TB
        LOWER["lowering.py"] --> PLAN["logical plan<br/>(relational/plan.py)"]
        PLAN --> COMP["compiler.py<br/>plan → lazy frames<br/>pure: nothing is read"]
        DR[("data<br/>parquet paths / any Arrow table")] --> EXEC
        COMP --> EXEC["executor.py<br/>bind sources, label, assemble<br/>the model frames"]
        EXEC --> LPS["lp_file sink (sinks/lp_file.py)<br/>portability, debugging<br/>(mps planned)"]
        EXEC --> DIRECT["solver_direct sink (sinks/highs.py)<br/>COO batches → highspy → HiGHS"]
        DIRECT --> SOL["solution tables<br/>(label join, never dense)"]
    end

    subgraph EAGER["Linopy lane (eager/oracle) — opt-in via farkas.linopy · the ONLY lane importing linopy · not a runtime dependency"]
        direction TB
        DE[("data<br/>parquet paths / pandas")] --> LOAD["linopy/loader.py<br/>coerce data → xr.Dataset"]
        LOAD --> BUILD["linopy/builder.py<br/>evaluate AST"]
        BUILD --> MODEL[linopy.Model] --> SOLVE["linopy solve / writers"]
    end

    classDef laneR fill:#f0f7f0,stroke:#3a7d44,stroke-width:2px,color:#111
    classDef laneE fill:#eef1fb,stroke:#4a5fc1,stroke-width:2px,color:#111
    class REL laneR
    class EAGER laneE

Eligibility is decided by attempting the lowering — lower_program returns a Program or raises fk.LanguageError — so it cannot drift from what the engine supports. Errors split model from run: everything under LanguageError is decidable without data, DataError is what a source failed to supply, and both are LinopyYamlError (errors.py). fk.check() is exactly parse → expand → validate → lower with no data bound, so a model repository can compile-check its math in CI. Expansion precedes validation in both lanes, because a formulation emits declarations and those are language too — a stray dim in generated math is the same error as a stray dim in a written one.

One contract, many consumers¶

The AST is a narrow waist. Everything upstream emits it, everything downstream reads it, and nothing else has to agree on anything — so the model you write once is the same model that gets checked, solved, typeset and read back.

flowchart TB
    Y(["your math, written once — one YAML file"])

    Y --> AST

    AST["the whole model, typed and checked<br/>before a byte of data is read"]

    AST --> SHOW
    AST --> CHECK
    AST --> RUN

    subgraph SHOW["show it — no data, no solver"]
        direction TB
        S1(["typeset the math for a paper or a review"])
        S2(["drive it from the command line"])
        S3(["watch what a build is doing"])
    end

    subgraph CHECK["check it — no data, no solver"]
        direction TB
        C1(["will this build, and is the math sayable?"])
        C2(["do the dimensions line up?"])
        C3(["will that solver take it, and how big is it?"])
    end

    subgraph RUN["run it — the only part that touches your data"]
        direction TB
        R1(["stream it straight into a solver"])
        R2(["write an LP file for anything else"])
        R3(["put the same math on a linopy model"])
    end

    R1 --> ANS

    subgraph ANS["your answers, as tables you can join"]
        direction TB
        A1(["values and shadow prices, by name"])
        A2(["derived results and diagnostics"])
        A3(["change a number, re-solve, keep the labels"])
    end

    classDef built fill:#eef6ee,stroke:#3a7d44,stroke-width:1.5px,color:#111
    classDef plan fill:#fdf4e8,stroke:#b7791f,stroke-width:1.5px,stroke-dasharray:5 4,color:#111
    classDef waist fill:#e9edfa,stroke:#4a5fc1,stroke-width:3px,color:#111
    classDef fam fill:#fcfcfb,stroke:#c9c5be,color:#111
    class Y,R1,R2,R3,C1,C2,A1 built
    class C3,S1,S2,S3,A2,A3 plan
    class AST waist
    class RUN,CHECK,SHOW,ANS fam

Solid is what ships today; dashed is what the shape makes cheap. None of the dashed boxes is a rewrite — each reads the same AST the engine reads, so a renderer is a tree walk, a check is a pass with no data bound, and a new output format is one function in relational/sinks/. Two properties carry it: data enters at exactly one place, which is why checking a model costs seconds and needs nothing but the file; and the waist is closed, which is what the ceiling in Two tiers protects — a new consumer is free, a new primitive is taxed. What is planned, and why, is docs/ROADMAP.md.

Hard rules¶

Enforced, not aspirational: tests/test_architecture.py encodes these as static checks and CI's bare-install job proves the dependency claims.

These rules constrain the language. What a construct may say, which layer may know what, and what a file means on its own — each survives any engine, and each decides what can enter docs/SPEC.md. How much a build costs is a property of the engine, measured in docs/benchmarks.md and not a rule. It was one once, phrased around a memory_limit that only one engine had, and that made an implementation choice load-bearing in the language's rulebook.

The layers are ordered, and imports prove it. Every module imports only downward, at module level, with exactly one declared exception: lowering.py reaches piecewise.py lazily, because a formulation must expand before lowering while expanding needs the subset test lowering defines. DELIBERATE_LAZY_IMPORTS in tests/test_architecture.py is the whole list, and an undeclared in-function import fails the build — a lazy import is how the one real cycle is broken, so decorative ones cannot be allowed to hide it.
Core AST is the whole language. Both backends consume only core AST; macros, named expressions and piecewise: are expanded away before dispatch, and the plan/query/xarray are backend-private. The AST crossing that seam is fully resolved — names are typed Variable/Parameter/Dimension nodes — so a backend cannot hold its own opinion about what a name refers to. Resolving independently is how the two lanes silently disagreed about scoping before.
The engine knows nothing about linopy, xarray or YAML. src/farkas/relational/ goes polars → highspy → solver, with linopy's semantics as a spec to match rather than code to share; it never sees the schema, the AST, or the eager builder. Engine-internal naming encodes neither "polars" nor "yaml". Enforced more strictly than stated — the engine imports nothing from the package at all, bar declared dependency-free leaves (errors.py today, listed in ENGINE_MAY_IMPORT) — because a near-zero import surface is what keeps the subpackage extractable. Widening that list is a decision, not an accident.
One language, two lanes — not fast-vs-slow versions of each other. The streaming engine builds models declared in YAML; the linopy lane attaches YAML math to a linopy.Model already in memory, which is structurally eager. Both accept exactly the same language, and no helper registry exists that could create a divergence — that equality is what makes the differential tests an oracle rather than a comparison of dialects. A construct outside the language is a load error naming the construct and its rewrite, never a redirection to the other lane.
Backend-visible YAML files are self-contained. No Python-side state (registries, session objects) may change what a file means.
The public interface is a declared model, not a Python API. YAML is the format we ship and document; the contract underneath it is MathSchema, and whether that seam is ever blessed is open (see Composition). The Python surface is the runner (api.py); the plan is internal, and a stable plan-construction API is a later possibility, not a current contract.

Two tiers, and the ceiling¶

Primitives (operators, sum, group_sum, roll/shift, where predicates) set the expressive ceiling, and each costs the full two-backend tax: eager implementation, plan node + locality class, executor case, lowering case, differential tests, SPEC entry. macros: / expressions: are pure AST substitution — every composition of primitives at zero marginal cost and zero divergence risk. Formulations (piecewise:) are taxed like a primitive but compose like a macro: they emit new declarations before dispatch and never enter as plan expression nodes.

For any request, triage: macro, primitive, or escape? Most are compositions. New primitives must be macro-friendly — anything a user might parameterise goes in a value position like over=/by=, never a kwarg key; the roll(x, snapshot=1) dim-as-key design is the language's own counterexample.

A candidate primitive is admissible iff it is all three of degree 1 (affine) — variable × parameter, never variable × variable, the one axis that is a scope choice rather than a consequence of streaming (ROADMAP); relational — filter / join / group-by-aggregate over tidy tables; and local — pointwise or bounded-halo, which compose under partition-wise execution where global operators do not. Locality is judged in data space: reductions over a coordinate space ("the last snapshot") read only the small, already materialised dim tables and stay admissible even though they look global.

Read the verdict off the plan. Rules 2 and 3 are one question asked twice, and the compiler already answers it — write the candidate's query over the term stream first and read .explain():

Shape of the emitted query	Locality	Rules 2–3
filter on a column already in the frame	pointwise	admissible
equi-join against a parameter or mapping table	pointwise	admissible
join on the dim table at `ord ± k`, `k` fixed	bounded-halo	admissible
dim table only, no data join	coordinate-space	admissible (free)
window over unbounded rows, or a recursive CTE	global	reject, with the rewrite

This is the case analysis _sum_fragment, _group_fragment and _translate_fragment already implement — each rewriting one fragment on its own, which is what pointwise and bounded-halo mean in code — so a candidate fitting none of those shapes has no executor to be written into. Two limits: degree is not a property of the plan, and it presumes the terminal sum(coeff) over (row, col) stays the only aggregate a term passes through. A primitive is finished when lowering.py accepts it and the differential test against the linopy oracle passes.

The ceiling is a claim, so it needs evidence. In docs/ports.md, math a ported model needed and this language could not state becomes a ledger row with its triage verdict — what docs/ROADMAP.md should be argued from. Those ports also cover a class no other test reaches: both lanes consume the same resolved AST by rule 1, so a shared misreading passes the differential suite green, and only an outside optimum catches it.

What is outside the closure splits three ways, and the split decides whether a request can ever be met:

Tier	Bounded by	Members	Can it move?
Capability-bounded	what a given sink can ingest	SOS / indicator (#23); quadratic	per sink — see below
Budget-bounded	the escape label budget — a cap on the rows and columns an island may emit	global operators, arbitrary Python, non-relational manipulation	already movable — that is what an island is
Design-bounded	our choice of where work belongs	data prep, domain helpers, Python declaring structure	movable any time; we don't want to

Impossible in the symbolic plan: conditionals, iteration, any data-dependent structure inside expressions. What is protected is that the plan's shape is fixed before any data is read — which declarations exist and which dims each spans. Cardinality is always data's to supply: foreach: [snapshot] does not know how many snapshots there are either.

That distinction decides more than it looks. A dimension whose members are computed in data prep is completely ordinary — a cycle basis for KVL, the subsets of a subtour-elimination family — because a graph algorithm run before the build is design-bounded, the row above. The line is temporal, not computational: it does not matter how clever the Python is or whether its output size depends on the data, only whether it can run before the model is built. What is outside is work that needs the solver's answer to decide the next row — lazy cut generation, a solve loop — because there is no "before" for it to happen in.

This is a different property from how much a build costs, though the two meet at the escape hatch. That is why an escape: island (#38) is admissible where a registered Python helper was not: its extent is fixed by the preceding where mask, it is terminal, and it is named in the file. Its label budget is what keeps it accountable — an island's cost is bounded by what it may emit, declared and enforced before any Python runs rather than discovered after it allocates.

An escape buys back the relational and local rules (it returns affine COO rows — a running-sum island still emits affine rows, just O(T²) of them) but never degree, because affine COO is what it returns. That refusal stands on what an island emits, not on what a sink accepts, so it is unaffected by the capability findings below.

Capability is not the ceiling¶

The ceiling above is about streamability and is solver-independent. What a sink can ingest is a separate axis, and conflating the two let one solver's limits read as architectural law — "no sink carries the stream" described HiGHS, not the architecture. Two findings, measured in docs/benchmarks.md: SOS is solver-bounded (HiGHS has no SOS concept at all, while lp_file carries it as a text section and Gurobi natively), and what blocks quadratic is a conjunction — HiGHS has integrality and a Hessian and refuses the pair — so capability is not a flat set. The whole-Hessian handoff is an implementation difference, not a rule-4 violation.

Making this a declared per-sink capability set, with check taking an optional sink, is ROADMAP Track 4.

The relational lane¶

Three modules, one per box above. compiler.py turns plan nodes into lazy frames and reads nothing; executor.py binds the data and fills the model frames; sinks/ drains them. The split is what makes the admissibility test below something you can perform rather than reason about — build a PolarsCompiler, hand it a node, read .explain() (tests/test_compiler.py does exactly that, over empty frames: a schema is all it takes to compile a query). It is also why a new sink is a function in one file instead of another method on the executor.

Tidy tables. Parameters are (dims…, value); a variable frame is (dims…, var_label), one row per existing variable; a linear expression is (frame dims…, var_label, coeff) plus a constant part; constraint rows are (row, sense, rhs); the coefficient matrix is COO (row, col, coeff). Masks are row absence — no NaN sentinels, no -1 labels. Broadcasting is a join, sum drops coordinate columns, group_sum joins the dim table and projects a declared coordinate in place of the grouped dim. Neither aggregates: both rewrite a fragment's dim tuple, and duplicates collapse in the terminal SUM(coeff) GROUP BY row, col at assembly. Labels are dense 0..n-1 by construction, so var_label is the solver column index and row the solver row index — no remapping. That is also why value-only re-solve is cheap and structural editing is out of scope.

Labels are also row-major over the coordinate product, and that is a contract rather than a side effect of how they are computed: it is what makes a build reproducible run to run. _label_frame reaches it three ways depending on how much of the product survives the mask — arithmetic, factored, counted — and they must agree integer for integer, because a label is a solver index.

The plan is affine-by-design. No node introduces variables or constraints as a side effect of an expression; formulations are model transformations. Variable types are not formulations — binary/integer are a vtype column, LP binary/general sections and HiGHS integrality, which keeps basic MILP inside the streaming lane. Reimplementing linopy's reformulation passes inside the plan is explicitly rejected: that duplicates the library this package consumes.

Labels are the one place order is load-bearing. var_label is the solver's column index and row is its row index, so both are assigned by sorting the masked coordinate product on its dimensions' declared ordinals and numbering the result. Variables and constraint rows are the same operation over different frames and it is written once (_label_frame) — twice is how the two would come to disagree about which coordinate gets which index. Everything else is order-free, which is what lets the query planner rearrange it.

That order is also what comes back. primal / dual / to_parquet sort on the label before handing rows over, so a read is row-major over the coordinate product — the same order the LP sink writes. Sorting is stated at the read rather than assumed from the inputs, because a where mask decides which rows of the product survive and a join decides nothing about the order they arrive in; without it two reads of one unchanged result came back differently.

A frame is the boundary in both directions. relational/frames.py recognises a caller's table through the Arrow PyCapsule protocol without importing any dataframe library, and Result.primal hands back a polars.DataFrame, which exports the same protocol. That symmetry is what keeps pandas and pyarrow off the dependency list: they are bridges out (to_pandas, to_dataarray), shipped with the [linopy] extra, not shapes the engine holds. The bare-install CI job runs the suite with neither present.

Sinks are capped, explicitly. Today every sink expresses the same three streams and no more: cols (bounds, objective coefficients, integrality), rows, and A in COO. The upgrade path is two further streams — sos_sets and genconstr — plus a semi-continuous threshold on cols. Unlike the three that exist, those two would land unevenly, because the destinations differ per sink (see "Capability is not the ceiling"); that unevenness is what Track 4 exists to make declared rather than discovered at solve time.

Composition (component libraries)¶

A component library is a fixed set of parametrised templates agreeing on a port/flow convention, merged into one program, wired through a data connectivity table and a single group_sum balance. Topology is data, not structure — wiring a specific system is rows in a connectivity table, never generated YAML, so structure is bounded by the number of component types while cardinality lives entirely in data. Schema merge is therefore a pure compose-then-build step producing one MathSchema before a single lower/stream pass (linopy.extend is a linopy-lane shim; native merge is #30). Namespacing via qualified names is the missing primitive (#29) — the port/flow surface stays deliberately shared, as the coupling contract between templates — and signs and bidirectional flows need bounds-as-expressions (#31).

Whatever genuinely is not data (variable port counts, runtime-unknown component types) belongs in a thin layer emitting more rows or more templates, never per-instance YAML — but that layer has nothing supported to call. A seam does exist: api.load_schema accepts dict | MathSchema, so a programmatically built model already goes through validation, expansion, resolution and dim checking. It is just undocumented and unversioned, while rule 6 refuses a Python modeling API and this section forbids generated YAML. Composition therefore forces that contract earlier than the roadmap has it: not a general modeling API, but a narrow, versioned way to emit declarations. Should anything ever be blessed, it is the schema level and not the plan level — that much is settled; what is not is whether to bless the seam at all, and namespacing (#29) or a native schema merge (#30) is what would force the question.

Module map¶

Module	Role
`schema.py`	pydantic schema incl. `expressions:` / `macros:` / `piecewise:`
`expression_parser.py`, `where_parser.py`	text → core AST; grammar only, dependency-free
`expansion.py`	named-expression / macro substitution (pre-dispatch)
`resolution.py`	one flat namespace; `NameNode` → typed `Variable`/`Parameter`/`Dimension` nodes
`dimensions.py`	static dim-set checking over the resolved AST
`validation.py`	load-time: parse, expand, resolve, check everything
`piecewise.py`	`piecewise:` → λ-formulation declarations + curvature guard
`api.py`	native entry point: `check` / `solve` / `write`, linopy-free
`sources.py`	bind runtime data (parquet paths / in-memory tables) to a validated schema
`lowering.py`	core AST → logical plan (defines the relational subset)
`helpers.py`	the closed set of built-in operators: their names and call shapes — no registry
`errors.py`	the exception hierarchy; the one module the engine may import
`relational/plan.py`	frozen logical-plan dataclasses
`relational/frames.py`	the boundary — caller tables in, via the Arrow PyCapsule protocol
`relational/compiler.py`	plan → lazy frames; pure, reads nothing
`relational/chunking.py`	how a batched pass sizes its chunk: budget ÷ the width of one unit
`relational/status.py`	solve outcome on two axes; linopy's vocabulary, copied not imported
`relational/executor.py`	bind sources, assign labels, assemble the model frames
`relational/sinks/`	how a built model leaves: `lp_file`, `solver_direct` (one module each, README)
`linopy/__init__.py`	opt-in shim: `build` / `extend` on a `linopy.Model`
`linopy/loader.py`	data coercion to `xr.Dataset`, master coords
`linopy/builder.py`	eager backend: core AST → `linopy.Model`
`linopy/semantics.py`	where this lane answers linopy's v1 arithmetic convention — one home, as linopy's own `semantics.py` is

Two subpackages, and the directory is the rule in both cases. Everything under relational/ is the engine and imports nothing else from the package; everything under linopy/ is the opt-in eager lane and is the only code allowed to import linopy or xarray. tests/test_architecture.py reads membership off the path, so neither fence can be stepped over by naming a file differently.

Naming across the layers¶

The same construct passes through three layers, and each names it in full — no abbreviations, so a name never has to be decoded. The layer is the suffix, which is what keeps the three vocabularies from colliding:

Layer	Suffix	Example
YAML block (`schema.py`)	`Block`	`VariableBlock`, `PiecewiseBlock`
Core AST (`*_parser.py`)	`Node`	`VariableNode`, `DimensionComparisonNode`
Logical plan (`relational/plan.py`)	none / `Declaration`	`Variable`, `VariableDeclaration`

Two rules follow from that table, and a PR that adds a construct keeps them:

A node names the coordinate map, not a surface spelling. roll and shift are one node, so it is Translate — naming it Shift would make one of the two spellings look privileged.
Nothing is abbreviated. Cmp became ParameterComparison, vtype became variable_type. The one place abbreviation survives is frame column names inside the engine, which are not Python identifiers.

Where a concept is already linopy's, use linopy's name¶

For anything this package shares with linopy — solve statuses, result shapes, solver metrics, duals — adopt linopy's primitive: its spelling, its field names, its decomposition. Result is the envelope (status + solution + report) and Solution the raw arrays, because that is what those words mean in linopy; status / termination_condition are two axes and is_ok is the rollup, because that is linopy's model. Our audience arrives from linopy/PyPSA, and a second vocabulary for one fact is a tax on every one of them. It also keeps the oracle honest: the two lanes can be compared exactly rather than through case-folding.

Copy it; do not import it. The engine may not import linopy (rule 2), so the tables live here — and a test imports linopy and asserts the copy still matches (tests/test_solve_status.py). A copy nobody checks is a copy that rots, and the failure should be a red test rather than a user handed two dialects.

This applies to vocabulary we share. Where the design genuinely differs it stays ours: we have no Solution of dense arrays to hold, because the values are read back by joining labels to coordinates.

Extension checklists¶

Add a macro or named expression: edit YAML. Nothing else.

Add a primitive: grammar (usually free — f(x, k=v) already parses) → signature in helpers.BUILTINS (arity and which arguments name dimensions — resolution, validation and lowering all read it from there, so the shape is declared once) → eager helper → plan node + locality class → executor → lowering case → differential test on both sinks → SPEC §5/§7, and this file if structural.

Two things are deliberately not per-primitive work, because they are one implementation each: a primitive's dim rule lives only in dimensions.py — both its dim set and its verdict on an operand that lacks the dim being reduced along, which lowering asks for rather than deciding again — and the dense-label assignment that gives a coordinate its solver index lives only in PolarsExecutor._label_frame, shared by variables and constraint rows. What a lowering case still owns is what is about the plan: which node the call becomes, and the shapes that node cannot represent.