rucc_codegen/

lower.rs

//! The selector: an IR function becomes a machine IR function.
//!
//! Design: `spec/10-backend.md` sections 10.2 and 10.3.
//!
//! What the matcher in [`crate::select`] does is answer one question about one term. What this
//! does is ask it: walk a function, decide which terms are worth asking about, and build machine
//! instructions out of what comes back. Nothing here decides what an IR term lowers to. That is
//! in `rules/x86-64.rules` and it is proved before it is used, which is the whole point of the
//! arrangement and the reason this file is short.
//!
//! # What it does with an instruction
//!
//! It tries the ways the instruction can be shown to the matcher, in order, and takes the first
//! that a rule fires on. [`crate::term`] is what a way of showing one is, and the order is the
//! most specific first: an operand that is a constant is offered as a constant before it is
//! offered as a register, and an operand computed by an instruction of its own is offered as
//! that instruction before it is offered as a register. A rule that wants an immediate too wide
//! for the machine has a guard that turns it down, and the search carries on to the way of
//! showing it that puts the constant in a register, which is the right answer and is one nobody
//! had to write down.
//!
//! A constant is not lowered where it is written. It is materialized where a register for it is
//! first wanted, which is what keeps a constant that every use folded into an immediate from
//! leaving a dead instruction behind, and it also gives the value the shortest live range it
//! could have. The instruction that materializes it comes from the rule set like everything else.
//!
//! # What it does not do yet
//!
//! Everything is in the general purpose registers, because every rule in the set is about an
//! integer, so a call that passes a `double` and a function that returns one are both reported
//! rather than lowered. So is an argument that travels on the stack, on either side of a call,
//! and so is a call through an address rather than to a name.
//!
//! # A call
//!
//! Not a rule, because a rule pattern sees one term and what a call's operands are is whatever
//! the signature made them. [`crate::abi`] builds one instead, out of the same description of the
//! convention the arguments come from: the values it passes are reads constrained to the
//! registers the convention places them in, what comes back is a write constrained to the
//! register it comes back in, and every other register the callee is free to destroy is a write
//! of that register and nothing else, which is all the allocator needs to keep a value out of it.
//!
//! What that costs the frame is an argument area, and nothing after selection could work out how
//! big, so the size of the widest call is given back with the function. A function that makes no
//! call at all is a leaf, and a leaf is the function that may use the red zone.
//!
//! # Where a block goes
//!
//! On the block, which is what machine IR does with an edge and is why the branches need no more
//! rule language than the arithmetic did. A rule never names a block, so an unconditional jump
//! has no rule at all and a conditional branch has one that is about its condition and nothing
//! else. The arms are copied across after the block is filled, arguments and all, because an
//! argument that is a constant is materialized where a register for it is first wanted and the
//! end of the block is where an edge wants it.
//!
//! What this leaves behind is a function whose blocks are in the order the IR held them and whose
//! branches are still branches on a register. Turning one into a `test` and a `jcc` is the block
//! layout's, since which of the two arms falls through is the layout's answer, and [`crate::split`]
//! has to run before allocation so that every edge carrying a value has somewhere to put it.
//!
//! A store and a return are the two things here that write no register. A store is emitted like
//! everything else and the only difference is that there is no result to put anywhere, so the
//! operands the target describes are all reads. A return is the same, and what it is for is its
//! one operand: the target constrains it to the register the caller reads the value out of, and
//! the allocator is what gets it there. The instruction that leaves is not chosen here at all,
//! because the epilogue has to give the frame back first and [`crate::finish`] writes that after
//! allocation, so a return of nothing is lowered to nothing.
//!
//! The entry block is the one block whose parameters are not block parameters here. They are the
//! function's arguments, they are already somewhere when it starts, and [`crate::abi`] is what
//! says where. An argument that arrives on the stack is reported rather than read, because where
//! the stack put it is a distance into a frame and no frame exists until after allocation.
//!
//! Blocks are walked in the order the function holds them and a value is expected to be defined
//! before it is used, which is true of the IR this is given because every pass before it keeps
//! definitions ahead of uses.

use std::collections::{HashMap, HashSet};
use std::fmt;

use rucc_base::{Interner, Symbol};
use rucc_diag::Span;
use rucc_ir::{
    Abi, AsmOperand, AsmOperands, AttrSet, Block, Def, Extra, Flags, FloatPred, Func, Inst,
    Linkage, MemOrder, Opcode, Param, PrefetchHint, RmwOp, Type, Value, Visibility,
};
use rucc_mir as mir;
use rucc_target::template::{template_name, template_reg};
use rucc_target::{
    Address, CallRegs, Constraint, OperandDesc, PhysReg, RegClass, Role, VaList, Variadic,
};
use rucc_target::{aarch64, x86_64};

use crate::abi::{self, Missing, Refused};
use crate::coverage::Fired;
use crate::elsewhere::Elsewhere;
use crate::frame::{Layout, Local};
use crate::select::{Match, Piece, Pointer, Reach, Rule, Selector};
use crate::term::{MAX_ARGS, PLAIN, Plan, Shown, Term, Terms};
use crate::varargs;

/// The instruction a template's `jmp` to a name outside it becomes.
///
/// The same instruction [`x86_64::FRAME`] names for the end of a tail call, named here as well
/// because what reaches this one is a template in a function with no prologue and no epilogue,
/// which is nothing to do with the frame.
/// See [`x86_64::Step::Away`].
const AWAY: &str = "jmp_away";

/// How wide an address is on this target, which is the width a cast between a pointer and an
/// integer has to be at for the cast to be nothing.
const ADDRESS_BITS: u32 = 64;

/// How much of a register an operand of an `asm` statement fills, which is the width of its type
/// with two exceptions. A pointer is an address, and a truth value is the byte it is stored in: a
/// program that writes `sete %0` into a `_Bool` is asking for exactly that byte, which is what tcc's
/// own test of the width of one checks.
fn held_bits(ty: Type) -> u32 {
    if ty.is_ptr() {
        ADDRESS_BITS
    } else if ty.bits() == 1 {
        8
    } else {
        ty.bits()
    }
}

/// How many bytes a `long double` takes in memory, and what it is aligned to, which are the same
/// number and are both more than the ten bytes that mean anything.
///
/// The psABI's answer rather than a choice here. `sizeof (long double)` is sixteen on this
/// machine, so an array of them is laid out this way whatever a slot holding one does, and a slot
/// that agreed with the array is one fewer thing to get wrong.
const X87_BYTES: u32 = 16;

/// How many values the x87 stack holds at once.
///
/// Eight, which is the machine's number rather than a choice here, and it matters in one place:
/// the parameters of a block are copied through the stack so that they all move at once, and a
/// block with more of them than this has nowhere to put the ninth.
const X87_DEPTH: usize = 8;

/// How far into the buffer of a `__builtin_setjmp` each of the four words it writes is.
///
/// The first three are gcc's, measured against gcc 16.2.0 on x86-64 at `-O0`: the frame pointer,
/// the address control comes back to, and the stack pointer, in that order. The fourth is this
/// compiler's own. gcc has no word for the answer because it writes a second block that sets the
/// answer to one and is arrived at from the restore, and this writes the answer through memory
/// instead, for the reason [`Lowering::saves_place`] gives.
///
/// None of the four is an interface. The buffer is the program's memory and its five words are
/// the front end's promise about how much of it there is, but nothing except the matching restore
/// ever reads a word of it, and a buffer written by one compiler was never going to be one another
/// compiler could come back through.
const JUMP_FRAME: i32 = 0;

/// Where the address control comes back to is. See [`JUMP_FRAME`].
const JUMP_PC: i32 = 8;

/// Where the stack pointer is. See [`JUMP_FRAME`].
const JUMP_STACK: i32 = 16;

/// Where the address of the word the answer arrives in is. See [`JUMP_FRAME`].
const JUMP_ANSWER: i32 = 24;

/// How many bytes the word a `__builtin_setjmp` answers with takes in the frame, and what it is
/// aligned to, which are the same number because it is one machine word.
const JUMP_WORD: u32 = 8;

/// How many registers the restore needs to hold things in while it puts the frame back.
///
/// Four, and every one of them is a register nothing else in the function may be in, which is why
/// they are counted here rather than asked for one at a time. See [`Lowering::comes_back`].
const JUMP_REGS: usize = 4;

/// How many bytes the block `__builtin_apply_args` answers takes, which is a word for where the
/// arguments in memory are, a word of nothing and then the register save area of a variadic
/// function. See [`Lowering::save_arguments`].
const APPLY_ARGS: u32 = 192;

/// How far into that block the registers start, which is how far the save area has moved up.
const APPLY_REGS: u32 = 16;

/// How many bytes the block `__builtin_apply` answers takes, which is two words and two vectors.
const APPLY_BACK: u32 = 48;

/// How many bytes a value passes through on its way between a register and the x87 stack.
///
/// Eight, because the widest thing that crosses is a `double` or a sixty four bit integer, and
/// nothing crosses at eighty bits: a value that wide is already in the frame and the stack reaches
/// it where it is.
const X87_CROSSING: u32 = 8;

/// Where the rounding field of the x87 control word is and what it has to be set to for the unit
/// to cut towards zero, which is the one rounding C asks for that the unit does not do by default.
///
/// Both bits on is truncate. The field is ORed into the word that was already there rather than
/// written over it, so the precision control and the exception masks somebody else set stay set.
const X87_TRUNCATE: i64 = 0x0c00;

/// Whether a type is the one this machine has no register for.
///
/// Only the eighty bit float is, and that is a fact about x86-64 rather than about floats: every
/// other scalar the front end produces is in a general purpose register or a vector one, and this
/// one is on the x87 stack while it is being worked on and in memory the rest of the time. So it
/// has no place in [`Lowering::class_of`] and no name in [`crate::term`], and every instruction
/// that touches one is written out by hand in this file.
fn on_x87(ty: Type) -> bool {
    ty.is_scalar() && ty.is_float() && ty.bits() == 80
}

/// Where one operand of an assembly statement is, on each side of the assembly.
///
/// Two registers rather than one, because an operand written `+` is a value that arrives and a
/// value that leaves and those are two values. The machine IR has one definition per register by
/// construction, so an instruction of the template that reads the operand and writes it has to name
/// a different register in each place, and what makes the two one register in the end is the
/// [`Constraint::Reuse`] the instruction's description carries: the allocator reads it, gives both
/// the same physical register, and copies the incoming value somewhere first when something else is
/// still using it.
///
/// Most operands have one of the two. An input has only a place it is read from and an output
/// written `=` has only a place it is written to, and asking either of them for the other is an
/// operand read where the opcode writes or written where it reads, which [`Lowering::placed`]
/// refuses.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
struct Place {
    /// The register the value arrives in, for an operand something reads.
    read: Option<mir::Reg>,
    /// The register the value leaves in, for an operand something writes.
    write: Option<mir::Reg>,
}

/// Whether that operand of the statement is one the assembly may read, and so where a read of it
/// gets its value from.
///
/// [`bound`] asks this question of an operand a constraint letter named and this asks it of one the
/// template numbered, which is the same question twice because a two-address instruction reaches
/// its first source both ways. `mulq %3` reaches `rax` by the letter on the output and libgmp says
/// what is in it with `"%0"` on an input. `addq %5,%q1` reaches its first source by numbering the
/// output, and libgmp says what is in it with `"0"` on an input in the same way.
///
/// So an output written `=` has no value of its own and is still readable when an input is tied to
/// it, and the value the read wants is that input's. An output written `+` carries its own value
/// and answers with that. An output nothing is tied to answers `None`, which is a program that told
/// the compiler the assembly only writes the operand while the instruction reads it before it
/// writes it, and is refused where it is asked.
fn read_as(list: &[AsmOperand<'_>], index: usize) -> Option<Value> {
    let operand = list.get(index)?;
    if operand.value.is_some() {
        return operand.value;
    }
    operand.result?;
    list.iter().find(|entry| entry.tied == Some(index)).and_then(|entry| entry.value)
}

/// Which of an assembly statement's operands is in that register, for an instruction that reaches
/// the register without its text saying so.
///
/// The constraint is what says so, and it is the only thing in such a statement that could:
/// `"=a"` is an output in `rax`, `"c"` is an input in `rcx`, an operand that is a local register
/// variable is in the register its declaration named, and a register nothing names is a register
/// nobody has said anything about. So a write looks among the outputs and a read among the inputs,
/// and an output written `+` answers for either, since it is read before it is written. See
/// [`pinned`], which is the one question asked of both ways of saying it.
///
/// The other way a read of such a register is said is a matching constraint. `"=a"` on an output
/// and `"0"` on an input is the program saying that one register holds the input on the way in and
/// the output on the way out, and it is how a statement fills a register the instruction reads and
/// writes without writing the register down twice. The letter is on the output, which has no value
/// to read, and the value is on the input, which has no letter, and the answer is the output: its
/// place is read out of the register the input arrived in, and in a template with a loop in it the
/// place moves on to wherever the last write left it, which is what a read on the next time round
/// wants. tcc steps a pointer along a string with `lodsb` and `"=&S"` tied to `"0"`, and a read of
/// the input would start the string again every time round.
///
/// And a read of a register an output alone is in is a read of that output, the same as a read of
/// an output the template numbered. tcc copies a string with `lodsb` and `stosb` and `"=&a"` on an
/// output nothing is tied to, and what `stosb` stores is what `lodsb` loaded one line up, which is
/// the output as the template left it rather than anything the statement handed in.
///
/// `None` is a register the instruction uses and the statement put nothing in, which is the usual
/// answer rather than an unusual one. `cpuid` writes four registers and a program that wanted one
/// of them names one. See [`Lowering::spare`], which is where that one goes.
fn bound(list: &[AsmOperand<'_>], reg: PhysReg, role: Role) -> Option<usize> {
    let output =
        list.iter().position(|operand| operand.result.is_some() && pinned(operand) == Some(reg));
    if role.is_def() {
        return output;
    }
    // The output first when something is in it on the way in, which is what `+` and a matching
    // constraint both say, since its place is where a write earlier in the template left it and
    // the read wants that. See [`read_as`] for what it holds before anything wrote it.
    let arrives = |at: usize| read_as(list, at).is_some();
    if let Some(at) = output.filter(|&at| arrives(at)) {
        return Some(at);
    }
    let named = list.iter().position(|operand| {
        operand.result.is_none() && operand.value.is_some() && pinned(operand) == Some(reg)
    });
    named.or(output)
}

/// The register one of an assembly statement's operands is in, whichever of the two ways said it.
///
/// A constraint letter is one way and is the only way a program can say one of the six registers
/// that have a letter. A local register variable is the other, and it is the only way to say any
/// of the rest: there is no letter for `r12`, which is the whole reason the extension exists, so
/// the declaration says it and the front end wrote the name into the constraint. The name is read
/// against this machine's table here, the same place the letter is read against it, and a name the
/// machine has not got answers nothing, which leaves the operand where an operand nobody placed
/// goes.
///
/// The sigil gcc allows in front of a name is taken off here, because what a name is written with
/// is syntax and which register it means is this question.
fn pinned(operand: &AsmOperand<'_>) -> Option<PhysReg> {
    match operand.named {
        Some(name) => {
            let (reg, _) = x86_64::gpr_named(name.strip_prefix('%').unwrap_or(name))?;
            Some(reg)
        }
        None => operand.fixed.and_then(x86_64::gpr_letter),
    }
}

/// Whether a constraint says nothing but what it says on every machine.
///
/// [`AsmOperands::read`] gives the x86 meaning to every letter it knows, and most of the letters
/// mean something else on AArch64: `Q` is an address in one register there rather than one of four
/// registers, and `a` to `d` name nothing. So an AArch64 statement is taken only with the letters
/// the two agree on, which are a register, a constant, memory, the immediate ranges and a matching
/// number, and anything else is refused rather than read as x86. `w` and `Q` are the exceptions.
/// `w` is a register on both, and which file it is in is decided by the caller with
/// [`vector_letter`]. `Q` is read as `m` by the caller before the list is read. A
/// register the front end named in braces is read against AArch64's own names, so what is inside
/// them is not a letter.
fn shared_letters(constraint: &str) -> bool {
    let mut inside = false;
    constraint.chars().all(|c| match c {
        '{' => {
            inside = true;
            true
        }
        '}' => {
            inside = false;
            true
        }
        _ if inside => true,
        _ => matches!(
            c,
            '=' | '+' | '&' | '%' | 'r' | 'w' | 'Q' | 'm' | 'o' | 'V' | 'g' | 'X' | 'i' | 'n'
                | 'p' | 'I'..='N' | '0'..='9'
        ),
    })
}

/// A constraint list with every letter outside braces put through `swap`, and what is inside them,
/// which is a register's name rather than letters, left alone.
fn letters_outside(constraints: &str, swap: impl Fn(char) -> char) -> String {
    let mut inside = false;
    constraints
        .chars()
        .map(|c| {
            match c {
                '{' => inside = true,
                '}' => inside = false,
                _ if !inside => return swap(c),
                _ => {}
            }
            c
        })
        .collect()
}

/// Whether an AArch64 constraint asks for a floating point or vector register, which is what `w`
/// means there. A register named in braces is not a letter, so a `w` inside one is not read.
fn vector_letter(constraint: &str) -> bool {
    let mut inside = false;
    constraint.chars().any(|c| {
        match c {
            '{' => inside = true,
            '}' => inside = false,
            _ => {}
        }
        !inside && c == 'w'
    })
}

/// The x86-64 vector register one entry of a clobber list names, spelled `xmm0` or `ymm0` with or
/// without the sigil, or nothing for any other entry. Only the sixteen there are without AVX-512,
/// so `zmm0` and `xmm16` are still refused as names this has no register for.
fn vector_named(entry: &str) -> Option<PhysReg> {
    let entry = entry.trim().trim_matches('"');
    let entry = entry.strip_prefix('%').unwrap_or(entry);
    let number = entry.strip_prefix("xmm").or_else(|| entry.strip_prefix("ymm"))?;
    if number.len() > 1 && number.starts_with('0') {
        return None;
    }
    let number: u8 = number.parse().ok()?;
    (number < 16).then(|| x86_64::xmm(number))
}

/// Whether a line of a template names, by number, an operand `wanted` says yes to.
///
/// `%%` is a percent sign rather than an operand, and a modifier letter may stand between the sign
/// and the number.
fn names_one(line: &str, wanted: impl Fn(usize) -> bool) -> bool {
    let mut rest = line;
    while let Some(at) = rest.find('%') {
        let after = &rest[at + 1..];
        if let Some(escaped) = after.strip_prefix('%') {
            rest = escaped;
            continue;
        }
        let after = after.strip_prefix(|c: char| c.is_ascii_alphabetic()).unwrap_or(after);
        let digits = after.len() - after.trim_start_matches(|c: char| c.is_ascii_digit()).len();
        if after[..digits].parse().is_ok_and(&wanted) {
            return true;
        }
        rest = &after[digits..];
    }
    false
}

/// Why a function could not be lowered.
///
/// One reason and then nothing. A function with no rule for something in it is a function this
/// cannot finish, and the second thing it could not lower is not news.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Unsupported {
    /// An instruction no rule fires on.
    Inst {
        /// The instruction that stopped it.
        inst: Inst,
        /// What the rule file would call it, or nothing if the rule language has no name for it
        /// at all, which is what an instruction at a width nothing is written about looks like.
        term: Option<&'static str>,
        /// The opcode, which is what gets named when the rule language has no word for it.
        ///
        /// An opcode the rule language has no word for is exactly the opcode no rule lowers, so
        /// without this the message would be empty in every case where somebody needs it.
        opcode: Opcode,
        /// What it produces, or nothing for an instruction that is only an effect.
        ty: Option<Type>,
    },
    /// A parameter that does not arrive somewhere this can bring it in from.
    ///
    /// Not an instruction, which is why it is a separate arm: it is a fact about the signature
    /// and there is nothing in the body of the function to point at.
    Argument {
        /// Its position in the signature.
        index: usize,
        /// What is wrong with where it arrives.
        missing: Missing,
    },
    /// A call that passes or gives back a value this cannot put where the convention wants it.
    Call {
        /// The call.
        inst: Inst,
        /// Which value, and what is wrong with where it travels.
        refused: Refused,
    },
    /// A `return` this cannot put where the convention wants it.
    ///
    /// A separate arm from [`Unsupported::Inst`] because it is not an instruction no rule fires
    /// on. A return of more than one value is built from the convention rather than matched, the
    /// same way a call is, so what goes wrong with one is what goes wrong with a call and not the
    /// absence of a rule.
    Returned {
        /// The `return`.
        inst: Inst,
        /// What is wrong with where one of the values travels.
        missing: Missing,
    },
    /// A stack slot the frame cannot give the bytes it asked for.
    ///
    /// Not an instruction no rule covers. An `alloca` is built here rather than matched, so what
    /// goes wrong with one is what the frame can and cannot hold rather than what the rules spell.
    Dynamic {
        /// The `alloca`.
        inst: Inst,
        /// What the frame could not do about it.
        growing: Growing,
    },
    /// More parameters of a type that travels on the x87 stack than the stack is deep.
    ///
    /// Not an instruction either, for the reason a function's parameter is not one: it is a fact
    /// about the block and there is nothing in the block to point at. What crosses an edge for one
    /// of these is the address of where the value is, and the block copies the bytes into a slot
    /// of its own, all of them through the stack at once so that a block carrying two of them
    /// swapped is copied in an order that is right. Eight is as many as the stack holds, and a
    /// ninth would have to be copied before or after the rest, which is the order that could be
    /// wrong.
    Phi {
        /// Which block it arrives at.
        block: Block,
        /// How many of them arrive there, which is the whole of what is wrong.
        count: usize,
        /// What they are.
        ty: Type,
    },
    /// An `asm` statement this cannot build.
    ///
    /// Not an instruction no rule fires on, for the reason a call is not one: what it stands for is
    /// whatever its template says, and no pattern over terms can read a string.
    Assembly {
        /// The `inline_asm`.
        inst: Inst,
        /// What about it is not built here yet.
        refused: Written,
    },
    /// A `register long x asm ("...")` naming something this machine has not got.
    ///
    /// Not an instruction no rule fires on. There is a rule's worth of instruction here and what
    /// is wrong is the string beside it, which is a name rather than a term, so the message says
    /// the name. Which names a machine has is the machine's own question and this is where it is
    /// asked, at the table a clobber list is read against.
    Register {
        /// The `register_value`.
        inst: Inst,
        /// The name the program wrote, as it wrote it.
        name: String,
    },
    /// A naked function whose frame is not empty.
    ///
    /// Not an instruction no rule fires on, and there is nothing in the body to point at: the
    /// function asked for no prologue and then wanted bytes only a prologue takes. Refused rather
    /// than given the bytes anyway, because an offset into a frame nothing set up reaches into
    /// whatever the caller left below its own stack pointer, which is wrong code that assembles.
    /// See [`crate::frame::Layout::naked`].
    Naked {
        /// How many bytes it wanted, which is the whole of what is wrong.
        bytes: u32,
    },
    /// Something the x86-64 lowering writes by hand and nothing has written for this machine yet.
    ///
    /// Refused rather than written with the x86 instructions, which is what the walk would do
    /// otherwise, since these are the places it names them itself.
    Unported {
        /// The instruction, or nothing for the one that is about a signature.
        inst: Option<Inst>,
        /// Which of them.
        what: Unported,
    },
}

/// What [`Unsupported::Unported`] is about.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Unported {
    /// The thread pointer on Apple's platforms, which keep it somewhere other than Linux does.
    Thread,
}

impl Unported {
    /// The whole message, since there is nothing to put in front of it.
    #[must_use]
    pub fn why(self) -> &'static str {
        match self {
            Unported::Thread => "the thread pointer is not written for this platform yet",
        }
    }
}

/// What about an `asm` statement is not built yet.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Written {
    /// A template with instructions in it.
    Template,
    /// An `asm goto`, whose labels make the statement a terminator.
    Goto,
    /// An operand this cannot put where the constraint says it goes.
    Operand,
    /// A clobber list naming something this has no register for.
    Clobber,
    /// A `jmp` out of the function in a function that has an epilogue behind it.
    Away,
}

impl Written {
    /// The rest of the sentence that starts with the statement.
    #[must_use]
    pub fn why(self) -> &'static str {
        match self {
            // The template is the assembler's to read and there is no assembler here yet, so a
            // template with anything in it is a string nothing can turn into bytes. An empty one is
            // no instructions, and no instructions is something this can write.
            Written::Template => "has instructions in its template, which nothing here assembles",
            Written::Goto => "jumps to a label, which nothing here builds an edge for",
            Written::Operand => "has an operand this cannot place",
            Written::Clobber => "says it destroys a register this has no name for",
            Written::Away => {
                "jumps out of the function, which only a function that is `naked` may do, since \
                 anywhere else there is an epilogue behind it to give the frame back"
            }
        }
    }
}

/// What the frame could not do about a stack slot.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Growing {
    /// An object of a size the number a frame counts bytes in does not reach.
    Huge,
    /// A variable length array wanting more alignment than a call leaves the stack pointer with.
    ///
    /// Rounding the stack pointer down again after the bytes have been taken would put it
    /// somewhere no constant reaches the rest of the frame from, so a frame like this needs a
    /// second base register held for the whole of the function. Nothing here holds one.
    ///
    /// [`crate::expand::rounds`] takes the array away before this sees it, by asking for the
    /// alignment in extra bytes and handing out an address inside them, so what is left of this
    /// is IR that arrived without going through that pass and the fixed local in
    /// [`crate::pipeline`] that wants the same thing from the other side.
    Aligned,
    /// A variable length array in a function written without a prologue.
    ///
    /// A frame that grows is reached from a frame pointer, and establishing one is the first two
    /// instructions of a prologue that `__attribute__((naked))` asked there be none of. See
    /// [`crate::frame::Layout::naked`].
    Naked,
}

impl Growing {
    /// The rest of the sentence that starts with the slot.
    #[must_use]
    pub fn why(self) -> &'static str {
        match self {
            Growing::Huge => "is more bytes than a frame counts",
            Growing::Aligned => {
                "wants more alignment than the stack pointer is left on, which needs a base \
                 register nothing here keeps"
            }
            Growing::Naked => {
                "is in a function that is `naked`, which has no prologue to point a frame pointer \
                 at it with"
            }
        }
    }
}

impl Unsupported {
    /// The instruction it is about, or nothing for the one arm that is about a signature.
    ///
    /// What a caller wants this for is the span. The function knows where every instruction in
    /// it came from, so a caller holding both can point a message at the line somebody wrote
    /// rather than at the file as a whole, and nothing here has to carry a span of its own.
    pub fn inst(&self) -> Option<Inst> {
        match *self {
            Unsupported::Inst { inst, .. }
            | Unsupported::Call { inst, .. }
            | Unsupported::Returned { inst, .. }
            | Unsupported::Dynamic { inst, .. }
            | Unsupported::Assembly { inst, .. }
            | Unsupported::Register { inst, .. } => Some(inst),
            Unsupported::Unported { inst, .. } => inst,
            Unsupported::Argument { .. } | Unsupported::Phi { .. } | Unsupported::Naked { .. } => {
                None
            }
        }
    }
}

impl fmt::Display for Unsupported {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match *self {
            Unsupported::Inst { term: Some(term), .. } => write!(f, "no rule lowers `{term}`"),
            Unsupported::Inst { term: None, opcode, ty: Some(ty), .. } => {
                write!(f, "no rule lowers a `{opcode}` producing a `{ty}`")
            }
            Unsupported::Inst { term: None, opcode, ty: None, .. } => {
                write!(f, "no rule lowers a `{opcode}`")
            }
            Unsupported::Argument { index, missing } => {
                write!(f, "parameter {index} {}", missing.why())
            }
            Unsupported::Call { refused: Refused { argument: Some(index), missing }, .. } => {
                write!(f, "argument {index} of this call {}", missing.why())
            }
            Unsupported::Call { refused: Refused { argument: None, missing }, .. } => {
                write!(f, "what this call gives back {}", missing.why())
            }
            Unsupported::Returned { missing, .. } => {
                write!(f, "what this function gives back {}", missing.why())
            }
            Unsupported::Dynamic { growing, .. } => {
                write!(f, "this local {}", growing.why())
            }
            Unsupported::Phi { block, count, ty } => {
                let block = block.index();
                write!(
                    f,
                    "block{block} takes {count} parameters of type `{ty}` and only {X87_DEPTH} can cross an edge at once"
                )
            }
            Unsupported::Assembly { refused, .. } => write!(f, "this `asm` {}", refused.why()),
            Unsupported::Unported { what, .. } => f.write_str(what.why()),
            Unsupported::Register { ref name, .. } => {
                write!(
                    f,
                    "this object is kept in `{name}`, which is not a register this machine has"
                )
            }
            Unsupported::Naked { bytes } => write!(
                f,
                "this function is `naked` and wants {bytes} bytes of frame, which there is no prologue to take"
            ),
        }
    }
}

impl std::error::Error for Unsupported {}

/// A lowered function, and what the frame needs that the machine IR does not hold.
#[derive(Debug)]
pub struct Lowered {
    /// The function, in machine instructions.
    pub func: mir::Func,
    /// What it wants its stack to look like, which is separate from the function so that the two
    /// can be read and written at the same time.
    pub stack: Stack,
    /// Which rules of the table lowered it, which is what `-Zrule-coverage` asks for and what
    /// `crate::coverage` writes down.
    pub fired: Fired,
    /// Which machine IR block each IR block became, indexed by the IR block's own index, and
    /// nothing for a block the walk never reached.
    ///
    /// Here because it is the only place the correspondence exists. Selection makes one block per
    /// block, in the same order and with the arms in the same order, so anything the IR knows
    /// about a block can be carried down through this and nothing else, and
    /// [`crate::weights::carry`] is what does.
    pub blocks: Vec<Option<mir::Block>>,
}

/// What a function's stack has to hold, as far as selection is able to say.
///
/// All of it is answered here because selection is where a call is built and where an `alloca`
/// is read, and nothing after it could tell what either of them needed.
#[derive(Debug, Default)]
pub struct Stack {
    /// How many bytes the widest call in the function needs below the stack pointer for the
    /// arguments it passes there, or `None` for a function that makes no call at all.
    ///
    /// `None` is a leaf, which is the function that may use the red zone and the one whose stack
    /// pointer does not have to be left aligned for anybody.
    pub calls: Option<u32>,
    /// The memory the function asked for itself, one entry for every `alloca` in it, in the order
    /// the walk reached them.
    pub locals: Vec<Local>,
    /// Which instruction computes the address of which of those locals.
    ///
    /// An address in the frame is a distance from the stack pointer, and there is no frame until
    /// after allocation, so the instruction is written here with nothing in its displacement and
    /// [`crate::finish`] writes the number in once [`crate::frame::Frame`] knows it.
    pub addresses: Vec<(mir::Inst, usize)>,
    /// Which of those locals is which declaration in the source, for the ones the program declared.
    ///
    /// The number is the one the IR function carries and means nothing here. What it is for is the
    /// debugging information, which has to say where a named local ended up and cannot ask the
    /// frame directly: the frame knows a local by the order the `alloca` for it was lowered in and
    /// by nothing else.
    ///
    /// Shorter than the list above rather than the same length, because most of what a function
    /// keeps in its frame is memory an expression wanted somewhere to put.
    pub declared: Vec<(usize, u32)>,
    /// Which instruction computes the address of a piece of memory whose size the function works
    /// out while it runs, which is what a variable length array is.
    ///
    /// Waiting on [`crate::finish`] for a different number from the one the addresses above are:
    /// the bytes were taken off the stack pointer by the instruction in front of this one, so where
    /// they start is however much of the bottom of the frame belongs to the arguments of a call,
    /// and that is not known until the frame is.
    pub dynamic: Vec<mir::Inst>,
    /// Which instruction takes those bytes off the stack pointer, one for every one of them, in the
    /// order the walk reached them.
    ///
    /// Read by [`crate::finish`] on a command line that asked for the stack to be touched a page at
    /// a time, which is the one thing that has to find these again: the bytes are in a register by
    /// then, so the walk down to them is a loop, and a loop is written around an instruction rather
    /// than in front of a block. Nothing else looks at them, because everything else about a frame
    /// that grows is answered by the address the instruction below this one computes.
    pub grown: Vec<mir::Inst>,
    /// Where the function first moves the stack pointer while it runs, if it does at all.
    ///
    /// Two things are read off this. One is whether at all, which is what [`crate::frame::Layout`]
    /// wants, because a frame that moves its stack pointer has a different shape from one that does
    /// not and the layout is built before the instructions are looked at again. See `Growing` in
    /// [`crate::frame`]. The other is where, so that a caller that cannot accept such a frame has
    /// somewhere to point when it says so.
    pub grown_at: Option<Inst>,
    /// Which instruction reads which of the arguments the caller passed on the stack, as how far up
    /// the caller's argument area it reads.
    ///
    /// Waiting on [`crate::finish`] for the same reason the addresses above are, and on one thing
    /// more: where the caller's argument area is from inside this function depends on whether the
    /// prologue had to force the stack pointer's alignment, so which register the load reads
    /// through is not settled here either.
    pub arguments: Vec<(mir::Inst, u32)>,
    /// Whether the function asked where its own frame is, which is what `__builtin_frame_address`
    /// and `__builtin_return_address` both start from.
    ///
    /// A function like that keeps a frame pointer whatever the flags say, because the register is
    /// the answer to the first of them and the start of the walk for every depth above zero. There
    /// is no other way to reach it: the distance from the stack pointer to the frame is a number
    /// the layout works out, and what a walk up the chain needs is the link the prologue saved.
    pub walks_frames: bool,
    /// Whether the function saved a place for a `__builtin_longjmp` to come back to, which is what
    /// `__builtin_setjmp` does.
    ///
    /// A function like that keeps a frame pointer whatever the flags say as well, and for a reason
    /// of the same shape: the two registers the restore puts back are the frame pointer and the
    /// stack pointer, and a frame that did not keep the first of them has nothing in it saying
    /// where the caller's frame is for the epilogue to find after control has come back.
    pub saves_place: bool,
    /// The calls a `tail_call` became that [`crate::tail::jumps`] may turn into a jump, which is
    /// the ones that passed everything in registers.
    pub tails: Vec<crate::tail::Tail>,
}

impl Stack {
    /// The layout given, with the three fields only the lowering knows the answer to filled in.
    ///
    /// Everything else in a layout comes from the flags the function is compiled under or from the
    /// allocation, so this takes one and returns it rather than building one.
    ///
    /// A function that saved a place is not a leaf whatever it called. What a leaf buys is the red
    /// zone, which is the words below the stack pointer nothing else may write, and a function
    /// control comes back into from a `__builtin_longjmp` has already had something else running
    /// down there: whatever it called and whatever that called, or a signal handler on the same
    /// stack. Every one of those has written over the red zone by the time control arrives, so a
    /// value this function left there would not be there any more.
    #[must_use]
    pub fn layout<'a>(&'a self, base: Layout<'a>) -> Layout<'a> {
        Layout {
            leaf: self.calls.is_none() && !self.saves_place,
            outgoing: self.calls.unwrap_or(0),
            locals: &self.locals,
            grows: self.grown_at.is_some(),
            ..base
        }
    }
}

/// The machine IR for that function, for the machine the selector describes.
///
/// # Errors
///
/// The first instruction no rule fires on, which today is anything at a width the rule set is not
/// written at, a parameter that does not arrive in a register this can read, or a call that
/// passes something this cannot put where the convention wants it.
pub fn func(
    source: &Func,
    names: &mut Interner,
    selector: &'static Selector,
    conv: &'static CallRegs,
    elsewhere: &Elsewhere,
) -> Result<Lowered, Unsupported> {
    func_for(source, names, selector, conv, elsewhere, true)
}

/// [`func`], for a build that says whether it writes debugging information. Without it the walk
/// leaves out which value each declaration holds on the way into each block, since that is read
/// only for the debugging information.
///
/// # Errors
///
/// The same as [`func`].
pub fn func_for(
    source: &Func,
    names: &mut Interner,
    selector: &'static Selector,
    conv: &'static CallRegs,
    elsewhere: &Elsewhere,
    debug: bool,
) -> Result<Lowered, Unsupported> {
    Lowering::new(source, names, selector, conv, elsewhere, debug).run()
}

/// What the matcher settled on for one block, indexed the way the block's instructions are.
struct Decided {
    /// What each instruction matched, and nothing for one that matched no rule or was folded
    /// into a later one.
    found: Vec<Option<Match<Term>>>,
    /// How each instruction showed its operands to the matcher, which is what says what it took.
    plans: Vec<Option<Plan>>,
    /// The instructions some other instruction took, which are the ones with nothing to write.
    folded: Vec<Inst>,
}

/// The instruction in front of an assignment that starts a declaration on a value, and the first
/// machine instruction after it once the block is filled.
type Mark = (Option<Inst>, Option<mir::Inst>);

/// One function being lowered.
struct Lowering<'a> {
    source: &'a Func,
    names: &'a mut Interner,
    out: mir::Func,
    /// The machine register each IR value is in, once it has one.
    regs: Vec<Option<mir::Reg>>,
    /// For a constant that has been written into a register, the block it was written into,
    /// which is the only block that register is any good in.
    written: Vec<Option<mir::Block>>,
    /// How many times each IR value is read, which is what says whether an instruction may be
    /// folded into the one that reads it.
    uses: Vec<u32>,
    /// The block being filled.
    at: Option<mir::Block>,
    /// The machine IR block each IR block became.
    blocks: Vec<Option<mir::Block>>,
    /// The class an address is in, which is the general purpose one and is not a question: every
    /// register an addressing mode names holds part of an address, and there is no machine here
    /// that computes an address anywhere but in this file. Which class a *value* is in is
    /// [`Lowering::class_of`], and it is a question, because a float is in the other one.
    gpr: RegClass,
    /// The machine this selects for.
    selector: &'static Selector,
    /// Where the convention this function is compiled for puts things, which is read for the
    /// arguments and for the calls.
    conv: &'static CallRegs,
    /// Which names this function may not work an address out for itself, which is a fact about the
    /// module and so is worked out before any of this and handed in.
    elsewhere: &'a Elsewhere,
    /// Whether the build writes debugging information, which is the one thing that reads which
    /// value a declaration holds on the way into each block.
    debug: bool,
    /// What the function wants its stack to look like, filled in as the walk finds out.
    stack: Stack,
    /// What a `va_start` in this function has to write, or nothing for a function that takes no
    /// arguments its signature does not name.
    ///
    /// Worked out once, when the entry block binds the parameters, because every number in it is
    /// about where those parameters left the walk over the argument registers and there is nowhere
    /// else that knows.
    varargs: Option<Varargs>,
    /// Which of the function's stack objects each eighty bit value lives in, once it has asked
    /// for one.
    ///
    /// One slot per value and it is never given back, which is what makes an eighty bit value
    /// behave like every other one: it is written once and read wherever it is read, and no two
    /// of them share a slot the way two of them would share a register. What is in a register is
    /// the address, and that is worked out again at every use rather than kept, so nothing here
    /// holds a general purpose register open across a whole function.
    slots: Vec<Option<usize>>,
    /// The eight bytes a value passes through between a register and the x87 stack, once
    /// something has wanted them.
    ///
    /// One for the whole function, because every group that uses it is a handful of instructions
    /// with nothing in between: the bytes are written, read straight back and never looked at
    /// again, so a second slot would be a second slot holding the same nothing.
    crossing: Option<usize>,
    /// The four bytes the control word is saved in and the changed copy written to, once
    /// something has wanted them.
    ///
    /// One for the whole function for the reason above, and four rather than two because it is
    /// two words: the one the unit had and the one with the rounding field turned to truncate.
    control: Option<usize>,
    /// The word a `__builtin_setjmp` in this function answers with, once one has asked for it.
    ///
    /// One for the whole function however many saves there are in it, because the word is written
    /// and read back with nothing in between: the save writes a zero into it and the instruction
    /// straight after reads it, and the only other thing that ever writes it is a restore arriving
    /// between those two. Two saves sharing it is two pairs each doing that, and neither can be
    /// inside the other.
    answer: Option<usize>,
    /// The block `__builtin_apply_args` answers the address of, or nothing in a function that holds
    /// none.
    ///
    /// Written once, in the prologue, because what it holds is every argument register as it was
    /// on the way in, and by the time the walk reaches the call the registers hold whatever the
    /// function has done since. Every `__builtin_apply_args` in the function answers the same one.
    applied: Option<usize>,
    /// Which rules have fired so far.
    fired: Fired,
    /// Where each assignment that starts a declaration on a value part of the way through is, by
    /// the IR block it is in and the instruction in front of it, and which machine instruction
    /// is the first one after it once the block has been filled. See
    /// [`rucc_ir::Func::declare_value_from`].
    marks: HashMap<Block, Vec<Mark>>,
    /// The frame slot each fixed size `alloca` was given, which a landing pad writes the address
    /// of again rather than reading the register the rest of the function has it in. See
    /// [`Self::pad`].
    frame_slots: HashMap<Value, usize>,
    /// The machine call each IR call with an unwind edge became, which [`Self::edges`] pairs with
    /// the pad the edge went to. See [`rucc_ir::Opcode::Unwound`].
    unwinding: HashMap<Inst, mir::Inst>,
}

/// What a `va_start` in a variadic function writes into the list it is given.
///
/// Two shapes, because two conventions describe a list two ways, and [`crate::varargs`] is where
/// both are written down. Neither is a set of numbers on its own: where the save area is and where
/// the caller's argument area is are distances into a frame that does not exist until after
/// allocation, so each is a `lea` [`crate::finish`] fills in.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Varargs {
    /// The four field list, whose two offsets are settled here and whose two addresses are not.
    Fields {
        /// Which of the function's stack objects is the register save area.
        save: usize,
        /// How far up the caller's argument area the first argument the signature does not name is,
        /// which is the whole of that area the named ones did not take.
        incoming: u32,
        /// What `gp_offset` starts at, which is past the general purpose registers the named
        /// arguments took.
        integers: u32,
        /// What `fp_offset` starts at, which is past the vector ones.
        floats: u32,
    },
    /// The AAPCS64 list, whose two offsets count up to zero from the top of each half of the save
    /// area. The two tops are addresses in the frame and so is the first field, like the SysV list.
    Aapcs {
        /// Which of the function's stack objects is the register save area.
        save: usize,
        /// How far up the caller's argument area the first argument the signature does not name is.
        incoming: u32,
        /// Where the general purpose half of the save area ends.
        integers_end: u32,
        /// Where the vector half ends, which is the end of the area.
        floats_end: u32,
        /// What `__gr_offs` starts at, which is minus the general purpose half the named arguments
        /// did not take.
        integers: i32,
        /// What `__vr_offs` starts at.
        floats: i32,
    },
    /// The list that is a pointer, which is the one address and nothing else.
    Pointer {
        /// How far up the caller's argument area the first argument the signature does not name is,
        /// which on this convention is the word belonging to the position the named ones stopped
        /// at.
        incoming: u32,
    },
}

/// How far a function's name reaches, narrowed from the linkage the IR gave it.
///
/// The IR has five and an object file says three, and the two the linker cannot tell apart are
/// the two weak ones: which of them a symbol had is a fact the optimizer reads and the linker has
/// no way to record. A function is never `Common`, since that is what a tentative definition of an
/// object is and there is no tentative definition of a function, and it is written here rather
/// than left out so that a linkage added later has to come past this.
const fn binding(linkage: Linkage) -> mir::Binding {
    match linkage {
        Linkage::Internal => mir::Binding::Local,
        Linkage::Weak | Linkage::LinkOnce => mir::Binding::Weak,
        Linkage::External | Linkage::Common => mir::Binding::Global,
    }
}

/// How far a function's name reaches outside a shared library, carried across unchanged.
///
/// Nothing is narrowed here the way [`binding`] narrows the linkage, because ELF records all
/// three of these and the two enumerations are the same three answers written twice: once in a
/// crate that is not allowed to know what an object file is and once in one that is.
const fn visibility(visibility: Visibility) -> mir::Visibility {
    match visibility {
        Visibility::Default => mir::Visibility::Default,
        Visibility::Hidden => mir::Visibility::Hidden,
        Visibility::Protected => mir::Visibility::Protected,
    }
}

impl<'a> Lowering<'a> {
    fn new(
        source: &'a Func,
        names: &'a mut Interner,
        selector: &'static Selector,
        conv: &'static CallRegs,
        elsewhere: &'a Elsewhere,
        debug: bool,
    ) -> Self {
        let counts = source.counts();
        let name = source.name;
        let mut uses = vec![0; counts.values];
        for block in source.blocks() {
            for inst in source.insts(block) {
                for &arg in &source[source[inst].args] {
                    uses[arg.index()] += 1;
                }
                for call in source.successors(inst) {
                    for &arg in &source[call.args] {
                        uses[arg.index()] += 1;
                    }
                }
            }
        }
        let mut out = mir::Func::new(name);
        out.align = source.align;
        // Carried rather than worked out here, because where a function was declared is a fact
        // about the source and this is a long way past it. What wants it is the line table.
        out.declared = source.declared;
        out.binding = binding(source.linkage);
        out.visibility = visibility(source.visibility);
        Self {
            source,
            names,
            out,
            regs: vec![None; counts.values],
            written: vec![None; counts.values],
            blocks: vec![None; counts.blocks],
            uses,
            at: None,
            gpr: selector.gpr,
            selector,
            conv,
            elsewhere,
            debug,
            stack: Stack::default(),
            varargs: None,
            slots: vec![None; counts.values],
            crossing: None,
            control: None,
            answer: None,
            applied: None,
            fired: Fired::new(),
            marks: HashMap::new(),
            frame_slots: HashMap::new(),
            unwinding: HashMap::new(),
        }
    }

    fn run(mut self) -> Result<Lowered, Unsupported> {
        for value in self.source.values() {
            for start in self.source.value_starts(value) {
                let Some((block, after)) = self.source.start_place(start) else { continue };
                let marks = self.marks.entry(block).or_default();
                if !marks.iter().any(|&(have, _)| have == after) {
                    marks.push((after, None));
                }
            }
        }
        // Every block before any of them is filled, because a block that jumps forward has to
        // name the block it jumps to and a machine IR block is named by a handle rather than by
        // the IR block it came from.
        for block in self.source.blocks() {
            let out = self.out.create_block();
            self.blocks[block.index()] = Some(out);
        }
        for block in self.order() {
            self.block(block)?;
        }
        // And the name each block an image holds the address of was given, which nothing in the
        // walk above would ask for: the `lea` a label address is inside the function needs no
        // symbol, and the one thing that does is a relocation in another section.
        let named: Vec<(Block, Symbol)> = self.source.named_blocks().collect();
        let labels: Vec<(mir::Block, Symbol)> =
            named.into_iter().map(|(block, name)| (self.out_block(block), name)).collect();
        self.out.labels = labels;
        self.naming();
        Ok(Lowered { func: self.out, stack: self.stack, fired: self.fired, blocks: self.blocks })
    }

    /// Which register each declaration the front end kept in a value ended up in, as far as this
    /// walk can say, which is the other half of what [`Lowering::new_reg`] writes down as it goes.
    ///
    /// Two halves because there are two ways a value gets a register here. Most of them ask for a
    /// fresh one and that is where `new_reg` catches them, and the rest are put in a register
    /// something else chose: a parameter arrives in whichever one the convention handed it, a block
    /// parameter in whichever one the edge agreed on, and a result of a rule that names its own
    /// registers in the one the rule named. None of those goes past the mint, so this is the map at
    /// the end read off the other side, and the two together are every value a declaration is
    /// behind.
    ///
    /// The map on its own would not do, which is why `new_reg` writes down what it writes down: the
    /// entry for a constant is cleared every time the walk leaves the block that wrote it, so a
    /// local a constant holds is in the map for one block of the function and nowhere else.
    fn naming(&mut self) {
        let mut named = std::mem::take(&mut self.out.named);
        for value in self.source.values() {
            let Some(reg) = self.regs[value.index()] else { continue };
            named.extend(self.source.value_decls(value).map(|decl| (decl, reg)));
            // A start in a block a pass took out was never reached above, and it says nothing
            // rather than something about another place.
            for start in self.source.value_starts(value) {
                let Some((block, after)) = self.source.start_place(start) else { continue };
                let first = self.marks.get(&block).and_then(|marks| {
                    marks.iter().find(|&&(have, _)| have == after).and_then(|&(_, at)| at)
                });
                if let Some(first) = first {
                    self.out.starts.push((start.decl, reg, first));
                }
            }
        }
        named.sort_unstable();
        named.dedup();
        self.out.named = named;
        self.out.starts.sort_unstable();
        self.out.starts.dedup();
        // Which of its values a declaration holds on the way into a block, for the blocks where
        // two of them are live at once. A block a pass took out says nothing, and neither does a
        // value the map above has lost the register of, since that is not the same as having none.
        // Only for a build that writes debugging information, since that is all that reads it,
        // and on a function of tens of thousands of blocks it is a walk of all of them for every
        // local.
        let mut entries = Vec::new();
        let held = if self.debug { crate::holding::on_entry(self.source) } else { Vec::new() };
        for (decl, block, value) in held {
            if let (Some(block), Some(reg)) = (self.blocks[block.index()], self.regs[value.index()])
            {
                entries.push((decl, block, reg));
            }
        }
        entries.sort_unstable();
        entries.dedup();
        self.out.entries = entries;
    }

    /// The order the blocks are filled in, which is not the order they are written in.
    ///
    /// Reverse postorder, because a value is written in a block that dominates every block that
    /// reads it and a block in reverse postorder comes before every block it dominates. The order
    /// the blocks are written in does not have that property: a block written early can read a
    /// value a block below it writes, and reading a value with no register yet mints one, so the
    /// register the definition writes later is not the register the read named. Nothing writes the
    /// one the read named, and what comes out is a function that loads a stack slot no store ever
    /// reached. It is the order this walk goes in rather than the order the blocks come out in,
    /// which is what the loop above fixes, so the machine function is still written the way the IR
    /// function was.
    ///
    /// Blocks the entry does not reach come last, in the order they are written in. Nothing runs
    /// them and nothing they name is read by anything that does, but they still have to be filled,
    /// because a machine block with no terminator is not one the passes below can read.
    fn order(&self) -> Vec<Block> {
        let Some(entry) = self.source.entry() else { return self.source.blocks().collect() };
        let count = self.blocks.len();
        let mut succs: Vec<Vec<Block>> = vec![Vec::new(); count];
        for block in self.source.blocks() {
            let Some(term) = self.source.terminator(block) else { continue };
            succs[block.index()] = self.source.successors(term).map(|call| call.block).collect();
        }
        // An explicit stack, because the depth of the walk is the number of blocks and a function
        // built by a generator has as many of those as it likes.
        let mut seen = vec![false; count];
        let mut order = Vec::with_capacity(count);
        let mut stack = vec![(entry, 0usize)];
        seen[entry.index()] = true;
        while let Some((block, at)) = stack.pop() {
            let Some(&next) = succs[block.index()].get(at) else {
                order.push(block);
                continue;
            };
            stack.push((block, at + 1));
            if !seen[next.index()] {
                seen[next.index()] = true;
                stack.push((next, 0));
            }
        }
        order.reverse();
        order.extend(self.source.blocks().filter(|block| !seen[block.index()]));
        order
    }

    /// One block: its parameters, then every instruction in it that is not folded into another.
    fn block(&mut self, block: Block) -> Result<(), Unsupported> {
        let out = self.out_block(block);
        self.at = Some(out);
        if self.source.entry() == Some(block) {
            self.arrive(block, out)?;
        } else {
            let mut arriving = Vec::new();
            for &param in &self.source[block].params {
                // A value with no register to arrive in, which the class would not say, since
                // `class_of` puts one of these in the general purpose file on purpose and what it
                // means by that is that nothing there can hold it. What crosses the edge for one
                // of those is the address of where the value already is, so the parameter is a
                // pointer here and the bytes it points at are copied below.
                let ty = self.source[param].ty;
                let reg = self.out.append_param(out, self.class_of(ty));
                self.regs[param.index()] = Some(reg);
                if on_x87(ty) {
                    arriving.push((param, reg));
                }
            }
            self.settle(block, &arriving)?;
        }
        let kept = self.pad(block)?;

        // What each instruction matched, and which instructions were folded into another. The
        // decision is made for the whole block before any of it is written, and it is made more
        // than once: a value that only some of its readers took has to be put back in a register
        // for all of them, and taking it away from those readers changes what they match.
        let insts: Vec<Inst> = self.source.insts(block).collect();
        let mut refused: HashSet<Value> = HashSet::new();
        let mut decided = self.decide(&insts, &refused);
        while let Some(value) = self.left_alive(&insts, &decided.plans) {
            refused.insert(value);
            decided = self.decide(&insts, &refused);
        }
        let Decided { found, folded, .. } = decided;

        // Where each assignment in this block that starts a declaration on a value is, as the
        // machine instruction in front of the place its IR instruction left off, or the block
        // for one where nothing has been written yet. What comes after it is not known until the
        // block is filled, so that is read below.
        let wanted: HashSet<Option<Inst>> =
            self.marks.get(&block).into_iter().flatten().map(|&(after, _)| after).collect();
        let mut reached: Vec<(Option<Inst>, mir::Block, Option<mir::Inst>)> = Vec::new();
        for (index, (&inst, matched)) in insts.iter().zip(found).enumerate() {
            let before = index.checked_sub(1).map(|index| insts[index]);
            if wanted.contains(&before) {
                let at = self.at.unwrap_or(out);
                reached.push((before, at, self.out.terminator(at)));
            }
            if folded.contains(&inst) || self.writes_nothing(inst) {
                continue;
            }
            // A call is built from the convention rather than matched, which is why it is the one
            // opcode looked at by name here. Through an address it is a different instruction and
            // the same convention, so the two arrive at the same place and differ in one line of
            // it.
            match self.source[inst].opcode {
                Opcode::Call | Opcode::CallIndirect => {
                    self.called(inst)?;
                    continue;
                }
                // The exception a landing pad was entered with, which the unwinder left in the
                // first return register. Built by name for the reason a named register is.
                Opcode::Landing => {
                    self.landing(inst)?;
                    continue;
                }
                // A call and the return behind it, which is what `crate::tail::mark` made it out
                // of, and both are built the way they would have been. What makes it a jump is
                // written at the very end, once the epilogue is there to jump from.
                Opcode::TailCall => {
                    self.tail_called(inst)?;
                    continue;
                }
                // Built from the frame rather than matched, for the same shape of reason a call
                // is built from the convention: what a rule replaces a term with is instructions,
                // and what an `alloca` needs first is bytes, which the rule language has no way
                // to ask for.
                Opcode::Alloca => {
                    self.reserve(inst)?;
                    continue;
                }
                // Reading the stack pointer and writing it back, which are the two ends of a scope
                // holding a variable length array. Built here for the reason an `alloca` is: the
                // value is a register the rule language has no way to name, because what it holds
                // is not a value the program computed but where the machine's stack had got to.
                // The arguments the function was handed, saved in the prologue, and a call made
                // out of them. Built here because neither is a value a rule could say anything
                // about: the first is a place in the frame and the second is a call, whose
                // arguments are a block of registers rather than values.
                Opcode::ApplyArgs => {
                    self.apply_args(inst)?;
                    continue;
                }
                Opcode::Apply => {
                    self.apply(inst)?;
                    continue;
                }
                Opcode::StackSave => {
                    self.stack_pointer(inst, false)?;
                    continue;
                }
                Opcode::StackRestore => {
                    self.stack_pointer(inst, true)?;
                    continue;
                }
                // The address of a name, built here for the same reason an `alloca` is: what a
                // rule replaces a term with is instructions over values, and the operand of this
                // one is a symbol, which is a thing the rule language has no way to bind and the
                // solver has no way to say anything about. There is nothing in `lea sym(%rip)` a
                // proof over bitvectors could discharge, because what makes it the right answer
                // is the relocation and what the linker does with it.
                Opcode::GlobalAddr => {
                    self.address_of(inst)?;
                    continue;
                }
                // The address of a label and the branch that reads one, built here for the same
                // reason and for one more. The reason is the same: what the first of them names is
                // a block, which is not a value a rule pattern can bind, and there is nothing in
                // the distance between two places in one function that a proof over bitvectors
                // could discharge. The extra one is that the second is a terminator whose arms are
                // not two and not fixed, and a rule says what an instruction reads rather than
                // where a block goes.
                Opcode::BlockAddr => {
                    self.block_address(inst)?;
                    continue;
                }
                Opcode::IndirectBr => {
                    self.indirect_branch(inst)?;
                    continue;
                }
                // A `switch` that `crate::switch` found dense enough for a table, which is a load
                // out of the table and the same jump. Built here for the reasons the jump above
                // is, and because what the load reads is a place in this function.
                Opcode::Switch => {
                    self.jump_table(inst)?;
                    continue;
                }
                // The pair that saves a place in this function and comes back to it. Built here
                // for the reason the address of a label is, and for two more. The reason is the
                // same: the first of them writes down where control comes back to, which is a
                // place in this function and not a value a rule pattern can bind. The extra ones
                // are that each of them is a group of instructions over a buffer the program owns
                // rather than one instruction, and that the first of them leaves the block it was
                // written in and carries on in a new one, which is a thing no rule can do.
                Opcode::SetjmpMarker => {
                    self.saves_place(inst)?;
                    continue;
                }
                Opcode::LongjmpMarker => {
                    self.comes_back(inst)?;
                    continue;
                }
                // Where this thread's own storage starts, built here for a reason of the same
                // shape: what it reads is `%fs`, which is not a register the rule language can
                // bind and not one a proof over bitvectors could say anything about, because what
                // makes the load the right answer is an agreement between the loader and the C
                // library rather than any arithmetic.
                Opcode::ThreadPointer => {
                    self.thread_pointer(inst)?;
                    continue;
                }
                // What a named machine register holds, built here for the reason above written
                // about any register rather than about one: which register it is is a string
                // beside the instruction, and a rule matches on an opcode and a type and could
                // not see it. There is nothing to prove either, since the answer is the register
                // and the instruction is the move that reads it.
                Opcode::RegisterValue => {
                    self.register_value(inst)?;
                    continue;
                }
                // Where a frame is and what it returns to, built here for the same reason and one
                // more. The reason is the same: what the walk starts from is the frame pointer,
                // which is not a register a rule pattern can bind, and there is nothing in reading
                // the link the prologue saved that a proof over bitvectors could discharge. The
                // extra one is that how long the walk is comes out of a number beside the
                // instruction, so one of these is not one instruction but however many the depth
                // says, and a rule replaces a term with a term.
                Opcode::FrameAddress | Opcode::ReturnAddress => {
                    self.frames(inst)?;
                    continue;
                }
                // Built from the frame for the reason an `alloca` is, and from the convention for
                // the reason a call is: three of the four fields it writes are distances that do
                // not exist until the frame does, and the fourth is where the walk over the
                // argument registers stopped. A function that is not variadic has no such walk to
                // report, so it has nothing here and is refused below, which is the right answer
                // for a `va_start` in one.
                Opcode::VaStart if self.varargs.is_some() => {
                    self.va_start(inst)?;
                    continue;
                }
                // A return of more than one value, which is a structure small enough to come
                // back in a pair of registers. Built from the convention for the reason a call
                // is: which register each half goes in depends on the halves in front of it,
                // because the two register files are walked separately, and a pattern over a term
                // cannot see them. A return of one value is a term with a name and a rule, and it
                // stays one.
                //
                // A return of none in a function whose answer went through memory is here too,
                // and for a different reason: what it gives back is not written in the IR at all.
                // The convention says the address the caller handed over comes back, and only the
                // signature says this function was handed one.
                //
                // And a return of one eighty bit value, for a third reason: what a rule would
                // write is an instruction leaving the value in a register, and this one is left on
                // the x87 stack instead. A rule could not name that stack any more than any other
                // rule about this type could.
                //
                // And a return the convention asks this side to extend, which a rule has no way to
                // know about since the signature is what says so and not the value.
                Opcode::Return
                    if self.source[self.source[inst].args].len() > 1
                        || self.sret().is_some()
                        || self.gives_back_x87(inst)
                        || self.widens_return() =>
                {
                    let values = self.source[self.source[inst].args].to_vec();
                    self.returned(inst, values)?;
                    continue;
                }
                // A cast between a pointer and an integer of the same width, which on this
                // machine is every one the front end writes. No instruction at all, so no rule
                // could name one.
                Opcode::PtrToInt | Opcode::IntToPtr => {
                    self.rename(inst)?;
                    continue;
                }
                // A barrier, which is one instruction or none depending on the ordering. Written
                // by name because there is nothing about it a rule could be proved against, the
                // way there is nothing to prove about the address of a symbol.
                Opcode::Fence => {
                    self.barrier(inst)?;
                    continue;
                }
                // An ordered load or store that `crate::expand::orderings` left alone, which on a
                // machine that is not total store order is every one stronger than relaxed. Written
                // by name for the barrier's reason: what it adds to the plain access is an ordering.
                Opcode::AtomicLoad | Opcode::AtomicStore if self.on_aarch64() => {
                    self.ordered(inst)?;
                    continue;
                }
                // A hint, written by name for the reason a barrier is and one step further: not
                // only is there no equality for a proof to discharge, there is nothing about the
                // program around it either. Which of the four instructions it is comes out of the
                // number the builtin was given, which is beside the instruction rather than in it.
                Opcode::Prefetch => {
                    self.hint(inst)?;
                    continue;
                }
                // Stopping, written by name for the first half of the barrier's reason: it
                // computes nothing, so there is no term for a rule to replace, and what makes it
                // right is what the operating system does with the fault rather than anything a
                // proof over bitvectors could discharge.
                Opcode::Trap => {
                    self.trap(inst);
                    continue;
                }
                // A compare and exchange, which is written by name because it produces two values
                // and a rule produces one. The replacement of a rule is one term, a term names the
                // value an instruction computes, and there is no way in that language to say that
                // an instruction leaves an answer in one place and a yes or no in another.
                Opcode::Cmpxchg => {
                    self.exchange(inst)?;
                    continue;
                }
                // A read modify write, which is written by name for a different reason: it produces
                // one value, so a rule could name it, and what it does is not in the head a rule
                // matches on. Every one of the thirteen operations is the same opcode at the same
                // type and differs only in what is carried beside it, so one pattern would be all
                // thirteen patterns. Of the thirteen only the three with an instruction reach here,
                // since `crate::retry` turned the rest into loops a long way above this.
                Opcode::AtomicRmw => {
                    self.modify(inst)?;
                    continue;
                }
                // An `asm` statement, whose lowering is its template and there is no term for a
                // string. Written by name for the reason a barrier is, and before the x87 arm
                // below so that an `asm` holding a `long double` is refused as the `asm` it is
                // rather than as an instruction nothing computes.
                Opcode::InlineAsm => {
                    // The template is read as x86 assembly, and that reader is the only one there
                    // is. AArch64 keeps every template as text, and any other machine's `asm` is
                    // refused here rather than read as the wrong language.
                    if self.on_aarch64() {
                        self.spelled(inst)?;
                        continue;
                    }
                    if !std::ptr::eq(self.selector.shapes, &x86_64::MACHINE) {
                        return Err(self.unsupported(inst));
                    }
                    if self.touches_x87(inst) {
                        self.x87_assembly(inst)?;
                        continue;
                    }
                    self.assembly(inst)?;
                    continue;
                }
                // Anything at all with an eighty bit float in it, which is the one arm here
                // chosen by a type rather than by an opcode, because what makes these different
                // is not what they do but where the value is. A `long double` has no register,
                // so it has no name in `crate::term` and no rule could bind one: every one of
                // these is a group of instructions over a frame slot, written out below.
                //
                // Last of the arms, so that a call and a return with one of these in them reach
                // the convention first and are refused by it, which is the truer answer: what is
                // wrong there is where the value has to travel and not that nothing can compute
                // it.
                _ if self.touches_x87(inst) => {
                    self.x87(inst)?;
                    continue;
                }
                _ => {}
            }
            let matched = matched.ok_or_else(|| self.unsupported(inst))?;
            self.emit(inst, &matched)?;
            // After it is built rather than when it matched, so that what is recorded is the rules
            // this function was lowered by and not the rules something was tried with.
            self.fired.mark(matched.rule);
        }
        // Whichever block the walk ended in rather than the one it started in. The two are the
        // same block for every function that does not save a place for a `__builtin_longjmp`, and
        // where they differ it is the last of them that the terminator and the arms belong to.
        // See [`Self::saves_place`].
        let last = self.at.expect("a block is being filled");
        self.edges(block, last)?;
        for (value, reg) in kept {
            self.regs[value.index()] = reg;
        }
        // Now that the block is filled, the instruction after each place an assignment was is the
        // first one it holds its value at. One with nothing after it, which a block ending in the
        // assignment would be, stays unanswered.
        if let Some(marks) = self.marks.get_mut(&block) {
            for &(before, at, last) in &reached {
                let first = match last {
                    Some(last) => self.out.next_inst(last),
                    None => self.out.insts(at).next(),
                };
                for mark in marks.iter_mut().filter(|(after, _)| *after == before) {
                    mark.1 = first;
                }
            }
        }
        Ok(())
    }

    /// One call, which is built from the convention rather than matched against the table for the
    /// same reason the arguments of the function itself are.
    ///
    /// The arguments are read before the call is built, which is what materializes a constant
    /// argument into a register, since no call passes an immediate.
    ///
    /// A call to a name and a call through an address are both here, and what tells them apart is
    /// the opcode rather than whether a callee was recorded, which is the same thing the verifier
    /// reads. Through an address the first operand is the address and the arguments are the ones
    /// behind it, and everything after that is the same: where each argument goes, where the value
    /// comes back and which registers are gone across it are the convention's answers and the
    /// convention does not ask what is being called.
    fn called(&mut self, inst: Inst) -> Result<u32, Unsupported> {
        let data = &self.source[inst];
        let Extra::Call(info) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[info];
        let indirect = data.opcode == Opcode::CallIndirect;

        let values: Vec<Value> = self.source[data.args].to_vec();
        let callee = if indirect {
            let &address = values.first().ok_or_else(|| self.unsupported(inst))?;
            abi::Callee::Through(self.reg_of(address)?)
        } else {
            abi::Callee::Named(info.callee.ok_or_else(|| self.unsupported(inst))?)
        };

        // What the ABI asks of each argument, read out before any of them is, because reading one
        // borrows the function this is a table in. The ones the signature names are the signature's
        // answer and the ones behind them are the call's, which is where a structure passed to a
        // variadic callee by value says that its bytes travel: there is no parameter to say it on.
        let signature = &self.source[info.signature];
        let variadic = signature.variadic;
        let named: Vec<Abi> = signature.params.iter().map(|param| param.abi).collect();
        let beyond: Vec<Abi> = self.source[info.varargs].to_vec();
        // Every value that comes back and not only the first. A structure small enough to travel
        // in registers comes back in up to two of them, and which register each half is in is the
        // convention's answer, which is why the whole list goes to the same place the arguments do
        // rather than to a rule.
        let returns: Vec<Type> = signature.return_types().collect();

        let mut args = Vec::with_capacity(values.len());
        for (index, value) in values.into_iter().skip(usize::from(indirect)).enumerate() {
            let abi = named.get(index).or_else(|| beyond.get(index - named.len()));
            let abi = abi.copied().unwrap_or_default();
            let ty = self.source[value].ty;
            // What travels for an eighty bit value is its bytes, so what the call is handed is
            // where they are rather than a register they are in, and there is no register they
            // could be in. Everything else about it is a sixteen byte object passed by value and
            // is built by the same code.
            let reg =
                if abi::on_the_stack(ty) { self.x87_slot(value) } else { self.reg_of(value)? };
            args.push(abi::Passing { ty, reg, abi });
        }
        let block = self.at.expect("a block is being filled");
        let what = abi::Calling {
            callee,
            args: &args,
            returns: &returns,
            variadic,
            named: named.len(),
            at: self.source.span(inst),
        };
        let made = abi::call(&mut self.out, block, &what, self.conv, self.selector.abi, self.names)
            .map_err(|refused| Unsupported::Call { inst, refused })?;
        if self.source.unwinds_to_pad(inst) {
            let call = self.out.insts(block).last().expect("the call just built");
            self.unwinding.insert(inst, call);
        }
        let calls = &mut self.stack.calls;
        *calls = Some(calls.unwrap_or(0).max(made.outgoing));
        // An eighty bit value came back on the x87 stack, and the one thing that has to happen
        // before anything else touches that stack is taking it off. So the `fstp` goes here, in
        // front of everything the block does next, and after it the value is in its slot and is
        // read the way every other one is. A complex one is two of them, the real half on top, so
        // taking them off in order leaves each in its own slot and the stack empty.
        let results: Vec<Value> = self.source[inst].results().collect();
        let types: Vec<Type> = results.iter().map(|&result| self.source[result].ty).collect();
        if abi::back_on_x87(&types) {
            let span = self.source.span(inst);
            for result in results {
                let into = self.x87_slot(result);
                let into = self.through(into);
                self.x87_at("fstp_t", span, into);
            }
            return Ok(made.outgoing);
        }
        for (result, &reg) in results.into_iter().zip(&made.results) {
            self.regs[result.index()] = Some(reg);
        }
        Ok(made.outgoing)
    }

    /// One `tail_call`, as the call and a return of what it gave back.
    ///
    /// The call is written down for [`crate::tail::jumps`] when it can be made after the frame is
    /// gone. That is when it put nothing in the argument area, which is the bottom of this frame,
    /// and when the answer comes back in registers, since one on the x87 stack is taken off and put
    /// back by instructions after the call. A call that is not written down stays a call and a
    /// return, which is what the IR said before `crate::tail::mark` read it.
    fn tail_called(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let outgoing = self.called(inst)?;
        let block = self.at.expect("a block is being filled");
        let call = self.out.insts(block).last().expect("the call just built");
        let values: Vec<Value> = self.source[inst].results().collect();
        let x87 = self.x87_values(&values);
        self.returned(inst, values)?;
        if outgoing == 0 && !x87 && self.sret().is_none() {
            let returns = self.out.insts(block).skip_while(|&at| at != call).skip(1).collect();
            self.stack.tails.push(crate::tail::Tail { call, returns });
        }
        Ok(())
    }

    /// The pointer a function returning through memory was handed, or nothing in a function that
    /// was not.
    ///
    /// It is the first parameter and the signature is what says so, since in the IR it is an
    /// ordinary pointer and reads like one everywhere in the body. A function with a signature
    /// like that and no entry block has nothing to give back and no body to give it back from.
    fn sret(&self) -> Option<Value> {
        let first = self.source.signature().params.first()?;
        if !matches!(first.abi, Abi::Sret { .. }) {
            return None;
        }
        self.source[self.source.entry()?].params.first().copied()
    }

    /// One `return` the convention has to write, as the place each value has to be in by the end.
    ///
    /// One pseudo per value, each a read constrained to a return register, which is what a return
    /// of one value already is and is the whole of what either does. The `ret` itself comes from
    /// the epilogue for both, long after this, because the frame has to be given back first.
    ///
    /// The two register files are counted separately, so a structure of a `double` and a `long`
    /// leaves the `double` in the first vector register and the `long` in the first integer one
    /// rather than in the second of either. That is the same walk `rucc_codegen::abi` makes on
    /// the other side of the call, which is what makes the two ends agree.
    ///
    /// A function whose answer went through memory gives back the address it was handed, in front
    /// of nothing else, because a signature that returns that way returns nothing else. That the
    /// caller already knows the address is not enough: it is allowed to read the register instead,
    /// and a caller that does gets whatever the allocator last left there. In a leaf function that
    /// is usually the right answer by accident, and one call in the body is enough to make it a
    /// wild pointer, which is why this is written rather than left to luck.
    ///
    /// Where everything goes is worked out before anything is written, so a return this cannot
    /// make leaves no half of one behind.
    /// Whether a value this function gives back has to be extended first, which is Apple's arm64
    /// asking the callee to fill the 32 bits above a `char` or a `short` by its sign.
    fn widens_return(&self) -> bool {
        let returns = &self.source.signature().returns;
        returns.iter().any(|it| (self.selector.abi.extend)(it.ty, it.abi).is_some())
    }

    /// Whether what a `return` gives back goes back on the x87 stack, per [`abi::back_on_x87`].
    fn gives_back_x87(&self, inst: Inst) -> bool {
        self.x87_values(&self.source[self.source[inst].args])
    }

    /// Whether those values go back on the x87 stack, per [`abi::back_on_x87`].
    fn x87_values(&self, values: &[Value]) -> bool {
        let types: Vec<Type> = values.iter().map(|&value| self.source[value].ty).collect();
        abi::back_on_x87(&types)
    }

    fn returned(&mut self, inst: Inst, values: Vec<Value>) -> Result<(), Unsupported> {
        let (mut ints, mut floats) = (0usize, 0usize);
        let mut parts = Vec::with_capacity(values.len() + 1);
        // An eighty bit value goes back on the x87 stack, which is where the convention says it is
        // and is the one place a value is left rather than put in a register. So the whole of the
        // return is an `fld` of its slot, and the stack it leaves the value on is not empty at the
        // `ret`, which is the one time in this file that is true and is what the convention asks
        // for. What comes after is the epilogue, which gives the frame back and touches nothing in
        // the unit. A complex one loads its imaginary half first so that the real half ends up on
        // top of it, in `st(0)`, with the imaginary half under it in `st(1)`.
        if self.x87_values(&values) && self.sret().is_none() {
            let span = self.source.span(inst);
            for &value in values.iter().rev() {
                let from = self.x87_slot(value);
                let from = self.through(from);
                self.x87_at("fld_t", span, from);
            }
            return Ok(());
        }
        // What the signature says about the bits above a narrow one, which on an ABI that extends
        // it is an obligation of this side: the caller reads the whole of the 32 bit register.
        let asked: Vec<Abi> = self.source.signature().returns.iter().map(|it| it.abi).collect();
        let asked = asked.into_iter().chain(std::iter::repeat(Abi::Plain));
        let sret = self.sret().map(|value| (value, Abi::Plain));
        for (value, abi) in sret.into_iter().chain(values.into_iter().zip(asked)) {
            let ty = self.source[value].ty;
            let at = if crate::term::in_vector_file(ty) { &mut floats } else { &mut ints };
            // Why it cannot come back, and not only that it cannot. A type that travels nowhere
            // says so itself, and a type that travels perfectly well ran out of registers.
            let missing = abi::refuses(ty, self.selector.abi).unwrap_or(Missing::NoRoom);
            let name =
                (self.selector.abi.ret)(ty, *at).ok_or(Unsupported::Returned { inst, missing })?;
            *at += 1;
            // The register is the target's answer and not one worked out here, the same as it is
            // for a return of one value, so that both halves of a pair and every rule that writes
            // half of one are reading the same table.
            let opcode =
                name.strip_prefix(self.selector.prefix()).ok_or_else(|| self.unsupported(inst))?;
            let descs = self.selector.operands(opcode).ok_or_else(|| self.unsupported(inst))?;
            let [desc] = descs else { return Err(self.unsupported(inst)) };
            let widen = (self.selector.abi.extend)(ty, abi).map(|name| self.names.intern(name));
            parts.push((self.names.intern(name), self.reg_of(value)?, *desc, widen));
        }

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        for (opcode, mut reg, desc, widen) in parts {
            if let Some(widen) = widen {
                let wide = self.out.new_vreg(desc.class);
                let build = self.out.build(block, mir::Opcode::new(widen)).at(span);
                build.def(wide, desc.class).uses(reg, desc.class).finish();
                reg = wide;
            }
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            self.out.build(block, mir::Opcode::new(opcode)).at(span).operand(operand).finish();
        }
        Ok(())
    }

    /// One `alloca`: the bytes it asks for go on the list the frame is laid out from, and the
    /// address of them is one instruction.
    ///
    /// The instruction is a `lea` off the stack pointer, which is the one register that reaches
    /// the frame in every function, and its displacement is left at nothing because there is no
    /// frame yet. Which instruction is waiting for which local is remembered, and
    /// [`crate::finish`] fills the numbers in after [`crate::frame::Frame`] has placed them.
    ///
    /// There is deliberately no rule for `alloca` and no name for one in [`crate::term`], and
    /// that is what stops it being folded into something else. An operand shown as the
    /// instruction that computed it is offered to the matcher by its name, so an `alloca` with no
    /// name is one no pattern can reach past, and the address it computes is always in a register
    /// by the time anything reads it.
    fn reserve(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        // A variable length array carries the size it wants as an operand rather than in the
        // instruction, which is the whole of what tells the two apart here.
        if let Some(&size) = self.source[data.args].first() {
            return self.grow(inst, size);
        }
        let Extra::Mem(mem) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[mem];
        let size = u32::try_from(info.size)
            .map_err(|_| Unsupported::Dynamic { inst, growing: Growing::Huge })?;
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;

        // At least one, because the frame divides by the alignment and an object with no
        // alignment at all is one the front end had nothing to say about rather than one that may
        // go anywhere.
        let index = self.stack.locals.len();
        self.stack.locals.push(Local { size, align: info.align.max(1) });
        if let Some(decl) = self.source.mem_decl(mem) {
            self.stack.declared.push((index, decl));
        }

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let lea = self.named(self.selector.frame.lea);
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let made =
            self.out.build(block, lea).at(span).def(reg, self.gpr).mem(mir::Mem::at(sp)).finish();
        self.stack.addresses.push((made, index));
        self.frame_slots.insert(result, index);
        Ok(())
    }

    /// The other kind of `alloca`: one whose size the function does not know until it runs, which
    /// is what a variable length array is.
    ///
    /// Nothing about it is a slot the frame laid out, because the frame is laid out once and this
    /// happens as often as control reaches the declaration. The bytes come off the stack pointer
    /// where the declaration stands, which is two instructions:
    ///
    /// ```text
    ///   sub sp, bytes     the stack pointer moves down over the memory, which is what takes it
    ///   lea reg, [sp+n]   where the memory starts, which is above the outgoing argument area
    /// ```
    ///
    /// The displacement is left at nothing for the reason the constant kind leaves its own at
    /// nothing, and for a different number: that area belongs to the arguments of whatever this
    /// function calls, it stays at the bottom of the frame wherever the bottom has moved to, and
    /// how big it is is not known until every call in the function has been seen.
    ///
    /// The bytes are already a multiple of the stack pointer's alignment by the time they arrive,
    /// because [`crate::expand::rounds`] rounded them up in the IR, so nothing here has to mask the
    /// stack pointer afterwards and the stack pointer stays somewhere a call can be made from.
    ///
    /// Two instructions here and not always two in the finished function. On a command line that
    /// asked for the stack to be touched a page at a time, the subtraction becomes a loop that
    /// walks the same distance a page at a time, which [`crate::finish`] writes. That is why the
    /// instruction is written down in [`Stack::grown`] as well as left where it is.
    ///
    /// An array wanting more alignment than the convention leaves the stack pointer with does not
    /// reach here asking for it: [`crate::expand::rounds`] gives it the alignment in extra bytes
    /// and turns the array into a `ptr_add` of the offset that lands inside them, so what arrives
    /// is a block asking for the convention's alignment like any other. The refusal below is what
    /// answers IR that came from somewhere other than that pass, since forcing the alignment here
    /// would be a second rounding of a register the frame already rounded, and after it no
    /// constant reaches the rest of the frame from anywhere. See `Growing` in [`crate::frame`].
    fn grow(&mut self, inst: Inst, size: Value) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Mem(mem) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[mem];
        if info.align > self.conv.stack_align {
            return Err(Unsupported::Dynamic { inst, growing: Growing::Aligned });
        }
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let bytes = self.reg_of(size)?;

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        let grow = self.named(self.selector.frame.grow);
        let took = self
            .out
            .build(block, grow)
            .at(span)
            .operand(mir::Operand::write(stack, self.gpr))
            .operand(mir::Operand::read(stack, self.gpr))
            .operand(mir::Operand::read(bytes, self.gpr))
            .finish();
        self.stack.grown.push(took);

        let reg = self.new_reg(result);
        let lea = self.named(self.selector.frame.lea);
        let sp = mir::Operand::read(stack, self.gpr);
        let made =
            self.out.build(block, lea).at(span).def(reg, self.gpr).mem(mir::Mem::at(sp)).finish();
        self.stack.dynamic.push(made);
        self.stack.grown_at.get_or_insert(inst);
        Ok(())
    }

    /// Where the stack pointer is, kept so that something later can put it back.
    ///
    /// One move out of the stack pointer and one move into it, which is the whole of what the two
    /// halves are. What makes them worth writing is where the front end puts them: a scope holding
    /// a variable length array saves the stack pointer as it opens and puts it back as it closes,
    /// so a loop declaring one takes its bytes once round rather than once per iteration, and a
    /// jump out of the scope gives the bytes back on the way out.
    ///
    /// The value travels in an ordinary register the allocator hands out, so it may be spilled like
    /// any other, and a spill slot in a frame that grows is reached through the frame pointer,
    /// which is exactly the register that still means something after the stack pointer has moved.
    fn stack_pointer(&mut self, inst: Inst, into: bool) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        let mov =
            self.selector.frame.moves(self.gpr).expect("a class the target says how to move").mov;
        let mov = self.named(mov);
        let (write, read) = if into {
            let &saved = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
            (stack, self.reg_of(saved)?)
        } else {
            let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
            (self.new_reg(result), stack)
        };
        self.out
            .build(block, mov)
            .at(span)
            .operand(mir::Operand::write(write, self.gpr))
            .operand(mir::Operand::read(read, self.gpr))
            .finish();
        // Only the write is a move of the stack pointer, and it is the one that makes the frame a
        // growing one. A read of it in a function that never writes it back is a function that
        // asked where the stack was and did nothing with the answer.
        if into {
            self.stack.grown_at.get_or_insert(inst);
        }
        Ok(())
    }

    /// Whether an instruction has an eighty bit float anywhere in it.
    ///
    /// Producing one and reading one are the same question here, because what makes one of these
    /// different from every other instruction is not the operation but where the value is. A
    /// `long double` is on the x87 stack while it is being worked on and in a frame slot the rest
    /// of the time, and neither of those is somewhere the operand of a rule could point.
    fn touches_x87(&self, inst: Inst) -> bool {
        let data = &self.source[inst];
        data.results().any(|value| on_x87(self.source[value].ty))
            || self.source[data.args].iter().any(|&arg| on_x87(self.source[arg].ty))
    }

    /// Everything that happens to an eighty bit float, as the group of instructions it is.
    ///
    /// The first six move one, and every one of those is a load, a store, or a load and a store at
    /// two different formats, because that is the whole of what this machine converts with: the
    /// x87 has no instruction that turns one thing on its stack into another, so a widening is
    /// `fld` of the narrow format and a narrowing is `fstp` of it.
    ///
    /// The rest work on one, and they are here rather than in a rule for the same reason the six
    /// are. An add is a push, a push, the add and a pop, and what passes between those four is the
    /// top of a stack nothing allocates from, so there is no value in the middle of the group for
    /// a pattern to bind or a replacement to name. The comparison is the same shape with its last
    /// two instructions folded into one opcode, which is where the byte it produces comes from.
    ///
    /// Every group leaves the stack as empty as it found it, which is what `spec/10-backend.md`
    /// section 10.8 asks of one and is why nothing in this file has to track a depth: each push
    /// below is answered by a pop a line or two later, so no two groups can ever be looking at
    /// the same eight registers.
    fn x87(&mut self, inst: Inst) -> Result<(), Unsupported> {
        match self.source[inst].opcode {
            Opcode::Load => self.x87_load(inst),
            Opcode::Store => self.x87_store(inst),
            Opcode::FPExt => self.x87_widen(inst),
            Opcode::FPTrunc => self.x87_narrow(inst),
            Opcode::SIToFP => self.x87_from_signed(inst),
            Opcode::FPToSI => self.x87_to_signed(inst),
            Opcode::FAdd => self.x87_arith(inst, "fadd_p"),
            Opcode::FSub => self.x87_arith(inst, "fsubr_p"),
            Opcode::FMul => self.x87_arith(inst, "fmul_p"),
            Opcode::FDiv => self.x87_arith(inst, "fdivr_p"),
            Opcode::FNeg => self.x87_flip(inst),
            Opcode::FCmp => self.x87_compare(inst),
            Opcode::FConst => self.x87_const(inst),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// The eighty bit parameters of a block, copied out of the addresses an edge handed over and
    /// into slots of the block's own.
    ///
    /// What crosses an edge for a value of this type is an address, because the value is sixteen
    /// bytes of the frame and no register holds any of it. The block cannot keep that address: a
    /// second edge into the same block hands over a second one, and a read after the block would
    /// then be a read of whichever edge was taken rather than of one place. So the block has a
    /// slot per parameter and the bytes are copied into it here, which is the move on an edge that
    /// every other type gets from the allocator.
    ///
    /// Every load runs before every store and the stores run backwards, so all of the values are
    /// on the x87 stack at once and nothing reads a slot another one has already written. That
    /// costs nothing in the ordinary case of one parameter and is what makes the back edge of a
    /// loop that swaps two of these work. It is also the reason for the limit: the stack is eight
    /// deep, and a block with more of these than that is refused rather than copied in an order
    /// that could be wrong.
    fn settle(&mut self, block: Block, arriving: &[(Value, mir::Reg)]) -> Result<(), Unsupported> {
        let Some(&(first, _)) = arriving.first() else { return Ok(()) };
        if arriving.len() > X87_DEPTH {
            let ty = self.source[first].ty;
            return Err(Unsupported::Phi { block, count: arriving.len(), ty });
        }
        // A block parameter comes from no instruction, so what this points at is the first thing
        // in the block, which is where a reader looking for the copy would look.
        let first_inst = self.source.insts(block).next();
        let span = first_inst.map_or(Span::DUMMY, |it| self.source.span(it));
        for &(_, reg) in arriving {
            let from = self.through(reg);
            self.x87_at("fld_t", span, from);
        }
        for &(param, _) in arriving.iter().rev() {
            let into = self.x87_slot(param);
            let into = self.through(into);
            self.x87_at("fstp_t", span, into);
        }
        Ok(())
    }

    /// The frame slot an eighty bit value lives in, as its address in a fresh register.
    ///
    /// The slot is the value's for the whole function and is taken the first time somebody asks.
    /// The address is worked out again every time, which is a `lea` per use and is deliberate: one
    /// address kept in a register from the definition to the last use would hold a general purpose
    /// register open across everything in between, and a function with a handful of these in it
    /// would spend its registers on addresses of things rather than on things.
    fn x87_slot(&mut self, value: Value) -> mir::Reg {
        // An argument of the function has a slot already and it is the caller's. The convention
        // puts the bytes in the argument area and hands over where they are, so the address that
        // arrived is the answer and no second copy of the value is made. Nothing ever writes to a
        // value of this type once it exists, so nothing writes to the caller's copy either. A
        // parameter of any other block is not this: what arrived there is an address a predecessor
        // chose, [`Lowering::settle`] has already copied the bytes out of it, and the slot those
        // bytes landed in is the one below.
        let entry = self.source.entry();
        if let (Def::Param { block, .. }, Some(reg)) =
            (self.source[value].def, self.regs[value.index()])
        {
            if entry == Some(block) {
                return reg;
            }
        }
        let index = match self.slots[value.index()] {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: X87_BYTES, align: X87_BYTES });
                self.slots[value.index()] = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// The bytes a value crosses between a register and the x87 stack through, as their address
    /// in a fresh register.
    fn x87_crossing(&mut self) -> mir::Reg {
        let index = match self.crossing {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: X87_CROSSING, align: X87_CROSSING });
                self.crossing = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// The two control words, as the address of the first of them in a fresh register.
    fn x87_control(&mut self) -> mir::Reg {
        let index = match self.control {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: 4, align: 4 });
                self.control = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// An address held in a register, as the addressing mode that reaches it.
    fn through(&self, reg: mir::Reg) -> mir::Mem {
        mir::Mem::at(mir::Operand::read(reg, self.gpr))
    }

    /// One instruction of a group, which names an address and nothing else.
    ///
    /// Every x87 instruction that moves a value is one of these. What it does to the stack is in
    /// the mnemonic rather than in an operand, so there is no register to write down and no
    /// register the allocator gets a say in.
    fn x87_at(&mut self, name: &str, span: Span, at: mir::Mem) {
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(name);
        self.out.build(block, opcode).at(span).mem(at).finish();
    }

    /// The one instruction of a group that reaches the program's own memory.
    ///
    /// A `long double` moves in two instructions with a frame slot at one end of them, and the
    /// other end is the address the program wrote. That end is the access, so it is the one that
    /// carries what the program said about it, and the trip through the slot is this compiler's
    /// own business the way a spill is. See [`Self::carried`].
    fn x87_touching(&mut self, name: &str, inst: Inst, at: mir::Mem) {
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(name);
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        self.out.build(block, opcode).at(span).flags(flags).mem(at).finish();
    }

    /// One instruction of a group that names nothing at all.
    ///
    /// The arithmetic is these. Both of an add's operands are already on the stack when it runs
    /// and so is where the answer goes, and the stack is not somewhere an instruction says, so
    /// `faddp` has an argument in the assembler's syntax and nothing here for the argument to come
    /// from. What it works on is which two pushes came before it, which is a fact about the order
    /// of the group and is why the group is written in one place.
    fn x87_only(&mut self, name: &str, span: Span) {
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(name);
        self.out.build(block, opcode).at(span).finish();
    }

    /// A `load` of a `long double`: onto the stack from where it was, and off it into the slot.
    ///
    /// Two instructions rather than the two general purpose moves the same sixteen bytes would
    /// take, because `fld` and `fstp` at this format neither convert nor look: the value goes on
    /// in the format it was already in and comes back off in it, so a signalling NaN stays one
    /// and nothing is raised. Which is what makes this a copy at all.
    fn x87_load(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &address = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.reg_of(address)?;
        let from = self.through(from);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_touching("fld_t", inst, from);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A `store` of a `long double`: the same pair the other way round.
    fn x87_store(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let args = self.source[self.source[inst].args].to_vec();
        let [value, address] = args[..] else { return Err(self.unsupported(inst)) };
        let span = self.source.span(inst);
        let from = self.x87_slot(value);
        let from = self.through(from);
        let into = self.reg_of(address)?;
        let into = self.through(into);
        self.x87_at("fld_t", span, from);
        self.x87_touching("fstp_t", inst, into);
        Ok(())
    }

    /// A `float`, a `double` or an integer becoming a `long double`.
    ///
    /// Through memory, because the x87 reads memory and nothing else: the value is in a register
    /// the machine has and the unit has no way to be handed one, so it is written to the crossing
    /// bytes and loaded back at the format that widens it. Every one of these is exact. Sixty four
    /// bits of significand and fifteen of exponent hold every `float`, every `double` and every
    /// sixty four bit integer outright, so none of the four can round and none can raise.
    fn x87_across(
        &mut self,
        inst: Inst,
        put: &'static str,
        class: RegClass,
        get: &'static str,
    ) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let value = self.reg_of(source)?;
        let across = self.x87_crossing();
        let across = self.through(across);
        let into = self.x87_slot(result);
        let into = self.through(into);

        let block = self.at.expect("a block is being filled");
        let store = self.named(put);
        self.out.build(block, store).at(span).uses(value, class).mem(across).finish();
        self.x87_at(get, span, across);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A `long double` becoming a `float`, a `double` or an integer.
    ///
    /// Through memory for the reason above and in the same three instructions backwards. The two
    /// that go to a float round to nearest, which is what the control word says unless somebody
    /// has changed it and is what C wants. The two that go to an integer do not, which is why they
    /// do not come here.
    fn x87_back(
        &mut self,
        inst: Inst,
        put: &'static str,
        get: &'static str,
        class: RegClass,
    ) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.x87_slot(source);
        let from = self.through(from);
        let across = self.x87_crossing();
        let across = self.through(across);

        self.x87_at("fld_t", span, from);
        self.x87_at(put, span, across);
        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let load = self.named(get);
        self.out.build(block, load).at(span).def(reg, class).mem(across).finish();
        Ok(())
    }

    /// An `fpext` up to a `long double`, which is the only direction this machine has one in.
    fn x87_widen(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let sse = self.conv.sse_class;
        match self.source[self.narrow(inst)?].ty.bits() {
            32 => self.x87_across(inst, "movss_mr", sse, "fld_s"),
            64 => self.x87_across(inst, "movsd_mr", sse, "fld_l"),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// An `fptrunc` down from a `long double`, which is the other direction of the same.
    fn x87_narrow(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let sse = self.conv.sse_class;
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        match self.source[result].ty.bits() {
            32 => self.x87_back(inst, "fstp_s", "movss_rm", sse),
            64 => self.x87_back(inst, "fstp_l", "movsd_rm", sse),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// A `sitofp` up to a `long double`.
    ///
    /// Thirty two bits and sixty four, and nothing narrower, because C widens an integer to `int`
    /// before it converts one and the front end writes that widening down. An unsigned integer is
    /// not here at all: `fild` reads its operand as signed, so a value above the signed range
    /// comes back short by two to the sixty fourth and has to be added back, which is arithmetic
    /// rather than a move and waits with the rest of it.
    fn x87_from_signed(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let gpr = self.gpr;
        match self.source[self.narrow(inst)?].ty.bits() {
            32 => self.x87_across(inst, "mov_mr_32", gpr, "fild_l"),
            64 => self.x87_across(inst, "mov_mr_64", gpr, "fild_ll"),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// An `fptosi` down from a `long double`, which is the one conversion here with no single
    /// instruction behind it.
    ///
    /// C cuts towards zero and the unit rounds the way its control word says, so the store that
    /// takes the value off the stack is wrapped in the control word being saved, changed and put
    /// back. Five instructions around the one that does the work, and three more moving the word
    /// through a register, because this machine has no way to OR a constant into memory at this
    /// width. The unit has a shorter answer in `fisttp`, and `spec/10-backend.md` section 10.8
    /// says why it is not used: it is SSE3, the x86-64 baseline is not, and there is nothing here
    /// that can gate an instruction on a feature yet.
    fn x87_to_signed(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let (put, get) = match self.source[result].ty.bits() {
            32 => ("fistp_l", "mov_rm_32"),
            64 => ("fistp_ll", "mov_rm_64"),
            _ => return Err(self.unsupported(inst)),
        };
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let from = self.x87_slot(source);
        let from = self.through(from);
        let across = self.x87_crossing();
        let across = self.through(across);
        let control = self.x87_control();
        let saved = self.through(control).plus(0);
        let cut = self.through(control).plus(2);

        // The word the unit has now, into the first of the two slots and into a register, with the
        // rounding field turned to truncate on the way to the second.
        self.x87_at("fnstcw", span, saved);
        let block = self.at.expect("a block is being filled");
        let was = self.out.new_vreg(gpr);
        let read = self.named("mov_rm_16");
        self.out.build(block, read).at(span).def(was, gpr).mem(saved).finish();
        let now = self.out.new_vreg(gpr);
        let set = self.named("or_ri_16");
        // Two address, which is written out here rather than taken from the two shorthands
        // because the shorthands leave an operand unconstrained: this machine ORs into the
        // register it read, so the two have to be the same one and only the constraint says so.
        self.out
            .build(block, set)
            .at(span)
            .operand(mir::Operand::write(now, gpr).with(Constraint::Reuse(1)))
            .operand(mir::Operand::read(was, gpr))
            .imm(X87_TRUNCATE)
            .finish();
        let write = self.named("mov_mr_16");
        self.out.build(block, write).at(span).uses(now, gpr).mem(cut).finish();

        // The conversion itself, under the changed word, and then the word the unit had put back
        // before anything else runs.
        self.x87_at("fldcw", span, cut);
        self.x87_at("fld_t", span, from);
        self.x87_at(put, span, across);
        self.x87_at("fldcw", span, saved);

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let load = self.named(get);
        self.out.build(block, load).at(span).def(reg, gpr).mem(across).finish();
        Ok(())
    }

    /// A constant of this type, as the bits of it written into its slot.
    ///
    /// No x87 instruction at all, which is the surprise here. A slot holding an eighty bit value is
    /// the value, so a constant is ten bytes put where the value lives, and the unit never has to
    /// see it: whatever reads it will `fld` it out of the slot the way it reads any other one.
    ///
    /// Ten bytes in two goes, because the machine stores eight at a time and there is no store of
    /// an immediate to memory, so each half is put in a register first. The six bytes above the ten
    /// are left alone, since nothing reads them: they are the padding that makes the type sixteen
    /// wide and they are unspecified in the psABI rather than zero.
    ///
    /// The other way is a constant pool, an `fldt` of a symbol, and a relocation, which is what a
    /// compiler with somewhere to put a literal does. This back end has nowhere to put one yet, and
    /// four instructions in the frame is what that costs until it does.
    fn x87_const(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Imm(imm) = self.source[inst].extra else { return Err(self.unsupported(inst)) };
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let bits = self.source[imm].bits();
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let slot = self.x87_slot(result);
        let low = self.through(slot).plus(0);
        let high = self.through(slot).plus(8);

        let block = self.at.expect("a block is being filled");
        for (bytes, at, into) in
            [(bits as u64 as i64, low, "64"), (((bits >> 64) & 0xffff) as i64, high, "16")]
        {
            let held = self.out.new_vreg(gpr);
            let put = self.named(&format!("mov_ri_{into}"));
            self.out.build(block, put).at(span).def(held, gpr).imm(bytes).finish();
            let store = self.named(&format!("mov_mr_{into}"));
            self.out.build(block, store).at(span).uses(held, gpr).mem(at).finish();
        }
        Ok(())
    }

    /// One arithmetic instruction on two eighty bit values, as the four it takes.
    ///
    /// The left operand is pushed first and the right one on top of it, so the left ends up
    /// underneath and the answer wanted is the one below against the top in that order. Which of
    /// the two mnemonics computes that is a question about the spelling rather than about the
    /// machine, and the two spellings disagree. Intel's `FSUBP ST(i), ST(0)` is `ST(i) - ST(0)`
    /// and is `DE E8+i`, and AT&T's `fsubp` is `DE E0+i`, which is the other subtraction. This
    /// compiler writes AT&T and encodes what gas encodes, so what it asks for here is `fsubr_p`
    /// and `fdivr_p`, and the `r` is not a reversal of anything the code generator decided.
    ///
    /// An addition and a multiplication have one form each and do not care, which is why a test
    /// that reads the mnemonic back would not have caught this and one that computes a subtraction
    /// and checks the answer does.
    ///
    /// The answer is left where the deeper of the two was and the shallower is gone, which is what
    /// the `p` on the mnemonic means, so one push has already been paid back by the time the
    /// `fstp` runs and the stack is level again after it.
    ///
    /// Nothing here is folded and nothing is reused. Two values that are the same value get two
    /// pushes of the same slot, and an operand that was just computed is read back out of the slot
    /// it was written to rather than left on the stack, which costs a store and a load per
    /// instruction in an expression. Keeping a partial result on the stack across the next
    /// instruction's operands means knowing how deep the stack is at every point in the block, and
    /// that is a different thing from writing a group.
    fn x87_arith(&mut self, inst: Inst, with: &'static str) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let [left, right] = args[..] else { return Err(self.unsupported(inst)) };
        let span = self.source.span(inst);
        let left = self.x87_slot(left);
        let left = self.through(left);
        let right = self.x87_slot(right);
        let right = self.through(right);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_at("fld_t", span, left);
        self.x87_at("fld_t", span, right);
        self.x87_only(with, span);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A negation, which is a push, the sign bit turned over and a pop.
    ///
    /// `fchs` does not read the value as a number, so this is right for a zero, for an infinity
    /// and for a NaN, and it raises nothing on any of them. Which is what C asks of a negation and
    /// is not what subtracting from zero would give: `0.0L - x` is a different answer at a
    /// negative zero and a signalling one at a NaN.
    fn x87_flip(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.x87_slot(source);
        let from = self.through(from);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_at("fld_t", span, from);
        self.x87_only("fchs", span);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A comparison of two eighty bit values, as the two pushes and the one opcode that reads them.
    ///
    /// The right operand is pushed first and the left one on top of it, which is the other way
    /// round from the arithmetic and is because `fucomip` asks about the top against what is under
    /// it: the comparison this machine can do is the top's, so the value the predicate is about
    /// has to be the top. The pop that gets the loser off the stack and the byte that reads the
    /// flags are both inside the opcode, since what passes between those and the comparison is the
    /// flags and the flags are not something anything here can name.
    ///
    /// Which of the ten opcodes, and which way round, is the same table the vector comparisons
    /// match against in `rules/x86-64.rules`, and it has to stay the same table: a predicate that
    /// picked a different condition here than there would be a `long double` comparison that
    /// disagreed with the `double` comparison of the same two numbers, which is the one thing a
    /// wider format is not allowed to do.
    ///
    /// The always false and the always true are refused rather than folded into a constant,
    /// because a comparison this machine never has to do is one the optimizer should have removed
    /// and an instruction here that quietly agreed with it would hide that it did not.
    fn x87_compare(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::FloatPred(pred) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let (args, result) = self.ends(inst)?;
        let [left, right] = args[..] else { return Err(self.unsupported(inst)) };
        // Two of the fourteen need a second byte and an instruction to put the two together,
        // because they are two conditions at once: an ordered equal is equal and not unordered,
        // and an unordered not equal is either. The opcode carries all of that and says here only
        // that it writes somewhere else as well.
        let (name, reversed, both) = match pred {
            FloatPred::Ogt => ("fucomip_set_a", false, false),
            FloatPred::Oge => ("fucomip_set_ae", false, false),
            FloatPred::Olt => ("fucomip_set_a", true, false),
            FloatPred::Ole => ("fucomip_set_ae", true, false),
            FloatPred::One => ("fucomip_set_ne", false, false),
            FloatPred::Ord => ("fucomip_set_np", false, false),
            FloatPred::Uno => ("fucomip_set_p", false, false),
            FloatPred::Ueq => ("fucomip_set_e", false, false),
            FloatPred::Ult => ("fucomip_set_b", false, false),
            FloatPred::Ule => ("fucomip_set_be", false, false),
            FloatPred::Ugt => ("fucomip_set_b", true, false),
            FloatPred::Uge => ("fucomip_set_be", true, false),
            FloatPred::Oeq => ("fucomip_set_e_and_np", false, true),
            FloatPred::Une => ("fucomip_set_ne_or_p", false, true),
            FloatPred::False | FloatPred::True => return Err(self.unsupported(inst)),
        };
        let (top, under) = if reversed { (right, left) } else { (left, right) };

        let span = self.source.span(inst);
        let gpr = self.gpr;
        let under = self.x87_slot(under);
        let under = self.through(under);
        let top = self.x87_slot(top);
        let top = self.through(top);
        self.x87_at("fld_t", span, under);
        self.x87_at("fld_t", span, top);

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        // Taken before the instruction is started rather than inside it, since both come from the
        // same function being built and only one thing at a time may be adding to it.
        let spare = both.then(|| self.out.new_vreg(gpr));
        let opcode = self.named(name);
        let mut build = self.out.build(block, opcode).at(span).def(reg, gpr);
        if let Some(spare) = spare {
            build = build.def(spare, gpr);
        }
        build.finish();
        Ok(())
    }

    /// The operands and the one result of an instruction that has exactly one.
    fn ends(&self, inst: Inst) -> Result<(&'a [Value], Value), Unsupported> {
        let data = &self.source[inst];
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        Ok((&self.source[data.args], result))
    }

    /// The operand of a conversion, which is the end of it that is not the `long double`.
    fn narrow(&self, inst: Inst) -> Result<Value, Unsupported> {
        let args = &self.source[self.source[inst].args];
        args.first().copied().ok_or_else(|| self.unsupported(inst))
    }

    /// One `va_start`, as the fields of the list it was handed.
    ///
    /// On the four field list, two of them are numbers this already knows, and each costs an
    /// instruction to put in a register before it can be stored, because the machine here has no
    /// store of an immediate to memory. The other two are addresses in the frame, and each is a
    /// `lea` [`crate::finish`] finishes: the save area is one of the function's own stack objects,
    /// and the caller's argument area is where the parameters that had no register came from, which
    /// is the same place and the same fixup a parameter past the sixth already uses.
    ///
    /// On the list that is a pointer it is the second of those four and nothing else, since the
    /// whole of what that list says is where the walk is and the walk starts at the first argument
    /// the signature does not name. One `lea` and one store.
    ///
    /// What is written is exactly the fields [`crate::varargs`] describes, in the order they are
    /// laid out, so that reading this beside that table is the whole of the check.
    fn va_start(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Some(&list) = self.source[self.source[inst].args].first() else {
            return Err(self.unsupported(inst));
        };
        let started = self.varargs.ok_or_else(|| self.unsupported(inst))?;
        let list = self.reg_of(list)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);

        let (save, incoming) = match started {
            Varargs::Pointer { incoming } => (None, incoming),
            Varargs::Fields { save, incoming, integers, floats } => {
                let counts = [(varargs::GP_OFFSET, integers), (varargs::FP_OFFSET, floats)];
                for (at, count) in counts {
                    self.store_small(list, at, i64::from(count), span);
                }
                (Some(save), incoming)
            }
            Varargs::Aapcs { save, incoming, integers_end, floats_end, integers, floats } => {
                let counts =
                    [(varargs::aapcs::GR_OFFS, integers), (varargs::aapcs::VR_OFFS, floats)];
                for (at, count) in counts {
                    self.store_small(list, at, i64::from(count), span);
                }
                let overflow = self.overflow(block, incoming, span);
                let integers_top = self.frame_address_plus(block, save, integers_end);
                let floats_top = self.frame_address_plus(block, save, floats_end);
                let fields = [
                    (varargs::aapcs::STACK, overflow),
                    (varargs::aapcs::GR_TOP, integers_top),
                    (varargs::aapcs::VR_TOP, floats_top),
                ];
                for (at, held) in fields {
                    self.store_word(list, at, held, span);
                }
                return Ok(());
            }
        };

        // At the front of the list when that address is the whole of it, and at the field the
        // layout gives it when there are four, with the save area behind it.
        let overflow = self.overflow(block, incoming, span);
        let fields = match save {
            None => vec![(0, overflow)],
            Some(save) => {
                let save = self.frame_address(block, save);
                vec![(varargs::OVERFLOW, overflow), (varargs::SAVE_AREA, save)]
            }
        };
        for (at, held) in fields {
            self.store_word(list, at, held, span);
        }
        Ok(())
    }

    /// The first argument the signature did not name, which is as far up the caller's argument
    /// area as the ones it did name reached. Nothing here knows where that area is, so the distance
    /// is recorded the way a parameter read out of it is and finished with it.
    fn overflow(&mut self, block: mir::Block, incoming: u32, span: Span) -> mir::Reg {
        let overflow = self.out.new_vreg(self.gpr);
        let lea = self.named(self.selector.frame.lea);
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let made = self
            .out
            .build(block, lea)
            .at(span)
            .def(overflow, self.gpr)
            .mem(mir::Mem::at(sp))
            .finish();
        self.stack.arguments.push((made, incoming));
        overflow
    }

    /// Writes a small constant into a 32 bit field of a list.
    fn store_small(&mut self, list: mir::Reg, at: i64, value: i64, span: Span) {
        let block = self.at.expect("a block is being filled");
        let held = self.out.new_vreg(self.gpr);
        let load = mir::Opcode::new(self.names.intern(self.selector.abi.small));
        self.out.build(block, load).at(span).def(held, self.gpr).imm(value).finish();

        let head = (self.selector.abi.store)(Type::int(32)).expect("a store of a word");
        let store = mir::Opcode::new(self.names.intern(head));
        let mem = self.field(list, at);
        self.out.build(block, store).at(span).uses(held, self.gpr).mem(mem).finish();
    }

    /// Writes an address into a pointer field of a list.
    fn store_word(&mut self, list: mir::Reg, at: i64, held: mir::Reg, span: Span) {
        let block = self.at.expect("a block is being filled");
        let head = (self.selector.abi.store)(Type::int(64)).expect("a store of an address");
        let store = mir::Opcode::new(self.names.intern(head));
        let mem = self.field(list, at);
        self.out.build(block, store).at(span).uses(held, self.gpr).mem(mem).finish();
    }

    /// One field of a list, as the addressing mode that reaches it.
    fn field(&self, list: mir::Reg, at: i64) -> mir::Mem {
        let base = mir::Operand::read(list, self.gpr);
        mir::Mem::at(base).plus(i32::try_from(at).expect("a field of a list is a small offset"))
    }

    /// The address of a name: one `lea` off the instruction pointer, with the name on it.
    ///
    /// That is x86-64, and [`Selector::symbols`] is what says so. AArch64 writes the same thing as
    /// an `adrp` of the page and an `add` of the low twelve bits, which is one opcode with the name
    /// as its own symbol and no addressing mode, and the table read is an `adrp` and an `ldr`.
    ///
    /// The same instruction an `alloca` gets and for a related reason. An address that is not in
    /// the program is a `lea` of an addressing mode that names no register, and the mode carries
    /// the symbol so that [`rucc_asm`] can write it relative to `%rip` and leave the relocation
    /// for the assembler. Both halves of that already existed: the printer writes `sym(%rip)` and
    /// the encoder emits the relocation, because a call to a name the file does not define needed
    /// them first.
    ///
    /// One `mov` and not one `lea` when the name is one [`Elsewhere`] holds, because the distance
    /// the `lea` adds to the instruction pointer is a number only a link that puts the name in
    /// this program can work out, and the address of a function this file merely declares is not
    /// such a number. The load reads the address out of the slot the linker fills in instead. The
    /// linker turns it back into the `lea` when the name turns out to have been here all along,
    /// so this is not slower in the case that was already right.
    ///
    /// There is deliberately no name for this in [`crate::term`], which is what stops the address
    /// being folded into the instruction that reads it. Folding it is the right thing to do and
    /// is what turns a load of a global from two instructions into one, but it is a separate
    /// question about addressing modes and issue #282 is it. Until then the address is in a
    /// register before anything uses it, which is correct and one instruction longer.
    ///
    /// What this does not do is give the name anything to refer to. A module carries its globals
    /// and nothing writes them out, so a file that defines the variable it reads compiles to a
    /// reference the linker cannot resolve. Issue #293 is the other half.
    ///
    /// A thread-local variable is neither of the two above and is [`Self::thread_address`].
    fn address_of(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Symbol(symbol) = data.extra else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        if self.elsewhere.thread(symbol) {
            return self.thread_address(inst, symbol, result);
        }

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let far = self.elsewhere.holds(symbol);
        let symbols = self.selector.symbols;
        match if far { symbols.far } else { symbols.near } {
            Reach::Mode(name) => {
                let mem = if far { mir::Mem::got(symbol) } else { mir::Mem::of(symbol) };
                let opcode = self.named(name);
                self.out.build(block, opcode).at(span).def(reg, self.gpr).mem(mem).finish();
            }
            Reach::Own(name) => {
                let opcode = self.named(name);
                self.out.build(block, opcode).at(span).def(reg, self.gpr).symbol(symbol).finish();
            }
        }
        Ok(())
    }

    /// The address of a thread-local variable, which is this thread's copy of it.
    ///
    /// Neither instruction the ordinary case writes would mean anything here. There is no distance
    /// to the variable for a `lea` to add, because there is no variable: there is one copy of it per
    /// thread and they are at different addresses, so a link asked for the distance to the name
    /// refuses rather than picking one. And there is no address for a table slot to hold either, for
    /// the same reason.
    ///
    /// What is the same in every thread is where the variable sits inside the block of storage a
    /// thread gets, so that offset is what the link writes down, and the address of the running
    /// thread's block is what turns it into an address. x86-64 keeps that address in `%fs`, at the
    /// front of the block, so the whole of this is three instructions:
    ///
    /// ```text
    /// movq  x@gottpoff(%rip), %off   # how far into the block x sits, which the link fills in
    /// movq  %fs:0, %tp               # where this thread's block is, which only the machine knows
    /// addq  %tp, %off                # this thread's copy of x
    /// ```
    ///
    /// That is the initial exec model. It is one instruction longer than what gcc writes at `-O2`
    /// in an executable, which folds the addition into the instruction that uses the address, and
    /// the difference is issue #282 rather than anything about threads: nothing here folds an
    /// address into its reader yet. The link relaxes the first instruction into an immediate when it
    /// is making an executable, since it lays the blocks out and therefore knows the number, so the
    /// table slot costs nothing in the case that is common.
    ///
    /// It is not the most general model. A library loaded by `dlopen` gets its storage after the
    /// program is already running, and the block this reaches was laid out before it started, so
    /// the loader has to find room in that block for the library's variables. glibc keeps a little
    /// spare room for exactly this and a library that fits in it loads and runs; one that does not
    /// fails to load, with a message saying so. The model with no such limit calls `__tls_get_addr`
    /// and is what gcc writes under `-fPIC` by default, and it is issue #1104.
    ///
    /// So this is the model gcc writes under `-ftls-model=initial-exec`: right for an executable,
    /// right for a library the program is linked against, and a load that either works or is
    /// refused out loud for a library something opens later. What it is never is quietly wrong.
    ///
    /// AArch64 Linux is the same three steps. The slot is reached with `adrp` and `ldr` against
    /// `:gottprel:`, the thread pointer is `tpidr_el0` read with `mrs`, and the add has three
    /// operands. Apple's platforms reach a thread-local variable through a descriptor call instead,
    /// which is [`Self::thread_descriptor`].
    fn thread_address(
        &mut self,
        inst: Inst,
        symbol: Symbol,
        result: Value,
    ) -> Result<(), Unsupported> {
        if self.elsewhere.described() {
            return self.thread_descriptor(inst, symbol, result);
        }
        if self.elsewhere.indexed() {
            return self.thread_indexed(inst, symbol, result);
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let gpr = self.gpr;

        let offset = self.out.new_vreg(gpr);
        match self.selector.symbols.thread {
            Reach::Mode(name) => {
                let load = self.named(name);
                let mem = mir::Mem::thread(symbol);
                self.out.build(block, load).at(span).def(offset, gpr).mem(mem).finish();
            }
            Reach::Own(name) => {
                let load = self.named(name);
                self.out.build(block, load).at(span).def(offset, gpr).symbol(symbol).finish();
            }
        }
        let pointer = self.out.new_vreg(gpr);
        self.read_thread_pointer(block, span, pointer);

        // Two address on x86-64, for the reason `x87_to_int` gives: that machine adds into the
        // register it read, and only the constraint says the two are the same one.
        let reg = self.new_reg(result);
        let jumps = self.selector.jumps;
        let add = self.named(jumps.add);
        let written = mir::Operand::write(reg, gpr);
        let written = if jumps.two_address { written.with(Constraint::Reuse(1)) } else { written };
        self.out
            .build(block, add)
            .at(span)
            .operand(written)
            .operand(mir::Operand::read(offset, gpr))
            .operand(mir::Operand::read(pointer, gpr))
            .finish();
        Ok(())
    }

    /// A thread-local variable on Mach-O, which is a call.
    ///
    /// The slot the machine's thread load reads holds the address of the variable's descriptor
    /// there, `_v@TLVP` on x86-64 and `_v@TLVPPAGE` with `_v@TLVPPAGEOFF` on AArch64. The first
    /// word of the descriptor is the function that finds this thread's copy, and it takes the
    /// descriptor's address as its one argument and gives back the copy's address. That is the
    /// sequence clang writes on both machines.
    ///
    /// The call is built as an ordinary call through an address, so it costs what any call costs:
    /// everything the convention does not preserve is taken to be gone across it. Apple's function
    /// keeps more than that, all but the result and the two scratch registers on AArch64, and
    /// taking the fewer registers as gone would be faster. What this gives up is speed, and a
    /// function that reads a thread-local is no longer a leaf.
    fn thread_descriptor(
        &mut self,
        inst: Inst,
        symbol: Symbol,
        result: Value,
    ) -> Result<(), Unsupported> {
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let gpr = self.gpr;

        let descriptor = self.out.new_vreg(gpr);
        match self.selector.symbols.thread {
            Reach::Mode(name) => {
                let load = self.named(name);
                let mem = mir::Mem::thread(symbol);
                self.out.build(block, load).at(span).def(descriptor, gpr).mem(mem).finish();
            }
            Reach::Own(name) => {
                let load = self.named(name);
                let build = self.out.build(block, load).at(span);
                build.def(descriptor, gpr).symbol(symbol).finish();
            }
        }
        let finder = self.out.new_vreg(gpr);
        let word = (self.selector.abi.load)(Type::PTR).ok_or_else(|| self.unsupported(inst))?;
        let word = mir::Opcode::new(self.names.intern(word));
        let mem = mir::Mem::at(mir::Operand::read(descriptor, gpr));
        self.out.build(block, word).at(span).def(finder, gpr).mem(mem).finish();

        let args = [abi::Passing { ty: Type::PTR, reg: descriptor, abi: Abi::default() }];
        let what = abi::Calling {
            callee: abi::Callee::Through(finder),
            args: &args,
            returns: &[Type::PTR],
            variadic: false,
            named: 1,
            at: span,
        };
        let made = abi::call(&mut self.out, block, &what, self.conv, self.selector.abi, self.names)
            .map_err(|refused| Unsupported::Call { inst, refused })?;
        let calls = &mut self.stack.calls;
        *calls = Some(calls.unwrap_or(0).max(made.outgoing));
        let &[reg] = &made.results[..] else { return Err(self.unsupported(inst)) };
        self.regs[result.index()] = Some(reg);
        Ok(())
    }

    /// A thread-local variable on Windows, which is four loads and no call.
    ///
    /// `_tls_index` is this image's slot in the array of `.tls` copies the thread block holds at
    /// `%gs:88`, and the variable is as far into this thread's copy as it is into the section. The
    /// C runtime defines the index and the linker writes the offset. See [`crate::select::Indexed`] for
    /// the four instructions, which are the ones gcc writes.
    fn thread_indexed(
        &mut self,
        inst: Inst,
        symbol: Symbol,
        result: Value,
    ) -> Result<(), Unsupported> {
        let Some(indexed) = self.selector.symbols.indexed.as_ref() else {
            return Err(Unsupported::Unported { inst: Some(inst), what: Unported::Thread });
        };
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let gpr = self.gpr;

        let slot = self.out.new_vreg(gpr);
        let tls_index = self.names.intern("_tls_index");
        let index = self.named(indexed.index);
        self.out.build(block, index).at(span).def(slot, gpr).mem(mir::Mem::of(tls_index)).finish();

        let array = self.out.new_vreg(gpr);
        let load = self.named(indexed.load);
        let at = mir::Mem::in_segment(indexed.segment, indexed.at);
        self.out.build(block, load).at(span).def(array, gpr).mem(at).finish();

        let copy = self.out.new_vreg(gpr);
        let mem =
            mir::Mem::at(mir::Operand::read(array, gpr)).indexed(mir::Operand::read(slot, gpr), 8);
        self.out.build(block, load).at(span).def(copy, gpr).mem(mem).finish();

        let reg = self.new_reg(result);
        let add = self.named(indexed.add);
        let mem = mir::Mem::section(mir::Operand::read(copy, gpr), symbol);
        self.out.build(block, add).at(span).def(reg, gpr).mem(mem).finish();
        Ok(())
    }

    /// Refuses the thread pointer where it is not written, which is Mach-O. Apple keeps it in a
    /// different register from the one Linux does on both machines, and nothing written for it
    /// has been checked on one.
    fn threads_written(&self, inst: Inst) -> Result<(), Unsupported> {
        if self.elsewhere.described() || self.elsewhere.indexed() {
            return Err(Unsupported::Unported { inst: Some(inst), what: Unported::Thread });
        }
        Ok(())
    }

    /// The front of this thread's block into `reg`.
    ///
    /// On x86-64 that is the one thing no instruction can work out: `%fs` is not a register a
    /// program can read, and what it points at is a word holding its own address, so reading
    /// through it at zero is how the address is come by. AArch64 keeps it in `tpidr_el0`, which
    /// `mrs` reads.
    fn read_thread_pointer(&mut self, block: mir::Block, span: Span, reg: mir::Reg) {
        let gpr = self.gpr;
        match self.selector.symbols.pointer {
            Pointer::Segment(name, segment) => {
                let load = self.named(name);
                let at = mir::Mem::in_segment(segment, 0);
                self.out.build(block, load).at(span).def(reg, gpr).mem(at).finish();
            }
            Pointer::Own(name) => {
                let read = self.named(name);
                self.out.build(block, read).at(span).def(reg, gpr).finish();
            }
        }
    }

    /// `&&label`, GNU's address of a label, which is the same `lea` a global gets against a place
    /// in this same function.
    ///
    /// What the two have in common is the whole of the instruction: an address worked out from
    /// where the instruction is, which is what `(%rip)` means and is the only way this compiler
    /// reaches anything. What they do not have in common is what fills the four bytes in. A
    /// global is a name, so the number is a relocation and the linker writes it. A block is a
    /// place in this function, so both ends are in one section and the number is known as soon as
    /// the blocks have been laid out, which is why `rucc_asm` fills it in the way it fills in a
    /// jump rather than leaving a relocation behind.
    ///
    /// Nothing here says the block is one control can arrive at. That is said by the
    /// [`Opcode::IndirectBr`] that reads the address, which lists every block it can arrive at,
    /// and by nothing else: an address on its own is a number.
    fn block_address(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let Some(call) = self.source.successors(inst).next() else {
            return Err(self.unsupported(inst));
        };
        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let opcode = self.named(self.selector.jumps.near);
        let mem = mir::Mem::block(self.out_block(call.block));
        self.out.build(block, opcode).at(span).def(reg, self.gpr).mem(mem).finish();
        Ok(())
    }

    /// `goto *p`, GNU's computed goto, which is a jump through a register.
    ///
    /// Where it goes is not written here and cannot be. Every block it can arrive at is on the
    /// block this ends, the way every other arm is, and which of them the address holds is decided
    /// while the program runs. So this is one instruction with one operand, and the arms are
    /// copied across by [`Self::edges`] like anybody else's.
    fn indirect_branch(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let &address = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let reg = self.reg_of(address)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let name = self.selector.branch.indirect;
        let opcode = self.named(name);
        self.out.build(block, opcode).at(span).operand(mir::Operand::read(reg, self.gpr)).finish();
        Ok(())
    }

    /// A `switch` on an index from zero up, as a jump through a table of this function.
    ///
    /// Every `switch` that reaches here is one `crate::switch` left behind on purpose: it has
    /// already checked the value is inside the table and taken the lowest case off it, so the
    /// operand is a 64 bit index, the cases are the values from zero up with gaps where the
    /// program had no case, and the default is only where those gaps go. What is written is the
    /// shape gcc writes for the same statement in position independent code:
    ///
    /// ```text
    /// leaq    table(%rip), %base
    /// movslq  (%base,%index,4), %offset
    /// addq    %base, %offset
    /// jmp     *%offset
    /// ```
    ///
    /// The table holds distances from itself to each arm rather than addresses, which is what
    /// lets it be filled in by the assembler with nothing left for a linker to do. Each cell is
    /// stored as the place of an arm among this block's successors, which [`Self::edges`] copies
    /// across in the IR's own order, the default first and then one per case. See
    /// [`mir::Table`] for why a place and not a block.
    fn jump_table(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Switch(info) = data.extra else { return Err(self.unsupported(inst)) };
        let &index = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let ty = self.source[index].ty;
        if ty != Type::int(u64::BITS) {
            return Err(self.unsupported(inst));
        }
        let cases = self.source[self.source[info].cases].to_vec();
        let mut cells: Vec<u32> = Vec::new();
        for (arm, case) in cases.iter().enumerate() {
            let at = usize::try_from(case.signed(ty)).map_err(|_| self.unsupported(inst))?;
            if at >= cells.len() {
                cells.resize(at + 1, 0);
            }
            cells[at] = u32::try_from(arm + 1).map_err(|_| self.unsupported(inst))?;
        }
        let reg = self.reg_of(index)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let table = u32::try_from(self.out.tables.len()).expect("fewer tables than that");

        let jumps = self.selector.jumps;

        let base = self.out.new_vreg(gpr);
        let near = self.named(jumps.near);
        self.out.build(block, near).at(span).def(base, gpr).mem(mir::Mem::table(table)).finish();
        let offset = self.out.new_vreg(gpr);
        let cell =
            mir::Mem::at(mir::Operand::read(base, gpr)).indexed(mir::Operand::read(reg, gpr), 4);
        let load = self.named(jumps.cell);
        self.out.build(block, load).at(span).def(offset, gpr).mem(cell).finish();
        // Two address on x86-64, for the reason `thread_pointer` gives.
        let to = self.out.new_vreg(gpr);
        let add = self.named(jumps.add);
        let written = mir::Operand::write(to, gpr);
        let written = if jumps.two_address { written.with(Constraint::Reuse(1)) } else { written };
        self.out
            .build(block, add)
            .at(span)
            .operand(written)
            .operand(mir::Operand::read(offset, gpr))
            .operand(mir::Operand::read(base, gpr))
            .finish();
        let jump = self.named(self.selector.branch.indirect);
        let jump =
            self.out.build(block, jump).at(span).operand(mir::Operand::read(to, gpr)).finish();
        self.out.tables.push(mir::Table { jump, cells });
        Ok(())
    }

    /// `__builtin_setjmp`, which writes down where the function is so that a `__builtin_longjmp`
    /// somewhere else can bring control back here, and answers zero on the way past.
    ///
    /// Four words of the buffer, the three gcc writes and one of this compiler's own, and then the
    /// block ends: everything after the save in the IR block is put into a new machine IR block,
    /// and the address of that block is what went into the buffer. That is the whole reason the
    /// block is split here. An address points at a label, a machine IR block is the only thing in
    /// this representation that has one, and a save is in the middle of a block rather than at the
    /// end of one.
    ///
    /// # How the answer gets back
    ///
    /// Through the frame rather than through a register. The save writes a zero into a word of its
    /// own frame, puts the address of that word in the buffer, and the new block reads the word
    /// back. The restore writes a one through the address it finds in the buffer before it goes.
    /// So one load answers zero on the way past and one on the way back, and neither path has to
    /// agree with the other about a register.
    ///
    /// gcc does it the other way round, with a second block that sets the answer to one and is
    /// what the restore arrives at. That block is one nothing in the function jumps to, and a
    /// machine IR whose blocks are walked from the entry has nowhere to put such a thing: the
    /// allocator lays a function out in the line it is going to be emitted in, and a block no edge
    /// reaches is not in that line. The word in the frame costs eight bytes of stack and one load,
    /// and it needs nothing said anywhere about a block arrived at from outside.
    ///
    /// # What the allocator is told
    ///
    /// That every register it hands out is gone at the end of the first block. That is what makes
    /// the rest of the function right on the way back: control arrives from a `__builtin_longjmp`
    /// in some other function, and the only two registers that puts back are the stack pointer and
    /// the frame pointer, so anything this function still wants has to be in the frame those two
    /// reach. It is said with a write of every one of those registers, which is the same thing a
    /// call says about the registers a callee may destroy, on an instruction with nothing else on
    /// it so that the stores above are not caught up in it.
    fn saves_place(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let &buffer = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let buf = self.reg_of(buffer)?;
        let at = self.at.expect("a block is being filled");
        let gpr = self.gpr;
        let moves = self.selector.frame.moves(gpr).expect("a class the target says how to move");
        let store = self.named(moves.store);
        let load = self.named(moves.load);
        let lea = self.named(self.selector.frame.lea);
        let put = self.named(self.selector.frame.imm);
        let nothing =
            self.selector.frame.pad.expect("a target with an instruction that does nothing");
        let nothing = self.named(nothing);
        self.stack.saves_place = true;
        let answer = self.answer_slot();
        let back = self.out.create_block();

        // The zero this answers with, into the word a restore writes a one into.
        let zero = self.out.new_vreg(gpr);
        self.out.build(at, put).at(span).def(zero, gpr).imm(0).finish();
        let mem = self.frame_mem();
        let made = self.out.build(at, store).at(span).uses(zero, gpr).mem(mem).finish();
        self.stack.addresses.push((made, answer));

        // The four words: where that word is, where control comes back to, and the two registers
        // the restore puts back.
        let found = self.frame_address(at, answer);
        self.write_word(at, span, store, found, buf, JUMP_ANSWER);
        let pc = self.out.new_vreg(gpr);
        self.out.build(at, lea).at(span).def(pc, gpr).mem(mir::Mem::block(back)).finish();
        self.write_word(at, span, store, pc, buf, JUMP_PC);
        let frame = mir::Reg::physical(self.conv.frame_pointer);
        self.write_word(at, span, store, frame, buf, JUMP_FRAME);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        self.write_word(at, span, store, stack, buf, JUMP_STACK);

        // Nothing is in a register past this point, which is what the rest of the function is
        // allowed to assume about the way back in.
        let gone = self.across_jump();
        let mut build = self.out.build(at, nothing).at(span);
        for (reg, class) in gone {
            build = build.operand(mir::Operand::write(reg, class));
        }
        build.finish();

        // And the rest of the block, which is the block the address above was of.
        *self.out.succs_mut(at) = vec![mir::BlockCall::to(back)];
        self.at = Some(back);
        let reg = self.new_reg(result);
        let mem = self.frame_mem();
        let made = self.out.build(back, load).at(span).def(reg, gpr).mem(mem).finish();
        self.stack.addresses.push((made, answer));
        Ok(())
    }

    /// `__builtin_longjmp`, which reads a buffer a `__builtin_setjmp` filled in and goes there.
    ///
    /// Everything comes out of the buffer before anything is put back, and the four registers it
    /// comes out into are physical ones rather than values the allocator places. Both of those are
    /// about the same moment. The stack pointer is one of the things being put back, a value the
    /// allocator sent to the stack is reached through the stack pointer, and between the
    /// instruction that moves it and the jump there is no stack this function owns any more. A
    /// register named outright is a register nothing reloads into and nothing else is in, which is
    /// the only way to hold something across that moment.
    ///
    /// Four of them because that is how many things are in the air at once: where to go, the frame
    /// pointer to put back, the one the matching save is to answer with, and one register used
    /// twice, first for the address that one is written through and then for the stack pointer.
    ///
    /// Nothing after this in the block is reached. The marker is not a terminator, for the reason
    /// `spec/08-ir.md` gives, so the block goes on and whatever the front end wrote after it is
    /// written out and never run.
    fn comes_back(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let &buffer = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let buf = self.reg_of(buffer)?;
        let at = self.at.expect("a block is being filled");
        let gpr = self.gpr;
        let moves = self.selector.frame.moves(gpr).expect("a class the target says how to move");
        let load = self.named(moves.load);
        let store = self.named(moves.store);
        let mov = self.named(moves.mov);
        let put = self.named(self.selector.frame.imm);
        let jump = self.named(self.selector.branch.indirect);

        let held = self.jump_regs();
        if held.len() < JUMP_REGS {
            return Err(self.unsupported(inst));
        }
        let pc = mir::Reg::physical(held[0]);
        let frame = mir::Reg::physical(held[1]);
        let spare = mir::Reg::physical(held[2]);
        let one = mir::Reg::physical(held[3]);

        self.read_word(at, span, load, pc, buf, JUMP_PC);
        self.read_word(at, span, load, frame, buf, JUMP_FRAME);
        self.read_word(at, span, load, spare, buf, JUMP_ANSWER);

        // What the matching save answers with, written through the address that came out of the
        // buffer, because the word it goes in is in the other function's frame and this one has no
        // way of knowing where that is.
        self.out.build(at, put).at(span).def(one, gpr).imm(1).finish();
        let mem = mir::Mem::at(mir::Operand::read(spare, gpr));
        self.out.build(at, store).at(span).uses(one, gpr).mem(mem).finish();

        // The stack last of the four, so that the register the buffer is reached through is done
        // with before the stack it may have been spilled to stops being this function's.
        self.read_word(at, span, load, spare, buf, JUMP_STACK);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        self.copy(at, span, mov, stack, spare);
        let base = mir::Reg::physical(self.conv.frame_pointer);
        self.copy(at, span, mov, base, frame);

        // And the jump, which reads the two registers just put back as well as the address it
        // goes through. Neither of those is printed, because the target's spelling of an indirect
        // jump has one argument and it is the first one read. They are there because the code
        // control arrives at reaches its frame through them, and because without them the two
        // instructions above write registers nothing reads: a scheduler is then free to put the
        // jump in front of them, and at `-O2` it does.
        self.out
            .build(at, jump)
            .at(span)
            .operand(mir::Operand::read(pc, gpr))
            .operand(mir::Operand::read(stack, gpr))
            .operand(mir::Operand::read(base, gpr))
            .finish();
        Ok(())
    }

    /// One word of the buffer of a `__builtin_setjmp`, written from a register.
    fn write_word(
        &mut self,
        at: mir::Block,
        span: Span,
        store: mir::Opcode,
        from: mir::Reg,
        buf: mir::Reg,
        word: i32,
    ) {
        let mem = mir::Mem::at(mir::Operand::read(buf, self.gpr)).plus(word);
        self.out.build(at, store).at(span).uses(from, self.gpr).mem(mem).finish();
    }

    /// One word of that buffer, read back into a register.
    fn read_word(
        &mut self,
        at: mir::Block,
        span: Span,
        load: mir::Opcode,
        into: mir::Reg,
        buf: mir::Reg,
        word: i32,
    ) {
        let mem = mir::Mem::at(mir::Operand::read(buf, self.gpr)).plus(word);
        self.out.build(at, load).at(span).def(into, self.gpr).mem(mem).finish();
    }

    /// One register into another, which is the one shape of instruction the builder has no word
    /// for because neither operand is a definition of a value or a read of memory.
    fn copy(
        &mut self,
        at: mir::Block,
        span: Span,
        mov: mir::Opcode,
        into: mir::Reg,
        from: mir::Reg,
    ) {
        self.out
            .build(at, mov)
            .at(span)
            .operand(mir::Operand::write(into, self.gpr))
            .operand(mir::Operand::read(from, self.gpr))
            .finish();
    }

    /// The word a `__builtin_setjmp` in this function answers with, asked for once and kept.
    fn answer_slot(&mut self) -> usize {
        match self.answer {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: JUMP_WORD, align: JUMP_WORD });
                self.answer = Some(index);
                index
            }
        }
    }

    /// An address in this function's frame with nothing in its displacement, which is what an
    /// instruction reaching one of its stack objects is written with until [`crate::finish`] knows
    /// where the object is.
    fn frame_mem(&self) -> mir::Mem {
        mir::Mem::at(mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr))
    }

    /// Every register the allocator hands out, which is what a `__builtin_setjmp` destroys.
    ///
    /// Both files, since a `double` live across a save has the same problem an integer does. The
    /// two registers a frame is reached through are not here: the restore puts both of them back,
    /// which is the whole of what it puts back, and a function whose frame pointer was destroyed
    /// by its own save would have nothing left to find its caller with.
    fn across_jump(&self) -> Vec<(mir::Reg, RegClass)> {
        let mut gone = Vec::new();
        for &reg in self.conv.int_order {
            if reg == self.conv.stack_pointer || reg == self.conv.frame_pointer {
                continue;
            }
            gone.push((mir::Reg::physical(reg), self.gpr));
        }
        for &reg in self.conv.sse_order {
            gone.push((mir::Reg::physical(reg), self.conv.sse_class));
        }
        gone
    }

    /// The registers a `__builtin_longjmp` may hold things in while it puts a frame back.
    ///
    /// The ones the allocator hands out, less the two a frame is reached through. The scratch
    /// registers are not among them on purpose: the rewriter writes a reload into one of those
    /// wherever it likes, and one of these has to survive from the load that fills it to the
    /// instruction that reads it however many instructions apart those are.
    fn jump_regs(&self) -> Vec<PhysReg> {
        self.conv
            .int_order
            .iter()
            .copied()
            .filter(|&reg| {
                reg != self.conv.stack_pointer
                    && reg != self.conv.frame_pointer
                    && !self.selector.scratch.contains(&reg)
            })
            .collect()
    }

    /// A machine opcode of this target from the name the target gives it.
    fn named(&mut self, name: &str) -> mir::Opcode {
        mir::Opcode::new(self.names.intern(&format!("{}{name}", self.selector.prefix())))
    }

    /// `__builtin_frame_address` and `__builtin_return_address`, which are a walk up the chain of
    /// saved frame pointers and then one thing read at the end of it.
    ///
    /// Every frame that kept a frame pointer holds the caller's at the address the register points
    /// at, and the address that frame returns to one word above that, which is where the call
    /// instruction put it and where the prologue's push left it. So the walk is a load through the
    /// register for each link, the frame address is wherever the walk stopped, and the return
    /// address is one more load from a word above it. gcc 16.2.0 writes exactly this, measured on
    /// x86-64 at `-O2` for depths zero to three of both builtins.
    ///
    /// The function is given a frame pointer because of this, which is what [`Stack::walks_frames`]
    /// carries out to the layout. A depth of zero needs it as the answer and every depth above zero
    /// needs it as the start, so there is no case here where it is not wanted.
    ///
    /// How far the chain actually reaches is the program's business and not this one's. A caller
    /// compiled without a frame pointer has no link in it for the walk to follow, so a depth above
    /// zero is a promise about how the whole program was built. That is why gcc documents a nonzero
    /// depth as unsafe rather than as an answer, and why the depth is refused above a limit in
    /// `check/builtin/frame.rs` rather than walked as far as it says.
    fn frames(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Depth(depth) = data.extra else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let returning = data.opcode == Opcode::ReturnAddress;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let moves =
            self.selector.frame.moves(self.gpr).expect("a class the target says how to move");
        let load = self.named(moves.load);
        self.stack.walks_frames = true;

        // Where the walk is up to. The frame pointer to begin with, and the register the last load
        // wrote after that.
        let reg = self.new_reg(result);
        let mut base = mir::Reg::physical(self.conv.frame_pointer);
        for link in 0..depth {
            // The last load of a walk that is looking for a frame writes the answer itself, which
            // is what keeps a walk of so many links that many instructions and not one more.
            let ends_here = link + 1 == depth && !returning;
            let next = if ends_here { reg } else { self.out.new_vreg(self.gpr) };
            let at = mir::Mem::at(mir::Operand::read(base, self.gpr));
            self.out.build(block, load).at(span).def(next, self.gpr).mem(at).finish();
            base = next;
        }

        if returning {
            let up = i32::try_from(self.conv.return_address).expect("a word above the frame");
            let at = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(up);
            self.out.build(block, load).at(span).def(reg, self.gpr).mem(at).finish();
        } else if depth == 0 {
            // The one case with no load in it at all: the frame this function is running in is the
            // register itself, and a physical register is not one the allocator hands out, so the
            // answer is a copy of it.
            let mov = self.named(moves.mov);
            self.out
                .build(block, mov)
                .at(span)
                .operand(mir::Operand::write(reg, self.gpr))
                .operand(mir::Operand::read(base, self.gpr))
                .finish();
        }
        Ok(())
    }

    /// `__builtin_thread_pointer`, which is the front of the block [`Self::thread_address`] adds
    /// an offset to.
    ///
    /// The same one instruction, on its own this time and with nothing to add to it. A program
    /// writes this when what it wants is a number that is different in every thread and cheap to
    /// come by, rather than a variable of its own in the block, so there is no relocation here and
    /// no name for the link to resolve.
    fn thread_pointer(&mut self, inst: Inst) -> Result<(), Unsupported> {
        self.threads_written(inst)?;
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let reg = self.new_reg(result);
        self.read_thread_pointer(block, span, reg);
        Ok(())
    }

    /// What a named machine register holds, which is `register long x asm ("rbx");`.
    ///
    /// One move out of that register, with the register named as itself the way a register a
    /// template wrote is named, which is [`Self::itself`] and is the thing #1653 built. What it
    /// buys here is what it buys there: the register is part of the instruction the allocator
    /// sees, so it is a use the allocator will not have written over first, and the value goes
    /// into an ordinary one of its own that everything downstream reads.
    ///
    /// The whole sixty four bits are moved whatever the type is, because the register is that
    /// wide and a narrower type reads the low end of the copy, which is the same low end. A type
    /// wider than the register is refused, since there is no register holding it to read. On
    /// AArch64 a float may be kept in a vector register, `register double x asm ("d8");`, and it is
    /// moved out of that file the same way.
    ///
    /// A name the machine has not got is refused too, and is the only thing that can be wrong
    /// with the string: which register a name means is this machine's question and this is where
    /// the question is asked, at the same table `asm` asks about clobbers at. The sigil gcc
    /// allows in front of it is taken off here, because what the name is written with is syntax.
    fn register_value(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Symbol(symbol) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let ty = self.source[result].ty;
        let bits = if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() };
        if bits > ADDRESS_BITS {
            return Err(self.unsupported(inst));
        }
        let spelled = self.names.resolve(symbol).to_owned();
        let bare = spelled.strip_prefix('%').unwrap_or(&spelled);
        let named = if self.on_aarch64() {
            aarch64::named(bare)
        } else if self.class_of(ty) != self.gpr {
            return Err(self.unsupported(inst));
        } else {
            x86_64::gpr_named(bare).map(|(reg, _)| (reg, self.gpr))
        };
        let Some((held, file)) = named else {
            return Err(Unsupported::Register { inst, name: spelled });
        };
        // A float in a general purpose register, or a number in a vector one, is a register the
        // machine has holding a type that is not kept there, and would need a move between the
        // files that nothing here makes yet.
        if on_x87(ty) || self.class_of(ty) != file {
            return Err(self.unsupported(inst));
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let mov = self.selector.frame.moves(file).expect("a class the target says how to move").mov;
        let mov = self.named(mov);
        let into = self.new_reg(result);
        self.out
            .build(block, mov)
            .at(span)
            .operand(mir::Operand::write(into, file))
            .operand(
                mir::Operand::read(mir::Reg::physical(held), file).with(Constraint::Fixed(held)),
            )
            .finish();
        Ok(())
    }

    /// A conversion that converts nothing: the result is the operand under another type.
    ///
    /// `ptrtoint` and `inttoptr` at one width are the whole of this. An address on this machine is
    /// an integer as wide as the machine addresses, so a cast between the two changes what the
    /// type system calls the value and changes nothing about the value, and the register holding
    /// it is the register that already held it. The front end never writes either of them at any
    /// other width, because it widens or narrows around the cast rather than through it, so the
    /// two widths disagreeing here means the IR came from somewhere else and is refused rather
    /// than guessed at.
    ///
    /// Reading the operand first is what materializes it when it is a constant, which is the case
    /// that matters: a null pointer is an `inttoptr` of zero, and that zero has to reach a
    /// register before anything can call it an address.
    fn rename(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let [arg] = self.source[data.args] else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        if !self.is_address_width(self.source[arg].ty)
            || !self.is_address_width(self.source[result].ty)
        {
            return Err(self.unsupported(inst));
        }
        let reg = self.reg_of(arg)?;
        self.regs[result.index()] = Some(reg);
        Ok(())
    }

    /// One barrier, which on this machine is one instruction at the strongest ordering and no
    /// instruction at all at every other one.
    ///
    /// x86-64 is total store order, so the only reordering the machine does is a store followed by
    /// a load of a different address, and the only ordering that forbids that is sequential
    /// consistency. An acquire, a release and an acquire release fence are therefore already true
    /// of every program running here, and what a program wanted from writing one is that the
    /// compiler not move memory accesses across it. The optimizer has finished by the time this
    /// runs and nothing below reorders one access past another, so the constraint is already
    /// discharged and there is nothing to write.
    ///
    /// The strongest one is `mfence`, which is what gcc 16.2.0 writes for
    /// `__atomic_thread_fence(__ATOMIC_SEQ_CST)` and for `__sync_synchronize`. A locked instruction
    /// on the stack is faster on most parts and is what some compilers write instead; it is also a
    /// write to memory the program did not ask for, and the plain barrier is the one that says what
    /// it means.
    ///
    /// Written here by name rather than by a rule, for the same reason a `lea` of a symbol is:
    /// there is nothing in a barrier that a proof over bitvectors could discharge. It computes
    /// nothing, so there is no equality to state, and what makes it the right answer is the memory
    /// model, which the rule language cannot talk about.
    fn barrier(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Order(order) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        // AArch64 is not total store order, so every ordering above relaxed is an instruction
        // there. An acquire fence only has to keep later accesses after earlier loads, which is
        // `dmb ishld`, and everything stronger is the full `dmb ish` gcc writes for it.
        let name = match order {
            MemOrder::NotAtomic | MemOrder::Relaxed => return Ok(()),
            MemOrder::Acquire if self.on_aarch64() => "fence_acquire",
            _ if self.on_aarch64() => self.selector.fence,
            MemOrder::SeqCst => self.selector.fence,
            _ => return Ok(()),
        };
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let fence = self.named(name);
        self.out.build(block, fence).at(span).finish();
        Ok(())
    }

    /// The instruction a program stops on, which is one byte pair and no operands.
    ///
    /// `ud2` is an opcode the manual promises will never be given a meaning, so a processor that
    /// reaches it raises the fault for an instruction it does not know, and on Linux that arrives
    /// at the program as `SIGILL`. That is what `__builtin_trap` is for: a stop that cannot be
    /// caught by anything the program installed for an ordinary error, cannot be returned from,
    /// and leaves the address of the fault in the core file.
    ///
    /// Why not a call to `abort`. It is two bytes against a call and a relocation, it needs no
    /// library, and it works in the places this one is written most, which are a kernel and a
    /// freestanding program that has no `abort` to call. gcc 16.2.0 writes `ud2` here too.
    fn trap(&mut self, inst: Inst) {
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stop = self.named(self.selector.trap);
        self.out.build(block, stop).at(span).finish();
    }

    /// One hint that an address is about to be used, which is one instruction and no promise.
    ///
    /// Four instructions on this machine and the locality picks between them, which is what the
    /// number means: how much of the data will still be wanted after the access. None of it wanted
    /// is `prefetchnta`, which brings the line in without keeping it, and all of it wanted is
    /// `prefetcht0`, which brings it as close as the machine can. The two in between are the levels
    /// between those. Measured against gcc 16.2.0 on x86-64 rather than read off the manual: zero
    /// gives `prefetchnta`, one `prefetcht2`, two `prefetcht1` and three `prefetcht0`.
    ///
    /// Whether the access will write is not read on x86-64, and that is the machine rather than an
    /// omission. The write hint is `prefetchw`, which is not in the base instruction set, and gcc
    /// writes it only when the command line said the part has it. So a prefetch for a write is the
    /// same instruction as a prefetch for a read, which is what gcc 16.2.0 writes without
    /// `-mprfchw`. AArch64 has it in the base set, so there a write picks the four `pst` forms of
    /// `prfm` in place of the `pld` ones, at the same levels.
    ///
    /// The address goes in the addressing mode rather than in an operand, the way a store's does.
    /// It is built here as the plainest one there is, a register and nothing else, because what
    /// arrives is a value and folding an addition into the mode is a rule's job and no rule reaches
    /// this instruction. An address the program computed is therefore one `lea` or one add in front
    /// of this, which is what it would have been for the load the hint is about anyway.
    fn hint(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Prefetch(hint) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let [address] = args[..] else { return Err(self.unsupported(inst)) };
        // AArch64 has the write hint in the base instruction set, and gcc 16.2.0 writes it there.
        let write = hint.write && self.on_aarch64();
        let name = match (hint.locality, write) {
            (0, false) => "prefetch_nta",
            (1, false) => "prefetch_t2",
            (2, false) => "prefetch_t1",
            (PrefetchHint::MOST, false) => "prefetch_t0",
            (0, true) => "prefetch_w_nta",
            (1, true) => "prefetch_w_t2",
            (2, true) => "prefetch_w_t1",
            (PrefetchHint::MOST, true) => "prefetch_w_t0",
            // Nothing else exists. The checker reads a locality outside the range as zero and the
            // verifier refuses one that got here another way, so this is a hint that was built
            // rather than checked, and the safe answer for a hint is to write no instruction.
            _ => return Err(self.unsupported(inst)),
        };
        let base = self.reg_of(address)?;
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(name);
        self.out
            .build(block, opcode)
            .at(self.source.span(inst))
            .mem(mir::Mem::at(mir::Operand::read(base, self.gpr)))
            .finish();
        Ok(())
    }

    /// One compare and exchange, which is the instruction every other atomic on this machine is
    /// built out of.
    ///
    /// What the IR asks for is: read what is at an address, compare it against a value the program
    /// expected, put a second value there if the two were equal, and say both what was read and
    /// whether the exchange happened. The machine has exactly that instruction, and the `lock` in
    /// front of it is what makes the whole of it one step as far as every other processor is
    /// concerned.
    ///
    /// The ordering is not read here, and that is the memory model rather than an omission. A
    /// locked instruction on x86-64 is a full barrier whatever the program asked for, so a relaxed
    /// compare and exchange and a sequentially consistent one are the same instruction, and there
    /// is nothing weaker to emit for the weaker orderings. The failure ordering is not read for the
    /// same reason.
    ///
    /// The two values it produces are why this is written by name. The one the program compares
    /// against and the one it gets back are both `rax`, which the instruction reads and writes
    /// without being told, and the table says so with a fixed constraint at each end rather than
    /// leaving the allocator to find out. The second value is the byte behind it, which is the zero
    /// flag read out by a `sete`, and it is a definition of the same instruction so that the
    /// allocator knows the two are live together and never gives the byte the register the answer
    /// is in.
    fn exchange(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let results: Vec<Value> = self.source[inst].results().collect();
        let [addr, expected, desired] = args[..] else { return Err(self.unsupported(inst)) };
        let [old, exchanged] = results[..] else { return Err(self.unsupported(inst)) };
        if self.on_aarch64() {
            return self.exchange_a64(inst, [addr, expected, desired], [old, exchanged]);
        }

        // A value the machine can compare in one instruction, which is an integer or an address at
        // one of the four widths it has a compare and exchange for. Anything else is a type this
        // has no instruction for rather than a program that is wrong, and the front end refuses it
        // before ever getting here.
        let ty = self.source[old].ty;
        let bits = if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() };
        if (!ty.is_int() && !ty.is_ptr()) || !matches!(bits, 8 | 16 | 32 | 64) {
            return Err(self.unsupported(inst));
        }

        let base = self.reg_of(addr)?;
        let want = self.reg_of(expected)?;
        let put = self.reg_of(desired)?;
        let got = self.new_reg(old);
        let flag = self.new_reg(exchanged);

        let name = format!("cmpxchg_{bits}");
        let descs = self.selector.operands(&name).ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(&name);
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in descs.iter().zip([got, flag, want, put]) {
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            build = build.operand(operand);
        }
        build.mem(mir::Mem::at(mir::Operand::read(base, self.gpr))).finish();
        Ok(())
    }

    /// One read modify write, for the three operations this machine does in a single instruction.
    ///
    /// What the IR asks for is: read what is at an address, do something to it, put the answer back,
    /// say what was there before, and let nothing get between the three steps. The machine has
    /// `xchg` for putting a value there and `lock xadd` for adding one, and both leave what they
    /// found in the register the operand arrived in, which is why the value that comes back and the
    /// value that went in are one register here.
    ///
    /// A subtraction is the add over the negated operand, which is right at every width because the
    /// machine's arithmetic wraps and negating then adding is subtracting in two's complement
    /// whatever the operands were. The negate is a separate instruction in front, over a register of
    /// its own, so that the value the program handed over is not the one written on: an operand may
    /// be live after this and a program that read it again would read the negation.
    ///
    /// The ordering is not read, for the reason the compare and exchange beside this does not read
    /// it. `xchg` with memory locks the bus whether it is asked to or not and `lock xadd` is asked
    /// to, so both are full barriers on this machine and there is nothing weaker to fall to.
    ///
    /// Eight of the other ten never arrive, because `crate::retry` turned each of them into a loop
    /// around a compare and exchange before anything here saw it. The two that do arrive are the
    /// ones on floating values, and they are refused: a compare and exchange of a float wants the
    /// value carried through an integer of the same width, and an eighty bit float has no such
    /// width. Neither family of builtins can write one yet either, so a program that reaches this
    /// refusal is a program that reached an unimplemented builtin first.
    fn modify(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Rmw(op, _) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let [addr, operand] = args[..] else { return Err(self.unsupported(inst)) };
        let old = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;

        // A value the machine can exchange in one instruction, which is an integer at one of the
        // four widths it has these for. A pointer arrives as an address, so it is an integer by the
        // time it is here, and anything else is a type this has no instruction for.
        let ty = self.source[old].ty;
        if !ty.is_int() || !matches!(ty.bits(), 8 | 16 | 32 | 64) {
            return Err(self.unsupported(inst));
        }
        if self.on_aarch64() {
            return self.modify_a64(inst, op, [addr, operand], old);
        }
        let name = match op {
            RmwOp::Xchg => format!("xchg_{}", ty.bits()),
            RmwOp::Add | RmwOp::Sub => format!("xadd_{}", ty.bits()),
            _ => return Err(self.unsupported(inst)),
        };

        let base = self.reg_of(addr)?;
        let mut put = self.reg_of(operand)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        if op == RmwOp::Sub {
            let negated = self.out.new_vreg(self.gpr);
            let negate = self.named(&format!("neg_r_{}", ty.bits()));
            let descs = self
                .selector
                .operands(&format!("neg_r_{}", ty.bits()))
                .ok_or_else(|| self.unsupported(inst))?;
            let mut build = self.out.build(block, negate).at(span);
            for (desc, reg) in descs.iter().zip([negated, put]) {
                build = build.operand(mir::Operand {
                    reg,
                    class: desc.class,
                    role: desc.role,
                    constraint: desc.constraint,
                });
            }
            build.finish();
            put = negated;
        }

        let got = self.new_reg(old);
        let descs = self.selector.operands(&name).ok_or_else(|| self.unsupported(inst))?;
        let opcode = self.named(&name);
        let flags = self.carried(inst);
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in descs.iter().zip([got, put]) {
            build = build.operand(mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            });
        }
        build.mem(mir::Mem::at(mir::Operand::read(base, self.gpr))).finish();
        Ok(())
    }

    /// The width an AArch64 atomic works at, which is an integer or an address of one of the four
    /// widths the exclusive loads and stores have. Anything else is refused.
    fn atomic_bits(&self, inst: Inst, ty: Type) -> Result<u32, Unsupported> {
        let bits = if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() };
        if (!ty.is_int() && !ty.is_ptr()) || !matches!(bits, 8 | 16 | 32 | 64) {
            return Err(self.unsupported(inst));
        }
        Ok(bits)
    }

    /// One instruction by name, with its operands in the order the table lists them.
    fn written_as(&mut self, inst: Inst, name: &str, regs: &[mir::Reg]) -> Result<(), Unsupported> {
        let descs = self.selector.operands(name).ok_or_else(|| self.unsupported(inst))?;
        if descs.len() != regs.len() {
            return Err(self.unsupported(inst));
        }
        let block = self.at.expect("a block is being filled");
        let opcode = self.named(name);
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, &reg) in descs.iter().zip(regs) {
            build = build.operand(mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            });
        }
        build.finish();
        Ok(())
    }

    /// An acquiring load or a releasing store on AArch64, which is `ldar` or `stlr`.
    ///
    /// Only a relaxed access became the plain one above this, so what arrives is acquire or
    /// stronger for a load and release or stronger for a store. `ldar` and `stlr` are also
    /// sequentially consistent with each other, which is why the strongest ordering needs no fence
    /// on either side, and is what gcc 16.2.0 writes for all of them.
    fn ordered(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        if self.source[inst].opcode == Opcode::AtomicLoad {
            let [addr] = args[..] else { return Err(self.unsupported(inst)) };
            let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
            let bits = self.atomic_bits(inst, self.source[result].ty)?;
            let base = self.reg_of(addr)?;
            let got = self.new_reg(result);
            return self.written_as(inst, &format!("ldar_{bits}"), &[got, base]);
        }
        let [value, addr] = args[..] else { return Err(self.unsupported(inst)) };
        let bits = self.atomic_bits(inst, self.source[value].ty)?;
        let put = self.reg_of(value)?;
        let base = self.reg_of(addr)?;
        self.written_as(inst, &format!("stlr_{bits}"), &[put, base])
    }

    /// A compare and exchange on AArch64, which is a loop of an exclusive load and store.
    ///
    /// The loop is one instruction as far as everything below is concerned, so that nothing can
    /// be spilled or reloaded between the two exclusive accesses, which would lose the reservation
    /// on some parts every time. Its definitions are all early, since they are written before the
    /// last read. The acquiring and releasing forms are used whatever the ordering, which is what
    /// gcc writes at the strongest one and is never wrong at a weaker one. The yes or no comes out
    /// of the status register the store wrote, read as a flag after the loop.
    fn exchange_a64(
        &mut self,
        inst: Inst,
        [addr, expected, desired]: [Value; 3],
        [old, exchanged]: [Value; 2],
    ) -> Result<(), Unsupported> {
        let bits = self.atomic_bits(inst, self.source[old].ty)?;
        let base = self.reg_of(addr)?;
        let want = self.reg_of(expected)?;
        let put = self.reg_of(desired)?;
        let got = self.new_reg(old);
        let flag = self.new_reg(exchanged);
        self.written_as(inst, &format!("cmpxchg_{bits}"), &[got, flag, base, want, put])
    }

    /// A read modify write on AArch64, for the three operations that reach here, each a loop of
    /// an exclusive load and store for the reason the compare and exchange above is.
    fn modify_a64(
        &mut self,
        inst: Inst,
        op: RmwOp,
        [addr, operand]: [Value; 2],
        old: Value,
    ) -> Result<(), Unsupported> {
        let bits = self.atomic_bits(inst, self.source[old].ty)?;
        let base = self.reg_of(addr)?;
        let put = self.reg_of(operand)?;
        let got = self.new_reg(old);
        let status = self.out.new_vreg(self.gpr);
        match op {
            RmwOp::Xchg => {
                self.written_as(inst, &format!("xchg_{bits}"), &[got, status, base, put])
            }
            RmwOp::Add | RmwOp::Sub => {
                let name = if op == RmwOp::Add { "xadd" } else { "xsub" };
                let new = self.out.new_vreg(self.gpr);
                self.written_as(inst, &format!("{name}_{bits}"), &[got, new, status, base, put])
            }
            _ => Err(self.unsupported(inst)),
        }
    }

    /// One `asm` statement.
    ///
    /// An empty template is most of the inline assembly in a test suite, and it is not a corner
    /// case somebody wrote by accident. A program that wants a value computed where it stands, or a
    /// loop the optimizer must not touch, writes `asm volatile ("" : : : "memory")`, and forty
    /// years of bug reports about optimizers are full of them. What such a statement asks for is
    /// the barrier and the operand places, and no instructions at all.
    ///
    /// So the operands are the half that is always real: a constraint says where a value has to be,
    /// and where it has to be is still true when the template between them is empty.
    ///
    /// What the constraints ask for, on an empty template, is only ever that two operands share a
    /// place. Nothing reads a register no text names, so `"r"` on its own asks for a register and
    /// no particular one, and any register at all answers it. A matching constraint is different,
    /// because it says the output the assembly leaves is the place the input arrived in, and with
    /// no instructions between them that is the input unchanged. So it is a rename and not a move:
    /// the value is already in a register and the result is that register.
    ///
    /// An output nothing is tied to and no instruction writes is whatever the assembly left there,
    /// which for a template that writes nothing is whatever was in the register. That is a value
    /// the program is not entitled to, and this writes a zero rather than reading one, because the
    /// allocator has to be given a definition before a use whatever the program is entitled to.
    ///
    /// # A template with instructions in it
    ///
    /// [`x86_64::read`] turns the text into the opcodes this backend already has, which is what
    /// `spec/11-asm-objects-debug.md` section 11.1 asks for: the machine is described once, and an
    /// instruction a program wrote is looked up in that description rather than copied through to
    /// an assembler that has one of its own. So nothing here assembles anything. What it does is
    /// put the statement's operands where the opcode holds them, and from there an `asm` statement
    /// is ordinary machine code: the allocator picks the registers, the listing and the object file
    /// are written from the same table as every other instruction, and a spill around one works
    /// because there is nothing left about it for a spill to get wrong.
    ///
    /// A register the template named in its own text is the one thing in there that is nobody's
    /// operand, and it is placed as itself. See [`Self::itself`] for why that is safer here than
    /// the thing gcc does, which is to copy the name out and leave the allocator none the wiser.
    ///
    /// Two things are refused, both for one reason, which is that placing them by a guess gives a
    /// program that assembles into something other than what it says.
    ///
    /// An output the template writes more than once, which is one place with two definitions in it,
    /// and the machine IR between here and the allocator has one definition per register by
    /// construction. An output tied to an input and written once is not that: it is two registers
    /// the description ties together, which is what [`Place`] is about.
    ///
    /// An operand read where the opcode writes, or written where it reads. An output that has not
    /// been written yet is not a value, and an input the assembly writes over is a value something
    /// else may still be using.
    ///
    /// # A register the instruction uses without being told
    ///
    /// An instruction may reach a register its text does not name, and `cpuid` is all of them at
    /// once: the leaf goes in `eax`, the subleaf in `ecx`, and the answer comes back in all four
    /// registers. The description holds every bit of that already, so what is left is to say which
    /// of the statement's operands is in each of those registers, and the constraint letter is the
    /// one thing in an assembly statement that says it. `"=a"` is an output in `rax` and `"c"` is
    /// an input in `rcx`, which is why a program writing `cpuid` writes its constraints that way
    /// and has no choice about it.
    ///
    /// A register no letter named is one the statement put nothing in, and that is the usual case
    /// rather than an unusual one, since an instruction that answers four questions is written by
    /// programs that asked one. A write of one is the register being destroyed and gets a register
    /// of its own, which is what tells the allocator to keep everything else out of it. A read of
    /// one is a register the instruction looks at and the program never filled, which gets a zero
    /// for the reason [`Self::undefined`] gives.
    ///
    /// # The clobber list
    ///
    /// Read now, as the registers it names being written by every instruction of the template. By
    /// every one rather than by one of them, because the list says the assembly as a whole leaves
    /// them ruined and nothing here knows which line did it. Every entry has to be a register this
    /// machine has a name for or the statement is refused, since a name nobody read is a register
    /// nobody is keeping out of.
    ///
    /// `memory` and `cc` are the two entries that are not registers and both are skipped. `memory`
    /// says the assembly touches storage, which is already true of every `asm` this writes and is
    /// nothing a register list could hold. `cc` says it ruins the condition flags, and the flag
    /// tracking already has that from the instructions the template was read into, since it takes
    /// every instruction it does not recognize as writing them and every instruction here is one
    /// this machine describes. `flags` is the name gcc's own register table gives the same thing on
    /// this machine, so a program writing it has written `cc` and is read that way: tcc's
    /// `tests/tcctest.c` lists both on one statement.
    ///
    /// A clobber the instruction already writes is left off it. `cpuid` writes all four registers
    /// by description, and a statement listing three of them as clobbers as well is saying the
    /// same thing twice, which the allocator would read as one register with two definitions.
    ///
    /// On a template with nothing in it the list is ignored, as it was before, since a template
    /// with no instructions ruins nothing whatever it said about what it ruins.
    fn assembly(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Asm(asm) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[asm];
        if self.jumps_from_text(inst) {
            return Err(Unsupported::Assembly { inst, refused: Written::Goto });
        }
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };

        let constraints = self.names.resolve(info.constraints).to_string();
        let results: Vec<Value> = data.results().collect();
        let operands = AsmOperands::read(&constraints, &results, &self.source[data.args])
            .ok_or_else(refused)?;
        let list: Vec<AsmOperand<'_>> = operands.iter().copied().collect();

        // Read after the constraints and not before them, because a mnemonic whose suffix the
        // program left off is read at the width of the operands it names, and the operands are
        // what the constraints are a list of.
        let widths: Vec<Option<x86_64::Width>> = list
            .iter()
            .map(|operand| {
                let ty = self.source[operand.result.or(operand.value)?].ty;
                if !ty.is_scalar() {
                    return None;
                }
                x86_64::Width::of_bits(held_bits(ty))
            })
            .collect();
        // An operand in memory is an address the statement holds and an object the template names,
        // so the reader is told which ones those are and spells `%0` for one as the object.
        let memory: Vec<bool> = list.iter().map(|operand| operand.memory).collect();
        let template = self.names.resolve(info.template).to_string();
        // A clobber list naming a vector register goes the way a template this cannot read does.
        // The instructions read here are all in the general purpose file, and what keeps the text
        // already takes every vector register a call may use away from the allocator across it.
        let clobbers = self.names.resolve(info.clobbers);
        if clobbers.split(',').any(|entry| vector_named(entry).is_some()) {
            return self.kept(inst, &template, &list, &widths, &memory);
        }
        let steps = if template.trim().is_empty() {
            Vec::new()
        } else {
            match x86_64::read_in(&template, &widths, &memory) {
                Some(steps) => steps,
                None => return self.kept(inst, &template, &list, &widths, &memory),
            }
        };

        // Which operands the template writes, counted before anything is placed, because the answer
        // decides where each of the three below comes from and one instruction may name an operand
        // that a later one writes. Which of them any instruction puts in a register at all is
        // counted in the same walk, since an operand no instruction reaches that way is one nothing
        // has to put anywhere: a constant a template names only as the distance into an address is
        // written into the instruction, and a register holding a copy of it would be one nobody
        // reads. An operand the address is counted from is reached that way and is counted here for
        // that reason, because the walk below it is over the opcode's operands and an address is
        // not one of those.
        //
        // Whether any instruction reads an operand an instruction above it wrote is counted in the
        // same walk too. Such a template is one whose instructions have to be written in order with
        // each read taken from wherever the last write left the operand, which is what
        // [`Self::woven`] does, and so is one that writes an operand twice.
        let mut writes = vec![0usize; list.len()];
        let mut reads = vec![false; list.len()];
        let mut held = vec![false; list.len()];
        let mut after = false;
        for step in &steps {
            // A call out of the template writes every register the convention lets the callee
            // leave anything in, and an output pinned to one of those is written by it.
            if let x86_64::Step::Call { .. } = step {
                for index in self.lost(&list).into_iter().filter_map(|(_, _, index)| index) {
                    *writes.get_mut(index).ok_or_else(refused)? += 1;
                }
                continue;
            }
            let x86_64::Step::Line(line) = step else { continue };
            match line.at.and_then(|at| at.base) {
                Some(x86_64::Piece::Operand { index, .. }) => {
                    *held.get_mut(index).ok_or_else(refused)? = true;
                    after |= writes[index] > 0;
                }
                Some(x86_64::Piece::Reg { reg, .. }) => {
                    if let Some(index) = bound(&list, reg, Role::Use) {
                        *held.get_mut(index).ok_or_else(refused)? = true;
                        after |= writes[index] > 0;
                    }
                }
                _ => {}
            }
            let mut written = Vec::new();
            let form = x86_64::form(line.opcode).ok_or_else(refused)?;
            // Which registers the instruction reaches, asked the same way it is asked again when
            // the instruction is written. See [`Self::lettered`] for the one opcode whose answer
            // comes from the constraint letters rather than from the description.
            let lettered = (line.opcode == x86_64::LITERAL).then(|| self.lettered(&list));
            let (described, pieces) = match &lettered {
                Some((described, pieces)) => (described.as_slice(), pieces.as_slice()),
                None => (form.operands(), line.operands.as_slice()),
            };
            for (desc, piece) in described.iter().zip(pieces) {
                // An operand the instruction reaches without its text saying so is the statement's
                // only when a constraint letter put something there. One that is nobody's writes
                // nothing of the program's, so it is counted nowhere and is dealt with where it is
                // placed.
                let index = match *piece {
                    x86_64::Piece::Operand { index, .. } => index,
                    x86_64::Piece::Implicit { reg } => match bound(&list, reg, desc.role) {
                        Some(index) => index,
                        None => continue,
                    },
                    x86_64::Piece::Reg { reg, .. } => match bound(&list, reg, desc.role) {
                        Some(index) => index,
                        None => continue,
                    },
                };
                *held.get_mut(index).ok_or_else(refused)? = true;
                if matches!(desc.role, Role::Def | Role::EarlyDef) {
                    written.push(index);
                } else {
                    *reads.get_mut(index).ok_or_else(refused)? = true;
                    after |= writes[index] > 0;
                }
            }
            for index in written {
                *writes.get_mut(index).ok_or_else(refused)? += 1;
            }
        }
        let woven = after
            || writes.iter().any(|&count| count > 1)
            || steps.iter().any(|step| !matches!(step, x86_64::Step::Line(_)));

        // Where every operand is. Worked out in full before the first instruction is written, since
        // reading a value may be what puts it in a register in the first place, and that has to
        // happen in front of the assembly rather than in the middle of it.
        let mut places: Vec<Place> = vec![Place::default(); list.len()];
        for (index, operand) in list.iter().copied().enumerate() {
            let Some(result) = operand.result else {
                // An input, or an output the assembly was handed the address of, and both are a
                // value that arrives in a register and is read out of it, unless no instruction of
                // the template reads it out of one.
                let value = operand.value.ok_or_else(refused)?;
                if held[index] {
                    places[index].read = Some(self.reg_of(value)?);
                }
                continue;
            };
            let ty = self.source[result].ty;
            if on_x87(ty) {
                return Err(refused());
            }
            let tied = operands.tied_to(index);
            if let Some(from) = tied {
                if self.class_of(self.source[from].ty) != self.class_of(ty) {
                    return Err(refused());
                }
                places[index].read = Some(self.reg_of(from)?);
            }
            if writes[index] > 0 {
                places[index].write = Some(self.new_reg(result));
                continue;
            }
            match tied {
                // The place the input arrived in, which the assembly wrote nothing over. One
                // register, so this is a rename rather than a move.
                Some(_) => {
                    let reg = places[index].read.ok_or_else(refused)?;
                    self.regs[result.index()] = Some(reg);
                    places[index].write = Some(reg);
                }
                None => {
                    self.undefined(inst, result)?;
                    places[index].write = self.regs[result.index()];
                }
            }
        }

        // An output an instruction of the template also reads, which the statement said nothing
        // about because an output is what a statement says the other thing about. What it holds
        // there is undefined, and a program writing one means it: `sbb %0, %0` in libgmp's
        // `add_mssaaaa` subtracts a register from itself and is asking for the borrow bit rather
        // than for the number, so whatever the register held, the answer is the same. Undefined is
        // not the same as absent though, since the allocator is owed a definition in front of every
        // use, so it gets the zero an output nothing wrote gets and for the same reason.
        //
        // Unless an input could have been in the same register, in which case gcc's allocator puts
        // it there whenever it can and a program may have been written against that. tcc's test of
        // a call from a template reads its output `"=a" (s)` to pass `"r" (str)` to `getenv`, which
        // is only the string because gcc gave the two of them `rax`. So an output nothing has
        // written yet reads the one input that could share its place, when there is exactly one.
        // One written `&` is written before the inputs are read and shares nothing.
        for index in 0..list.len() {
            if !reads[index] || places[index].read.is_some() || places[index].write.is_none() {
                continue;
            }
            let reg = match self.shared(&list, index) {
                Some(value) => self.reg_of(value)?,
                None => self.seeded(inst, list[index])?,
            };
            places[index].read = Some(reg);
        }

        // Worked out once for the whole template, since the list is one list and every instruction
        // of the template gets it. Not worked out at all for a template with no instructions, which
        // is where there is nothing for it to go on.
        let clobbers = self.names.resolve(info.clobbers).to_string();
        let clobbered =
            if steps.is_empty() { Vec::new() } else { Self::clobbered(inst, &clobbers)? };

        // A template with a label in it is not one run of instructions, and what it is instead is
        // in [`Self::woven`], which is also where a template goes whose instructions read what the
        // ones above them wrote. Every other template is what it has always been, which is every
        // instruction of it written into the block the statement stands in.
        if woven {
            return self.woven(inst, &steps, &mut places, &list, &clobbered, &writes);
        }
        for step in &steps {
            let x86_64::Step::Line(line) = step else { continue };
            self.instruction(inst, line, &places, &list, &clobbered)?;
        }
        Ok(())
    }

    /// A template the reader could not take apart, kept as its text. See [`x86_64::Form::Template`].
    ///
    /// What the text names is spelled into it here, the way gcc prints it into its listing: a
    /// constant as `$5`, or as `5` under the `c` modifier, and the address of a name as the name.
    /// An object in memory is the one thing that cannot be spelled yet, since where it is depends on
    /// registers nothing has chosen, so it is left as a hole the writer fills and its address is the
    /// instruction's memory operand. One is all an instruction has room for, and every template this
    /// has met names one at most. A template that names an operand by name rather than by number is
    /// refused for now.
    ///
    /// # An operand in a register
    ///
    /// Which register is not known until the allocator has run, and the text is written down before
    /// then, so an operand in a register is a hole too. It names the instruction's own operand and
    /// the width the modifier asked for, or the width of the operand's type when there was none,
    /// and the writer spells whatever register the operand ended up in. What the text writes goes
    /// in first as definitions and what it reads goes in last as uses, with the registers below in
    /// between, so the allocator sees the statement as one instruction with every operand said. An
    /// output tied to an input, by `+` or by a number, reuses the input's register, and one written
    /// `&` is written early. Anything wider than a general purpose register is refused.
    ///
    /// A statement written with no colons is basic assembly, where `%` is a character like any
    /// other and a register is written `%eax`. The front end keeps no mark of which kind a statement
    /// was, so one with no operands and no clobbers is read as basic, which is what gcc would do for
    /// every such template but one written with empty colons around it.
    ///
    /// The registers a call may write are taken as written, see below for why.
    fn kept(
        &mut self,
        inst: Inst,
        template: &str,
        list: &[AsmOperand<'_>],
        widths: &[Option<x86_64::Width>],
        memory: &[bool],
    ) -> Result<(), Unsupported> {
        // Refused as the template it is, since keeping it is what was tried after reading it
        // failed, and what could not be kept is what it names rather than any one operand.
        let refused = || Unsupported::Assembly { inst, refused: Written::Template };
        let data = &self.source[inst];
        let Extra::Asm(asm) = data.extra else { return Err(self.unsupported(inst)) };
        let clobbers = self.names.resolve(self.source[asm].clobbers).to_string();
        let basic = list.is_empty() && clobbers.trim().is_empty();

        // Every register a call may leave anything in, as well as the ones the list names. The
        // text can write any register it likes without saying so, and tcc's tests do: gcc gets
        // away with that at `-O0` because nothing lives in a register between two statements
        // there, and taking these away from the allocator across the template is what gives the
        // same answer here. Nothing is written to them by this, so a register one template leaves
        // a value in is still holding it when the next template reads it.
        let a64 = self.on_aarch64();
        let mut clobbered: Vec<(PhysReg, RegClass)> =
            self.lost(list).into_iter().map(|(reg, class, _)| (reg, class)).collect();
        let named = if a64 {
            Self::clobbered_a64(inst, &clobbers)?
        } else {
            Self::clobbered_x86(inst, &clobbers, self.gpr, self.conv.sse_class)?
        };
        for &(reg, class) in &named {
            if !clobbered.iter().any(|&(had, of)| had == reg && of == class) {
                clobbered.push((reg, class));
            }
        }

        // The file each operand is in. On AArch64 `w` is a floating point or vector register and an
        // input tied to an output is in that output's file. A value whose type puts it in the other
        // file would need a move into this one first, which gcc makes and this does not yet, so
        // that is refused below.
        let mut files = vec![self.gpr; list.len()];
        if a64 {
            let constraints = self.names.resolve(self.source[asm].constraints);
            for (file, entry) in files.iter_mut().zip(constraints.split(',')) {
                if vector_letter(entry) {
                    *file = self.conv.sse_class;
                }
            }
            for index in 0..list.len() {
                if let Some(&file) = list[index].tied.and_then(|output| files.get(output)) {
                    files[index] = file;
                }
            }
        }
        let pins: Vec<_> = list.iter().map(|operand| self.pinned_here(operand)).collect();
        let pin = |index: usize, file: RegClass| match pins[index] {
            Some((at, class)) if class == file => Ok(Some(Constraint::Fixed(at))),
            Some(_) => Err(refused()),
            None => Ok(None),
        };

        // The operands in a register, as the instruction's own. An input the text is handed as a
        // constant or as the address of a name is spelled into the text instead, when its
        // constraint allows a constant at all and no output is tied to it. `"a" (0x1234)` is a
        // register holding the number, the way gcc loads it, since the text may ask for `%h0`.
        let mut defs: Vec<mir::Operand> = Vec::new();
        let mut uses: Vec<mir::Operand> = Vec::new();
        let mut def_of: Vec<Option<usize>> = vec![None; list.len()];
        let mut use_of: Vec<Option<usize>> = vec![None; list.len()];
        if !basic {
            for (index, operand) in list.iter().enumerate() {
                let Some(result) = operand.result else { continue };
                let (ty, file) = (self.source[result].ty, files[index]);
                if on_x87(ty) || self.class_of(ty) != file {
                    return Err(refused());
                }
                let reg = self.new_reg(result);
                let written = if operand.early {
                    mir::Operand::write_early(reg, file)
                } else {
                    mir::Operand::write(reg, file)
                };
                def_of[index] = Some(defs.len());
                defs.push(match pin(index, file)? {
                    Some(fixed) => written.with(fixed),
                    None => written,
                });
            }
            for (index, operand) in list.iter().enumerate() {
                let Some(value) = operand.value else { continue };
                let spelled = operand.result.is_none()
                    && operand.tied.is_none()
                    && operand.immediate
                    && (self.number(value).is_some() || self.named_address(value).is_some());
                // An operand in memory is spelled on AArch64 as the register its address is in,
                // which is `[x3]` and is an address every instruction that takes one reads.
                if (operand.memory && !a64) || spelled {
                    continue;
                }
                let (ty, file) = (self.source[value].ty, files[index]);
                if on_x87(ty) || self.class_of(ty) != file {
                    return Err(refused());
                }
                let read = mir::Operand::read(self.reg_of(value)?, file);
                use_of[index] = Some(uses.len());
                uses.push(match pin(index, file)? {
                    Some(fixed) => read.with(fixed),
                    None => read,
                });
            }
        }
        // Every register a call may write is more than a template can give up when it has more
        // operands in registers than the convention keeps across a call. `sodium_sub` in
        // libsodium's `utils.c` is one: eight outputs written early and two inputs pinned, against
        // the five registers SysV preserves, so an output has nowhere to go and nothing is left to
        // carry one to its slot either. gcc gives that template ten registers, and a program that
        // writes a register it did not name is only owed what gcc would have done, which here is
        // one of the ten. So the registers taken as written without being named are handed back,
        // from the end of the convention's order, until the operands fit in what is left. One the
        // list names or an operand is pinned to stays where it is.
        let fixed_to: Vec<PhysReg> = defs
            .iter()
            .chain(&uses)
            .filter_map(|operand| match operand.constraint {
                Constraint::Fixed(at) => Some(at),
                _ => None,
            })
            .collect();
        let wanted = defs.iter().chain(&uses).count() - fixed_to.len();
        let int = self.conv.int_class;
        let free = |clobbered: &[(PhysReg, RegClass)]| {
            self.conv
                .int_order
                .iter()
                .filter(|&&reg| !fixed_to.contains(&reg) && !clobbered.contains(&(reg, int)))
                .count()
        };
        while free(&clobbered) < wanted {
            let Some(at) = clobbered.iter().rposition(|&(reg, class)| {
                class == int && !named.contains(&(reg, class)) && !fixed_to.contains(&reg)
            }) else {
                break;
            };
            clobbered.remove(at);
        }

        // A register an output is pinned to is that output's definition and not a clobber as well.
        // One an input is pinned to is written as the instruction finishes, the way a call writes
        // the register its argument came in, and every other one is written early, since the text
        // may write it before it has read its inputs and an input must not be in it.
        let mut written: Vec<mir::Operand> = Vec::new();
        for (reg, class) in clobbered {
            let fixed = |operand: &mir::Operand| {
                operand.class == class && operand.constraint == Constraint::Fixed(reg)
            };
            if defs.iter().any(fixed) {
                continue;
            }
            let reg = mir::Reg::physical(reg);
            written.push(if uses.iter().any(fixed) {
                mir::Operand::write(reg, class)
            } else {
                mir::Operand::write_early(reg, class)
            });
        }
        // An output tied to an input is one register, which the definition says by reusing the
        // use, or by both being fixed to the same one when the output was pinned.
        //
        // A reused register is kept from every other input already, since the allocator counts the
        // output as taken from where the instruction reads. So `+&` asks for nothing more than `+`,
        // and saying it as an early write as well costs a register: the allocator only hands an
        // output the register of the input it reuses when the output starts at the instruction, and
        // an early one starts a point sooner, so it gets one of its own and a copy in front. Eleven
        // operands written that way in xz's range decoder need seventeen registers and run out. The
        // one case where `&` still means something is an input reading the same value as the one
        // tied, which would be in the same register and read after the output was written.
        let first_use = defs.len() + written.len();
        for (output, operand) in list.iter().enumerate() {
            let Some(def) = def_of[output] else { continue };
            let input = if operand.value.is_some() {
                Some(output)
            } else {
                list.iter().position(|entry| entry.tied == Some(output))
            };
            let Some(read) = input.and_then(|input| use_of[input]) else { continue };
            match defs[def].constraint {
                Constraint::Fixed(_) => uses[read].constraint = defs[def].constraint,
                _ => {
                    let at = u8::try_from(first_use + read).map_err(|_| refused())?;
                    defs[def].constraint = Constraint::Reuse(at);
                    let source = uses[read].reg;
                    let shared = uses
                        .iter()
                        .enumerate()
                        .any(|(other, operand)| other != read && operand.reg == source);
                    if defs[def].role == Role::EarlyDef && !shared {
                        defs[def].role = Role::Def;
                    }
                }
            }
        }

        // A line naming an operand in a register, with an instruction on it the reader knows, is
        // one the reader refused for a reason of its own, and keeping it as text would hand the
        // assembler what the reader already said no to. `addq %1, %k0` is that: a quadword add
        // into half a register. What is kept is a line with an instruction nothing here knows.
        let registered = |index: usize| def_of[index].is_some() || use_of[index].is_some();
        if !a64 && (0..list.len()).any(registered) {
            for line in template.split(['\n', ';']) {
                if names_one(line, registered)
                    && x86_64::known(line, widths, memory)
                    && x86_64::read_in(line, widths, memory).is_none()
                {
                    return Err(refused());
                }
            }
        }

        let mut text = String::with_capacity(template.len());
        let mut memory: Option<usize> = None;
        if basic {
            text.push_str(template);
        } else {
            let mut chars = template.chars().peekable();
            // Inside `{att|intel}`, and past the `|` in it, which is the half nobody reads. AArch64
            // has one dialect, and a brace there is a list of vector registers.
            let mut dialect = false;
            let mut skipped = false;
            while let Some(c) = chars.next() {
                match c {
                    '{' if !a64 => {
                        dialect = true;
                        continue;
                    }
                    '|' if dialect => {
                        skipped = true;
                        continue;
                    }
                    '}' if dialect => {
                        dialect = false;
                        skipped = false;
                        continue;
                    }
                    _ if skipped => continue,
                    '%' => {}
                    _ => {
                        text.push(c);
                        continue;
                    }
                }
                match chars.peek().copied() {
                    Some(c @ ('%' | '{' | '|' | '}')) => {
                        chars.next();
                        text.push(c);
                        continue;
                    }
                    Some('=') => {
                        chars.next();
                        text.push_str(&inst.index().to_string());
                        continue;
                    }
                    _ => {}
                }
                let modifier = match chars.peek().copied() {
                    Some(c) if c.is_ascii_alphabetic() => {
                        chars.next();
                        Some(c)
                    }
                    _ => None,
                };
                let mut digits = String::new();
                while let Some(c) = chars.peek().copied().filter(char::is_ascii_digit) {
                    digits.push(c);
                    chars.next();
                }
                let index: usize = digits.parse().map_err(|_| refused())?;
                let operand = list.get(index).ok_or_else(refused)?;
                if operand.memory && a64 {
                    let at = use_of[index].map(|at| first_use + at).ok_or_else(refused)?;
                    if modifier.is_some() {
                        return Err(refused());
                    }
                    text.push('[');
                    text.push_str(&template_reg(at, 'x'));
                    text.push(']');
                    continue;
                }
                if operand.memory {
                    if modifier.is_some() || memory.is_some_and(|had| had != index) {
                        return Err(refused());
                    }
                    memory = Some(index);
                    text.push_str(x86_64::TEMPLATE_MEM);
                    continue;
                }
                let placed = def_of[index].or(use_of[index].map(|at| first_use + at));
                if let Some(at) = placed {
                    let value = operand.result.or(operand.value).ok_or_else(refused)?;
                    let bits = held_bits(self.source[value].ty);
                    // `w` and `x` are the two names every general purpose register has, and one
                    // with no modifier is named at the width of its type, as gcc names it. A
                    // vector register with no modifier is `v`, which is what gcc writes for one
                    // whatever is in it, and the modifiers name the scalar views of it.
                    let width = if a64 && files[index] != self.gpr {
                        match modifier {
                            None => 'v',
                            Some(view @ ('b' | 'h' | 's' | 'd' | 'q')) => view,
                            Some(_) => return Err(refused()),
                        }
                    } else if a64 {
                        match (modifier, bits) {
                            (None, 8 | 16 | 32) | (Some('w'), _) => 'w',
                            (None, 64) | (Some('x'), _) => 'x',
                            _ => return Err(refused()),
                        }
                    } else {
                        match modifier {
                            None => match held_bits(self.source[value].ty) {
                                8 => 'b',
                                16 => 'w',
                                32 => 'k',
                                64 => 'q',
                                _ => return Err(refused()),
                            },
                            Some(width @ ('b' | 'w' | 'k' | 'q')) => width,
                            // The second byte is a name only four registers have, so it is taken for
                            // an operand pinned to one of them and for nothing the allocator chose.
                            Some('h') if pinned(operand).and_then(x86_64::gpr_high).is_some() => {
                                'h'
                            }
                            Some(_) => return Err(refused()),
                        }
                    };
                    text.push_str(&template_reg(at, width));
                    continue;
                }
                let value = operand.value.ok_or_else(refused)?;
                let bare = match modifier {
                    None => false,
                    Some('c' | 'P' | 'p') => true,
                    Some(_) => return Err(refused()),
                };
                // A constant is bare on AArch64 whatever the modifier, which is how gcc prints one
                // there and a form GNU as takes wherever `#` would go.
                if !bare && !a64 {
                    text.push('$');
                }
                if let Some(number) = self.number(value) {
                    text.push_str(&number.to_string());
                } else if let Some(symbol) = self.named_address(value) {
                    text.push_str(&template_name(self.names.resolve(symbol)));
                } else {
                    return Err(refused());
                }
            }
        }

        // An object in this function's frame is named by where it is in the frame, the way gcc
        // names it, rather than by a register its address was put in first. The text may write
        // registers it does not declare, and tcc's tests do: one that writes `%ecx` behind the
        // compiler's back would otherwise take the address with it.
        let mut local = None;
        let at = match memory.filter(|_| !a64) {
            Some(index) => {
                let value = list[index].value.ok_or_else(refused)?;
                local = self.local_of(value);
                let base = match local {
                    Some(_) => mir::Reg::physical(self.conv.stack_pointer),
                    None => self.reg_of(value)?,
                };
                Some(mir::Mem::at(mir::Operand::read(base, self.gpr)))
            }
            None => None,
        };
        let symbol = self.names.intern(&text);
        let opcode = self.named(if a64 { aarch64::TEMPLATE } else { x86_64::TEMPLATE });
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let mut build = self.out.build(block, opcode).at(span).symbol(symbol);
        for operand in defs.into_iter().chain(written).chain(uses) {
            build = build.operand(operand);
        }
        if let Some(mem) = at {
            build = build.mem(mem);
        }
        let made = build.finish();
        if let Some(local) = local {
            self.stack.addresses.push((made, local));
        }
        Ok(())
    }

    /// The object in this function's frame a value is the address of, for one an `alloca` of a
    /// size known here made. See [`Self::reserve`], which is where the `lea` it is found by came
    /// from.
    fn local_of(&self, value: Value) -> Option<usize> {
        let Def::Result { inst, .. } = self.source[value].def else { return None };
        if self.source[inst].opcode != Opcode::Alloca
            || !self.source[self.source[inst].args].is_empty()
        {
            return None;
        }
        let reg = self.regs[value.index()]?;
        self.stack.addresses.iter().find_map(|&(made, local)| {
            let data = &self.out[made];
            let defined = self.out[data.operands].first()?;
            (defined.reg == reg).then_some(local)
        })
    }

    /// The name a value is the address of, for one a `global_addr` defined.
    fn named_address(&self, value: Value) -> Option<Symbol> {
        let Def::Result { inst, .. } = self.source[value].def else { return None };
        if self.source[inst].opcode != Opcode::GlobalAddr {
            return None;
        }
        let Extra::Symbol(symbol) = self.source[inst].extra else { return None };
        Some(symbol)
    }

    /// A register holding a zero, for an operand of a template that is read before anything filled
    /// it.
    ///
    /// Two things ask for this and they are the same thing twice. An output the template reads has
    /// nothing to be read out of until the instruction that writes it has run, and a loop carries
    /// an operand into a block before the instruction that fills it, so both are a use in front of
    /// every definition. What the program is owed there is nothing, since the value is undefined
    /// either way, and what the allocator is owed is a register something wrote.
    fn seeded(&mut self, inst: Inst, operand: AsmOperand<'_>) -> Result<mir::Reg, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let value = operand.result.or(operand.value).ok_or_else(refused)?;
        let class = self.class_of(self.source[value].ty);
        if class != self.gpr {
            return Err(refused());
        }
        let block = self.at.expect("a block is being filled");
        let reg = self.out.new_vreg(class);
        let put = self.named("mov_ri_64");
        self.out.build(block, put).at(self.source.span(inst)).def(reg, class).imm(0).finish();
        Ok(reg)
    }

    /// A template with labels in it, as the blocks its jumps leave and arrive at.
    ///
    /// A statement is an instruction of the IR and stands inside one block, so a template that
    /// jumps has to stop being one thing. Each label becomes a block, each jump ends the block it
    /// stands in and gives it two arms, and whatever follows the statement goes into whichever
    /// block the walk finished in, which is what [`Self::block`] already reads off `self.at` and
    /// what [`Self::saves_place`] already does for the same reason.
    ///
    /// # What is carried between them
    ///
    /// The machine IR here is in the form where a register is written once, so an operand written
    /// inside a loop and read again at the top of it cannot be one register. What arrives at the
    /// top is a parameter of that block, and every jump to it carries whichever register held the
    /// operand where the jump stands. That is the whole of the bookkeeping: every block a label
    /// made takes one parameter for each operand that is in a register at all, in one order, so an
    /// arm's arguments and a block's parameters are the same list read twice.
    ///
    /// Which register an operand is in at each point is kept in the read half of its place, since
    /// that is what the instructions below read it out of. An instruction that writes an operand
    /// leaves it in the register it wrote, and a jump below carries that one. The block an
    /// untaken jump falls into is arrived at one way only and so takes no parameters, and nothing
    /// about where the operands are changes there.
    ///
    /// An operand written by the template and filled by nothing is written as a zero first, for
    /// the reason [`Self::undefined`] gives and one more: a jump may carry it before the
    /// instruction that fills it has run, and an argument has to be a register something wrote.
    ///
    /// # The condition state
    ///
    /// Nothing carries it and nothing has to. The instruction that sets it and the jump that reads
    /// it are both written here, next to each other in one block, and what the allocator may put
    /// between them is a move, which on this machine leaves the condition state alone. The edge
    /// into a block a loop goes back to is a critical edge and `crate::split` gives it a block of
    /// its own, so the moves an arm turns into land behind the jump rather than in front of it.
    fn woven(
        &mut self,
        inst: Inst,
        steps: &[x86_64::Step],
        places: &mut [Place],
        list: &[AsmOperand<'_>],
        clobbered: &[PhysReg],
        writes: &[usize],
    ) -> Result<(), Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let span = self.source.span(inst);

        // Which operands are carried, which is every one that is in a register at all. An operand
        // the template never puts in one, such as a constant it names only as the distance into an
        // address, is in the instruction and has nowhere to be carried from.
        let mut carried: Vec<(usize, RegClass)> = Vec::new();
        for (index, operand) in list.iter().enumerate() {
            if places[index].read.is_none() && places[index].write.is_none() {
                continue;
            }
            let value = operand.result.or(operand.value).ok_or_else(refused)?;
            let ty = self.source[value].ty;
            if on_x87(ty) {
                return Err(refused());
            }
            carried.push((index, self.class_of(ty)));
        }

        // What each of them holds where the template starts.
        for &(index, _) in &carried {
            if places[index].read.is_some() {
                continue;
            }
            if writes[index] == 0 {
                places[index].read = places[index].write;
                continue;
            }
            places[index].read = Some(self.seeded(inst, list[index])?);
        }

        // The blocks, made before the walk because a jump forwards names a label the walk has not
        // reached yet.
        let mut labels: Vec<(&str, mir::Block, Vec<mir::Reg>)> = Vec::new();
        for step in steps {
            let x86_64::Step::Label(name) = step else { continue };
            let block = self.out.create_block();
            let mut params = Vec::with_capacity(carried.len());
            for &(_, class) in &carried {
                params.push(self.out.append_param(block, class));
            }
            labels.push((name.as_str(), block, params));
        }

        let mut wrote: Vec<usize> = Vec::new();
        for step in steps {
            match step {
                x86_64::Step::Label(name) => {
                    let (block, params) = Self::went(&labels, name).ok_or_else(refused)?;
                    let from = self.at.expect("a block is being filled");
                    let args = Self::held(places, &carried).ok_or_else(refused)?;
                    *self.out.succs_mut(from) = vec![mir::BlockCall::with(block, args)];
                    self.at = Some(block);
                    for (at, &(index, _)) in carried.iter().enumerate() {
                        places[index].read = params.get(at).copied();
                    }
                }
                x86_64::Step::Jump { opcode, to } => {
                    let (block, _) = Self::went(&labels, to).ok_or_else(refused)?;
                    let from = self.at.expect("a block is being filled");
                    let args = Self::held(places, &carried).ok_or_else(refused)?;
                    let opcode = self.named(opcode);
                    self.out.build(from, opcode).at(span).finish();
                    let next = self.out.create_block();
                    *self.out.succs_mut(from) =
                        vec![mir::BlockCall::with(block, args), mir::BlockCall::to(next)];
                    self.at = Some(next);
                }
                x86_64::Step::Away { symbol } => {
                    // Only in a function that is written without a prologue, which is the one
                    // place the jump means what it says. Anywhere else there is an epilogue behind
                    // the statement that puts the registers back and gives the frame up, and a
                    // jump over it goes to the next function with this function's frame still
                    // taken. The reader already made sure it is the last step of the template, so
                    // what is left to ask is about the function around it.
                    if !self.source.attrs.set.contains(AttrSet::NAKED) {
                        return Err(Unsupported::Assembly { inst, refused: Written::Away });
                    }
                    let from = self.at.expect("a block is being filled");
                    let opcode = self.named(AWAY);
                    let symbol = self.names.intern(symbol);
                    self.out.build(from, opcode).at(span).symbol(symbol).finish();
                    // Nowhere, which is what a jump out of the function leaves behind it and is
                    // the same list a `ret` leaves. The block after it is made for the walk above
                    // rather than for the program: the statement may be in the middle of a body
                    // that goes on being lowered, and what that lowering writes is reached by
                    // nothing and thrown away with the block.
                    *self.out.succs_mut(from) = Vec::new();
                    self.at = Some(self.out.create_block());
                }
                x86_64::Step::Call { symbol } => {
                    self.call_out(inst, symbol, places, list, clobbered, &carried, &mut wrote)?;
                }
                x86_64::Step::Line(line) => {
                    let form = x86_64::form(line.opcode).ok_or_else(refused)?;
                    let mut written = Vec::new();
                    for (desc, piece) in form.operands().iter().zip(&line.operands) {
                        if !desc.role.is_def() {
                            continue;
                        }
                        let index = match *piece {
                            x86_64::Piece::Operand { index, .. } => index,
                            x86_64::Piece::Implicit { reg } => match bound(list, reg, desc.role) {
                                Some(index) => index,
                                None => continue,
                            },
                            x86_64::Piece::Reg { reg, .. } => match bound(list, reg, desc.role) {
                                Some(index) => index,
                                None => continue,
                            },
                        };
                        written.push(index);
                    }
                    // A register is written once in this form of the machine IR, so an operand
                    // an instruction above already wrote is written into a new one here, and what
                    // reads it below reads that one.
                    for &index in &written {
                        if !wrote.contains(&index) {
                            wrote.push(index);
                            continue;
                        }
                        let &(_, class) =
                            carried.iter().find(|&&(at, _)| at == index).ok_or_else(refused)?;
                        let place = places.get_mut(index).ok_or_else(refused)?;
                        place.write = Some(self.out.new_vreg(class));
                    }
                    self.instruction(inst, line, places, list, clobbered)?;
                    for index in written {
                        let place = places.get_mut(index).ok_or_else(refused)?;
                        if place.write.is_some() {
                            place.read = place.write;
                        }
                    }
                }
            }
        }

        // Where the walk left each output, which is the parameter of the block a label made when
        // the template ends in one and the register an instruction wrote when it does not.
        for (index, operand) in list.iter().enumerate() {
            let Some(result) = operand.result else { continue };
            if let Some(reg) = places[index].read {
                self.regs[result.index()] = Some(reg);
            }
        }
        Ok(())
    }

    /// A template's call to a function somewhere else, as the call the convention makes.
    ///
    /// The opcode is the one a call written in C becomes, so everything that asks whether a
    /// function calls anything gets the answer it would for one: the stack pointer is left aligned
    /// at the statement and nothing is kept in the red zone. What is not the same is the operands.
    /// Nothing is passed by the convention, since the template put the arguments where it wanted
    /// them, and what comes back is whatever an output is pinned to, since that is the only thing
    /// the template says about it. Every other register the callee may leave anything in is
    /// written here, which is what a program that calls from a template never says and always
    /// means.
    #[allow(clippy::too_many_arguments)]
    fn call_out(
        &mut self,
        inst: Inst,
        symbol: &str,
        places: &mut [Place],
        list: &[AsmOperand<'_>],
        clobbered: &[PhysReg],
        carried: &[(usize, RegClass)],
        wrote: &mut Vec<usize>,
    ) -> Result<(), Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let mut operands = Vec::new();
        let mut written = Vec::new();
        let lost = self.lost(list);
        for &(reg, class, index) in &lost {
            let Some(index) = index else {
                operands.push(mir::Operand::write(mir::Reg::physical(reg), class));
                continue;
            };
            // Written once in this form of the machine IR, so a second write is a new register,
            // the same as for an instruction in [`Self::woven`].
            if wrote.contains(&index) {
                let &(_, class) =
                    carried.iter().find(|&&(at, _)| at == index).ok_or_else(refused)?;
                places.get_mut(index).ok_or_else(refused)?.write = Some(self.out.new_vreg(class));
            } else {
                wrote.push(index);
            }
            let place = places.get(index).ok_or_else(refused)?.write.ok_or_else(refused)?;
            operands.push(mir::Operand::write(place, class).with(Constraint::Fixed(reg)));
            written.push(index);
        }
        for &reg in clobbered {
            if lost.iter().all(|&(gone, class, _)| gone != reg || class != self.gpr) {
                operands.push(mir::Operand::write(mir::Reg::physical(reg), self.gpr));
            }
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let opcode = mir::Opcode::new(self.names.intern(abi::CALL));
        let symbol = self.names.intern(symbol);
        let mut build = self.out.build(block, opcode).at(span).symbol(symbol);
        for operand in operands {
            build = build.operand(operand);
        }
        build.finish();
        let calls = &mut self.stack.calls;
        *calls = Some(calls.unwrap_or(0));
        for index in written {
            let place = places.get_mut(index).ok_or_else(refused)?;
            place.read = place.write;
        }
        Ok(())
    }

    /// Every register a call may leave anything in, with its file and the output pinned to it if
    /// one is.
    ///
    /// A register is asked about with its file, since the two files are numbered from nought alike
    /// and a question about `v8` alone would find an output pinned to `x8`.
    fn lost(&self, list: &[AsmOperand<'_>]) -> Vec<(PhysReg, RegClass, Option<usize>)> {
        let conv = self.conv;
        let ints = conv.int_order.iter().filter(|&&reg| !conv.preserves_int(reg));
        let sses = conv.sse_order.iter().filter(|&&reg| !conv.preserves_sse(reg));
        let written = |reg, class| {
            list.iter().position(|operand| {
                operand.result.is_some() && self.pinned_here(operand) == Some((reg, class))
            })
        };
        ints.map(|&reg| (reg, conv.int_class, written(reg, conv.int_class)))
            .chain(sses.map(|&reg| (reg, conv.sse_class, written(reg, conv.sse_class))))
            .collect()
    }

    /// The input an output read before anything wrote it shares its register with, which is the
    /// one input that could be in that register, or nothing when there is none or more than one.
    ///
    /// Could be means nothing ties it elsewhere: it is in a register rather than in memory, no
    /// constraint pins it anywhere the output is not, and it is not tied to another output. An
    /// output written `&` shares nothing, since the assembly writes it before it reads the inputs.
    fn shared(&self, list: &[AsmOperand<'_>], index: usize) -> Option<Value> {
        let output = list.get(index)?;
        if output.early || output.tied.is_some() {
            return None;
        }
        let class = self.class_of(self.source[output.result?].ty);
        let mut fits = list.iter().filter(|operand| {
            operand.result.is_none()
                && !operand.memory
                && operand.tied.is_none()
                && operand.value.is_some_and(|value| self.class_of(self.source[value].ty) == class)
                && pinned(operand).is_none_or(|reg| pinned(output) == Some(reg))
        });
        let value = fits.next()?.value;
        if fits.next().is_some() {
            return None;
        }
        value
    }

    /// The block one of the template's labels made, and the parameters it takes.
    fn went<'b>(
        labels: &'b [(&str, mir::Block, Vec<mir::Reg>)],
        name: &str,
    ) -> Option<(mir::Block, &'b [mir::Reg])> {
        labels
            .iter()
            .find(|(had, ..)| *had == name)
            .map(|(_, block, params)| (*block, params.as_slice()))
    }

    /// The register each carried operand is in, which is what an arm to a label carries.
    fn held(places: &[Place], carried: &[(usize, RegClass)]) -> Option<Vec<mir::Reg>> {
        carried.iter().map(|&(index, _)| places.get(index)?.read).collect()
    }

    /// The registers a clobber list names, in the order it named them.
    ///
    /// Nothing is dropped. A name this has no register for is refused, because the list is the
    /// program telling the compiler which registers it may not leave anything in, and an entry
    /// nobody read is a register something may still be left in. See [`Self::assembly`] for the
    /// two entries that are not registers and for why they are skipped rather than refused.
    fn clobbered(inst: Inst, clobbers: &str) -> Result<Vec<PhysReg>, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Clobber };
        let mut named = Vec::new();
        for entry in clobbers.split(',') {
            let entry = entry.trim().trim_matches('"');
            // The sigil is optional in a clobber list and means nothing when it is there, unlike
            // in a template, where it is what tells a register from an operand.
            let entry = entry.strip_prefix('%').unwrap_or(entry);
            if entry.is_empty() || matches!(entry, "memory" | "cc" | "flags") {
                continue;
            }
            let (reg, _) = x86_64::gpr_named(entry).ok_or_else(refused)?;
            if !named.contains(&reg) {
                named.push(reg);
            }
        }
        Ok(named)
    }

    /// [`Self::clobbered`] for a template kept as text, where a clobber may also name a vector
    /// register, `xmm0` or its wider spelling `ymm0`, which busybox's `xorbuf16_aligned_long` does.
    /// Each comes back with the file it is in, since `xmm0` and `rax` are both register nought.
    fn clobbered_x86(
        inst: Inst,
        clobbers: &str,
        gpr: RegClass,
        sse: RegClass,
    ) -> Result<Vec<(PhysReg, RegClass)>, Unsupported> {
        let mut named = Vec::new();
        let mut general = Vec::new();
        for entry in clobbers.split(',') {
            match vector_named(entry) {
                Some(reg) => {
                    if !named.contains(&(reg, sse)) {
                        named.push((reg, sse));
                    }
                }
                None => general.push(entry),
            }
        }
        for reg in Self::clobbered(inst, &general.join(","))? {
            named.push((reg, gpr));
        }
        Ok(named)
    }

    /// [`Self::clobbered`] on AArch64, where a clobber may name a vector register as well as a
    /// general purpose one, so each comes back with the file it is in. See [`aarch64::named`].
    fn clobbered_a64(inst: Inst, clobbers: &str) -> Result<Vec<(PhysReg, RegClass)>, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Clobber };
        let mut named = Vec::new();
        for entry in clobbers.split(',') {
            let entry = entry.trim().trim_matches('"');
            if entry.is_empty() || matches!(entry, "memory" | "cc") {
                continue;
            }
            let reg = aarch64::named(entry).ok_or_else(refused)?;
            if !named.contains(&reg) {
                named.push(reg);
            }
        }
        Ok(named)
    }

    /// Whether the machine being lowered for is AArch64.
    fn on_aarch64(&self) -> bool {
        std::ptr::eq(self.selector.shapes, &aarch64::MACHINE)
    }

    /// The register an operand is pinned to on the machine being lowered for.
    ///
    /// [`pinned`] on x86, where it is always a general purpose register. AArch64 has no constraint
    /// letter for one register, so there only a local register variable pins anything, and its name
    /// is read against [`aarch64::named`], which may put it in either file. The file comes back with
    /// the register because the two are numbered from nought alike, and `x8` is not `v8`.
    fn pinned_here(&self, operand: &AsmOperand<'_>) -> Option<(PhysReg, RegClass)> {
        if !self.on_aarch64() {
            return pinned(operand).map(|reg| (reg, self.gpr));
        }
        let name = operand.named?;
        aarch64::named(name.strip_prefix('%').unwrap_or(name))
    }

    /// An `asm` statement whose operands are `long double` values on the x87 stack.
    ///
    /// `t` is the top of the stack and `u` is the register under it, and those two letters, or a
    /// number tying an input to an output in one of them, are the only places taken here. That is
    /// what glibc's old `<bits/mathinline.h>` writes, `fpatan` with `=t`, `0` and `u` and `st(1)`
    /// in the clobber list, and it is gcc-torture `execute/990413-2.c`.
    ///
    /// The group is the shape every other one in [`Self::x87`] has. The inputs are pushed from the
    /// deepest up, so the `t` one is pushed last and ends up on top, then the template runs, then
    /// the outputs are popped into their slots from the top down. That leaves the stack as empty
    /// as it was found only when the template popped every input it was handed and pushed every
    /// output it says it leaves, and gcc's rule for these statements says when that is: an input
    /// tied to an output or named in the clobber list is one the template pops. So a statement
    /// with an input it leaves behind is refused, as is one with an operand anywhere other than
    /// `st(0)` and `st(1)`, since nothing here would know what to do with the stack after it.
    fn x87_assembly(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Asm(asm) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[asm];
        if !self.source[info.targets].is_empty() {
            return Err(Unsupported::Assembly { inst, refused: Written::Goto });
        }
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let constraints = self.names.resolve(info.constraints).to_string();
        let results: Vec<Value> = data.results().collect();
        let operands = AsmOperands::read(&constraints, &results, &self.source[data.args])
            .ok_or_else(refused)?;
        let list: Vec<AsmOperand<'_>> = operands.iter().copied().collect();

        // Where on the stack each operand is, as a depth from the top.
        let letters: Vec<&str> = constraints.split(',').collect();
        let mut depths = Vec::with_capacity(list.len());
        for (operand, letter) in list.iter().zip(&letters) {
            let value = operand.result.or(operand.value).ok_or_else(refused)?;
            if operand.memory || !on_x87(self.source[value].ty) {
                return Err(refused());
            }
            let depth = match operand.tied {
                Some(output) => *depths.get(output).ok_or_else(refused)?,
                None => match letter.trim_start_matches(['=', '+', '&']) {
                    "t" => 0,
                    "u" => 1,
                    _ => return Err(refused()),
                },
            };
            depths.push(depth);
        }

        // Which depths the clobber list says the template pops.
        let clobbers = self.names.resolve(info.clobbers).to_string();
        let mut popped = [false; 2];
        for entry in clobbers.split(',') {
            let entry = entry.trim().trim_matches('"');
            let entry = entry.strip_prefix('%').unwrap_or(entry);
            match entry {
                "" | "memory" | "cc" | "flags" => {}
                "st" | "st(0)" => popped[0] = true,
                "st(1)" => popped[1] = true,
                _ => return Err(Unsupported::Assembly { inst, refused: Written::Clobber }),
            }
        }

        // The inputs, one per depth and from the top down with no gap, and each one popped.
        let mut inputs: Vec<Option<Value>> = vec![None; 2];
        let mut outputs: Vec<Option<Value>> = vec![None; 2];
        for (index, operand) in list.iter().enumerate() {
            let depth = depths[index];
            if let Some(result) = operand.result {
                if outputs[depth].replace(result).is_some() {
                    return Err(refused());
                }
            }
            let Some(value) = operand.value else { continue };
            // An output written `+` is an input tied to itself.
            let consumed = operand.result.is_some() || operand.tied.is_some() || popped[depth];
            if !consumed {
                return Err(refused());
            }
            if inputs[depth].replace(value).is_some() {
                return Err(refused());
            }
        }
        let gapless =
            |held: &[Option<Value>]| held.iter().skip_while(|it| it.is_some()).all(Option::is_none);
        if !gapless(&inputs) || !gapless(&outputs) {
            return Err(refused());
        }

        // The text, with an operand spelled as the register it is in.
        let template = self.names.resolve(info.template).to_string();
        let mut text = String::with_capacity(template.len());
        let mut chars = template.chars().peekable();
        while let Some(c) = chars.next() {
            if c != '%' {
                text.push(c);
                continue;
            }
            match chars.peek().copied() {
                Some('%') => {
                    chars.next();
                    text.push('%');
                }
                Some('=') => {
                    chars.next();
                    text.push_str(&inst.index().to_string());
                }
                Some(digit) if digit.is_ascii_digit() => {
                    chars.next();
                    if chars.peek().is_some_and(char::is_ascii_digit) {
                        return Err(refused());
                    }
                    let index = digit.to_digit(10).map_or(usize::MAX, |it| it as usize);
                    match depths.get(index).ok_or_else(refused)? {
                        0 => text.push_str("%st"),
                        depth => text.push_str(&format!("%st({depth})")),
                    }
                }
                _ => return Err(refused()),
            }
        }

        let span = self.source.span(inst);
        for value in inputs.iter().rev().flatten() {
            let from = self.x87_slot(*value);
            let from = self.through(from);
            self.x87_at("fld_t", span, from);
        }
        let symbol = self.names.intern(&text);
        let opcode = self.named(x86_64::TEMPLATE);
        let block = self.at.expect("a block is being filled");
        self.out.build(block, opcode).at(span).symbol(symbol).finish();
        for value in outputs.iter().flatten() {
            let into = self.x87_slot(*value);
            let into = self.through(into);
            self.x87_at("fstp_t", span, into);
        }
        Ok(())
    }

    /// An `asm` statement on AArch64, which is kept as text whatever is in it.
    ///
    /// Nothing reads AArch64 assembly back into instructions yet, so every template goes the way
    /// one the x86 reader could not take apart goes, which is [`Self::kept`]: the text is carried
    /// to the listing with a hole for each operand, and the operands are the instruction's own. A
    /// constraint with a letter whose meaning differs between the two machines is refused first.
    /// See [`shared_letters`].
    fn spelled(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Asm(asm) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[asm];
        if self.jumps_from_text(inst) {
            return Err(Unsupported::Assembly { inst, refused: Written::Goto });
        }
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let constraints = self.names.resolve(info.constraints).to_string();
        if !constraints.split(',').all(shared_letters) {
            return Err(refused());
        }
        // `Q` is memory addressed by one register and nothing else, which is how every operand in
        // memory is spelled here already, so it is read as `m`. See [`shared_letters`].
        let constraints = letters_outside(&constraints, |c| if c == 'Q' { 'm' } else { c });
        let results: Vec<Value> = data.results().collect();
        let operands = AsmOperands::read(&constraints, &results, &self.source[data.args])
            .ok_or_else(refused)?;
        let list: Vec<AsmOperand<'_>> = operands.iter().copied().collect();
        let widths = vec![None; list.len()];
        let memory: Vec<bool> = list.iter().map(|operand| operand.memory).collect();
        let template = self.names.resolve(info.template).to_string();
        self.kept(inst, &template, &list, &widths, &memory)
    }

    /// One instruction of a template, as the machine instruction it was read back into.
    fn instruction(
        &mut self,
        inst: Inst,
        line: &x86_64::Line,
        places: &[Place],
        list: &[AsmOperand<'_>],
        clobbered: &[PhysReg],
    ) -> Result<(), Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let form = x86_64::form(line.opcode).ok_or_else(refused)?;
        // What the instruction reaches and what is in each of them. The description answers the
        // first for every opcode but one, and the pieces the template was read into answer the
        // second. Bytes a program wrote out itself are the one, since nothing in a number is a
        // register anybody could read, so the constraint letters answer both. See
        // [`Self::lettered`].
        let lettered = (line.opcode == x86_64::LITERAL).then(|| self.lettered(list));
        let (described, pieces) = match &lettered {
            Some((described, pieces)) => (described.as_slice(), pieces.as_slice()),
            None => (form.operands(), line.operands.as_slice()),
        };
        let mut built = Vec::with_capacity(pieces.len() + clobbered.len());
        for (desc, piece) in described.iter().zip(pieces) {
            built.push(self.placed(inst, *desc, *piece, places, list)?);
        }
        // The clobbers go in among the definitions rather than behind the reads, because an operand
        // vector in the machine IR is every definition and then every use and what counts them
        // reads that order rather than each operand's role.
        let defs = built.iter().take_while(|operand| operand.role.is_def()).count();
        let mut added = 0usize;
        for &reg in clobbered {
            if described.iter().any(|desc| desc.constraint == Constraint::Fixed(reg)) {
                continue;
            }
            built.insert(defs, mir::Operand::write(mir::Reg::physical(reg), self.gpr));
            added += 1;
        }
        // A constraint tying one operand to another names it by its place in this vector, and the
        // clobbers were put in the middle of the vector, so everything behind them moved. The
        // description is written against an instruction with no clobbers in it and cannot know
        // that, which makes this the one place the two numberings have to be reconciled.
        for operand in &mut built {
            if let Constraint::Reuse(at) = operand.constraint {
                if usize::from(at) >= defs {
                    let moved = usize::from(at) + added;
                    operand.constraint =
                        Constraint::Reuse(u8::try_from(moved).map_err(|_| refused())?);
                }
            }
        }
        let at = match line.at {
            Some(at) => Some(self.addressed(inst, at, places, list)?),
            None => None,
        };

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let opcode = self.named(line.opcode);
        let mut build = self.out.build(block, opcode).at(span);
        for operand in built {
            build = build.operand(operand);
        }
        if let Some(value) = line.imm {
            build = build.imm(value);
        }
        if let Some(mem) = at {
            build = build.mem(mem);
        }
        build.finish();
        Ok(())
    }

    /// The registers a run of bytes reaches, taken from the constraint letters rather than from the
    /// description of an opcode.
    ///
    /// Every other instruction of a template has a description saying which registers it reaches
    /// without naming them, and [`Self::assembly`] matches the letters against that. Bytes a program
    /// wrote out itself have no such description and could not have one: what the instruction is, is
    /// a number, and nothing in a number is a register anything could read. So the letters are the
    /// whole of what is known, and they are enough, because a program writing an instruction this
    /// way has to say where its operands go for exactly the reason a program writing `cpuid` does.
    ///
    /// Each register named by a letter gets one entry for the write and one for the read, the same
    /// two `cpuid` has, and only the half the statement asked for: a register no output names is not
    /// written here and one no input names is not read. The writes come first because that is the
    /// order an operand vector in the machine IR is counted in. A register named by nothing is left
    /// out rather than given a spare one, which is the difference from `cpuid` and is right for the
    /// same reason: `cpuid` writes four registers whatever the program said, and what these bytes
    /// touch is known only from what the program said.
    fn lettered(&self, list: &[AsmOperand<'_>]) -> (Vec<OperandDesc>, Vec<x86_64::Piece>) {
        let mut named: Vec<PhysReg> = Vec::new();
        for operand in list {
            if let Some(reg) = pinned(operand) {
                if !named.contains(&reg) {
                    named.push(reg);
                }
            }
        }
        let mut described = Vec::with_capacity(named.len() * 2);
        let mut pieces = Vec::with_capacity(named.len() * 2);
        for role in [Role::Def, Role::Use] {
            for &reg in &named {
                if bound(list, reg, role).is_none() {
                    continue;
                }
                let desc = if role.is_def() {
                    OperandDesc::write(self.gpr)
                } else {
                    OperandDesc::read(self.gpr)
                };
                described.push(desc.with(Constraint::Fixed(reg)));
                pieces.push(x86_64::Piece::Implicit { reg });
            }
        }
        (described, pieces)
    }

    /// One operand of one instruction of a template, in the register the statement put it in.
    fn placed(
        &mut self,
        inst: Inst,
        desc: OperandDesc,
        piece: x86_64::Piece,
        places: &[Place],
        list: &[AsmOperand<'_>],
    ) -> Result<mir::Operand, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        // A register the instruction reaches without its text naming it belongs to whichever of the
        // statement's operands a constraint letter put there, and to nobody when no letter did.
        // There is no width to check in that case: the operand is the register the letter named and
        // the instruction does what it does to it, which is what a program writing `"=a"` asked for.
        let (index, spelled) = match piece {
            x86_64::Piece::Operand { index, width, stated } => (index, Some((width, stated))),
            x86_64::Piece::Implicit { reg } => match bound(list, reg, desc.role) {
                Some(index) => (index, None),
                None => return self.spare(inst, desc),
            },
            // A register the template named, which belongs to one of the statement's operands when
            // a constraint letter put that operand there and to nobody otherwise. Asked in that
            // order rather than placed straight away, because `"D" (p)` with `%rdi` in the text is
            // the program saying one thing twice, and answering it twice would hand the allocator
            // one register holding two values.
            x86_64::Piece::Reg { reg, .. } => match bound(list, reg, desc.role) {
                Some(index) => (index, None),
                None => return self.itself(inst, desc, reg),
            },
        };
        let operand = list.get(index).copied().ok_or_else(refused)?;
        // The two halves of an operand written `+`, which arrives in one register and leaves in
        // another with the allocator told to make them the same one. Everything else has one of
        // the two and asking for the other is the refusal below.
        let place = places.get(index).copied().ok_or_else(refused)?;
        let reg = match desc.role {
            Role::Use => place.read,
            Role::Def | Role::EarlyDef => place.write,
        }
        .ok_or_else(refused)?;

        // Read where the opcode reads and written where it writes, which is what the first half of
        // this asks. An output has a result and an input has a value, an output written `+` has
        // both because it is read before it is written, and an output a matching constraint names
        // is read as the input that named it. See [`read_as`].
        // An output with neither is read as well, and what it holds there is undefined, which
        // [`Self::assembly`] says why and puts a zero in a register for.
        let placeable = match desc.role {
            Role::Use => read_as(list, index).is_some() || operand.result.is_some(),
            Role::Def | Role::EarlyDef => operand.result.is_some(),
        };
        let ty = match (operand.result, operand.value) {
            (Some(result), _) => self.source[result].ty,
            (None, Some(value)) => self.source[value].ty,
            (None, None) => return Err(refused()),
        };
        let bits = held_bits(ty);
        if !placeable || self.class_of(ty) != desc.class {
            return Err(refused());
        }
        if let Some((width, stated)) = spelled {
            // An operand the template wrote a width on may be written by an instruction that fills
            // more of the register than the object in it does, and the object is then the low part
            // of what was written. That is what gmp asks for when it counts the low zero bits of a
            // limb into an `unsigned` and spells the count `%q0`: one quadword instruction writes
            // the whole register and the `unsigned` is the bottom of it, which is every bit of an
            // answer that cannot exceed sixty four anyway.
            //
            // An operand read at a width the template wrote is the other way round: the object is
            // in the register and the instruction looks at the bottom of it. tcc tests the low bits
            // of a `size_t` count with `testb $2,%b4`, and every bit that test reads is one the
            // object put there.
            //
            // A write of less of a register than the object fills is right in one case, which is
            // an instruction that reads the register it writes and an operand that arrives with
            // the object in it. The top of the register is then the top of the object, and the
            // instruction leaves it alone. tcc swaps the bytes of an `unsigned` with `xchgb
            // %b0,%h0` and a rotate between two of them, and the swap only ever touches the low
            // half.
            //
            // The two that stay refused are a read of more of a register than its type fills,
            // which hands an instruction bits nothing ever put there, and a write of less of one
            // that nothing carried the object into, which leaves the top of the object holding
            // whatever the register held before. An operand the template left plain is refused
            // either way, because what gets spelled for that one is the register at the width of
            // its type and no other instruction is the one written down.
            let carried = matches!(desc.constraint, Constraint::Reuse(_) | Constraint::Fixed(_))
                && read_as(list, index).is_some();
            // The other case is the one the machine settles by itself: a write of the low four
            // bytes of a register clears the four above them, so a sixty four bit object written
            // that way holds the thirty two bit answer and nothing else. tcc loads a word through
            // `movl 4(%0),%k0` into a `long` and means exactly that.
            let cleared = desc.class == self.gpr && width == x86_64::Width::Long && bits == 64;
            let widened = stated && desc.role.is_def() && width.bits() > bits;
            let narrowed =
                stated && width.bits() < bits && (!desc.role.is_def() || carried || cleared);
            if bits != width.bits() && !widened && !narrowed {
                return Err(refused());
            }
        }
        // An operand the program pinned is in that register and nowhere else, whatever the opcode
        // would have allowed it. That is the whole of what a local register variable asks for, and
        // it is the same shape a division already has: the allocator is told the register, puts a
        // move in front or behind where it has to, and leaves it out where it does not.
        let constraint = match pinned(&operand) {
            Some(reg) => Constraint::Fixed(reg),
            None => desc.constraint,
        };
        Ok(mir::Operand { reg, class: desc.class, role: desc.role, constraint })
    }

    /// A register the template named in its own text.
    ///
    /// Not one of the statement's operands and not something the allocator handed out. The program
    /// wrote `%rbx` in the middle of a template and meant that register, which is what code doing
    /// something the constraint letters cannot say is made of: micropython saves the callee-saved
    /// registers into a buffer by name because the whole point of the buffer is that those exact
    /// registers are in it, and there is no constraint letter for `%rsp`.
    ///
    /// So it is placed as itself, fixed to the register the template named. What that buys is the
    /// thing gcc does not do: the register becomes part of the instruction the allocator sees, so a
    /// write of one is a definition it knows about and will not leave anything of the program's
    /// across, and a read of one is a use it will not have put something else in first. gcc copies
    /// the text out and a register two things believe they own is a wrong program nothing reports.
    /// Here the allocator is told, and a program that also named the register in its clobber list
    /// says the same thing twice rather than something new.
    fn itself(
        &mut self,
        inst: Inst,
        desc: OperandDesc,
        reg: PhysReg,
    ) -> Result<mir::Operand, Unsupported> {
        let refused = Unsupported::Assembly { inst, refused: Written::Operand };
        if desc.class != self.gpr {
            return Err(refused);
        }
        Ok(mir::Operand {
            reg: mir::Reg::physical(reg),
            class: self.gpr,
            role: desc.role,
            constraint: Constraint::Fixed(reg),
        })
    }

    /// A register an instruction of a template uses and the statement put nothing in.
    ///
    /// A write of one is the register being destroyed, which is what a clobber list is usually
    /// written to say and what an instruction with more answers than the program asked for does
    /// anyway: `cpuid` writes all four registers whether or not the statement wanted all four. A
    /// register of its own is the whole of what that needs, since a value nothing reads is one the
    /// allocator may put anywhere and is told about so that nothing else is put there.
    ///
    /// A read of one is a register the instruction looks at and the program never filled, which
    /// gcc leaves as whatever happened to be there. A zero is written instead, for the reason
    /// [`Self::undefined`] gives: the allocator has to be given a definition before a use, and a
    /// zero is the one answer that reads the same on every run.
    fn spare(&mut self, inst: Inst, desc: OperandDesc) -> Result<mir::Operand, Unsupported> {
        let refused = Unsupported::Assembly { inst, refused: Written::Operand };
        if desc.class != self.gpr {
            return Err(refused);
        }
        let reg = self.out.new_vreg(desc.class);
        if !desc.role.is_def() {
            let block = self.at.expect("a block is being filled");
            let span = self.source.span(inst);
            let put = self.named("mov_ri_64");
            self.out.build(block, put).at(span).def(reg, desc.class).imm(0).finish();
        }
        Ok(mir::Operand { reg, class: desc.class, role: desc.role, constraint: desc.constraint })
    }

    /// The address one instruction of a template reads or writes.
    fn addressed(
        &mut self,
        inst: Inst,
        at: x86_64::At,
        places: &[Place],
        list: &[AsmOperand<'_>],
    ) -> Result<mir::Mem, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let base = match at.base {
            None => None,
            Some(x86_64::Piece::Operand { index, .. }) => {
                // The register an address is counted from is read and never written, whatever the
                // instruction does to what it finds there.
                let reg = places.get(index).and_then(|place| place.read).ok_or_else(refused)?;
                Some(mir::Operand::read(reg, self.gpr))
            }
            // A register the template named, counted from as itself. See [`Self::itself`], and note
            // that this is the half of it every one of these templates needs: `movq %rax, 16(%rdi)`
            // names one register as the thing being stored and another as where to store it. An
            // operand a constraint letter put in that register is that operand, for the reason
            // [`Self::placed`] gives.
            Some(x86_64::Piece::Reg { reg, .. }) => match bound(list, reg, Role::Use) {
                Some(index) => {
                    let reg = places.get(index).and_then(|place| place.read).ok_or_else(refused)?;
                    Some(mir::Operand::read(reg, self.gpr))
                }
                None => Some(
                    mir::Operand::read(mir::Reg::physical(reg), self.gpr)
                        .with(Constraint::Fixed(reg)),
                ),
            },
            // An address counted from a register the instruction reaches without being told is
            // not something this machine has: every addressing mode is written out in the text it
            // is part of, so a base that got here another way is a base nothing wrote down.
            Some(x86_64::Piece::Implicit { .. }) => return Err(refused()),
        };
        // A distance the template wrote, or the one in an operand the template pointed at, which is
        // the same distance said by something that knows how big a thing is. It has to be a number
        // the compiler can read at translation time, since it goes in the instruction rather than
        // in a register, and an operand holding anything else is refused rather than put somewhere.
        let disp = match at.disp {
            x86_64::Disp::Number(disp) => disp,
            x86_64::Disp::Operand(index) => {
                let value =
                    list.get(index).and_then(|operand| operand.value).ok_or_else(refused)?;
                let number = self.number(value).ok_or_else(refused)?;
                i32::try_from(number).map_err(|_| refused())?
            }
        };
        Ok(mir::Mem { base, scale: 1, disp, segment: at.segment, ..mir::Mem::default() })
    }

    /// The number in that value, for one an `iconst` defined, read at the width of its own type.
    ///
    /// Signed, because the two things a template asks this for are a distance into an address and
    /// the number on an instruction, and both of those are signed wherever they land. A constant
    /// whose type is unsigned and whose top bit is set therefore reads as a negative number here,
    /// which is the same number and is the reading that fits in the thirty two bits an addressing
    /// mode has room for.
    fn number(&self, value: Value) -> Option<i128> {
        let Def::Result { inst, .. } = self.source[value].def else { return None };
        if self.source[inst].opcode != Opcode::IConst {
            return None;
        }
        let Extra::Imm(imm) = self.source[inst].extra else { return None };
        let bits = self.source[imm].bits();
        let width = self.source[value].ty.bits();
        if width == 0 || width > 128 {
            return None;
        }
        let spare = 128 - width;
        Some(((bits << spare) as i128) >> spare)
    }

    /// A register holding a value the program has no claim on, written as a zero.
    ///
    /// Every other way of saying it costs the same instruction or needs a word the machine IR does
    /// not have, and a zero is the one that reads the same on every run.
    fn undefined(&mut self, inst: Inst, result: Value) -> Result<(), Unsupported> {
        let ty = self.source[result].ty;
        let refused = Unsupported::Assembly { inst, refused: Written::Operand };
        let bits = held_bits(ty);
        if self.class_of(ty) != self.gpr || !matches!(bits, 8 | 16 | 32 | 64) {
            return Err(refused);
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let reg = self.new_reg(result);
        let put = self.named(&format!("mov_ri_{bits}"));
        self.out.build(block, put).at(span).def(reg, self.gpr).imm(0).finish();
        Ok(())
    }

    /// Whether a type is the width an address is, which is what makes a cast to or from one free.
    fn is_address_width(&self, ty: Type) -> bool {
        ty.is_ptr() || (ty.is_int() && ty.bits() == ADDRESS_BITS)
    }

    /// Where a block goes, which in machine IR is on the block rather than on its terminator.
    ///
    /// That is why no rule ever names a block: a branch is selected for what it reads and the
    /// edges are copied across here, arguments and all. The arguments are read last, after every
    /// instruction of the block is written, because an argument that is a constant is
    /// materialized where it is first wanted and the end of the block is where an edge wants it.
    ///
    /// Which is not quite the end. A block that leaves two ways has the branch as its last
    /// instruction, and a block that leaves through a register has the indirect jump as its last,
    /// and anything appended after either is something it has already jumped past, so a constant
    /// materialized here would be a register the block below reads and nothing ever writes. The
    /// one that was there is put back on the end when that happened, which is the only reordering
    /// anything in this crate does and is why it is remembered before a single argument is read.
    fn edges(&mut self, block: Block, out: mir::Block) -> Result<(), Unsupported> {
        let Some(term) = self.source.terminator(block) else { return Ok(()) };
        // An `asm goto` whose template has nothing in it can only fall through, since there is no
        // instruction in it to jump with, so the only edge the machine block gets is the first
        // one. The labels it names are still arms in the IR, which is what kept the passes above
        // from assuming anything about the way into them, and here they are blocks nothing jumps
        // to, the same as a label no `goto` names. One that does have instructions was refused by
        // [`Self::jumps_from_text`] before this.
        if self.source[term].opcode == Opcode::InlineAsm {
            let Some(call) = self.source.successors(term).next() else { return Ok(()) };
            let args: Vec<Value> = self.source[call.args].to_vec();
            let regs =
                args.into_iter().map(|value| self.reg_of(value)).collect::<Result<_, _>>()?;
            *self.out.succs_mut(out) = vec![mir::BlockCall::with(self.out_block(call.block), regs)];
            return Ok(());
        }
        // A call's unwind edge, which is not an edge of the machine function at all. The branch was
        // never written, so what the block has is the arm control takes when the call returns, and
        // the pad is a block with nothing in front of it that the call site table is what reaches.
        // See [`Self::pad`] for why that is a block the allocator can be handed.
        if let Some(unwound) = self.unwind_edge(term) {
            let arms: Vec<rucc_ir::BlockCall> = self.source.successors(term).collect();
            let next = arms[1];
            let args: Vec<Value> = self.source[next.args].to_vec();
            let regs =
                args.into_iter().map(|value| self.reg_of(value)).collect::<Result<_, _>>()?;
            *self.out.succs_mut(out) = vec![mir::BlockCall::with(self.out_block(next.block), regs)];
            let call = self.source.prev_inst(unwound).and_then(|call| self.unwinding.get(&call));
            if let Some(&call) = call {
                let pad = self.out_block(arms[0].block);
                self.out.landings.push((call, pad));
            }
            return Ok(());
        }
        let leaves =
            matches!(self.source[term].opcode, Opcode::BrIf | Opcode::IndirectBr | Opcode::Switch);
        let branch = if leaves { self.out.terminator(out) } else { None };

        let calls: Vec<rucc_ir::BlockCall> = self.source.successors(term).collect();
        let mut succs = Vec::with_capacity(calls.len());
        for call in calls {
            let args: Vec<Value> = self.source[call.args].to_vec();
            let mut regs = Vec::with_capacity(args.len());
            for value in args {
                // The address of where the value is rather than the value, for the one type a
                // register holds none of. The block on the other side copies the bytes out of it
                // into a slot of its own, which is what makes a second edge into the same block
                // safe.
                let reg = if on_x87(self.source[value].ty) {
                    self.x87_slot(value)
                } else {
                    self.reg_of(value)?
                };
                regs.push(reg);
            }
            succs.push(mir::BlockCall::with(self.out_block(call.block), regs));
        }
        if let Some(branch) = branch {
            if self.out.terminator(out) != Some(branch) {
                self.out.remove_inst(branch);
                self.out.append_inst(out, branch);
            }
        }
        *self.out.succs_mut(out) = succs;
        Ok(())
    }

    /// The `unwound` a branch reads, when the branch is a call's unwind edge.
    fn unwind_edge(&self, inst: Inst) -> Option<Inst> {
        let data = &self.source[inst];
        if data.opcode != Opcode::BrIf {
            return None;
        }
        let &cond = self.source[data.args].first()?;
        match self.source[cond].def {
            Def::Result { inst, .. } if self.source[inst].opcode == Opcode::Unwound => Some(inst),
            _ => None,
        }
    }

    /// The exception a landing pad was entered with, as a copy out of the register the unwinder
    /// left it in, which is the first register a value comes back in.
    fn landing(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let held = *self.conv.int_returns.first().ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let mov = self.selector.frame.moves(self.gpr).expect("a class the target says how to move");
        let mov = self.named(mov.mov);
        let into = self.new_reg(result);
        self.out
            .build(block, mov)
            .at(span)
            .operand(mir::Operand::write(into, self.gpr))
            .operand(
                mir::Operand::read(mir::Reg::physical(held), self.gpr)
                    .with(Constraint::Fixed(held)),
            )
            .finish();
        Ok(())
    }

    /// Makes a landing pad a block that reads nothing from the blocks around it, and says what to
    /// put back once it has been filled.
    ///
    /// The pad has no machine block in front of it, because the edge into it is not one the machine
    /// takes: control arrives from the unwinder, with the registers the frame rules at the call put
    /// back. So nothing the allocator keeps in a register can reach it, and a value it reads from
    /// elsewhere is made again inside it. What a pad reads is the address of each object a handler
    /// is owed, which is a slot of the frame, the address of a name, or a constant, and each of
    /// those can be written a second time from nothing. Anything else is refused.
    ///
    /// The registers the rest of the function knows those values by are put back afterwards,
    /// which is what the answer is for: the pad's copies are its own.
    fn pad(&mut self, block: Block) -> Result<Vec<(Value, Option<mir::Reg>)>, Unsupported> {
        let mut kept = Vec::new();
        let first = self.source.insts(block).next();
        if !first.is_some_and(|inst| self.source[inst].opcode == Opcode::Landing) {
            return Ok(kept);
        }
        let out = self.at.expect("a block is being filled");
        let insts: Vec<Inst> = self.source.insts(block).collect();
        for inst in insts {
            let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
            for value in args {
                let Def::Result { inst: def, .. } = self.source[value].def else {
                    return Err(self.unsupported(inst));
                };
                if self.source.block_of(def) == Some(block)
                    || kept.iter().any(|&(done, _)| done == value)
                {
                    continue;
                }
                match self.source[def].opcode {
                    Opcode::IConst => {}
                    Opcode::Alloca => {
                        let &index =
                            self.frame_slots.get(&value).ok_or_else(|| self.unsupported(def))?;
                        kept.push((value, self.regs[value.index()]));
                        let reg = self.out.new_vreg(self.gpr);
                        self.regs[value.index()] = Some(reg);
                        let lea = self.named(self.selector.frame.lea);
                        let sp = mir::Reg::physical(self.conv.stack_pointer);
                        let sp = mir::Operand::read(sp, self.gpr);
                        let span = self.source.span(def);
                        let made = self
                            .out
                            .build(out, lea)
                            .at(span)
                            .def(reg, self.gpr)
                            .mem(mir::Mem::at(sp))
                            .finish();
                        self.stack.addresses.push((made, index));
                    }
                    Opcode::GlobalAddr => {
                        kept.push((value, self.regs[value.index()]));
                        self.regs[value.index()] = None;
                        self.address_of(def)?;
                    }
                    _ => return Err(self.unsupported(def)),
                }
            }
        }
        Ok(kept)
    }

    /// Whether an `asm goto` has instructions in its template, which is what it would jump with.
    ///
    /// One with an empty template is what a program writes to tell the optimizer that control may
    /// arrive at a label without saying how, and the torture suite has several of them. It never
    /// jumps, so it is written as the statement it would be without its labels and a fall through
    /// into its first arm. See [`Self::edges`]. One with anything in it needs the labels it names
    /// written into the text and an edge for each of them the allocator knows about, and that is
    /// still refused.
    fn jumps_from_text(&self, inst: Inst) -> bool {
        let Extra::Asm(asm) = self.source[inst].extra else { return false };
        let info = self.source[asm];
        !self.source[info.targets].is_empty()
            && !self.names.resolve(info.template).trim().is_empty()
    }

    /// The machine IR block an IR block became.
    fn out_block(&self, block: Block) -> mir::Block {
        self.blocks[block.index()].expect("every block was created before any was filled")
    }

    /// The parameters of the entry block, which are the function's arguments.
    ///
    /// They are not block parameters in the machine IR and they cannot be. A block parameter is
    /// given its value by a move on the edge into the block, and there is no edge into an entry
    /// block, so what arrives in a function is the convention's to say. [`crate::abi`] is what
    /// says it.
    ///
    /// The ones past the last register arrived in the caller's memory and are read out of it, and
    /// the loads that read them come back here so that the frame can finish them the way it
    /// finishes an `alloca`.
    fn arrive(&mut self, block: Block, out: mir::Block) -> Result<(), Unsupported> {
        let params = self.source[block].params.clone();
        // The type of each is the block's answer and what the ABI asks of it is the signature's,
        // and the two lists are the same list: a parameter the classification turned into a
        // pointer is a pointer in the block too. A block with more parameters than the signature
        // names is not one the front end writes, and each of those is taken as a plain value.
        let asked: Vec<Abi> = self.source.signature().params.iter().map(|it| it.abi).collect();
        let types: Vec<Param> = params
            .iter()
            .enumerate()
            .map(|(index, &value)| {
                let abi = asked.get(index).copied().unwrap_or_default();
                Param { ty: self.source[value].ty, abi }
            })
            .collect();
        // A save area for a function that takes arguments its signature does not name, which is a
        // block of this function's frame on one convention and the shadow space the caller already
        // reserved on the other. Which of the two it is is [`varargs::Area::of`]'s answer and
        // [`Self::save_area`] is where the difference is spent.
        //
        // Apple's AArch64 is neither. Every argument a signature does not name is in the caller's
        // memory, so there is nothing to save and the list starts at the first word past the named
        // ones.
        //
        // A function holding `__builtin_apply_args` asks for the same area whether it is variadic
        // or not, because what it saves is every argument register, and the area is where the
        // walk that binds them says where each one goes.
        let variadic = self.source.signature().variadic;
        let in_memory = self.conv.abi.variadic == Variadic::AlwaysMemory;
        let applies = self.saves_arguments();
        let area = (variadic && !in_memory || applies).then(|| varargs::Area::of(self.conv));
        let arrived =
            abi::entry(&mut self.out, out, &types, self.conv, self.selector.abi, self.names, area)
                .map_err(|(index, missing)| Unsupported::Argument { index, missing })?;
        for (&param, reg) in params.iter().zip(&arrived.regs) {
            self.regs[param.index()] = Some(*reg);
        }
        if applies {
            self.save_arguments(out, &arrived);
        }
        if let (true, Some(area)) = (variadic && !in_memory, area) {
            self.save_area(out, &arrived, area);
        } else if variadic {
            let incoming = arrived.beyond.next_multiple_of(self.conv.word);
            self.varargs = Some(Varargs::Pointer { incoming });
        }
        self.stack.arguments.extend(arrived.stack);
        Ok(())
    }

    /// The prologue of a variadic function, which is every argument register it was handed written
    /// into the frame.
    ///
    /// Every one the signature did not name, that is. Which of those hold anything is a thing only
    /// the caller knew and there is nothing here to ask, so all of them are written, and the ones a
    /// named parameter took are not, because `va_start` sets the two offsets past them and nothing
    /// ever reads their slots.
    ///
    /// What that costs is up to fourteen stores in the prologue of a function that may read none of
    /// them, and the convention's answer to that is the count of vector registers in `%al`, which
    /// lets a callee skip the eight vector stores when the call passed no floats. Skipping them is a
    /// branch in a prologue, and a prologue is written long after this by [`crate::finish`], which
    /// has no blocks to branch between. So they are all written every time, which is correct and is
    /// what `-O0` costs. Issue #323 is the branch.
    ///
    /// A vector register is written all sixteen bytes at a time, because a `_Float128` fills one and
    /// a `va_arg` of a quad reads the slot back whole. gcc writes the same sixteen with the same
    /// instruction, which is what [`crate::varargs`] says a list has to be built out of.
    ///
    /// The address is computed once into a register rather than written as a displacement off the
    /// stack pointer, because a displacement into a frame is not known until after allocation and
    /// one `lea` costs less than a fixup list for a dozen stores. It is the same `lea` an `alloca`
    /// gets and [`crate::finish`] fills it in the same way.
    ///
    /// A convention that homes its register arguments has none of that. Its area is the shadow
    /// space the caller reserved above the return address, so there is no object to make and no
    /// address to work out: each store reaches into the caller's argument area the way the load of
    /// a parameter the registers ran out before does, which is the same waiting list and the same
    /// fixup. There are at most four of them and none is a vector register, since a float the
    /// signature does not name arrived in a general purpose register too and that is the copy the
    /// walk reads.
    fn save_area(&mut self, out: mir::Block, arrived: &abi::Arrived, area: varargs::Area) {
        if self.conv.shared_positions {
            self.varargs = Some(Varargs::Pointer { incoming: arrived.beyond });
            let store = self.named("mov_mr_64");
            for &(reg, class, at) in &arrived.spare {
                let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
                let made =
                    self.out.build(out, store).uses(reg, class).mem(mir::Mem::at(sp)).finish();
                self.stack.arguments.push((made, at));
            }
            return;
        }

        let save = self.stack.locals.len();
        self.stack.locals.push(Local { size: area.size, align: varargs::VECTOR_SLOT });
        let took = |count: usize, float: bool| {
            let count = u32::try_from(count).unwrap_or(0).min(area.holds(float));
            area.starts_at(float) + count * area.stride(float)
        };
        let integers = took(arrived.took.0, false);
        let floats = took(arrived.took.1, true);
        self.varargs = Some(if self.conv.list == VaList::Aapcs {
            // Minus what is left of each half, since the two offsets count up to its top.
            let left = |at: u32, float: bool| {
                i32::try_from(at).unwrap_or(0) - i32::try_from(area.ends_at(float)).unwrap_or(0)
            };
            Varargs::Aapcs {
                save,
                incoming: arrived.beyond,
                integers_end: area.ends_at(false),
                floats_end: area.ends_at(true),
                integers: left(integers, false),
                floats: left(floats, true),
            }
        } else {
            Varargs::Fields { save, incoming: arrived.beyond, integers, floats }
        });

        // A vector register is saved all sixteen bytes wide, as a quad is, whatever it held.
        let base = self.frame_address(out, save);
        for &(reg, class, at) in &arrived.spare {
            let ty =
                if class == self.gpr { Type::int(64) } else { Type::float(rucc_ir::Float::F128) };
            let head = (self.selector.abi.store)(ty).expect("a store of a whole register");
            let store = mir::Opcode::new(self.names.intern(head));
            let up = i32::try_from(at).expect("a register save area under two gigabytes");
            let mem = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(up);
            self.out.build(out, store).uses(reg, class).mem(mem).finish();
        }
    }

    /// Whether the function holds a `__builtin_apply_args`, on a convention this can save the
    /// arguments of.
    ///
    /// Only the one that keeps the two register files apart and saves them the way a SysV list
    /// does, since the block is that layout with one word in front of it. On any other the call is
    /// refused where it stands, which is [`Self::apply_args`] finding nothing saved.
    fn saves_arguments(&self) -> bool {
        if self.conv.list != VaList::SysV || self.conv.shared_positions {
            return false;
        }
        let source = self.source;
        source
            .blocks()
            .any(|block| source.insts(block).any(|inst| source[inst].opcode == Opcode::ApplyArgs))
    }

    /// The prologue of a function holding `__builtin_apply_args`, which is every argument register
    /// it was handed and where the arguments in memory start, written into a block of its frame.
    ///
    /// The block is the one gcc lays out on this convention, so that a program reading it the way
    /// gcc's manual says reads the same bytes:
    ///
    /// ```text
    ///   0        where the arguments that came in memory are
    ///   8        nothing, so that what follows is sixteen byte aligned
    ///   16..64   the six general purpose argument registers, a word each
    ///   64..192  the eight vector argument registers, sixteen bytes each
    /// ```
    ///
    /// Which is the register save area of a variadic function with a word and a pad in front, so
    /// the offsets are that area's plus sixteen. What is different is that every register is
    /// written and not only the ones no parameter took: the one a parameter arrived in is written
    /// from the register the parameter was bound to, which holds it untouched because nothing has
    /// run yet, and the rest from the pseudos the walk made for them.
    fn save_arguments(&mut self, out: mir::Block, arrived: &abi::Arrived) {
        let applied = self.stack.locals.len();
        self.stack.locals.push(Local { size: APPLY_ARGS, align: varargs::VECTOR_SLOT });
        self.applied = Some(applied);
        let base = self.frame_address(out, applied);
        let overflow = self.overflow(out, 0, Span::DUMMY);
        let head = (self.selector.abi.store)(Type::int(64)).expect("a store of an address");
        let store = mir::Opcode::new(self.names.intern(head));
        let mem = mir::Mem::at(mir::Operand::read(base, self.gpr));
        self.out.build(out, store).uses(overflow, self.gpr).mem(mem).finish();

        let named = arrived.named.iter().map(|&(index, at)| {
            let reg = arrived.regs[index];
            let class = self.out.class_of(reg).unwrap_or(self.gpr);
            (reg, class, at)
        });
        let every: Vec<_> = named.chain(arrived.spare.iter().copied()).collect();
        for (reg, class, at) in every {
            let ty =
                if class == self.gpr { Type::int(64) } else { Type::float(rucc_ir::Float::F128) };
            let head = (self.selector.abi.store)(ty).expect("a store of a whole register");
            let store = mir::Opcode::new(self.names.intern(head));
            let up = i32::try_from(at + APPLY_REGS).expect("a block of under two gigabytes");
            let mem = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(up);
            self.out.build(out, store).uses(reg, class).mem(mem).finish();
        }
    }

    /// One `__builtin_apply_args`, which is the address of the block the prologue wrote.
    fn apply_args(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Some(applied) = self.applied else { return Err(self.unsupported(inst)) };
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let reg = self.frame_address(block, applied);
        self.regs[result.index()] = Some(reg);
        Ok(())
    }

    /// One `__builtin_apply`, which is a call whose arguments are every register in a block
    /// `__builtin_apply_args` answered and some bytes of the memory it says the arguments in
    /// memory were in.
    ///
    /// Built as a call of fourteen arguments, six words and eight vectors, which puts each in the
    /// register it came out of, and one object of the size the program gave, which is copied into
    /// the bottom of the outgoing area the way a structure passed by value is. The call is made as
    /// to a variadic function, so the count of vector registers is eight and a variadic callee
    /// saves all of them.
    ///
    /// What comes back is every register a value can come back in, which is two of each file, and
    /// they are written into a block of this function's frame whose address is the answer: the two
    /// words at 0 and 8 and the two vectors at 16 and 32. An eighty bit value comes back on the x87
    /// stack and is not in it, which is the one thing gcc's block holds that this one does not.
    fn apply(&mut self, inst: Inst) -> Result<(), Unsupported> {
        if self.conv.list != VaList::SysV || self.conv.shared_positions {
            return Err(self.unsupported(inst));
        }
        let values: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let [function, saved, size] = values[..] else { return Err(self.unsupported(inst)) };
        let size = self.number(size).ok_or_else(|| self.unsupported(inst))?;
        let size = u32::try_from(size).map_err(|_| self.unsupported(inst))?;
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let function = self.reg_of(function)?;
        let saved = self.reg_of(saved)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);

        let word = Type::int(64);
        let vector = Type::float(rucc_ir::Float::F128);
        let area = varargs::Area::of(self.conv);
        let load = |ty: Type| (self.selector.abi.load)(ty).expect("a load of a whole register");
        let (load_word, load_vector) = (load(word), load(vector));
        let mut read = |ty: Type, head: &str, class: RegClass, at: u32| {
            let reg = self.out.new_vreg(class);
            let opcode = mir::Opcode::new(self.names.intern(head));
            let at = i32::try_from(at).expect("a block of under two gigabytes");
            let mem = mir::Mem::at(mir::Operand::read(saved, self.gpr)).plus(at);
            self.out.build(block, opcode).at(span).def(reg, class).mem(mem).finish();
            abi::Passing { ty, reg, abi: Abi::Plain }
        };
        let sse = self.conv.sse_class;
        let gpr = self.gpr;
        let mut args = Vec::with_capacity(15);
        for (float, ty, head, class) in
            [(false, word, load_word, gpr), (true, vector, load_vector, sse)]
        {
            for index in 0..area.holds(float) {
                let at = APPLY_REGS + area.starts_at(float) + index * area.stride(float);
                args.push(read(ty, head, class, at));
            }
        }
        if size > 0 {
            let memory = read(word, load_word, gpr, 0);
            let object =
                Abi::ByVal { size: u64::from(size), align: 8, drains: rucc_ir::Drains::Nothing };
            args.push(abi::Passing { abi: object, ..memory });
        }
        let returns = [word, word, vector, vector];
        let what = abi::Calling {
            callee: abi::Callee::Through(function),
            args: &args,
            returns: &returns,
            variadic: true,
            named: args.len(),
            at: span,
        };
        let made = abi::call(&mut self.out, block, &what, self.conv, self.selector.abi, self.names)
            .map_err(|refused| Unsupported::Call { inst, refused })?;
        let calls = &mut self.stack.calls;
        *calls = Some(calls.unwrap_or(0).max(made.outgoing));

        let back = self.stack.locals.len();
        self.stack.locals.push(Local { size: APPLY_BACK, align: varargs::VECTOR_SLOT });
        let base = self.frame_address(block, back);
        for ((&reg, ty), at) in made.results.iter().zip(returns).zip([0, 8, 16, 32]) {
            let class = if ty == word { gpr } else { sse };
            let head = (self.selector.abi.store)(ty).expect("a store of a whole register");
            let store = mir::Opcode::new(self.names.intern(head));
            let mem = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(at);
            self.out.build(block, store).at(span).uses(reg, class).mem(mem).finish();
        }
        let answer = self.frame_address(block, back);
        self.regs[result.index()] = Some(answer);
        Ok(())
    }

    /// The address of one of the function's stack objects, in a fresh register.
    ///
    /// Written with nothing in its displacement, because where an object is in a frame is not known
    /// until after allocation, and given to [`crate::finish`] to fill in the way an `alloca` is.
    fn frame_address(&mut self, out: mir::Block, local: usize) -> mir::Reg {
        self.frame_address_plus(out, local, 0)
    }

    /// The address some way into a local, which the frame finishes the same way, adding where the
    /// local is to what is already there.
    fn frame_address_plus(&mut self, out: mir::Block, local: usize, plus: u32) -> mir::Reg {
        let reg = self.out.new_vreg(self.gpr);
        let lea = self.named(self.selector.frame.lea);
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let plus = i32::try_from(plus).expect("an offset into a local under two gigabytes");
        let mem = mir::Mem::at(sp).plus(plus);
        let made = self.out.build(out, lea).def(reg, self.gpr).mem(mem).finish();
        self.stack.addresses.push((made, local));
        reg
    }

    /// Whether an instruction is one no machine instruction is written for where it stands.
    ///
    /// Four of them, and none is a lowering decision, which is why none is a rule. A constant is
    /// written where a register for it is first wanted rather than where the IR put it, and every
    /// reader of one may have folded it into an immediate, in which case nowhere is the right
    /// place. A return of nothing has nothing to put anywhere: the epilogue gives the frame back
    /// and leaves, and it is appended to every block with no successors long after this has
    /// finished, so a return with a value is one instruction here and a return without one is
    /// none. Unless the value went back through memory, in which case there is something to put
    /// somewhere after all and the IR does not carry it: the address the caller handed over has
    /// to be in `rax` on the way out, and [`Lowering::returned`] is what writes that.
    ///
    /// An unconditional jump is the third, and there is even less of it: the edge is on the
    /// block, and whether the block it goes to is the next one and needs no jump at all is the
    /// block layout's answer rather than this one's.
    ///
    /// The fourth is a point control does not arrive at, in both of the forms the IR has for it:
    /// the `unreachable` terminator the front end puts at the end of a function whose body can run
    /// off the bottom, and the `unreachable_hint` a call to `__builtin_unreachable` becomes. What
    /// to write for a place nothing reaches is a question with no wrong answer, and nothing is the
    /// smallest one and the one gcc 16.2.0 gives at `-O0`. The terminator leaves the block with no
    /// successors, so the epilogue lands at the end of it the way it does on any other block that
    /// goes nowhere, and the function cannot fall out of its own last instruction into whatever
    /// the assembler puts next.
    fn writes_nothing(&self, inst: Inst) -> bool {
        let data = &self.source[inst];
        match data.opcode {
            Opcode::IConst | Opcode::Jump | Opcode::Unreachable | Opcode::UnreachableHint => true,
            // The question of whether a call unwound and the branch on its answer, neither of which
            // is an instruction. See [`Self::edges`].
            Opcode::Unwound => true,
            Opcode::BrIf => self.unwind_edge(inst).is_some(),
            Opcode::Return => self.source[data.args].is_empty() && self.sret().is_none(),
            _ => false,
        }
    }

    /// What every instruction in one block matched, with a set of values nobody may take.
    ///
    /// Backwards, because an instruction that has been folded into a later one does not get to
    /// fold anything into itself: the rule that took it only reached one level down, so what is
    /// under it is not in the term the matcher saw and cannot be replaced.
    fn decide(&self, insts: &[Inst], refused: &HashSet<Value>) -> Decided {
        let mut found: Vec<Option<Match<Term>>> = (0..insts.len()).map(|_| None).collect();
        let mut plans: Vec<Option<Plan>> = vec![None; insts.len()];
        let mut folded: Vec<Inst> = Vec::new();
        for (index, &inst) in insts.iter().enumerate().rev() {
            if folded.contains(&inst) {
                continue;
            }
            if let Some((plan, matched)) = self.select(inst, refused) {
                folded.extend(self.folds(inst, plan));
                found[index] = Some(matched);
                plans[index] = Some(plan);
            }
        }
        Decided { found, plans, folded }
    }

    /// A value some of its readers took and some of them did not, which is the one case folding
    /// buys nothing.
    ///
    /// Folding does not delete the instruction that computed a value for anybody else, so a
    /// reader that did not take it still needs it in a register and the instruction stays. The
    /// reader that did take it now does that work again. Either all of them take it, in which
    /// case nothing is left to read it and the instruction goes, or none of them do.
    ///
    /// The count is over the whole function rather than over the block, since a value read from
    /// another block is read from a register there whatever this block decides. An instruction
    /// built by name rather than matched, a call being the one that matters, has no plan and so
    /// takes nothing, which is the right answer for it as well.
    ///
    /// The count is kept only for the values this block's instructions take. It used to be a slot
    /// for every value in the function, cleared for every block, and on a function of thirty
    /// thousand blocks and a hundred and seventy thousand values that clearing was four percent of
    /// an optimized compile.
    fn left_alive(&self, insts: &[Inst], plans: &[Option<Plan>]) -> Option<Value> {
        let mut taken: HashMap<Value, u32> = HashMap::new();
        for (&inst, plan) in insts.iter().zip(plans) {
            let Some(plan) = plan else { continue };
            let args = &self.source[self.source[inst].args];
            for (index, &arg) in args.iter().take(MAX_ARGS).enumerate() {
                if plan[index] == Shown::Expand {
                    *taken.entry(arg).or_default() += 1;
                }
            }
        }
        for (&inst, plan) in insts.iter().zip(plans) {
            let Some(plan) = plan else { continue };
            let args = &self.source[self.source[inst].args];
            for (index, &arg) in args.iter().take(MAX_ARGS).enumerate() {
                if plan[index] == Shown::Expand && taken[&arg] < self.uses[arg.index()] {
                    return Some(arg);
                }
            }
        }
        None
    }

    /// The rule that fires on an instruction, and what it bound.
    ///
    /// The plans are tried in order and the first that matches wins, which is the maximal munch
    /// `spec/10-backend.md` asks for: a plan that offers more to the matcher is tried before one
    /// that offers less.
    fn select(&self, inst: Inst, refused: &HashSet<Value>) -> Option<(Plan, Match<Term>)> {
        for plan in self.plans(inst, refused) {
            let terms = Terms::new(self.source, inst, plan);
            if let Some(matched) = self.selector.table.find(&terms, Term::Root) {
                return Some((plan, matched));
            }
        }
        None
    }

    /// Every way this instruction can be shown to the matcher, most offered first.
    fn plans(&self, inst: Inst, refused: &HashSet<Value>) -> Vec<Plan> {
        let args = &self.source[self.source[inst].args];
        let mut plans = vec![PLAIN];
        for (index, &arg) in args.iter().enumerate().take(MAX_ARGS) {
            let mut ways = Vec::new();
            if self.foldable(inst, arg, refused) {
                ways.push(Shown::Expand);
            }
            if Terms::new(self.source, inst, PLAIN).constant(arg).is_some() {
                ways.push(Shown::Const);
            }
            ways.push(Shown::Reg);
            plans = plans
                .into_iter()
                .flat_map(|plan| {
                    ways.iter().map(move |&way| {
                        let mut next = plan;
                        next[index] = way;
                        next
                    })
                })
                .collect();
        }
        plans
    }

    /// Whether an operand may be shown as the instruction that computed it.
    ///
    /// It has to be in the same block, because a rule that folds one instruction into another
    /// moves the work to where the second one is. It has to be something rather than a block
    /// parameter, and not a constant, which is shown as a constant instead. And it has to be a
    /// value [`Lowering::left_alive`] has not put back, which is how the one reader at a time
    /// question is asked here: this says yes to a value with any number of readers, and a value
    /// only some of them could take is refused after the fact and asked again.
    ///
    /// A value with several readers used to be refused outright, on the reasoning that folding
    /// does not delete the instruction for anybody else. That reasoning is about the set of
    /// readers and was being applied to one reader at a time, which is stricter than it needs to
    /// be: when every reader takes it there is nobody left to read it and the instruction goes.
    /// An address a store and a load share is the shape that matters, since a memory operand has
    /// room for the whole of it and both readers have a memory operand.
    fn foldable(&self, into: Inst, value: Value, refused: &HashSet<Value>) -> bool {
        let Def::Result { inst, .. } = self.source[value].def else { return false };
        if self.source[inst].opcode == Opcode::IConst || refused.contains(&value) {
            return false;
        }
        self.source.block_of(inst).is_some()
            && self.source.block_of(inst) == self.source.block_of(into)
    }

    /// The instructions a match folded into the one it matched.
    ///
    /// The plan is what says this, not the bindings: a binding is a register or a number either
    /// way, and an operand shown as the instruction that computed it is one no rule could have
    /// matched without taking that instruction, because the plan offered the matcher nothing
    /// else to call it.
    fn folds(&self, inst: Inst, plan: Plan) -> Vec<Inst> {
        let args = &self.source[self.source[inst].args];
        args.iter()
            .take(MAX_ARGS)
            .enumerate()
            .filter(|&(index, _)| plan[index] == Shown::Expand)
            .filter_map(|(_, &arg)| match self.source[arg].def {
                Def::Result { inst, .. } => Some(inst),
                Def::Param { .. } => None,
            })
            .collect()
    }

    /// What the IR instruction said about itself that the machine instruction has to keep saying.
    ///
    /// One flag today. `volatile` says the access happens exactly once and is never moved or
    /// merged, and nothing below here can work that out again: a `volatile` load and an ordinary
    /// one are the same instruction over the same address, so a pass that puts two accesses
    /// together would put these together too. Carried rather than checked here, because the pass
    /// that has to refuse is a long way down and this is the last place the answer is known.
    ///
    /// The instructions this compiler writes for itself get nothing, which is the right answer
    /// for all of them: a prologue, a spill and the moves around a call were asked for by the
    /// machine rather than by the program.
    ///
    /// Every access the flag is legal on carries it: the loads and the stores a rule matched,
    /// the two ends of a `long double` copy that are the program's own memory, and the compare
    /// and exchange and the read modify write. An `asm` statement does not, and it is the one
    /// exception on purpose. What the flag says there is that the statement stays even when
    /// nothing reads what it wrote, which is a different sentence about a different thing, and
    /// every `asm` is already fixed where it stands whether the word was written or not.
    fn carried(&self, inst: Inst) -> mir::Flags {
        if self.source[inst].flags.contains(Flags::VOLATILE) {
            mir::Flags::VOLATILE
        } else {
            mir::Flags::NONE
        }
    }

    /// Build the machine instructions a match calls for.
    fn emit(&mut self, inst: Inst, matched: &Match<Term>) -> Result<(), Unsupported> {
        let rule: &Rule = self.selector.table.rule(matched);
        self.build(inst, rule.replacement, 0, &matched.bindings, true).map(|_| ())
    }

    /// Build the machine term that starts at `at`, and give back the position after it and the
    /// register it wrote, if it wrote one.
    ///
    /// The outermost term computes what the IR instruction does, so what it writes is the
    /// register of the instruction's result. A term inside another is a step on the way and
    /// writes a register of its own, which the term around it then reads. Its operands are read
    /// before it is built and it is built before the term around it, so the instructions come
    /// out in the order the values are needed.
    fn build(
        &mut self,
        inst: Inst,
        pieces: &'static [Piece],
        at: usize,
        bindings: &[Term],
        outermost: bool,
    ) -> Result<(usize, Option<mir::Reg>), Unsupported> {
        let Some(Piece::App { head, arity }) = pieces.get(at) else {
            return Err(self.unsupported(inst));
        };
        let opcode =
            head.strip_prefix(self.selector.prefix()).ok_or_else(|| self.unsupported(inst))?;
        let descs = self.selector.operands(opcode).ok_or_else(|| self.unsupported(inst))?;

        let mut read = Read::default();
        let mut at = at + 1;
        for _ in 0..*arity {
            at = self.read(inst, pieces, at, bindings, &mut read)?;
        }

        let writes = descs.iter().take_while(|desc| desc.role.is_def()).count();
        if descs.len() - writes != read.regs.len() {
            return Err(self.unsupported(inst));
        }

        // The first thing the instruction writes is what it computes, and any others are
        // registers the machine destroys on the way, which are fresh because nothing else is in
        // them and nothing reads them. An instruction that writes nothing at all is one whose
        // whole purpose is its effect, which is what a store is, and there is no result to put
        // anywhere.
        let mut regs = Vec::new();
        if writes > 0 {
            // A term inside another computes a step rather than the result, into a register only
            // the term around it reads.
            let first = match outermost {
                true => {
                    let result =
                        self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
                    self.new_reg(result)
                }
                false => self.out.new_vreg(descs[0].class),
            };
            regs.push(first);
            // The rest are the registers the machine destroys on the way, and the class each is in
            // is the one the instruction's description gives it rather than a guess, so that an
            // instruction that wrecks a register in the other file says so.
            regs.extend(descs[1..writes].iter().map(|desc| self.out.new_vreg(desc.class)));
        } else if !outermost || self.source[inst].first_result.is_some() {
            // A rule that throws away a value the IR gave a name to would leave every reader of
            // that name with nothing to read, so it is a rule this and the target disagree about.
            // So is a term inside another that writes nothing for the one around it to read.
            return Err(self.unsupported(inst));
        }
        let written = regs.first().copied();
        regs.extend(read.regs.iter().copied());

        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(head));
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in descs.iter().zip(regs) {
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            build = build.operand(operand);
        }
        if let Some(mem) = read.mem {
            build = build.mem(mem);
        }
        if let Some(imm) = read.imm {
            build = build.imm(imm);
        }
        build.finish();
        Ok((at, written))
    }

    /// Read one argument of a replacement, which is a register, a number, an address or another
    /// machine term.
    ///
    /// Gives back the position after it, because a replacement is flat and an address or a term
    /// takes arguments of its own. A machine term is built on the spot, and what is read is the
    /// register it wrote.
    fn read(
        &mut self,
        inst: Inst,
        pieces: &'static [Piece],
        at: usize,
        bindings: &[Term],
        out: &mut Read,
    ) -> Result<usize, Unsupported> {
        match pieces.get(at) {
            Some(Piece::Int(value)) => {
                out.imm = i64::try_from(*value).ok();
                Ok(at + 1)
            }
            // A number the rule worked out of the ones it matched rather than one it wrote down,
            // which is an immediate once it has been worked out and is read here as one. It gives
            // nothing back when a binding it reads is a register, and a replacement that cannot be
            // built is a rule this file and the matcher disagree about, which is what `unsupported`
            // is for.
            Some(Piece::Computed { work, .. }) => {
                let matched: Vec<Option<i128>> = bindings
                    .iter()
                    .map(|term| match *term {
                        Term::Num(value) => Some(value),
                        _ => None,
                    })
                    .collect();
                let number = work(&matched).ok_or_else(|| self.unsupported(inst))?;
                out.imm = i64::try_from(number).ok();
                Ok(at + 1)
            }
            Some(Piece::Var { index, .. }) => {
                match bindings.get(*index) {
                    Some(&Term::Reg(value)) => {
                        let reg = self.reg_of(value)?;
                        out.regs.push(reg);
                    }
                    Some(&Term::Num(value)) => out.imm = i64::try_from(value).ok(),
                    // A pattern binds a register or a number and nothing else, so this is a
                    // rule the matcher and this file disagree about.
                    _ => return Err(self.unsupported(inst)),
                }
                Ok(at + 1)
            }
            Some(Piece::App { head, .. }) if (self.selector.address)(head).is_none() => {
                let (next, reg) = self.build(inst, pieces, at, bindings, false)?;
                out.regs.push(reg.ok_or_else(|| self.unsupported(inst))?);
                Ok(next)
            }
            Some(Piece::App { head, arity }) => {
                let kind = (self.selector.address)(head).ok_or_else(|| self.unsupported(inst))?;
                let mut inner = Read::default();
                let mut next = at + 1;
                for _ in 0..*arity {
                    next = self.read(inst, pieces, next, bindings, &mut inner)?;
                }
                let mem = address(kind, &inner, self.gpr).ok_or_else(|| self.unsupported(inst))?;
                out.mem = Some(mem);
                Ok(next)
            }
            None => Err(self.unsupported(inst)),
        }
    }

    /// The register a value is in, materializing it if it is a constant that has not been put in
    /// one yet.
    ///
    /// A constant is written where it is wanted rather than where the IR defined it, and where it
    /// is wanted is a block that need not be the one the IR defined it in. So the register holding
    /// one is only good inside the block it was written into, and a second block that wants the
    /// same constant gets its own. Anything else is a register read where nothing wrote it: the
    /// IR guarantees a definition dominates its uses, and this moved the definition.
    ///
    /// Writing the number again is also the right answer and not merely the safe one. It is one
    /// instruction that reads nothing, which is cheaper than holding a register live across a
    /// branch for it, and it is what a rematerializing allocator would do with the value anyway.
    fn reg_of(&mut self, value: Value) -> Result<mir::Reg, Unsupported> {
        let constant = match self.source[value].def {
            Def::Result { inst, .. } => {
                (self.source[inst].opcode == Opcode::IConst).then_some(inst)
            }
            Def::Param { .. } => None,
        };
        let here = self.at.expect("a block is being filled");
        if let Some(reg) = self.regs[value.index()] {
            if constant.is_none() || self.written[value.index()] == Some(here) {
                return Ok(reg);
            }
        }
        if let Some(inst) = constant {
            // Cleared so that the register the constant is written into is a new one rather than
            // the one the block above wrote, which is still being read up there.
            self.regs[value.index()] = None;
            // Nothing is refused here. A constant is written on its own, out of the loop over the
            // block, and the operands of the rule that writes one are the number and nothing else.
            let matched = self
                .select(inst, &HashSet::new())
                .map(|(_, matched)| matched)
                .ok_or_else(|| self.unsupported(inst))?;
            self.emit(inst, &matched)?;
            // The same mark the loop over the instructions makes, and it has to be made here as
            // well because this is the only place a constant is ever selected: the loop skips one
            // where the IR wrote it, so a rule that lowers a constant fires from nowhere else and
            // would be reported as a rule nothing reaches.
            self.fired.mark(matched.rule);
            self.written[value.index()] = Some(here);
            return Ok(self.regs[value.index()].expect("a constant is written into a register"));
        }
        Ok(self.new_reg(value))
    }

    /// Which register file a value of that type lives in.
    ///
    /// The vector one for the two float widths the machine has scalar instructions for and for the
    /// one it only moves, and the general purpose one for everything else. An eighty bit `long
    /// double` is in neither, and it is here rather than in the vector class on purpose: it would
    /// be put in a register that cannot hold it, and there is no rule that names one, so the
    /// instruction computing it is reported. The wrong class would make that a wrong program
    /// instead of a refused one.
    ///
    /// A hundred and twenty eight bit float is in the vector class and fits it exactly, which is
    /// the difference. Nothing computes in it, so every arithmetic on one is still reported, and
    /// what the class buys is the moves: a register that holds the whole value is a register a
    /// spill, a reload and a copy are each one instruction for.
    fn class_of(&self, ty: Type) -> RegClass {
        if crate::term::in_vector_file(ty) { self.conv.sse_class } else { self.gpr }
    }

    /// A fresh register for a value, which is what the instruction computing it writes.
    ///
    /// Any declaration the value is a value of comes with it. Here rather than once at the end over
    /// the whole map, because a constant is written again in every block that wants one and the map
    /// only remembers the last of those registers, and a local held in a constant is a local that
    /// would otherwise be findable in one block of the function and nowhere else.
    fn new_reg(&mut self, value: Value) -> mir::Reg {
        if let Some(reg) = self.regs[value.index()] {
            return reg;
        }
        let reg = self.out.new_vreg(self.class_of(self.source[value].ty));
        self.regs[value.index()] = Some(reg);
        let source = self.source;
        for decl in source.value_decls(value) {
            self.out.named.push((decl, reg));
        }
        reg
    }

    fn unsupported(&self, inst: Inst) -> Unsupported {
        let data = &self.source[inst];
        Unsupported::Inst {
            inst,
            term: Terms::new(self.source, inst, PLAIN).name(inst),
            opcode: data.opcode,
            ty: data.first_result.map(|result| self.source[result].ty),
        }
    }
}

/// What the arguments of one replacement came to.
#[derive(Debug, Default)]
struct Read {
    regs: Vec<mir::Reg>,
    imm: Option<i64>,
    mem: Option<mir::Mem>,
}

/// The addressing mode an address constructor's arguments make.
///
/// One arm per constructor rather than a question asked of the kind, because what the arguments
/// mean is the whole of what tells the four apart: the same register is a base in one and an
/// index in another, and the same constant is a scale in one and a displacement in another.
fn address(kind: Address, read: &Read, gpr: RegClass) -> Option<mir::Mem> {
    let mut regs = read.regs.iter().copied().map(|reg| mir::Operand::read(reg, gpr));
    match kind {
        Address::BaseIndexScale => {
            let base = regs.next()?;
            let index = regs.next()?;
            Some(mir::Mem::at(base).indexed(index, u8::try_from(read.imm?).ok()?))
        }
        Address::IndexScale => Some(mir::Mem {
            base: None,
            index: Some(regs.next()?),
            scale: u8::try_from(read.imm?).ok()?,
            disp: 0,
            symbol: None,
            block: None,
            table: None,
            reach: mir::Reach::Itself,
            segment: None,
        }),
        Address::Base => Some(mir::Mem::at(regs.next()?)),
        // The rule that writes this has a guard saying the constant fits, so a displacement that
        // does not is a rule and a target that disagree rather than a program this cannot compile.
        Address::BaseOffset => {
            Some(mir::Mem { disp: i32::try_from(read.imm?).ok()?, ..mir::Mem::at(regs.next()?) })
        }
    }
}

#[cfg(test)]
mod tests {
    use rucc_ir::{
        AsmInfo, Builder, CallInfo, Flags, InstData, MemInfo, MemOrder, Restrict, Signature, Type,
    };
    use rucc_regalloc::assign::Env;
    use rucc_target::x86_64::{FRAME, REGS, SYSV};

    use super::*;
    use crate::finish::{Convention, finish};
    use crate::frame::{Frame, Incoming, Layout};
    use crate::select::x86_64::SELECTOR;

    /// A function of as many 64 bit parameters as the test wants, and the block they are in.
    fn blank(params: &[Type]) -> (Interner, Func, Block, Vec<Value>) {
        let mut names = Interner::new();
        let mut func = Func::new(names.intern("f"), Signature::new());
        let block = func.create_block();
        let values = params.iter().map(|&ty| func.append_param(block, ty)).collect();
        (names, func, block, values)
    }

    /// An ordinary access: not atomic, and aligned enough that nothing here has an opinion.
    /// Neither field reaches selection, which is the point of saying it once here.
    fn plain() -> MemInfo {
        MemInfo {
            size: 0,
            align: 1,
            order: MemOrder::NotAtomic,
            tbaa: None,
            owns: 0,
            restrict: Restrict::NONE,
        }
    }

    /// What the allocator is given: every integer register the convention offers except two, held
    /// back so that a move on an edge has somewhere to break a cycle and a spilled value has
    /// somewhere to be read into. Which two does not matter, and holding back the last two the
    /// convention would reach for leaves every expectation below unchanged.
    fn env() -> Env {
        const SCRATCH: [PhysReg; 2] = [x86_64::R10, x86_64::R11];
        let order: Vec<PhysReg> =
            SYSV.int_order.iter().copied().filter(|reg| !SCRATCH.contains(reg)).collect();
        Env::new().with(x86_64::GPR, &order, &SCRATCH)
    }

    /// The machine IR text a function lowers to.
    fn lower(names: &mut Interner, source: &Func) -> String {
        let out = func(source, names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        mir::print_func(&out.func, names, &REGS)
    }

    /// The same function lowered for AArch64, which is the first thing this file writes for a
    /// machine other than x86-64. Nothing past selection runs here, so what is checked is that the
    /// arguments, the rule and the return all come out named for the machine that was asked for.
    #[test]
    fn an_addition_lowers_for_aarch64_with_its_own_names() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        let sum = build.binary(Opcode::Add, args[0], args[1], Flags::default());
        build.ret(&[sum]);

        let conv = &aarch64::AAPCS64;
        let selector = &crate::select::aarch64::SELECTOR;
        let out = super::func(&func, &mut names, selector, conv, &Elsewhere::default())
            .expect("an addition and a return have AArch64 rules");
        let text = mir::print_func(&out.func, &names, &aarch64::REGS);
        assert!(!text.contains("x64."), "{text}");
        assert!(text.contains("= a64.arg_val_32"), "{text}");
        assert!(text.contains("= a64.add_rr_32 %0, %1"), "{text}");
        assert!(text.contains("a64.ret_val_32 %2"), "{text}");
    }

    /// Lowers one function for AArch64 and prints it, or says why it could not.
    fn lower_a64(names: &mut Interner, func: &Func) -> Result<String, String> {
        let conv = &aarch64::AAPCS64;
        let selector = &crate::select::aarch64::SELECTOR;
        let out = super::func(func, names, selector, conv, &Elsewhere::default())
            .map_err(|why| why.to_string())?;
        Ok(mir::print_func(&out.func, names, &aarch64::REGS))
    }

    /// Nothing reads AArch64 assembly back into instructions, so every template there is kept as
    /// its text. The operands are the instruction's own, with the output first and the inputs
    /// last, a hole in the text asks for the `w` or the `x` name of one, and a vector register the
    /// clobber list names is written by it as well as every register a call may leave anything in.
    #[test]
    fn a_template_on_aarch64_is_kept_as_text_with_its_operands_in_registers() {
        let (i32, i64) = (Type::int(32), Type::int(64));
        let (mut names, mut source, block, args) = blank(&[i32, i64]);
        let out = clobbering(
            &mut source,
            block,
            &mut names,
            "add %w0, %w1, #1\n\tstr %2, [sp]",
            "=r,r,r",
            "d8",
            &[args[0], args[1]],
            &[i32],
        );
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // Forty one registers between the output and the inputs: `x0` to `x15`, the sixteen vector
        // registers a call does not keep, and `v8`, which is the one the program named.
        let text = lower_a64(&mut names, &source).expect("kept as text");
        assert!(text.contains("%2:gpr, early $x0, early $x1,"), "{text}");
        assert!(text.contains(
            "early $v31, early $v8 = a64.template %0, %1, \
             @add \u{1}r0w\u{2}, \u{1}r42w\u{2}, #1\n\tstr \u{1}r43x\u{2}, [sp]\n"
        ));
    }

    /// A letter that means one thing on x86 and another on AArch64 is refused there rather than
    /// read as x86. `a` to `d` and `S` name one register each on x86 and nothing on AArch64.
    #[test]
    fn a_constraint_letter_the_two_machines_disagree_about_is_refused_on_aarch64() {
        let i64 = Type::int(64);
        for constraints in ["=a,r", "=r,S", "=r,c"] {
            let (mut names, mut source, block, args) = blank(&[i64]);
            let out = clobbering(
                &mut source,
                block,
                &mut names,
                "mov %0, %1",
                constraints,
                "",
                &[args[0]],
                &[i64],
            );
            let produced = source[out].results().next().expect("one result");
            Builder::new(&mut source, block).ret(&[produced]);
            let refused = lower_a64(&mut names, &source).expect_err(constraints);
            assert!(refused.contains("has an operand this cannot place"), "{refused}");
        }
    }

    /// `Q` on AArch64 is memory addressed by one register, which is `[x3]` and is how an operand in
    /// memory is spelled there already.
    #[test]
    fn a_q_operand_on_aarch64_is_its_address_in_brackets() {
        let (i64, ptr) = (Type::int(64), Type::PTR);
        let (mut names, mut source, block, args) = blank(&[ptr]);
        let out =
            clobbering(&mut source, block, &mut names, "ldr %x0, %1", "=r,Q", "", &args, &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);
        let text = lower_a64(&mut names, &source).expect("kept as text");
        assert!(text.contains("@ldr \u{1}r0x\u{2}, [\u{1}r"), "{text}");
    }

    /// `w` on AArch64 is a vector register, named `v` with no modifier the way gcc names it and by
    /// its scalar view with one. An integer asked for in one is refused, since it would need a move
    /// into that file first.
    #[test]
    fn a_vector_operand_on_aarch64_is_in_the_vector_file() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let out = clobbering(
            &mut source,
            block,
            &mut names,
            "fadd %d0, %d1, %d2\n\tmov %0.16b, %0.16b",
            "=w,w,w",
            "",
            &[args[0], args[1]],
            &[f64],
        );
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);
        let text = lower_a64(&mut names, &source).expect("kept as text");
        assert!(text.contains("%2:fpr, early $x0,"), "{text}");
        assert!(text.contains("@fadd \u{1}r0d\u{2}, \u{1}r"), "{text}");
        assert!(text.contains("\n\tmov \u{1}r0v\u{2}.16b, \u{1}r0v\u{2}.16b\n"), "{text}");

        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64]);
        let out =
            clobbering(&mut source, block, &mut names, "fmov %d0, %d1", "=w,w", "", &args, &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);
        assert!(lower_a64(&mut names, &source).is_err());
    }

    #[test]
    fn an_addition_of_two_registers_is_one_instruction() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::Add, args[0], args[1], Flags::default());

        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr(reuse 1) = x64.add_rr_32 %0, %1\n}\n"
        );
    }

    #[test]
    fn a_constant_operand_becomes_an_immediate() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32]);
        let mut build = Builder::new(&mut func, block);
        let seven = build.iconst(i32, 7);
        build.binary(Opcode::Add, args[0], seven, Flags::default());

        // The constant is in the instruction and nothing was written to hold it, which is what
        // materializing one where a register for it is wanted buys.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr(reuse 1) = x64.add_ri_32 %0, 7\n}\n"
        );
    }

    #[test]
    fn a_constant_too_wide_for_an_immediate_goes_into_a_register() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let big = build.iconst(i64, i128::from(i32::MAX) + 1);
        build.binary(Opcode::Add, args[0], big, Flags::default());

        // Nobody wrote this fallback down. The rule that takes an immediate has a guard that
        // turns a number this wide down, so it does not fire, and the next way of showing the
        // operand puts it in a register.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_ri_64 2147483648\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n}\n"
        );
    }

    #[test]
    fn an_index_calculation_folds_into_an_address() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        build.binary(Opcode::Add, args[0], scaled, Flags::default());

        // Three IR instructions and one machine instruction. The multiply is gone because the
        // rule that matched reached down and took it.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.lea_64 [%0 + %1*4]\n}\n"
        );
    }

    #[test]
    fn an_instruction_every_reader_can_take_is_folded_into_all_of_them() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        let first = build.binary(Opcode::Add, args[0], scaled, Flags::default());
        build.binary(Opcode::Add, first, scaled, Flags::default());

        // Both readers have room for a scaled index, so both of them take it and nothing is left
        // to read the multiply. Three IR instructions become two machine ones, where refusing to
        // fold into either reader would have left three.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.lea_64 [%0 + %1*4]\n    \
             %3:gpr = x64.lea_64 [%2 + %1*4]\n}\n"
        );
    }

    #[test]
    fn an_instruction_one_of_its_readers_cannot_take_is_folded_into_none_of_them() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        build.binary(Opcode::Add, args[0], scaled, Flags::default());
        build.store(scaled, args[0], plain(), Flags::default());

        // The addition has room for the multiply and the store does not: what a store writes is
        // a register, and no rule reaches through it. Folding into the addition alone would
        // leave the multiply where it is for the store to read and do the work twice, so the
        // multiply is put back and both readers read the register it wrote.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.lea_64 [%1*4]"), "{text}");
        assert!(text.contains("x64.add_rr_64"), "{text}");
    }

    #[test]
    fn a_shift_by_a_register_asks_for_it_in_cl() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::Shl, args[0], args[1], Flags::default());

        // The fixed register is not in the rule. It is what the target says the instruction does
        // with its operands, and the allocator is what will act on it.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.shl_rcl_32 %0, %1($rcx)"), "{text}");
    }

    #[test]
    fn a_division_names_the_registers_and_the_register_it_destroys() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::SDiv, args[0], args[1], Flags::default());

        // Two definitions, because a division writes the remainder whether anybody wanted it or
        // not, and the second one is early because it is destroyed before the operands are read.
        let text = lower(&mut names, &func);
        assert!(
            text.contains("%2:gpr($rax), early %3:gpr($rdx) = x64.idiv_quo_32 %0($rax), %1"),
            "{text}"
        );
    }

    #[test]
    fn a_load_reads_through_the_register_the_address_is_in() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        build.load(Type::int(32), args[0], plain(), Flags::default());

        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_rm_32 [%0]\n}\n"
        );
    }

    #[test]
    fn a_store_writes_no_register_and_the_value_it_writes_is_the_one_the_ir_gave_it() {
        let (mut names, mut func, block, args) = blank(&[Type::int(32), Type::int(64)]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[0], args[1], plain(), Flags::default());

        // The value is the first parameter and the address is the second, and the instruction
        // takes them the other way round. Getting that backwards would compile to a store of the
        // address into the value, which is a program that runs and does the wrong thing.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    x64.mov_mr_32 %0, [%1]\n}\n"
        );
    }

    #[test]
    fn an_address_with_a_constant_added_folds_into_the_access() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let twelve = build.iconst(i64, 12);
        let field = build.binary(Opcode::Add, args[0], twelve, Flags::default());
        build.load(Type::int(64), field, plain(), Flags::default());

        // Two IR instructions and one machine instruction, which is what every read of a field
        // of a structure comes to.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_rm_64 [%0 + 12]\n}\n"
        );
    }

    #[test]
    fn a_displacement_too_wide_to_encode_leaves_the_addition_where_it_is() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let big = build.iconst(i64, i128::from(i32::MAX) + 1);
        let far = build.binary(Opcode::Add, args[0], big, Flags::default());
        build.load(Type::int(32), far, plain(), Flags::default());

        // A displacement is signed and 32 bits. The rule that folds one has a guard that turns
        // this down, so the addition stays and the load reads through what it produced. Nobody
        // wrote that fallback: it is the next way of showing the operand.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.mov_rm_32 [%2]"), "{text}");
        assert!(text.contains("x64.add_rr_64"), "{text}");
    }

    #[test]
    fn a_store_of_a_value_that_was_loaded_is_two_instructions_and_no_arithmetic() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let got = build.load(Type::int(8), args[0], plain(), Flags::default());
        build.store(got, args[1], plain(), Flags::default());

        // A load feeding a store is the one place folding would be wrong: an x86-64 `mov` has at
        // most one memory operand, and there is no rule that takes two, so the load is left where
        // it is and the store reads the register it wrote.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.mov_rm_8 [%0]\n    \
             x64.mov_mr_8 %2, [%1]\n}\n"
        );
    }

    #[test]
    fn an_access_at_a_width_no_rule_is_written_at_is_reported() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut source, block);
        build.load(Type::int(128), args[0], plain(), Flags::default());

        // The width is the whole of what is wrong here, so the width is in the message: `load`
        // on its own is written about at every other width and would send a reader looking in
        // the wrong place.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("nothing loads 128 bits");
        assert_eq!(failed.to_string(), "no rule lowers a `load` producing a `i128`");
    }

    #[test]
    fn a_return_asks_for_the_value_in_the_register_the_caller_reads() {
        let (mut names, mut func, block, args) = blank(&[Type::int(32)]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0]]);

        // The register is not in the rule, the same way `cl` is not in the rule for a shift. It
        // is what the target says the instruction does with its operand, and the allocator is
        // what will act on it. There is no `ret` here, because giving the frame back has to
        // happen between this and leaving and the frame is not worked out yet.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_return_of_two_values_asks_for_the_second_register_as_well() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { long a, b; } f(long a, long b)`, after the front end has classified it. Both
        // halves are integers, so the second is in the second integer return register, and both
        // pseudos say so the same way the one for a single value does.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    x64.ret_val_64 %0($rax)\n    \
             x64.ret_val2_64 %1($rdx)\n}\n"
        );
    }

    #[test]
    fn two_values_back_in_different_files_are_both_the_first_of_their_own() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut func, block, args) = blank(&[f64, Type::int(64)]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { double a; long b; } f(double a, long b)`. The two files are counted apart, so
        // neither half is the second of anything and the `double` is in `xmm0` rather than in the
        // register a second `double` would have been in. Getting this wrong is not a crash: the
        // caller reads a register nobody wrote, and this is where that is ruled out.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:gpr($rdi) = x64.arg_val_64\n    x64.ret_val_f64 %0($xmm0)\n    \
             x64.ret_val_64 %1($rax)\n}\n"
        );
    }

    #[test]
    fn two_of_the_same_file_back_take_the_first_two_of_it() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut func, block, args) = blank(&[f64, f64]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { double x, y; } f(double x, double y)`, which is the vector half of the pair
        // above and counts in its own file the same way.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:xmm($xmm1) = x64.arg_val_f64\n    x64.ret_val_f64 %0($xmm0)\n    \
             x64.ret_val2_f64 %1($xmm1)\n}\n"
        );
    }

    /// A function whose answer goes back through memory, with the pointer to the space for it in
    /// front of whatever else it takes. Only the signature says it is one.
    fn returning_through_memory(params: &[Type]) -> (Interner, Func, Block, Vec<Value>) {
        let mut names = Interner::new();
        let sret = Abi::Sret { size: 32, align: 8 };
        let mut signature = Signature::new().and_param(Param::with_abi(Type::PTR, sret));
        signature.params.extend(params.iter().copied().map(Param::new));
        let mut func = Func::new(names.intern("f"), signature);
        let block = func.create_block();
        let space = func.append_param(block, Type::PTR);
        let values = std::iter::once(space)
            .chain(params.iter().map(|&ty| func.append_param(block, ty)))
            .collect();
        (names, func, block, values)
    }

    #[test]
    fn the_space_a_return_through_memory_was_given_goes_back_in_the_first_return_register() {
        let (mut names, mut func, block, _) = returning_through_memory(&[]);
        Builder::new(&mut func, block).ret(&[]);

        // `struct big f(void)`, where `big` is too large to come back in registers. The `return`
        // carries nothing, because the value went into the space the caller handed over, and the
        // document still says that address comes back in `rax`. Nothing in the IR says it, so the
        // convention says it, and the pseudo is the one any other pointer return would use.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn what_the_function_did_in_between_does_not_take_the_register_off_it() {
        let (mut names, mut func, block, args) = returning_through_memory(&[Type::int(32)]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[1], args[0], plain(), Flags::default());
        build.ret(&[]);

        // The register is a read at the end and not a move at the start, so it is live across
        // everything between the two and the allocator has to keep it somewhere. In a function
        // with a call in it that somewhere is a callee saved register, and the address comes back
        // into `rax` here rather than whatever the last instruction happened to leave there. That
        // is issue #333, and a store is enough to show the value outlives the entry block.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.mov_mr_32 %1, [%0]"), "{text}");
        assert!(text.ends_with("    x64.ret_val_64 %0($rax)\n}\n"), "{text}");
    }

    #[test]
    fn a_pointer_that_is_only_a_pointer_is_not_given_back() {
        let (mut names, mut func, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[0], args[0], plain(), Flags::default());
        build.ret(&[]);

        // `void f(void **p)`. It takes a pointer first and returns nothing, which is the shape of
        // the one above and none of its meaning, and what tells them apart is the signature. A
        // `void` function leaves `rax` alone.
        assert!(!lower(&mut names, &func).contains("ret_val"));
    }

    #[test]
    fn a_return_of_a_constant_puts_it_in_a_register_first() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        // No rule returns an immediate, so the plan that offers one is turned down and the next
        // one materializes it. That is `int main(void) { return 0; }` in full, once the epilogue
        // is appended to it.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_rule_that_writes_a_constant_down_is_recorded_as_a_rule_that_fired() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        // The loop over the instructions passes a constant by, because a constant is written where
        // a register for it is first wanted rather than where the IR put it. So the only place a
        // rule about one is ever selected is the materialization, and a mark made in the loop
        // alone would report every rule about a constant as a rule nothing reaches.
        let out = super::func(&func, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let rules = &crate::select::x86_64::TABLE.rules;
        let fired: Vec<&str> = rules
            .iter()
            .enumerate()
            .filter(|(index, _)| out.fired.has(*index))
            .map(|(_, rule)| rule.pattern)
            .collect();
        assert!(fired.contains(&"(iconst.i32 k)"), "{fired:?}");
    }

    #[test]
    fn a_return_of_nothing_is_no_instruction_at_all() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[]);

        // Every part of leaving a function that returns nothing is the epilogue's, and the
        // epilogue goes in after allocation. A block with nothing in it is the right answer here
        // rather than a function that could not be lowered.
        assert_eq!(lower(&mut names, &func), "mfunc @f {\nblock0:\n}\n");
    }

    #[test]
    fn the_allocator_is_what_moves_the_answer_into_the_return_register() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        let mut out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // `int main(void) { return 0; }` end to end. Nothing here asked for `rax`: the rule said
        // the value goes back, the target said where, and the allocator is what made it true. The
        // epilogue is what leaves, and this function needs no frame, so it is the return alone.
        //
        // Two instructions and no copy, which is what a hint buys. The return insists on `rax`,
        // so `rax` is the register the allocator tries first for the value the return reads, and
        // the constant is written straight into it.
        assert_eq!(
            mir::print_func(&out, &names, &REGS),
            "mfunc @f {\nblock0:\n    $rax = x64.mov_ri_32 0\n    \
             x64.ret_val_32 $rax($rax)\n    x64.ret\n}\n"
        );
    }

    #[test]
    fn a_function_of_two_arguments_is_a_whole_function_now() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, args[0], args[1], Flags::default());
        build.ret(&[sum]);

        let mut out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // `int f(int a, int b) { return a + b; }` end to end, and this is the test the argument
        // side exists for. Before it there was no way to write one: the allocator refuses a
        // function whose entry block takes parameters, because there is no edge into an entry
        // block for the moves that give a block parameter its value to go on.
        //
        // One move, and it is the one the machine's addition needs rather than one the allocator
        // owes anybody. Each argument stays in the register it arrived in, because the pseudo
        // that defines it insists on that register and the allocator now tries it first, and the
        // sum stays in the register the addition wrote it to until the return reads it out. The
        // copy in front of a two address instruction is what makes its destination one of the
        // registers it reads, and the source operand keeps its own name because the destination
        // is what the encoder writes.
        assert_eq!(
            mir::print_func(&out, &names, &REGS),
            "mfunc @f {\nblock0:\n    $rdi($rdi) = x64.arg_val_32\n    \
             $rsi($rsi) = x64.arg_val_32\n    \
             $rdi(reuse 1) = x64.add_rr_32 $rdi, $rsi\n    $rax = x64.mov_rr_64 $rdi\n    \
             x64.ret_val_32 $rax($rax)\n    x64.ret\n}\n"
        );
    }

    #[test]
    fn an_argument_with_no_register_left_for_it_is_read_out_of_the_caller_s_stack() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 7]);
        let mut build = Builder::new(&mut source, block);
        build.ret(&[args[6]]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("the seventh is read from memory");

        // SysV passes six integers in registers and the seventh in the caller's memory, so six of
        // these are pseudos that encode to nothing and the seventh is a load that encodes to real
        // bytes. Its displacement is nothing here for the reason a local's is: there is no frame
        // yet. What the walk hands on is which instruction is waiting, and for how far up the
        // caller's argument area, which is the bottom of it because it is the first one there.
        assert_eq!(lowered.stack.arguments.len(), 1);
        assert_eq!(lowered.stack.arguments[0].1, 0);
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("%6:gpr = x64.mov_rm_64 [$rsp]"), "{text}");
        assert_eq!(text.matches("x64.arg_val_64").count(), 6, "{text}");
    }

    #[test]
    fn the_frame_is_what_says_how_far_up_the_caller_s_stack_an_argument_is() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 8]);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, args[6], args[7], Flags::default());
        build.ret(&[sum]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("both are read from memory");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // A leaf that takes no frame, so the stack pointer never moves and the only thing between
        // it and the caller's arguments is the return address the call pushed. The seventh
        // parameter is at the bottom of the caller's argument area and the eighth is one word
        // further up, which is the eight bytes between the two offsets.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(frame.size(), 0);
        assert_eq!(frame.incoming(), Incoming::from_stack(8));
        assert!(text.contains("x64.mov_rm_64 [$rsp + 8]"), "{text}");
        assert!(text.contains("x64.mov_rm_64 [$rsp + 16]"), "{text}");
    }

    #[test]
    fn a_realigned_frame_reaches_the_caller_s_arguments_through_the_frame_pointer() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 7]);
        let wide = slot(&mut source, block, 64, 32);
        let mut build = Builder::new(&mut source, block);
        build.store(args[6], wide, plain(), Flags::default());
        build.ret(&[args[6]]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // A local wanting thirty two byte alignment makes the prologue force the stack pointer,
        // which throws away how far the caller's stack was. So the load the lowering wrote off the
        // stack pointer is rewritten to read through the frame pointer, at the one distance that
        // survives: the word the prologue pushed the frame pointer into, and the return address
        // above it.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(frame.realign(), Some(32));
        assert_eq!(frame.incoming(), Incoming::from_frame(16));
        assert!(text.contains("x64.mov_rm_64 [$rbp + 16]"), "{text}");
        assert!(!text.contains("x64.mov_rm_64 [$rsp"), "{text}");
    }

    #[test]
    fn a_jump_is_the_edge_and_nothing_else() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32]);
        let next = source.create_block();
        let got = source.append_param(next, i32);
        Builder::new(&mut source, entry).jump(next, &[args[0]]);
        Builder::new(&mut source, next).ret(&[got]);

        // Two blocks and two instructions, and the jump is neither of them. What it was is the
        // arm on the first block, and what the arm carries is the argument it was called with.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32 block1(%0)\n\n\
             block1(%1:gpr):\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    /// A block that reads what a block below it writes is filled after it, not before it.
    ///
    /// The blocks are written entry, `early`, `late`, `exit`, and the entry jumps straight past
    /// `early` to `late`, so `late` dominates `early` while sitting below it in the function.
    /// Filling them in the order they are written reaches the read in `early` first, and reading
    /// a value with no register yet mints one. The cast in `late` is no instruction at all, so
    /// what it does is give its answer the register its operand is already in, and that is not
    /// the register the read minted. Nothing writes the register the read minted. The printer
    /// says `%?` for a register nothing defines, which is what this looks for, and what came out
    /// of the real bug was SQLite loading a stack slot no store ever reached.
    #[test]
    fn a_block_that_reads_what_a_block_below_it_writes_is_filled_after_it() {
        let i64 = Type::int(64);
        let (mut names, mut source, entry, args) = blank(&[i64, i64]);
        let early = source.create_block();
        let late = source.create_block();
        let exit = source.create_block();

        Builder::new(&mut source, entry).jump(late, &[]);
        let ptr = cast(&mut source, late, Opcode::IntToPtr, args[0], Type::PTR);
        Builder::new(&mut source, early).ret(&[ptr]);
        let mut build = Builder::new(&mut source, late);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, early, &[], exit, &[]);
        Builder::new(&mut source, exit).ret(&[args[1]]);

        let text = lower(&mut names, &source);
        assert!(!text.contains("%?"), "every register has something that writes it: {text}");
    }

    /// A constant is written where it is wanted rather than where the IR defined it, and two
    /// blocks wanting the same one is two places. Writing it once and reading it in both is a
    /// register read where nothing wrote it, unless the block it was written in happens to
    /// dominate the other, which nothing here checks and which the second arm of a branch never
    /// does. Each block gets its own copy of the number instead.
    #[test]
    fn a_constant_two_blocks_want_is_written_in_both_of_them() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);

        let mut build = Builder::new(&mut source, entry);
        let seven = build.iconst(i32, 7);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        // Both arms want the seven in a register, because a block argument is never an immediate,
        // and neither arm dominates the other.
        Builder::new(&mut source, then).jump(join, &[seven]);
        Builder::new(&mut source, other).jump(join, &[seven]);
        Builder::new(&mut source, join).ret(&[got]);

        let text = lower(&mut names, &source);
        assert_eq!(text.matches("x64.mov_ri_32 7").count(), 2, "one seven per block: {text}");
    }

    /// An argument on an edge out of a block that leaves two ways is read after every instruction
    /// of the block is written, and reading one can write an instruction, which would land after
    /// the branch that has already jumped past it. The branch goes back on the end.
    #[test]
    fn a_constant_an_edge_wants_is_written_before_the_branch_and_not_after_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);

        let mut build = Builder::new(&mut source, entry);
        let nine = build.iconst(i32, 9);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], join, &[nine]);
        Builder::new(&mut source, then).jump(join, &[args[0]]);
        Builder::new(&mut source, join).ret(&[got]);

        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let entry = out.entry().expect("an entry block");
        let last = out.terminator(entry).expect("a block that leaves two ways has a branch");
        let branch = names.intern("x64.br_cond_8");
        assert_eq!(
            out[last].opcode,
            mir::Opcode::new(branch),
            "the branch is last: {}",
            mir::print_func(&out, &names, &REGS)
        );
    }

    #[test]
    fn a_conditional_branch_is_lowered_to_the_condition_and_nothing_about_where_it_goes() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        Builder::new(&mut source, then).ret(&[args[0]]);
        Builder::new(&mut source, other).ret(&[args[1]]);

        // The comparison writes a byte and the branch reads it, and neither says a block. Both
        // arms are on the entry block, in the order the branch took them, so the arm that runs
        // when the condition holds is the first.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr = x64.cmp_set_l_32 %0, %1\n    \
             x64.br_cond_8 %2, block1, block2\n\n\
             block1:\n    x64.ret_val_32 %0($rax)\n\n\
             block2:\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    /// A choice between two values, which is one instruction and no blocks at all.
    ///
    /// The arms come out the other way round from the IR, because a conditional move overwrites its
    /// destination and the destination is the arm taken when the condition does not hold. The
    /// condition arrives last for the same reason: it is read by the test in front of the move
    /// rather than by the move.
    #[test]
    fn a_select_is_lowered_to_a_test_and_a_conditional_move() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        let picked = build.select(cond, args[0], args[1]);
        build.ret(&[picked]);

        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr = x64.cmp_set_l_32 %0, %1\n    \
             %3:gpr(reuse 1) = x64.test_cmov_ne_32 %1, %0, %2\n    \
             x64.ret_val_32 %3($rax)\n}\n"
        );
    }

    #[test]
    fn a_branch_over_a_block_is_a_whole_function_now() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        let mut build = Builder::new(&mut source, then);
        let sum = build.binary(Opcode::Add, args[0], args[1], Flags::default());
        build.jump(join, &[sum]);
        Builder::new(&mut source, other).jump(join, &[args[1]]);
        Builder::new(&mut source, join).ret(&[got]);

        // `int f(int a, int b) { if (a < b) return a + b; else return b; }` end to end, written
        // the way a front end writes it: both arms of the branch are blocks of their own and the
        // return is the block they meet at. No edge here is critical, because the two arms out of
        // the entry carry nothing and the two arms into the join each leave a block that goes
        // nowhere else, so each has its own end to put its move at.
        let mut out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        assert_eq!(crate::split::critical(&mut out), 0, "no edge here is critical");
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // One epilogue, on the join, which is the one block the function leaves from, and the
        // moves that give the join its parameter are at the end of each arm. Every register is
        // physical and the branch is still a branch on a register, because turning it into a
        // `test` and a `jcc` is the block layout's and there is no block layout yet.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(text.matches("x64.ret\n").count(), 1, "{text}");
        assert!(text.contains("x64.br_cond_8"), "{text}");
        assert!(text.contains("x64.add_rr_32"), "{text}");
        assert!(!text.contains('%'), "{text}");
    }

    #[test]
    fn a_critical_edge_is_split_before_the_allocator_ever_sees_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], join, &[args[1]]);
        Builder::new(&mut source, then).jump(join, &[args[0]]);
        let mut build = Builder::new(&mut source, join);
        let twice = build.binary(Opcode::Add, got, got, Flags::default());
        build.ret(&[twice]);

        // The else arm is critical: the entry block leaves two ways and the join is arrived at
        // two ways, and the arm carries a value. Without splitting it the allocator asserts,
        // because the move that gives the join its parameter would have to run at the end of a
        // block that also goes to the other arm.
        let mut out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        assert_eq!(crate::split::critical(&mut out), 1);
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // The block the split added is where the move went, and it is the whole of that block.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(out.block_count(), 4, "{text}");
        assert_eq!(text.matches("x64.ret\n").count(), 1, "{text}");
    }

    #[test]
    fn a_call_passes_what_the_convention_says_and_takes_back_what_it_says() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32, i32]);
        let sig =
            source.add_signature(Signature::new().with_params(&[i32, i32]).with_returns(&[i32]));
        let callee = names.intern("g");
        let call = Builder::new(&mut source, block).call(callee, sig, &[args[0], args[1]]);
        let got = source[call].first_result.expect("an integer comes back");
        Builder::new(&mut source, block).ret(&[got]);

        // `int f(int a, int b) { return g(a, b); }`. The arguments arrived where the call wants
        // them, so what the call reads is what arrived, and the whole of the convention is in the
        // constraints rather than in a move.
        let text = lower(&mut names, &source);
        assert!(text.contains("= x64.call %0($rdi), %1($rsi), @g"), "{text}");
        assert!(text.contains("x64.ret_val_32 %2($rax)"), "{text}");
        // What the call writes is the value that comes back and then every register the callee is
        // free to destroy, in both classes, which is the whole of what stops the allocator from
        // leaving something in one of them.
        assert!(text.contains("%2:gpr($rax), $rcx, $rdx, $r8, $r9, $r10, $r11, $xmm0,"), "{text}");
        assert!(text.contains("$xmm15 = x64.call"), "{text}");
    }

    #[test]
    fn what_the_frame_owes_a_call_comes_back_with_the_function() {
        let i32 = Type::int(32);
        let sig = |source: &mut Func| source.add_signature(Signature::new().with_params(&[i32]));

        let (mut names, mut source, block, args) = blank(&[i32]);
        let sig = sig(&mut source);
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[args[0]]);
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // Nothing on the stack, so nothing owed, but not a leaf either: a function that calls
        // owes the callee an aligned stack pointer and may not use the red zone.
        assert_eq!(out.stack.calls, Some(0));
        let layout = out.stack.layout(Layout::new(&SYSV, REGS));
        assert!(!layout.leaf);
        assert_eq!(layout.outgoing, 0);

        // The same call under the other convention owes thirty two bytes for the callee to spill
        // its register arguments into, which is a fact about the convention and not about the call.
        let out = func(&source, &mut names, &SELECTOR, &x86_64::WIN64, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(out.stack.calls, Some(32));

        // And a function that calls nothing is a leaf, which is what says it may use the red zone.
        let (mut names, mut source, block, args) = blank(&[i32]);
        Builder::new(&mut source, block).ret(&[args[0]]);
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(out.stack.calls, None);
        assert!(out.stack.layout(Layout::new(&SYSV, REGS)).leaf);
    }

    /// A Windows variadic prologue writes the argument registers the signature did not name into
    /// the shadow space the caller already reserved, which makes every argument one run of words up
    /// there and a `va_start` the address of the first of them. One `lea` and one store, and no
    /// counts, because a list that is a pointer has nowhere to put one and nothing that reads one.
    #[test]
    fn a_windows_variadic_function_homes_its_spare_registers_in_the_callers_area() {
        let mut names = Interner::new();
        let params = [Type::int(32), Type::PTR];
        let signature = Signature::new().with_params(&params).variadic();
        let mut source = Func::new(names.intern("f"), signature);
        let block = source.create_block();
        let values: Vec<Value> = params.iter().map(|&ty| source.append_param(block, ty)).collect();
        let mut build = Builder::new(&mut source, block);
        let args = build.func().push_values(&values[1..]);
        build.inst(InstData { args, ..InstData::new(Opcode::VaStart) }, &[]);
        build.ret(&[]);

        let out = func(&source, &mut names, &SELECTOR, &x86_64::WIN64, &Elsewhere::default())
            .expect("every instruction has a rule");
        let text = mir::print_func(&out.func, &names, &REGS);

        // Two named parameters, so the registers at the next two positions hold arguments nobody
        // named and both are written up into the caller's area. The displacement is empty here and
        // `finish` fills it in, the same way it does for a parameter the registers ran out before.
        assert!(text.contains("($r8) = x64.arg_val_64"), "{text}");
        assert!(text.contains("($r9) = x64.arg_val_64"), "{text}");
        assert_eq!(text.matches("x64.mov_mr_64").count(), 3, "two homed and one stored: {text}");
        assert!(!text.contains("x64.mov_ri_32"), "and no field holds a count: {text}");

        // All three waiting on the same fixup, and the last of them is the `lea` the list is given,
        // sixteen bytes up, which is where the two arguments the signature does name stopped.
        assert_eq!(out.stack.arguments.len(), 3);
        assert_eq!(out.stack.arguments[2].1, 16);
    }

    #[test]
    fn a_value_that_outlives_a_call_is_not_left_where_the_call_destroys_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let sig = source.add_signature(Signature::new().with_params(&[i32]).with_returns(&[i32]));
        let callee = names.intern("g");
        let call = Builder::new(&mut source, block).call(callee, sig, &[args[0]]);
        let got = source[call].first_result.expect("an integer comes back");
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, got, args[0], Flags::default());
        build.ret(&[sum]);

        // `int f(int a) { return g(a) + a; }`, which is the smallest program that asks the
        // question: `a` is read after the call and `rdi` is a register the call destroys.
        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let layout = lowered.stack.layout(Layout::new(&SYSV, REGS));
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &layout);
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // It went to a register the callee has to put back, and the prologue and epilogue are what
        // put it back, which is the whole bargain the two halves of a convention make.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(text.contains("$rbx"), "{text}");
        assert!(!text.contains('%'), "{text}");
        assert_eq!(text.matches("x64.call").count(), 1, "{text}");
    }

    #[test]
    fn a_call_with_more_arguments_than_registers_writes_the_rest_into_the_outgoing_area() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64]);
        let seven = vec![i64; 7];
        let sig = source.add_signature(Signature::new().with_params(&seven));
        let callee = names.intern("g");
        let passed = vec![args[0]; 7];
        Builder::new(&mut source, block).call(callee, sig, &passed);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("the seventh goes to memory");
        // The bytes the call needs are on the layout the frame is worked out from, so that the
        // frame reserves as many as the widest call in the function asked for.
        assert_eq!(lowered.stack.calls, Some(8));
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("x64.mov_mr_64 %0, [$rsp]\n"), "{text}");
    }

    #[test]
    fn a_call_this_cannot_make_is_reported_rather_than_made() {
        let (mut names, mut source, block, _) = blank(&[]);
        let returns = [Type::float(rucc_ir::Float::F80), Type::int(64)];
        let sig = source.add_signature(Signature::new().with_returns(&returns));
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[]);
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("a long double is on the x87");
        assert_eq!(failed.to_string(), "what this call gives back is on the x87 stack");
    }

    /// A `long double` on its own is a different answer, because on its own it comes back on the
    /// x87 stack rather than in a register, which is somewhere the call cannot be said to write.
    ///
    /// So the call gives back nothing at all and the value is taken off the stack by the `fstp`
    /// straight after it. That instruction has to be straight after it: the stack is one place and
    /// anything else that touched it before this ran would be looking at the value still on it.
    #[test]
    fn a_call_that_gives_back_a_long_double_takes_it_off_the_stack_at_once() {
        let (mut names, mut source, block, _) = blank(&[]);
        let long_double = Type::float(rucc_ir::Float::F80);
        let sig = source.add_signature(Signature::new().with_returns(&[long_double]));
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("the value comes back in st0");
        let text = mir::print_func(&lowered.func, &names, &REGS);
        let after: Vec<&str> =
            text.lines().skip_while(|line| !line.contains("x64.call")).skip(1).collect();
        assert_eq!(after[0].trim(), "%0:gpr = x64.lea_64 [$rsp]", "{text}");
        assert_eq!(after[1].trim(), "x64.fstp_t [%0]", "{text}");
        // And the slot it went into is the sixteen bytes the type takes, like every other one.
        assert_eq!(lowered.stack.locals.len(), 1, "{text}");
        assert_eq!(lowered.stack.locals[0].size, X87_BYTES);
    }

    #[test]
    fn a_call_through_an_address_goes_through_the_register_the_address_is_in() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[Type::PTR, i32]);
        let sig = source.add_signature(Signature::new().with_params(&[i32]).with_returns(&[i32]));
        let varargs = source.push_abis(&[]);
        let info = source.add_call(CallInfo { callee: None, signature: sig, varargs });
        let mut build = Builder::new(&mut source, block);
        let inst = InstData {
            args: build.func().push_values(&[args[0], args[1]]),
            extra: Extra::Call(info),
            ..InstData::new(Opcode::CallIndirect)
        };
        let called = build.inst(inst, &[i32]);
        let got = source[called].first_result.expect("an integer comes back");
        Builder::new(&mut source, block).ret(&[got]);

        // `int f(int (*g)(int), int a) { return g(a); }`. The first operand is the address and
        // the arguments are the ones behind it, and everything else about the call is what a call
        // to a name would have been.
        let text = lower(&mut names, &source);
        assert!(text.contains("= x64.call_reg %0, %1($rdi)"), "{text}");
        assert!(text.contains("x64.ret_val_32 %2($rax)"), "{text}");
        assert!(!text.contains("@g"), "a call through an address names nobody: {text}");
    }

    #[test]
    fn an_instruction_no_rule_covers_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let operands = build.func().push_values(&[args[0]]);
        build.inst(InstData { args: operands, ..InstData::new(Opcode::MetaBegin) }, &[]);

        // The mark that an object has come into being, which nothing writes an instruction for
        // yet: what it needs is a write over a range of the lifetime plane, and that is
        // `tamnd/rucc#856`. Nothing about it is a width or a register, so there is nothing for the
        // message to add beyond the name.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("no rule writes the beginning of a lifetime");
        assert_eq!(failed.to_string(), "no rule lowers a `meta_begin`");

        // It produces nothing, so there is no type in the message and nothing invents one, and the
        // instruction comes back so a caller can ask the function where it was.
        let inst = failed.inst().expect("the instruction it is about");
        assert_eq!(source[inst].opcode, Opcode::MetaBegin);
    }

    /// A barrier is written by name here, and what it is depends on the ordering and on nothing
    /// else. `crate::expand` is where the reasoning about this machine's memory model lives.
    #[test]
    fn a_barrier_is_one_instruction_at_the_strongest_ordering_and_none_below_it() {
        for order in MemOrder::all().filter(|&order| order != MemOrder::NotAtomic) {
            let (mut names, mut source, block, _) = blank(&[]);
            let mut build = Builder::new(&mut source, block);
            build
                .inst(InstData { extra: Extra::Order(order), ..InstData::new(Opcode::Fence) }, &[]);

            let text = lower(&mut names, &source);
            assert_eq!(text.contains("x64.mfence"), order == MemOrder::SeqCst, "{order:?}: {text}");
        }
    }

    /// A compare and exchange is written by name too, and at the width of the value rather than at
    /// the width of the address, which is the mistake worth pinning: everything here is a pointer
    /// and only the value says how many bytes the instruction touches.
    #[test]
    fn a_compare_and_exchange_is_one_instruction_at_the_width_of_the_value() {
        for bits in [8, 16, 32, 64] {
            let ty = Type::int(bits);
            let (mut names, mut source, block, args) = blank(&[Type::PTR, ty, ty]);
            let mut build = Builder::new(&mut source, block);
            let mem = build.func().add_mem(MemInfo {
                size: u64::from(bits / 8),
                align: bits / 8,
                order: MemOrder::SeqCst,
                ..plain()
            });
            let operands = build.func().push_values(&[args[0], args[1], args[2]]);
            build.inst(
                InstData {
                    args: operands,
                    extra: Extra::Mem(mem),
                    ..InstData::new(Opcode::Cmpxchg)
                },
                &[ty, Type::I1],
            );

            // Two values out of one instruction, the first of them in the register the machine
            // reads the expected value out of, the second free for the allocator to place. The
            // address is the memory operand and neither of the two values is.
            let text = lower(&mut names, &source);
            let written = format!("%3:gpr($rax), %4:gpr = x64.cmpxchg_{bits} %1($rax), %2, [%0]");
            assert!(text.contains(&written), "{bits}: {text}");
        }
    }

    #[test]
    fn more_values_back_than_the_convention_has_registers_for_is_reported() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64, i64, i64]);
        let mut build = Builder::new(&mut source, block);
        build.ret(&[args[0], args[1], args[2]]);

        // Two integers come back in `rax` and `rdx` and a third has nowhere to go, which is not a
        // gap in the rules but the convention saying no. The front end classifies before it gets
        // here, so this is the shape that would mean the classification went wrong.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("only two come back");
        assert_eq!(
            failed.to_string(),
            "what this function gives back takes more registers than this convention has for it"
        );

        let inst = failed.inst().expect("the instruction it is about");
        assert_eq!(source[inst].opcode, Opcode::Return);
    }

    /// A refusal about a signature has no instruction, which is what makes it the one arm apart.
    ///
    /// Everything else is about something written somewhere in the body and hands it back so a
    /// caller can ask the function where it came from. A parameter arrives before the first
    /// instruction runs, so there is nothing in the body to point at and the message is about
    /// the function.
    #[test]
    fn a_refusal_about_a_parameter_has_no_instruction_to_point_at() {
        let missing = Unsupported::Argument { index: 0, missing: Missing::OnX87 };
        assert_eq!(missing.inst(), None);
    }

    /// An `alloca` of a fixed size, which is what every local whose address is taken becomes.
    fn slot(source: &mut Func, block: Block, size: u64, align: u32) -> Value {
        let info = MemInfo { size, align, ..plain() };
        let mut build = Builder::new(source, block);
        let mem = build.func().add_mem(info);
        build.value(InstData { extra: Extra::Mem(mem), ..InstData::new(Opcode::Alloca) }, Type::PTR)
    }

    #[test]
    fn a_local_is_memory_in_the_frame_and_one_instruction_that_says_where() {
        let (mut names, mut source, block, _) = blank(&[]);
        let slot = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, slot, plain(), Flags::default());
        let loaded = build.load(Type::int(32), slot, plain(), Flags::default());
        build.ret(&[loaded]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // Four bytes on the list the frame is laid out from, and the one instruction that reads
        // where they went. Its displacement is nothing here because there is no frame yet, and
        // which instruction is waiting for which local is what `finish` is handed.
        assert_eq!(lowered.stack.locals, vec![Local { size: 4, align: 4 }]);
        assert_eq!(lowered.stack.addresses.len(), 1);
        assert_eq!(lowered.stack.addresses[0].1, 0);
        assert_eq!(
            mir::print_func(&lowered.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.lea_64 [$rsp]\n    \
             %1:gpr = x64.mov_ri_32 9\n    x64.mov_mr_32 %1, [%0]\n    \
             %2:gpr = x64.mov_rm_32 [%0]\n    x64.ret_val_32 %2($rax)\n}\n"
        );
    }

    #[test]
    fn a_local_the_program_declared_says_which_declaration_it_is_and_the_rest_say_nothing() {
        let (mut names, mut source, block, _) = blank(&[]);
        let scratch = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let mem = build.func().add_mem(MemInfo { size: 8, align: 8, ..plain() });
        let declared = build
            .value(InstData { extra: Extra::Mem(mem), ..InstData::new(Opcode::Alloca) }, Type::PTR);
        build.func().declare_mem(mem, 41);
        build.store(scratch, declared, MemInfo { size: 8, align: 8, ..plain() }, Flags::default());
        build.ret(&[]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // Two locals and one declaration, held against the order the allocas were lowered in,
        // which is the only name a local has by the time the frame places it. The scratch one was
        // reached first and is local zero, so the declared one is local one.
        assert_eq!(lowered.stack.locals.len(), 2);
        assert_eq!(lowered.stack.declared, vec![(1, 41)]);
    }

    /// A local the program kept in a value comes out saying which register holds it.
    ///
    /// The other half of the local above, which had a slot. This one has none, so what carries the
    /// declaration is the register the instruction computing it writes into.
    #[test]
    fn a_local_the_program_kept_in_a_value_says_which_register_holds_it() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        let ten = build.iconst(Type::int(32), 10);
        let sum = build.binary(Opcode::Add, nine, ten, Flags::default());
        build.func().declare_value(sum, 41);
        build.ret(&[sum]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // One pair and not three. The constants are values the program never declared, and a
        // register holding one of those is nobody's. The register is the one the addition writes,
        // which the listing under it is what pins down.
        assert_eq!(lowered.func.named, vec![(41, mir::Reg::virtual_reg(1))]);
        assert_eq!(
            mir::print_func(&lowered.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 9\n    \
             %1:gpr(reuse 1) = x64.add_ri_32 %0, 10\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    /// A local held in a constant two blocks want is two registers and both of them are it.
    ///
    /// Why the declaration is written down as each register is handed out rather than once at the
    /// end over the map from values to registers. That map remembers the last register a value was
    /// written into, and a constant is written again in every block that wants one, so a local held
    /// in one would come out findable in the last block of the function and nowhere else.
    #[test]
    fn a_local_held_in_a_constant_two_blocks_want_is_named_in_both_of_them() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);

        let mut build = Builder::new(&mut source, entry);
        let seven = build.iconst(i32, 7);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.func().declare_value(seven, 41);
        build.br_if(cond, then, &[], other, &[]);
        Builder::new(&mut source, then).jump(join, &[seven]);
        Builder::new(&mut source, other).jump(join, &[seven]);
        Builder::new(&mut source, join).ret(&[got]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        let held = &lowered.func.named;
        assert_eq!(held.len(), 2, "one register per block that wanted the seven: {held:?}");
        assert!(held.iter().all(|&(decl, _)| decl == 41), "{held:?}");
        assert_ne!(held[0].1, held[1].1, "the same register in two blocks: {held:?}");
    }

    /// A parameter the program declared comes out named too, in the register it arrived in.
    ///
    /// The case the walk over the map at the end is for. A parameter is put in a register the
    /// convention chose rather than in a fresh one, so nothing asks the mint for it and the pair
    /// would otherwise never be written down.
    #[test]
    fn a_parameter_the_program_declared_says_which_register_it_arrived_in() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let mut build = Builder::new(&mut source, block);
        build.func().declare_value(args[0], 41);
        build.ret(&[args[0]]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        let held = &lowered.func.named;
        assert_eq!(held.len(), 1, "one pair for the one parameter: {held:?}");
        assert_eq!(held[0].0, 41);
    }

    /// A function with nothing declared in it says nothing, which is every function compiled
    /// without debugging information asked for.
    #[test]
    fn a_function_the_front_end_named_nothing_in_names_no_registers() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.ret(&[nine]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert!(lowered.func.named.is_empty(), "{:?}", lowered.func.named);
    }

    #[test]
    fn the_frame_is_what_fills_the_address_of_a_local_in() {
        let (mut names, mut source, block, _) = blank(&[]);
        let slot = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, slot, plain(), Flags::default());
        let loaded = build.load(Type::int(32), slot, plain(), Flags::default());
        build.ret(&[loaded]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // `int f(void) { int x; x = 9; return x; }` with the address of `x` taken, end to end.
        // A leaf small enough to live in the red zone takes no frame at all, so the stack pointer
        // never moves and the four bytes are below it, which is what the negative offset is. The
        // instruction the lowering left with nothing in its displacement now has the answer in it.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(text.contains("$rax = x64.lea_64 [$rsp - 8]"), "{text}");
        assert!(!text.contains("x64.sub_ri_64"), "{text}");
        assert_eq!(frame.size(), 0);
        assert_eq!(frame.local(0), Some(-8));
    }

    /// An `alloca` whose size is an operand, which is a variable length array.
    fn growing(source: &mut Func, block: Block, size: Value, align: u32) -> Value {
        let info = MemInfo { size: 0, align, ..plain() };
        let mut build = Builder::new(source, block);
        let mem = build.func().add_mem(info);
        let args = build.func().push_values(&[size]);
        build.value(
            InstData { args, extra: Extra::Mem(mem), ..InstData::new(Opcode::Alloca) },
            Type::PTR,
        )
    }

    #[test]
    fn a_stack_slot_whose_size_is_not_known_until_it_runs_takes_the_bytes_off_the_stack_pointer() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let slot = growing(&mut source, block, args[0], 16);
        Builder::new(&mut source, block).ret(&[slot]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // The bytes come off the stack pointer where the declaration stands and the address is
        // where the stack pointer then is, which is one subtraction and one `lea` rather than a
        // slot the frame laid out. Nothing is on the list of locals, because there is nothing
        // about this the frame could place.
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("$rsp = x64.sub_rr_64 $rsp, %0"), "{text}");
        assert!(text.contains("x64.lea_64 [$rsp]"), "{text}");
        assert!(lowered.stack.locals.is_empty(), "{text}");
        assert_eq!(lowered.stack.dynamic.len(), 1);
        assert!(lowered.stack.grown_at.is_some());
    }

    #[test]
    fn a_growing_slot_wanting_more_alignment_than_the_stack_pointer_has_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let slot = growing(&mut source, block, args[0], 32);
        Builder::new(&mut source, block).ret(&[slot]);

        // Thirty two is more than a call leaves the stack pointer on, so giving it what it asked
        // for means masking the stack pointer after moving it, and after that no constant reaches
        // the rest of the frame from the frame pointer either. A second pointer held for the
        // purpose is what fixes it and there is not one yet.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("nothing realigns a frame that grows");
        assert_eq!(
            failed.to_string(),
            "this local wants more alignment than the stack pointer is left on, which needs a \
             base register nothing here keeps"
        );
    }

    #[test]
    fn a_frame_that_grows_reaches_its_own_locals_through_the_frame_pointer() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let fixed = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, fixed, plain(), Flags::default());
        let grown = growing(&mut source, block, args[0], 16);
        Builder::new(&mut source, block).ret(&[grown]);

        let lowered = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // The stack pointer moves in the middle of the function, so the four bytes of the fixed
        // local are not a constant away from it any more and the frame pointer is what reaches
        // them. The frame keeps one whatever the flags asked for, takes its bytes rather than
        // living in the red zone, and the address of the growing slot is off the stack pointer as
        // it stands after the subtraction rather than off anything the prologue left.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(frame.grows());
        assert!(frame.frame_pointer());
        assert!(frame.size() > 0, "{text}");
        assert!(text.contains("x64.lea_64 [$rbp"), "{text}");
        assert!(text.contains("$rsp = x64.sub_rr_64 $rsp"), "{text}");
        assert!(text.contains("x64.lea_64 [$rsp]"), "{text}");
    }

    #[test]
    fn an_address_is_read_written_and_added_to_like_the_integer_it_is() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR, Type::int(64)]);
        let mut build = Builder::new(&mut source, block);
        let stepped = build.func().push_values(&[args[0], args[1]]);
        let next =
            build.value(InstData { args: stepped, ..InstData::new(Opcode::PtrAdd) }, Type::PTR);
        let loaded = build.load(Type::int(32), next, plain(), Flags::default());
        build.ret(&[loaded]);

        // `int f(int *p, long i) { return *(int *)((char *)p + i); }`. Nothing about this is new
        // in the rule set, which is the point: the two addresses arrive in registers because an
        // address is an integer as wide as one, and the arithmetic on them is the add it always
        // was, so every rule written about an add reaches it.
        //
        // The add stays its own instruction here rather than folding into the address the load
        // reads from. Two registers with no scale on either is the one addressing mode the rules
        // have no load through, because the folds that exist are the displacement one and the
        // scaled ones, and this is neither. `crate::fold` is what puts the two together, after
        // selection, and this is the pair it is handed.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n    \
             %3:gpr = x64.mov_rm_32 [%2]\n    x64.ret_val_32 %3($rax)\n}\n"
        );
    }

    /// The address of a file scope name, which is what every use of a global and every string
    /// literal starts from.
    fn address_of(source: &mut Func, block: Block, names: &mut Interner, name: &str) -> Value {
        let symbol = names.intern(name);
        let mut build = Builder::new(source, block);
        build.value(
            InstData { extra: Extra::Symbol(symbol), ..InstData::new(Opcode::GlobalAddr) },
            Type::PTR,
        )
    }

    #[test]
    fn the_address_of_a_name_is_one_instruction_carrying_the_name() {
        let (mut names, mut source, block, _) = blank(&[]);
        let counter = address_of(&mut source, block, &mut names, "counter");
        let mut build = Builder::new(&mut source, block);
        let loaded = build.load(Type::int(32), counter, plain(), Flags::default());
        build.ret(&[loaded]);

        // `extern int counter; int f(void) { return counter; }`. The address is an addressing mode
        // that names no register and carries the symbol, which is what the assembler writes
        // relative to `%rip` and what the object writer leaves a relocation for.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.lea_64 [@counter]\n    \
             %1:gpr = x64.mov_rm_32 [%0]\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    #[test]
    fn the_address_of_a_name_outside_the_file_is_read_out_of_the_offset_table() {
        let (mut names, mut source, block, _) = blank(&[]);
        let away = address_of(&mut source, block, &mut names, "away");
        Builder::new(&mut source, block).ret(&[away]);
        let elsewhere: Elsewhere = [names.intern("away")].into_iter().collect();

        // `extern void away(void); void *f(void) { return away; }`. A load and not an address
        // computation, because the distance from here to a name a shared library may be the one
        // that defines is not a number any link can work out, and the slot the linker fills in is
        // in this program and so is a distance it has.
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &elsewhere)
            .expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [got @away]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_address_of_a_thread_local_is_an_offset_out_of_the_table_plus_where_this_thread_starts() {
        let (mut names, mut source, block, _) = blank(&[]);
        let own = address_of(&mut source, block, &mut names, "own");
        Builder::new(&mut source, block).ret(&[own]);
        let elsewhere = Elsewhere::default().with_threads([names.intern("own")]);

        // `extern _Thread_local int own; void *f(void) { return &own; }`. Three instructions where
        // the two cases above are one, because there is no address to load or to work out: the
        // slot holds how far into a thread's block the variable sits, `%fs:0` is where this
        // thread's block starts, and the sum of the two is this thread's copy.
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &elsewhere)
            .expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [thread @own]\n    \
             %1:gpr = x64.mov_rm_64 [fs:0]\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n    \
             x64.ret_val_64 %2($rax)\n}\n"
        );
    }

    /// The same load with nothing added to it, which is the whole of `__builtin_thread_pointer`.
    #[test]
    fn the_start_of_this_thread_s_own_storage_is_the_one_load_and_no_arithmetic() {
        let (mut names, mut source, block, _) = blank(&[]);
        let here =
            Builder::new(&mut source, block).value(InstData::new(Opcode::ThreadPointer), Type::PTR);
        Builder::new(&mut source, block).ret(&[here]);

        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [fs:0]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// One `asm` statement, with its template and its constraint list written as a program does.
    fn assembly(
        source: &mut Func,
        block: Block,
        names: &mut Interner,
        template: &str,
        constraints: &str,
        args: &[Value],
        results: &[Type],
    ) -> Inst {
        clobbering(source, block, names, template, constraints, "memory", args, results)
    }

    /// The same with a clobber list of its own, for the statements that are about one.
    #[allow(clippy::too_many_arguments)]
    fn clobbering(
        source: &mut Func,
        block: Block,
        names: &mut Interner,
        template: &str,
        constraints: &str,
        clobbers: &str,
        args: &[Value],
        results: &[Type],
    ) -> Inst {
        let info = AsmInfo {
            template: names.intern(template),
            constraints: names.intern(constraints),
            clobbers: names.intern(clobbers),
            targets: rucc_ir::BlockCallList::EMPTY,
        };
        Builder::new(source, block).inline_asm(info, args, results, Flags::VOLATILE)
    }

    /// What a program asking the processor what it can do writes, which is the instruction whose
    /// every operand is a register its text does not name.
    #[test]
    fn a_template_whose_registers_are_named_by_the_constraints_places_them_from_the_letters() {
        let u32 = Type::int(32);
        let (mut names, mut source, block, _) = blank(&[]);
        let zero = Builder::new(&mut source, block).iconst(u32, 0);
        let out = clobbering(
            &mut source,
            block,
            &mut names,
            "cpuid",
            "=a,a",
            "ebx,ecx,edx",
            &[zero],
            &[u32],
        );
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("cpuid" : "=a" (n) : "a" (0) : "ebx", "ecx", "edx")`, which is the first thing
        // every program that has a faster path on some machines writes. Four registers written and
        // two read, none of them in the template, all of them out of the description, and the two
        // that the letters named are the statement's own. The subleaf is a zero because the
        // instruction reads `ecx` and the program said nothing about what is in it. The three
        // clobbers are gone because `cpuid` writes those three anyway, and saying it twice is one
        // register with two definitions.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    \
             %1:gpr = x64.mov_ri_64 0\n    \
             %2:gpr($rax), %3:gpr($rbx), %4:gpr($rcx), %5:gpr($rdx) = x64.cpuid %0($rax), \
             %1($rcx)\n    x64.ret_val_32 %2($rax)\n}\n"
        );
    }

    /// An operand the program pinned, by declaring the object it comes from `register long x asm
    /// ("r12")`. The letter on its own leaves the allocator to pick, and a template that reads the
    /// register by name needs the two to be the same register, so the brace is what ties them
    /// together. That is the one use of a local register variable the GNU manual calls reliable,
    /// and it is what tcc's `tests/tcctest.c` counts on.
    #[test]
    fn an_operand_the_program_pinned_is_placed_in_the_register_it_named() {
        let u64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let out =
            assembly(&mut source, block, &mut names, "mov $0x4542, %r12", "=r{r12}", &[], &[u64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // The template is one instruction the table already has, so it lowers to that instruction
        // rather than to text nobody read, and the register it names is the statement's own output
        // because the brace put the output there. Without the brace the letter would have let the
        // allocator pick, the two `%r12` would have been different registers, and the program would
        // have come back with whatever was in the one it picked.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($r12) = x64.mov_ri_64 17730\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// A clobber the instruction does not write itself, which is the case the list is there for.
    /// It goes on as a definition of the register, in among the other definitions, because that is
    /// the whole of how a machine function says a register is not worth anything after this.
    #[test]
    fn a_clobber_the_instruction_does_not_write_itself_is_a_definition_of_that_register() {
        let (mut names, mut source, block, _) = blank(&[]);
        clobbering(&mut source, block, &mut names, "pause", "", "rsi,cc,memory", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n    $rsi = x64.pause\n}\n");
    }

    /// A clobber naming something this has no register for. Refused rather than dropped, since the
    /// list is the program saying which registers it may not leave anything in, and an entry
    /// nobody read is a register something may still be left in.
    #[test]
    fn a_clobber_this_has_no_register_for_is_refused() {
        let (mut names, mut source, block, _) = blank(&[]);
        clobbering(&mut source, block, &mut names, "pause", "", "zmm0", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("there is no such register here");
        assert_eq!(
            failed.to_string(),
            "this `asm` says it destroys a register this has no name for"
        );
    }

    #[test]
    fn an_asm_with_an_empty_template_and_no_operands_is_no_instructions() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        // `asm volatile ("" : : : "memory")`, which is a barrier and nothing else. The barrier was
        // spent on the optimizer, which has finished by now, so what is left is nothing.
        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n}\n");
    }

    #[test]
    fn an_output_an_input_is_tied_to_is_the_register_that_input_arrived_in() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let out = assembly(&mut source, block, &mut names, "", "=r,0", &args, &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "=r" (x) : "0" (x))`, which is how a program stops the optimizer following a
        // value without changing it. The two share a place and the template writes nothing over
        // it, so the value comes back out of the register it went in.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn an_output_written_plus_is_the_same_rename() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let out = assembly(&mut source, block, &mut names, "", "+r", &args, &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "+r" (x))`, which says the same thing in one operand instead of two.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn an_output_nothing_is_tied_to_is_a_zero() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "", "=r", &[], &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "=r" (y))`, whose answer is whatever the assembly left in the register, and
        // an empty template leaves nothing. A definite value rather than a register nothing wrote,
        // because the allocator is owed a definition before the use however little the program is.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_template_that_is_one_instruction_becomes_that_instruction() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "pause", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        // `asm volatile ("pause")`, which is what every spin lock in every allocator writes. One
        // instruction, no operands, and nothing between the template and the machine but the table
        // that already says what a `pause` is.
        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n    x64.pause\n}\n");
    }

    #[test]
    fn a_template_that_reads_a_segment_becomes_the_load_it_already_was() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "movq %%fs:0, %0", "=r", &[], &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("movq %%fs:0, %0" : "=r" (tid))`, which is how a program finds the block its own
        // thread owns. The same instruction `crate::lower` already writes for a thread-local
        // variable, reached this time because a program wrote it out by hand.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [fs:0]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// A template this cannot read is kept as its text, which is what gcc does with every template.
    /// Whether the text is an instruction is the assembler's question, asked when the unit is
    /// assembled from its listing.
    #[test]
    fn a_template_naming_an_instruction_this_machine_has_not_got_is_kept_as_text() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "hcf", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        let printed = lower(&mut names, &source);
        assert!(printed.contains("x64.template"), "{printed}");
        assert!(printed.contains("@hcf"), "{printed}");
    }

    /// A template kept as text with an operand in a register reads the operand, and its text holds
    /// a hole naming that operand of the instruction, which the writer fills with the register the
    /// allocator chose. The input is the instruction's only use, behind every register a call may
    /// write.
    #[test]
    fn a_template_kept_as_text_reads_an_operand_in_a_register_through_a_hole() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        assembly(&mut source, block, &mut names, "hcf %0", "r", &[args[0]], &[]);
        Builder::new(&mut source, block).ret(&[]);

        let printed = lower(&mut names, &source);
        let line = printed.lines().find(|line| line.contains("x64.template")).unwrap_or_default();
        // Twenty five registers are written ahead of it, so the operand read is the twenty sixth,
        // spelled at the width of an `int`.
        assert!(line.contains("x64.template %0, @hcf \u{1}r25k\u{2}"), "{printed}");
        assert!(line.contains("early $rax"), "{printed}");
    }

    /// A template kept as text with more outputs than the convention keeps registers across a call
    /// gets back as many of the registers a call may write as it needs, from the end of the order,
    /// and keeps the rest. Six outputs against five preserved registers is one handed back, which is
    /// `r11`. The shape is `sodium_sub` in libsodium, whose `sbbq` into memory the reader has no
    /// form for, and before this the allocator ran out of registers on it.
    #[test]
    fn a_template_kept_as_text_with_more_outputs_than_are_kept_gets_registers_back() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let outputs = [i64; 6];
        let asm = assembly(
            &mut source,
            block,
            &mut names,
            "hcf %0, %1, %2, %3, %4, %5",
            "=&r,=&r,=&r,=&r,=&r,=&r",
            &[],
            &outputs,
        );
        let produced: Vec<Value> = source[asm].results().collect();
        Builder::new(&mut source, block).ret(&produced[..1]);

        let printed = lower(&mut names, &source);
        let line = printed.lines().find(|line| line.contains("x64.template")).unwrap_or_default();
        assert!(line.contains("early $r10"), "{printed}");
        assert!(!line.contains("early $r11"), "{printed}");
    }

    /// A register the template named is placed as itself, fixed to the register the program wrote
    /// down. A register a constraint letter names is a different thing and is placed too, which the
    /// test above is about: there the statement said which of its own operands is in the register,
    /// and a name in the middle of a template says the register and nothing about any operand.
    #[test]
    fn a_template_naming_a_register_gets_that_register() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "movq %%rax, %0", "=r", &[], &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("movq %%rax, %0" : "=r" (x))`, which is a program reading whatever is in `%rax`.
        // The source is the register itself and the destination is one the allocator picks.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rr_64 $rax($rax)\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// The half of the same thing every register saving template needs. micropython writes the
    /// callee-saved registers into a buffer one `movq %%r12, 48(%%rdi)` at a time, and both halves
    /// of that line are a register the template named: the one being stored and the one the address
    /// is counted from.
    #[test]
    fn a_template_counting_an_address_from_a_register_it_named_gets_that_register() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "movq %%r12, 48(%%rdi)", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    x64.mov_mr_64 $r12($r12), [$rdi + 48]\n}\n"
        );
    }

    /// A local kept in a named register, which is the same register named as itself and reached
    /// from the other side. micropython's collector writes six of these and reads them with
    /// ordinary C rather than with a template.
    #[test]
    fn a_local_kept_in_a_named_register_is_one_move_out_of_it() {
        let (mut names, mut source, block, _) = blank(&[]);
        let held = names.intern("rbx");
        let value = Builder::new(&mut source, block).value(
            InstData { extra: Extra::Symbol(held), ..InstData::new(Opcode::RegisterValue) },
            Type::int(64),
        );
        Builder::new(&mut source, block).ret(&[value]);

        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rr_64 $rbx($rbx)\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// The sigil gcc allows in front of the name is syntax and comes off, and a name that is not
    /// a register of this machine is refused in words that say which name it was.
    #[test]
    fn a_register_name_is_read_with_or_without_its_sigil_and_refused_when_there_is_no_such_one() {
        for written in ["%r12", "r12"] {
            let (mut names, mut source, block, _) = blank(&[]);
            let held = names.intern(written);
            let value = Builder::new(&mut source, block).value(
                InstData { extra: Extra::Symbol(held), ..InstData::new(Opcode::RegisterValue) },
                Type::int(64),
            );
            Builder::new(&mut source, block).ret(&[value]);
            assert!(lower(&mut names, &source).contains("$r12($r12)"), "{written} is not read");
        }

        let (mut names, mut source, block, _) = blank(&[]);
        let held = names.intern("nowhere");
        let value = Builder::new(&mut source, block).value(
            InstData { extra: Extra::Symbol(held), ..InstData::new(Opcode::RegisterValue) },
            Type::int(64),
        );
        Builder::new(&mut source, block).ret(&[value]);

        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("there is no such register");
        assert_eq!(
            failed.to_string(),
            "this object is kept in `nowhere`, which is not a register this machine has"
        );
    }

    #[test]
    fn a_constraint_list_that_does_not_describe_the_operands_is_refused() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        assembly(&mut source, block, &mut names, "", "=r", &args, &[]);
        Builder::new(&mut source, block).ret(&[]);

        // An output with no result to be, which is what the front end never writes and what a
        // hand written module can. Refused rather than placed by a guess.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("the list and the instruction disagree");
        assert_eq!(failed.to_string(), "this `asm` has an operand this cannot place");
    }

    /// A cast between a pointer and an integer, at whatever width the result is asked for.
    fn cast(source: &mut Func, block: Block, opcode: Opcode, from: Value, to: Type) -> Value {
        let mut build = Builder::new(source, block);
        let args = build.func().push_values(&[from]);
        build.value(InstData { args, ..InstData::new(opcode) }, to)
    }

    #[test]
    fn a_cast_between_a_pointer_and_an_integer_as_wide_is_no_instruction_at_all() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let number = cast(&mut source, block, Opcode::PtrToInt, args[0], Type::int(64));
        Builder::new(&mut source, block).ret(&[number]);

        // `long f(void *p) { return (long)p; }`. An address on this machine is an integer as wide
        // as the machine addresses, so the cast changes what the type system calls the value and
        // changes nothing about the value, and the register holding it is the one that held it.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_null_pointer_is_a_constant_that_reaches_a_register_before_anything_reads_it() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let zero = build.iconst(Type::int(64), 0);
        let null = cast(&mut source, block, Opcode::IntToPtr, zero, Type::PTR);
        Builder::new(&mut source, block).ret(&[null]);

        // `void *f(void) { return 0; }`. The cast is nothing, and reading its operand is what
        // writes the zero down: a constant is materialized where it is wanted rather than where
        // the IR defined it, and without the read there would be no instruction at all.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_64 0\n    x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_five_linkages_the_ir_has_narrow_to_the_three_an_object_file_can_say() {
        let readings = [
            (Linkage::External, mir::Binding::Global),
            (Linkage::Common, mir::Binding::Global),
            (Linkage::Internal, mir::Binding::Local),
            (Linkage::Weak, mir::Binding::Weak),
            (Linkage::LinkOnce, mir::Binding::Weak),
        ];
        for (linkage, wanted) in readings {
            let (mut names, mut source, block, _) = blank(&[]);
            source.linkage = linkage;
            Builder::new(&mut source, block).ret(&[]);
            let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
                .expect("a return");
            // The narrowing is done here rather than where the object is written, because a
            // machine function is all the assembler and the writer are ever handed.
            assert_eq!(out.func.binding, wanted, "{linkage:?}");
        }
    }

    /// The visibility makes the same trip and is not narrowed on the way, because ELF says all
    /// three of them.
    ///
    /// Here for the reason the linkage above is here. A machine function is the whole of what the
    /// assembler and the object writer are handed, so a fact about the symbol that does not get
    /// onto one is a fact that is gone by the time anything could write it down, and the way that
    /// shows up is a shared library exporting the wrong set of names with nothing said anywhere.
    #[test]
    fn the_visibility_survives_the_trip_from_the_ir_to_a_machine_function() {
        let readings = [
            (Visibility::Default, mir::Visibility::Default),
            (Visibility::Hidden, mir::Visibility::Hidden),
            (Visibility::Protected, mir::Visibility::Protected),
        ];
        for (visibility, wanted) in readings {
            let (mut names, mut source, block, _) = blank(&[]);
            source.visibility = visibility;
            Builder::new(&mut source, block).ret(&[]);
            let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
                .expect("a return");
            assert_eq!(out.func.visibility, wanted, "{visibility:?}");
        }
    }

    #[test]
    fn a_cast_between_a_pointer_and_a_narrower_integer_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let number = cast(&mut source, block, Opcode::PtrToInt, args[0], Type::int(32));
        Builder::new(&mut source, block).ret(&[number]);

        // The front end never writes one: it casts at the address width and truncates or extends
        // around it, so both of those are the rules they always were. IR from somewhere else that
        // does write one is refused rather than compiled to a move that keeps the high half.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("no rule narrows an address");
        assert_eq!(failed.to_string(), "no rule lowers a `ptrtoint` producing a `i32`");
    }

    /// The type this machine has no register for.
    fn long_double() -> Type {
        Type::float(rucc_ir::Float::F80)
    }

    #[test]
    fn a_double_widened_and_narrowed_again_goes_out_through_the_frame_and_back() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let back = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(double d) { long double x = d; return x; }`. The x87 reads memory and nothing
        // else, so the value is written to the crossing slot, loaded at the format that widens it
        // and put in the slot the eighty bit value lives in. Coming back is the same three the
        // other way. Both slots are addressed by a `lea` with nothing in it yet, which is what
        // every address in a frame looks like here until `finish` has the numbers.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    \
             %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:gpr = x64.lea_64 [$rsp]\n    \
             %2:gpr = x64.lea_64 [$rsp]\n    \
             x64.movsd_mr %0, [%1]\n    \
             x64.fld_l [%1]\n    \
             x64.fstp_t [%2]\n    \
             %3:gpr = x64.lea_64 [$rsp]\n    \
             %4:gpr = x64.lea_64 [$rsp]\n    \
             x64.fld_t [%3]\n    \
             x64.fstp_l [%4]\n    \
             %5:xmm = x64.movsd_rm [%4]\n    \
             x64.ret_val_f64 %5($xmm0)\n}\n"
        );
    }

    #[test]
    fn a_long_double_has_sixteen_bytes_of_its_own_and_keeps_them() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let once = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        let twice = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::FAdd, once, twice, Flags::default());
        build.ret(&[sum]);

        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");

        // Two slots and not four: sixteen bytes for the one eighty bit value, which is what the
        // psABI says one takes and is aligned to, and eight for the crossing, which every group
        // in the function shares because nothing is ever left in it. The value's slot is its own
        // for the whole function, so reading it twice reads the same sixteen bytes.
        assert_eq!(
            out.stack.locals,
            vec![Local { size: 8, align: 8 }, Local { size: 16, align: 16 }]
        );
    }

    #[test]
    fn an_integer_becomes_a_long_double_by_being_loaded_as_one() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let wide = cast(&mut source, block, Opcode::SIToFP, args[0], long_double());
        let back =
            cast(&mut source, block, Opcode::FPTrunc, wide, Type::float(rucc_ir::Float::F64));
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(long n) { long double x = n; return x; }`. `fild` is the same push at another
        // format, so the conversion is the load and there is no instruction that converts.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_mr_64 %0, [%1]"), "{text}");
        assert!(text.contains("x64.fild_ll [%1]"), "{text}");
    }

    #[test]
    fn a_long_double_becoming_an_integer_cuts_towards_zero_with_the_control_word() {
        let (mut names, mut source, block, args) = blank(&[Type::float(rucc_ir::Float::F64)]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let whole = cast(&mut source, block, Opcode::FPToSI, wide, Type::int(32));
        Builder::new(&mut source, block).ret(&[whole]);

        // The one conversion here with no single instruction behind it. C cuts towards zero and
        // the unit rounds the way its control word says, so the word is saved, ORed with the two
        // bits that mean truncate, loaded, used and put back. Nine instructions for what `fisttp`
        // does in one, and `spec/10-backend.md` section 10.8 says why that one is not used.
        let text = lower(&mut names, &source);
        let group: Vec<&str> = text
            .lines()
            .map(str::trim)
            .filter(|line| line.starts_with("x64.f") || line.contains("_16"))
            .collect();
        assert_eq!(
            group,
            [
                "x64.fld_l [%1]",
                "x64.fstp_t [%2]",
                "x64.fnstcw [%5]",
                "%6:gpr = x64.mov_rm_16 [%5]",
                "%7:gpr(reuse 1) = x64.or_ri_16 %6, 3072",
                "x64.mov_mr_16 %7, [%5 + 2]",
                "x64.fldcw [%5 + 2]",
                "x64.fld_t [%3]",
                "x64.fistp_l [%4]",
                "x64.fldcw [%5]",
            ],
            "{text}"
        );
    }

    #[test]
    fn a_long_double_is_read_and_written_as_the_bits_it_already_is() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR, Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let value = build.load(long_double(), args[0], plain(), Flags::default());
        build.store(value, args[1], plain(), Flags::default());
        build.ret(&[]);

        // `void f(long double *a, long double *b) { *b = *a; }`. A copy is a push and a pop at the
        // format the value is already in, which neither converts nor looks: a signalling NaN stays
        // one and nothing is raised, which is the whole of what makes it a copy.
        let text = lower(&mut names, &source);
        let group: Vec<&str> =
            text.lines().map(str::trim).filter(|line| line.starts_with("x64.f")).collect();
        assert_eq!(
            group,
            ["x64.fld_t [%0]", "x64.fstp_t [%2]", "x64.fld_t [%3]", "x64.fstp_t [%1]"],
            "{text}"
        );
    }

    /// Two `long double` values, from two `double` parameters, and the instructions that made
    /// them, which every test below this one throws away.
    fn two_long_doubles(source: &mut Func, block: Block, args: &[Value]) -> (Value, Value) {
        let left = cast(source, block, Opcode::FPExt, args[0], long_double());
        let right = cast(source, block, Opcode::FPExt, args[1], long_double());
        (left, right)
    }

    /// The x87 instructions of a function, in order, with everything else dropped.
    fn stack_only(text: &str) -> Vec<&str> {
        text.lines().map(str::trim).filter(|line| line.contains("x64.f")).collect()
    }

    /// The two frame slots the last two addresses of a function were taken of, which in a
    /// comparison are the two operands in the order they go on the stack.
    fn pushed(out: &Lowered) -> Vec<usize> {
        let taken: Vec<usize> = out.stack.addresses.iter().map(|&(_, local)| local).collect();
        taken[taken.len() - 2..].to_vec()
    }

    #[test]
    fn adding_two_long_doubles_pushes_both_and_leaves_the_answer_in_a_slot() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let sum =
            Builder::new(&mut source, block).binary(Opcode::FAdd, left, right, Flags::default());
        let back = cast(&mut source, block, Opcode::FPTrunc, sum, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(double a, double b) { return (long double) a + (long double) b; }`. The last
        // four lines are the add: both operands pushed, the instruction that names neither of
        // them because they are the top two of a stack, and the answer taken off into its slot.
        let text = lower(&mut names, &source);
        assert_eq!(
            stack_only(&text),
            [
                "x64.fld_l [%2]",
                "x64.fstp_t [%3]",
                "x64.fld_l [%4]",
                "x64.fstp_t [%5]",
                "x64.fld_t [%6]",
                "x64.fld_t [%7]",
                "x64.fadd_p",
                "x64.fstp_t [%8]",
                "x64.fld_t [%9]",
                "x64.fstp_l [%10]",
            ],
            "{text}"
        );
    }

    #[test]
    fn a_subtraction_pushes_the_left_operand_first_and_asks_for_the_att_spelling() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let less =
            Builder::new(&mut source, block).binary(Opcode::FSub, left, right, Flags::default());
        let back = cast(&mut source, block, Opcode::FPTrunc, less, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // The left one goes on first, so it ends up under the right one, and the answer wanted is
        // the one below minus the top. In AT&T that is `fsubrp`, since `fsubp` there is `DE E0+i`
        // and computes the other one. The `r` says which spelling this is and not which order the
        // pushes were in. `crates/rucc/tests/x87.rs` is what says the answer is right, because a
        // name is what got this wrong the first time.
        let text = lower(&mut names, &source);
        assert_eq!(
            &stack_only(&text)[4..8],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "x64.fsubr_p", "x64.fstp_t [%8]"],
            "{text}"
        );
    }

    #[test]
    fn negating_a_long_double_turns_the_sign_over_and_reads_nothing() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let flipped = Builder::new(&mut source, block).unary(Opcode::FNeg, wide, long_double());
        let back = cast(&mut source, block, Opcode::FPTrunc, flipped, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `fchs` and not a subtraction from zero, which would give a different answer at a negative
        // zero and would signal at a NaN. It does not read the value as a number at all.
        let text = lower(&mut names, &source);
        assert_eq!(
            &stack_only(&text)[2..5],
            ["x64.fld_t [%3]", "x64.fchs", "x64.fstp_t [%4]"],
            "{text}"
        );
    }

    #[test]
    fn comparing_two_long_doubles_puts_the_left_one_on_top() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Ogt, left, right, Flags::default());
        build.ret(&[]);

        // `a > b`. `fucomip` asks about the top of the stack against what is under it, so the
        // operand the predicate is about has to go on last, which is the other way round from the
        // arithmetic above. The pop that clears the loser and the byte that reads the flags are
        // both inside the one opcode.
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");
        let slots = pushed(&out);
        assert_eq!(slots, [2, 1], "the right operand goes on first and the left one on top");
        let text = mir::print_func(&out.func, &names, &REGS);
        assert_eq!(
            &stack_only(&text)[4..],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "%8:gpr = x64.fucomip_set_a"],
            "{text}"
        );
    }

    #[test]
    fn a_comparison_that_the_machine_has_backwards_swaps_the_two_pushes() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Olt, left, right, Flags::default());
        build.ret(&[]);

        // `a < b` is `b > a` and this machine has the one condition, so the same opcode runs with
        // the operands the other way round. The same trade the vector rules make, and it has to
        // be the same one: a `long double` comparison that picked a different condition from the
        // `double` comparison of the same two numbers would be wrong at exactly the unordered
        // cases the two conditions differ on.
        //
        // Which slot each push names is the whole of the difference from the test above, and the
        // text does not show it, since an address in a frame is a `lea` with nothing in it until
        // `finish` has the numbers. So the slots are what is read here.
        let out = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");
        let slots = pushed(&out);
        assert_eq!(slots, [1, 2], "the left operand goes on first and the right one on top");
        let text = mir::print_func(&out.func, &names, &REGS);
        assert_eq!(
            &stack_only(&text)[4..],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "%8:gpr = x64.fucomip_set_a"],
            "{text}"
        );
    }

    #[test]
    fn an_ordered_equal_needs_a_second_byte_to_put_the_two_conditions_together() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Oeq, left, right, Flags::default());
        build.ret(&[]);

        // Equal and ordered are two conditions and the flags carry both, so the opcode writes a
        // second register as well as the one the value is in and ANDs them together. Said here by
        // handing it a spare, since an instruction that wrote a register nothing knew about would
        // be an instruction the allocator could put a live value in the way of.
        let text = lower(&mut names, &source);
        assert!(text.contains("%8:gpr, %9:gpr = x64.fucomip_set_e_and_np"), "{text}");
    }

    #[test]
    fn a_comparison_that_is_never_asked_is_reported() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::False, left, right, Flags::default());
        build.ret(&[]);

        // Always false is a constant and not a comparison, so there is no condition to pick and
        // nothing here folds it into one: an instruction that quietly agreed with it would hide
        // that the optimizer left a comparison in that it should have taken out.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("no condition is always false");
        assert_eq!(failed.to_string(), "no rule lowers a `fcmp` producing a `i1`");
    }

    #[test]
    fn a_long_double_constant_is_the_bits_of_it_put_where_the_value_lives() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        // `1.5L`, which is the leading bit and one more of significand, and an exponent of zero.
        let one_and_a_half = build.fconst(long_double(), 0x3fff_c000_0000_0000_0000);
        build.store(one_and_a_half, args[0], plain(), Flags::default());
        build.ret(&[]);

        // No x87 instruction at all. A slot holding one of these is the value, so a constant is
        // its ten bytes written where the value lives, and whatever reads it does the `fld`.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_ri_64 -4611686018427387904"), "{text}");
        assert!(text.contains("x64.mov_ri_16 16383"), "{text}");
        assert!(text.contains("x64.mov_mr_16 %3, [%1 + 8]"), "{text}");
        // The six bytes above the ten are the padding that makes the type sixteen wide, and they
        // are unspecified rather than zero, so nothing writes them.
        assert_eq!(text.matches("x64.mov_mr").count(), 2, "{text}");
    }

    #[test]
    fn a_negative_long_double_constant_keeps_the_bit_above_its_exponent() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let minus = build.fconst(long_double(), 0xbfff_c000_0000_0000_0000);
        build.store(minus, args[0], plain(), Flags::default());
        build.ret(&[]);

        // `-1.5L`. The sign is the top bit of the two byte half, so the immediate that half is put
        // in a register with is above the signed range of sixteen bits and has to stay there: read
        // as a number it would be negative, and it is not a number, it is two bytes.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_ri_16 49151"), "{text}");
    }

    #[test]
    fn a_long_double_crosses_an_edge_as_an_address_and_is_copied_where_it_lands() {
        let (mut names, mut source, block, args) = blank(&[Type::float(rucc_ir::Float::F64)]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let next = source.create_block();
        let param = source.append_param(next, long_double());
        Builder::new(&mut source, block).jump(next, &[wide]);
        Builder::new(&mut source, next).ret(&[param]);

        // What the edge carries is the address of the slot the value is already in, which is an
        // ordinary register the allocator has an opinion about. The block on the other side copies
        // the sixteen bytes into a slot of its own before anything reads them, so a second edge
        // handing over a second address would still leave one place for a reader to look.
        let text = lower(&mut names, &source);
        let second: Vec<&str> = text
            .lines()
            .skip_while(|line| !line.starts_with("block1"))
            .skip(1)
            .take(3)
            .map(str::trim)
            .collect();
        assert_eq!(
            second,
            ["x64.fld_t [%4]", "%5:gpr = x64.lea_64 [$rsp]", "x64.fstp_t [%5]"],
            "{text}"
        );
    }

    #[test]
    fn more_long_doubles_at_a_block_than_the_stack_is_deep_are_reported() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let next = source.create_block();
        let params: Vec<Value> =
            (0..=X87_DEPTH).map(|_| source.append_param(next, long_double())).collect();
        let carried: Vec<Value> = params.iter().map(|_| wide).collect();
        Builder::new(&mut source, block).jump(next, &carried);
        Builder::new(&mut source, next).ret(&[params[0]]);

        // The copies go through the x87 stack so that every one of them is read before any of them
        // is written, which is what makes a block that swaps two of these right. Nine of them do
        // not fit on the stack, and copying the ninth before or after the rest is the order that
        // could be wrong, so it is refused instead.
        let failed = func(&source, &mut names, &SELECTOR, &SYSV, &Elsewhere::default())
            .expect_err("nine do not fit on the stack");
        assert_eq!(
            failed.to_string(),
            "block1 takes 9 parameters of type `f80` and only 8 can cross an edge at once"
        );
        assert_eq!(failed.inst(), None);
    }
}