rucc_codegen/

lower.rs

//! The selector: an IR function becomes a machine IR function.
//!
//! Design: `spec/10-backend.md` sections 10.2 and 10.3.
//!
//! What the matcher in [`crate::select`] does is answer one question about one term. What this
//! does is ask it: walk a function, decide which terms are worth asking about, and build machine
//! instructions out of what comes back. Nothing here decides what an IR term lowers to. That is
//! in `rules/x86-64.rules` and it is proved before it is used, which is the whole point of the
//! arrangement and the reason this file is short.
//!
//! # What it does with an instruction
//!
//! It tries the ways the instruction can be shown to the matcher, in order, and takes the first
//! that a rule fires on. [`crate::term`] is what a way of showing one is, and the order is the
//! most specific first: an operand that is a constant is offered as a constant before it is
//! offered as a register, and an operand computed by an instruction of its own is offered as
//! that instruction before it is offered as a register. A rule that wants an immediate too wide
//! for the machine has a guard that turns it down, and the search carries on to the way of
//! showing it that puts the constant in a register, which is the right answer and is one nobody
//! had to write down.
//!
//! A constant is not lowered where it is written. It is materialized where a register for it is
//! first wanted, which is what keeps a constant that every use folded into an immediate from
//! leaving a dead instruction behind, and it also gives the value the shortest live range it
//! could have. The instruction that materializes it comes from the rule set like everything else.
//!
//! # What it does not do yet
//!
//! Everything is in the general purpose registers, because every rule in the set is about an
//! integer, so a call that passes a `double` and a function that returns one are both reported
//! rather than lowered. So is an argument that travels on the stack, on either side of a call,
//! and so is a call through an address rather than to a name.
//!
//! # A call
//!
//! Not a rule, because a rule pattern sees one term and what a call's operands are is whatever
//! the signature made them. [`crate::abi`] builds one instead, out of the same description of the
//! convention the arguments come from: the values it passes are reads constrained to the
//! registers the convention places them in, what comes back is a write constrained to the
//! register it comes back in, and every other register the callee is free to destroy is a write
//! of that register and nothing else, which is all the allocator needs to keep a value out of it.
//!
//! What that costs the frame is an argument area, and nothing after selection could work out how
//! big, so the size of the widest call is given back with the function. A function that makes no
//! call at all is a leaf, and a leaf is the function that may use the red zone.
//!
//! # Where a block goes
//!
//! On the block, which is what machine IR does with an edge and is why the branches need no more
//! rule language than the arithmetic did. A rule never names a block, so an unconditional jump
//! has no rule at all and a conditional branch has one that is about its condition and nothing
//! else. The arms are copied across after the block is filled, arguments and all, because an
//! argument that is a constant is materialized where a register for it is first wanted and the
//! end of the block is where an edge wants it.
//!
//! What this leaves behind is a function whose blocks are in the order the IR held them and whose
//! branches are still branches on a register. Turning one into a `test` and a `jcc` is the block
//! layout's, since which of the two arms falls through is the layout's answer, and [`crate::split`]
//! has to run before allocation so that every edge carrying a value has somewhere to put it.
//!
//! A store and a return are the two things here that write no register. A store is emitted like
//! everything else and the only difference is that there is no result to put anywhere, so the
//! operands the target describes are all reads. A return is the same, and what it is for is its
//! one operand: the target constrains it to the register the caller reads the value out of, and
//! the allocator is what gets it there. The instruction that leaves is not chosen here at all,
//! because the epilogue has to give the frame back first and [`crate::finish`] writes that after
//! allocation, so a return of nothing is lowered to nothing.
//!
//! The entry block is the one block whose parameters are not block parameters here. They are the
//! function's arguments, they are already somewhere when it starts, and [`crate::abi`] is what
//! says where. An argument that arrives on the stack is reported rather than read, because where
//! the stack put it is a distance into a frame and no frame exists until after allocation.
//!
//! Blocks are walked in the order the function holds them and a value is expected to be defined
//! before it is used, which is true of the IR this is given because every pass before it keeps
//! definitions ahead of uses.

use std::collections::HashSet;
use std::fmt;

use rucc_base::{Interner, Symbol};
use rucc_diag::Span;
use rucc_ir::{
    Abi, AsmOperand, AsmOperands, Block, Def, Extra, Flags, FloatPred, Func, Inst, Linkage,
    MemOrder, Opcode, Param, PrefetchHint, RmwOp, Type, Value, Visibility,
};
use rucc_mir as mir;
use rucc_target::x86_64;
use rucc_target::{CallRegs, Constraint, OperandDesc, PhysReg, RegClass, Role, Segment};

use crate::abi::{self, Missing, Refused};
use crate::coverage::Fired;
use crate::elsewhere::Elsewhere;
use crate::frame::{Layout, Local};
use crate::select::{Match, Piece, Rule, Table};
use crate::term::{MAX_ARGS, PLAIN, Plan, Shown, Term, Terms};
use crate::varargs;

/// The prefix a rule file puts in front of a machine term, which says which target it belongs
/// to and is not part of the opcode.
pub(crate) const PREFIX: &str = "x64.";

/// The instruction a global offset table slot is read with.
///
/// Not in [`x86_64::FRAME`] with the other opcodes this file names, because a frame has no use for
/// it. It is spelled out here because the relocation it takes is only legal on a `mov` with a REX
/// prefix, so the width is part of the requirement rather than a choice.
const GOT_LOAD: &str = "mov_rm_64";

/// How wide an address is on this target, which is the width a cast between a pointer and an
/// integer has to be at for the cast to be nothing.
const ADDRESS_BITS: u32 = 64;

/// How many bytes a `long double` takes in memory, and what it is aligned to, which are the same
/// number and are both more than the ten bytes that mean anything.
///
/// The psABI's answer rather than a choice here. `sizeof (long double)` is sixteen on this
/// machine, so an array of them is laid out this way whatever a slot holding one does, and a slot
/// that agreed with the array is one fewer thing to get wrong.
const X87_BYTES: u32 = 16;

/// How many values the x87 stack holds at once.
///
/// Eight, which is the machine's number rather than a choice here, and it matters in one place:
/// the parameters of a block are copied through the stack so that they all move at once, and a
/// block with more of them than this has nowhere to put the ninth.
const X87_DEPTH: usize = 8;

/// How far into the buffer of a `__builtin_setjmp` each of the four words it writes is.
///
/// The first three are gcc's, measured against gcc 16.2.0 on x86-64 at `-O0`: the frame pointer,
/// the address control comes back to, and the stack pointer, in that order. The fourth is this
/// compiler's own. gcc has no word for the answer because it writes a second block that sets the
/// answer to one and is arrived at from the restore, and this writes the answer through memory
/// instead, for the reason [`Lowering::saves_place`] gives.
///
/// None of the four is an interface. The buffer is the program's memory and its five words are
/// the front end's promise about how much of it there is, but nothing except the matching restore
/// ever reads a word of it, and a buffer written by one compiler was never going to be one another
/// compiler could come back through.
const JUMP_FRAME: i32 = 0;

/// Where the address control comes back to is. See [`JUMP_FRAME`].
const JUMP_PC: i32 = 8;

/// Where the stack pointer is. See [`JUMP_FRAME`].
const JUMP_STACK: i32 = 16;

/// Where the address of the word the answer arrives in is. See [`JUMP_FRAME`].
const JUMP_ANSWER: i32 = 24;

/// How many bytes the word a `__builtin_setjmp` answers with takes in the frame, and what it is
/// aligned to, which are the same number because it is one machine word.
const JUMP_WORD: u32 = 8;

/// How many registers the restore needs to hold things in while it puts the frame back.
///
/// Four, and every one of them is a register nothing else in the function may be in, which is why
/// they are counted here rather than asked for one at a time. See [`Lowering::comes_back`].
const JUMP_REGS: usize = 4;

/// How many bytes a value passes through on its way between a register and the x87 stack.
///
/// Eight, because the widest thing that crosses is a `double` or a sixty four bit integer, and
/// nothing crosses at eighty bits: a value that wide is already in the frame and the stack reaches
/// it where it is.
const X87_CROSSING: u32 = 8;

/// Where the rounding field of the x87 control word is and what it has to be set to for the unit
/// to cut towards zero, which is the one rounding C asks for that the unit does not do by default.
///
/// Both bits on is truncate. The field is ORed into the word that was already there rather than
/// written over it, so the precision control and the exception masks somebody else set stay set.
const X87_TRUNCATE: i64 = 0x0c00;

/// Whether a type is the one this machine has no register for.
///
/// Only the eighty bit float is, and that is a fact about x86-64 rather than about floats: every
/// other scalar the front end produces is in a general purpose register or a vector one, and this
/// one is on the x87 stack while it is being worked on and in memory the rest of the time. So it
/// has no place in [`Lowering::class_of`] and no name in [`crate::term`], and every instruction
/// that touches one is written out by hand in this file.
fn on_x87(ty: Type) -> bool {
    ty.is_scalar() && ty.is_float() && ty.bits() == 80
}

/// Where one operand of an assembly statement is, on each side of the assembly.
///
/// Two registers rather than one, because an operand written `+` is a value that arrives and a
/// value that leaves and those are two values. The machine IR has one definition per register by
/// construction, so an instruction of the template that reads the operand and writes it has to name
/// a different register in each place, and what makes the two one register in the end is the
/// [`Constraint::Reuse`] the instruction's description carries: the allocator reads it, gives both
/// the same physical register, and copies the incoming value somewhere first when something else is
/// still using it.
///
/// Most operands have one of the two. An input has only a place it is read from and an output
/// written `=` has only a place it is written to, and asking either of them for the other is an
/// operand read where the opcode writes or written where it reads, which [`Lowering::placed`]
/// refuses.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
struct Place {
    /// The register the value arrives in, for an operand something reads.
    read: Option<mir::Reg>,
    /// The register the value leaves in, for an operand something writes.
    write: Option<mir::Reg>,
}

/// Whether that operand of the statement is one the assembly may read, and so where a read of it
/// gets its value from.
///
/// [`bound`] asks this question of an operand a constraint letter named and this asks it of one the
/// template numbered, which is the same question twice because a two-address instruction reaches
/// its first source both ways. `mulq %3` reaches `rax` by the letter on the output and libgmp says
/// what is in it with `"%0"` on an input. `addq %5,%q1` reaches its first source by numbering the
/// output, and libgmp says what is in it with `"0"` on an input in the same way.
///
/// So an output written `=` has no value of its own and is still readable when an input is tied to
/// it, and the value the read wants is that input's. An output written `+` carries its own value
/// and answers with that. An output nothing is tied to answers `None`, which is a program that told
/// the compiler the assembly only writes the operand while the instruction reads it before it
/// writes it, and is refused where it is asked.
fn read_as(list: &[AsmOperand], index: usize) -> Option<Value> {
    let operand = list.get(index)?;
    if operand.value.is_some() {
        return operand.value;
    }
    operand.result?;
    list.iter().find(|entry| entry.tied == Some(index)).and_then(|entry| entry.value)
}

/// Which of an assembly statement's operands is in that register, for an instruction that reaches
/// the register without its text saying so.
///
/// The constraint letter is what says so, and it is the only thing in such a statement that could:
/// `"=a"` is an output in `rax` and `"c"` is an input in `rcx`, and a register nothing names is a
/// register nobody has said anything about. So a write looks among the outputs and a read among the
/// inputs, and an output written `+` answers for either, since it is read before it is written.
///
/// The other way a read of such a register is said is a matching constraint. `"=a"` on an output
/// and `"0"` on an input is the program saying that one register holds the input on the way in and
/// the output on the way out, and it is how a statement fills a register the instruction reads and
/// writes without writing the register down twice. The letter is on the output, which has no value
/// to read, and the value is on the input, which has no letter, so neither of them answers this on
/// its own and the answer is the input: what a read wants is the register the value arrived in, and
/// that is the input's place.
///
/// `None` is a register the instruction uses and the statement put nothing in, which is the usual
/// answer rather than an unusual one. `cpuid` writes four registers and a program that wanted one
/// of them names one. See [`Lowering::spare`], which is where that one goes.
fn bound(list: &[AsmOperand], reg: PhysReg, role: Role) -> Option<usize> {
    let letter = |operand: &AsmOperand| operand.fixed.and_then(x86_64::gpr_letter);
    let named = list.iter().position(|operand| {
        letter(operand) == Some(reg)
            && if role.is_def() { operand.result.is_some() } else { operand.value.is_some() }
    });
    if named.is_some() || role.is_def() {
        return named;
    }
    list.iter().position(|operand| {
        operand.value.is_some()
            && operand
                .tied
                .is_some_and(|at| list.get(at).is_some_and(|out| letter(out) == Some(reg)))
    })
}

/// Why a function could not be lowered.
///
/// One reason and then nothing. A function with no rule for something in it is a function this
/// cannot finish, and the second thing it could not lower is not news.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Unsupported {
    /// An instruction no rule fires on.
    Inst {
        /// The instruction that stopped it.
        inst: Inst,
        /// What the rule file would call it, or nothing if the rule language has no name for it
        /// at all, which is what an instruction at a width nothing is written about looks like.
        term: Option<&'static str>,
        /// The opcode, which is what gets named when the rule language has no word for it.
        ///
        /// An opcode the rule language has no word for is exactly the opcode no rule lowers, so
        /// without this the message would be empty in every case where somebody needs it.
        opcode: Opcode,
        /// What it produces, or nothing for an instruction that is only an effect.
        ty: Option<Type>,
    },
    /// A parameter that does not arrive somewhere this can bring it in from.
    ///
    /// Not an instruction, which is why it is a separate arm: it is a fact about the signature
    /// and there is nothing in the body of the function to point at.
    Argument {
        /// Its position in the signature.
        index: usize,
        /// What is wrong with where it arrives.
        missing: Missing,
    },
    /// A call that passes or gives back a value this cannot put where the convention wants it.
    Call {
        /// The call.
        inst: Inst,
        /// Which value, and what is wrong with where it travels.
        refused: Refused,
    },
    /// A `return` this cannot put where the convention wants it.
    ///
    /// A separate arm from [`Unsupported::Inst`] because it is not an instruction no rule fires
    /// on. A return of more than one value is built from the convention rather than matched, the
    /// same way a call is, so what goes wrong with one is what goes wrong with a call and not the
    /// absence of a rule.
    Returned {
        /// The `return`.
        inst: Inst,
        /// What is wrong with where one of the values travels.
        missing: Missing,
    },
    /// A stack slot the frame cannot give the bytes it asked for.
    ///
    /// Not an instruction no rule covers. An `alloca` is built here rather than matched, so what
    /// goes wrong with one is what the frame can and cannot hold rather than what the rules spell.
    Dynamic {
        /// The `alloca`.
        inst: Inst,
        /// What the frame could not do about it.
        growing: Growing,
    },
    /// More parameters of a type that travels on the x87 stack than the stack is deep.
    ///
    /// Not an instruction either, for the reason a function's parameter is not one: it is a fact
    /// about the block and there is nothing in the block to point at. What crosses an edge for one
    /// of these is the address of where the value is, and the block copies the bytes into a slot
    /// of its own, all of them through the stack at once so that a block carrying two of them
    /// swapped is copied in an order that is right. Eight is as many as the stack holds, and a
    /// ninth would have to be copied before or after the rest, which is the order that could be
    /// wrong.
    Phi {
        /// Which block it arrives at.
        block: Block,
        /// How many of them arrive there, which is the whole of what is wrong.
        count: usize,
        /// What they are.
        ty: Type,
    },
    /// An `asm` statement this cannot build.
    ///
    /// Not an instruction no rule fires on, for the reason a call is not one: what it stands for is
    /// whatever its template says, and no pattern over terms can read a string.
    Assembly {
        /// The `inline_asm`.
        inst: Inst,
        /// What about it is not built here yet.
        refused: Written,
    },
}

/// What about an `asm` statement is not built yet.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Written {
    /// A template with instructions in it.
    Template,
    /// An `asm goto`, whose labels make the statement a terminator.
    Goto,
    /// An operand this cannot put where the constraint says it goes.
    Operand,
    /// A clobber list naming something this has no register for.
    Clobber,
}

impl Written {
    /// The rest of the sentence that starts with the statement.
    #[must_use]
    pub fn why(self) -> &'static str {
        match self {
            // The template is the assembler's to read and there is no assembler here yet, so a
            // template with anything in it is a string nothing can turn into bytes. An empty one is
            // no instructions, and no instructions is something this can write.
            Written::Template => "has instructions in its template, which nothing here assembles",
            Written::Goto => "jumps to a label, which nothing here builds an edge for",
            Written::Operand => "has an operand this cannot place",
            Written::Clobber => "says it destroys a register this has no name for",
        }
    }
}

/// What the frame could not do about a stack slot.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Growing {
    /// An object of a size the number a frame counts bytes in does not reach.
    Huge,
    /// A variable length array wanting more alignment than a call leaves the stack pointer with.
    ///
    /// Rounding the stack pointer down again after the bytes have been taken would put it
    /// somewhere no constant reaches the rest of the frame from, so a frame like this needs a
    /// second base register held for the whole of the function. Nothing here holds one.
    ///
    /// [`crate::expand::rounds`] takes the array away before this sees it, by asking for the
    /// alignment in extra bytes and handing out an address inside them, so what is left of this
    /// is IR that arrived without going through that pass and the fixed local in
    /// [`crate::pipeline`] that wants the same thing from the other side.
    Aligned,
}

impl Growing {
    /// The rest of the sentence that starts with the slot.
    #[must_use]
    pub fn why(self) -> &'static str {
        match self {
            Growing::Huge => "is more bytes than a frame counts",
            Growing::Aligned => {
                "wants more alignment than the stack pointer is left on, which needs a base \
                 register nothing here keeps"
            }
        }
    }
}

impl Unsupported {
    /// The instruction it is about, or nothing for the one arm that is about a signature.
    ///
    /// What a caller wants this for is the span. The function knows where every instruction in
    /// it came from, so a caller holding both can point a message at the line somebody wrote
    /// rather than at the file as a whole, and nothing here has to carry a span of its own.
    pub fn inst(&self) -> Option<Inst> {
        match *self {
            Unsupported::Inst { inst, .. }
            | Unsupported::Call { inst, .. }
            | Unsupported::Returned { inst, .. }
            | Unsupported::Dynamic { inst, .. }
            | Unsupported::Assembly { inst, .. } => Some(inst),
            Unsupported::Argument { .. } | Unsupported::Phi { .. } => None,
        }
    }
}

impl fmt::Display for Unsupported {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match *self {
            Unsupported::Inst { term: Some(term), .. } => write!(f, "no rule lowers `{term}`"),
            Unsupported::Inst { term: None, opcode, ty: Some(ty), .. } => {
                write!(f, "no rule lowers a `{opcode}` producing a `{ty}`")
            }
            Unsupported::Inst { term: None, opcode, ty: None, .. } => {
                write!(f, "no rule lowers a `{opcode}`")
            }
            Unsupported::Argument { index, missing } => {
                write!(f, "parameter {index} {}", missing.why())
            }
            Unsupported::Call { refused: Refused { argument: Some(index), missing }, .. } => {
                write!(f, "argument {index} of this call {}", missing.why())
            }
            Unsupported::Call { refused: Refused { argument: None, missing }, .. } => {
                write!(f, "what this call gives back {}", missing.why())
            }
            Unsupported::Returned { missing, .. } => {
                write!(f, "what this function gives back {}", missing.why())
            }
            Unsupported::Dynamic { growing, .. } => {
                write!(f, "this local {}", growing.why())
            }
            Unsupported::Phi { block, count, ty } => {
                let block = block.index();
                write!(
                    f,
                    "block{block} takes {count} parameters of type `{ty}` and only {X87_DEPTH} can cross an edge at once"
                )
            }
            Unsupported::Assembly { refused, .. } => write!(f, "this `asm` {}", refused.why()),
        }
    }
}

impl std::error::Error for Unsupported {}

/// A lowered function, and what the frame needs that the machine IR does not hold.
#[derive(Debug)]
pub struct Lowered {
    /// The function, in machine instructions.
    pub func: mir::Func,
    /// What it wants its stack to look like, which is separate from the function so that the two
    /// can be read and written at the same time.
    pub stack: Stack,
    /// Which rules of the table lowered it, which is what `-Zrule-coverage` asks for and what
    /// `crate::coverage` writes down.
    pub fired: Fired,
    /// Which machine IR block each IR block became, indexed by the IR block's own index, and
    /// nothing for a block the walk never reached.
    ///
    /// Here because it is the only place the correspondence exists. Selection makes one block per
    /// block, in the same order and with the arms in the same order, so anything the IR knows
    /// about a block can be carried down through this and nothing else, and
    /// [`crate::weights::carry`] is what does.
    pub blocks: Vec<Option<mir::Block>>,
}

/// What a function's stack has to hold, as far as selection is able to say.
///
/// All of it is answered here because selection is where a call is built and where an `alloca`
/// is read, and nothing after it could tell what either of them needed.
#[derive(Debug, Default)]
pub struct Stack {
    /// How many bytes the widest call in the function needs below the stack pointer for the
    /// arguments it passes there, or `None` for a function that makes no call at all.
    ///
    /// `None` is a leaf, which is the function that may use the red zone and the one whose stack
    /// pointer does not have to be left aligned for anybody.
    pub calls: Option<u32>,
    /// The memory the function asked for itself, one entry for every `alloca` in it, in the order
    /// the walk reached them.
    pub locals: Vec<Local>,
    /// Which instruction computes the address of which of those locals.
    ///
    /// An address in the frame is a distance from the stack pointer, and there is no frame until
    /// after allocation, so the instruction is written here with nothing in its displacement and
    /// [`crate::finish`] writes the number in once [`crate::frame::Frame`] knows it.
    pub addresses: Vec<(mir::Inst, usize)>,
    /// Which instruction computes the address of a piece of memory whose size the function works
    /// out while it runs, which is what a variable length array is.
    ///
    /// Waiting on [`crate::finish`] for a different number from the one the addresses above are:
    /// the bytes were taken off the stack pointer by the instruction in front of this one, so where
    /// they start is however much of the bottom of the frame belongs to the arguments of a call,
    /// and that is not known until the frame is.
    pub dynamic: Vec<mir::Inst>,
    /// Which instruction takes those bytes off the stack pointer, one for every one of them, in the
    /// order the walk reached them.
    ///
    /// Read by [`crate::finish`] on a command line that asked for the stack to be touched a page at
    /// a time, which is the one thing that has to find these again: the bytes are in a register by
    /// then, so the walk down to them is a loop, and a loop is written around an instruction rather
    /// than in front of a block. Nothing else looks at them, because everything else about a frame
    /// that grows is answered by the address the instruction below this one computes.
    pub grown: Vec<mir::Inst>,
    /// Where the function first moves the stack pointer while it runs, if it does at all.
    ///
    /// Two things are read off this. One is whether at all, which is what [`crate::frame::Layout`]
    /// wants, because a frame that moves its stack pointer has a different shape from one that does
    /// not and the layout is built before the instructions are looked at again. See `Growing` in
    /// [`crate::frame`]. The other is where, so that a caller that cannot accept such a frame has
    /// somewhere to point when it says so.
    pub grown_at: Option<Inst>,
    /// Which instruction reads which of the arguments the caller passed on the stack, as how far up
    /// the caller's argument area it reads.
    ///
    /// Waiting on [`crate::finish`] for the same reason the addresses above are, and on one thing
    /// more: where the caller's argument area is from inside this function depends on whether the
    /// prologue had to force the stack pointer's alignment, so which register the load reads
    /// through is not settled here either.
    pub arguments: Vec<(mir::Inst, u32)>,
    /// Whether the function asked where its own frame is, which is what `__builtin_frame_address`
    /// and `__builtin_return_address` both start from.
    ///
    /// A function like that keeps a frame pointer whatever the flags say, because the register is
    /// the answer to the first of them and the start of the walk for every depth above zero. There
    /// is no other way to reach it: the distance from the stack pointer to the frame is a number
    /// the layout works out, and what a walk up the chain needs is the link the prologue saved.
    pub walks_frames: bool,
    /// Whether the function saved a place for a `__builtin_longjmp` to come back to, which is what
    /// `__builtin_setjmp` does.
    ///
    /// A function like that keeps a frame pointer whatever the flags say as well, and for a reason
    /// of the same shape: the two registers the restore puts back are the frame pointer and the
    /// stack pointer, and a frame that did not keep the first of them has nothing in it saying
    /// where the caller's frame is for the epilogue to find after control has come back.
    pub saves_place: bool,
}

impl Stack {
    /// The layout given, with the three fields only the lowering knows the answer to filled in.
    ///
    /// Everything else in a layout comes from the flags the function is compiled under or from the
    /// allocation, so this takes one and returns it rather than building one.
    ///
    /// A function that saved a place is not a leaf whatever it called. What a leaf buys is the red
    /// zone, which is the words below the stack pointer nothing else may write, and a function
    /// control comes back into from a `__builtin_longjmp` has already had something else running
    /// down there: whatever it called and whatever that called, or a signal handler on the same
    /// stack. Every one of those has written over the red zone by the time control arrives, so a
    /// value this function left there would not be there any more.
    #[must_use]
    pub fn layout<'a>(&'a self, base: Layout<'a>) -> Layout<'a> {
        Layout {
            leaf: self.calls.is_none() && !self.saves_place,
            outgoing: self.calls.unwrap_or(0),
            locals: &self.locals,
            grows: self.grown_at.is_some(),
            ..base
        }
    }
}

/// The x86-64 machine IR for that function.
///
/// # Errors
///
/// The first instruction no rule fires on, which today is anything at a width the rule set is not
/// written at, a parameter that does not arrive in a register this can read, or a call that
/// passes something this cannot put where the convention wants it.
pub fn func(
    source: &Func,
    names: &mut Interner,
    conv: &'static CallRegs,
    elsewhere: &Elsewhere,
) -> Result<Lowered, Unsupported> {
    Lowering::new(source, names, conv, elsewhere).run()
}

/// What the matcher settled on for one block, indexed the way the block's instructions are.
struct Decided {
    /// What each instruction matched, and nothing for one that matched no rule or was folded
    /// into a later one.
    found: Vec<Option<Match<Term>>>,
    /// How each instruction showed its operands to the matcher, which is what says what it took.
    plans: Vec<Option<Plan>>,
    /// The instructions some other instruction took, which are the ones with nothing to write.
    folded: Vec<Inst>,
}

/// One function being lowered.
struct Lowering<'a> {
    source: &'a Func,
    names: &'a mut Interner,
    out: mir::Func,
    /// The machine register each IR value is in, once it has one.
    regs: Vec<Option<mir::Reg>>,
    /// For a constant that has been written into a register, the block it was written into,
    /// which is the only block that register is any good in.
    written: Vec<Option<mir::Block>>,
    /// How many times each IR value is read, which is what says whether an instruction may be
    /// folded into the one that reads it.
    uses: Vec<u32>,
    /// The block being filled.
    at: Option<mir::Block>,
    /// The machine IR block each IR block became.
    blocks: Vec<Option<mir::Block>>,
    /// The class an address is in, which is the general purpose one and is not a question: every
    /// register an addressing mode names holds part of an address, and there is no machine here
    /// that computes an address anywhere but in this file. Which class a *value* is in is
    /// [`Lowering::class_of`], and it is a question, because a float is in the other one.
    gpr: RegClass,
    /// Where the convention this function is compiled for puts things, which is read for the
    /// arguments and for the calls.
    conv: &'static CallRegs,
    /// Which names this function may not work an address out for itself, which is a fact about the
    /// module and so is worked out before any of this and handed in.
    elsewhere: &'a Elsewhere,
    /// What the function wants its stack to look like, filled in as the walk finds out.
    stack: Stack,
    /// What a `va_start` in this function has to write, or nothing for a function that takes no
    /// arguments its signature does not name.
    ///
    /// Worked out once, when the entry block binds the parameters, because every number in it is
    /// about where those parameters left the walk over the argument registers and there is nowhere
    /// else that knows.
    varargs: Option<Varargs>,
    /// Which of the function's stack objects each eighty bit value lives in, once it has asked
    /// for one.
    ///
    /// One slot per value and it is never given back, which is what makes an eighty bit value
    /// behave like every other one: it is written once and read wherever it is read, and no two
    /// of them share a slot the way two of them would share a register. What is in a register is
    /// the address, and that is worked out again at every use rather than kept, so nothing here
    /// holds a general purpose register open across a whole function.
    slots: Vec<Option<usize>>,
    /// The eight bytes a value passes through between a register and the x87 stack, once
    /// something has wanted them.
    ///
    /// One for the whole function, because every group that uses it is a handful of instructions
    /// with nothing in between: the bytes are written, read straight back and never looked at
    /// again, so a second slot would be a second slot holding the same nothing.
    crossing: Option<usize>,
    /// The four bytes the control word is saved in and the changed copy written to, once
    /// something has wanted them.
    ///
    /// One for the whole function for the reason above, and four rather than two because it is
    /// two words: the one the unit had and the one with the rounding field turned to truncate.
    control: Option<usize>,
    /// The word a `__builtin_setjmp` in this function answers with, once one has asked for it.
    ///
    /// One for the whole function however many saves there are in it, because the word is written
    /// and read back with nothing in between: the save writes a zero into it and the instruction
    /// straight after reads it, and the only other thing that ever writes it is a restore arriving
    /// between those two. Two saves sharing it is two pairs each doing that, and neither can be
    /// inside the other.
    answer: Option<usize>,
    /// Which rules have fired so far.
    fired: Fired,
}

/// What a `va_start` in a variadic function writes into the list it is given.
///
/// Two shapes, because two conventions describe a list two ways, and [`crate::varargs`] is where
/// both are written down. Neither is a set of numbers on its own: where the save area is and where
/// the caller's argument area is are distances into a frame that does not exist until after
/// allocation, so each is a `lea` [`crate::finish`] fills in.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Varargs {
    /// The four field list, whose two offsets are settled here and whose two addresses are not.
    Fields {
        /// Which of the function's stack objects is the register save area.
        save: usize,
        /// How far up the caller's argument area the first argument the signature does not name is,
        /// which is the whole of that area the named ones did not take.
        incoming: u32,
        /// What `gp_offset` starts at, which is past the general purpose registers the named
        /// arguments took.
        integers: u32,
        /// What `fp_offset` starts at, which is past the vector ones.
        floats: u32,
    },
    /// The list that is a pointer, which is the one address and nothing else.
    Pointer {
        /// How far up the caller's argument area the first argument the signature does not name is,
        /// which on this convention is the word belonging to the position the named ones stopped
        /// at.
        incoming: u32,
    },
}

/// How far a function's name reaches, narrowed from the linkage the IR gave it.
///
/// The IR has five and an object file says three, and the two the linker cannot tell apart are
/// the two weak ones: which of them a symbol had is a fact the optimizer reads and the linker has
/// no way to record. A function is never `Common`, since that is what a tentative definition of an
/// object is and there is no tentative definition of a function, and it is written here rather
/// than left out so that a linkage added later has to come past this.
const fn binding(linkage: Linkage) -> mir::Binding {
    match linkage {
        Linkage::Internal => mir::Binding::Local,
        Linkage::Weak | Linkage::LinkOnce => mir::Binding::Weak,
        Linkage::External | Linkage::Common => mir::Binding::Global,
    }
}

/// How far a function's name reaches outside a shared library, carried across unchanged.
///
/// Nothing is narrowed here the way [`binding`] narrows the linkage, because ELF records all
/// three of these and the two enumerations are the same three answers written twice: once in a
/// crate that is not allowed to know what an object file is and once in one that is.
const fn visibility(visibility: Visibility) -> mir::Visibility {
    match visibility {
        Visibility::Default => mir::Visibility::Default,
        Visibility::Hidden => mir::Visibility::Hidden,
        Visibility::Protected => mir::Visibility::Protected,
    }
}

impl<'a> Lowering<'a> {
    fn new(
        source: &'a Func,
        names: &'a mut Interner,
        conv: &'static CallRegs,
        elsewhere: &'a Elsewhere,
    ) -> Self {
        let counts = source.counts();
        let name = source.name;
        let mut uses = vec![0; counts.values];
        for block in source.blocks() {
            for inst in source.insts(block) {
                for &arg in &source[source[inst].args] {
                    uses[arg.index()] += 1;
                }
                for call in source.successors(inst) {
                    for &arg in &source[call.args] {
                        uses[arg.index()] += 1;
                    }
                }
            }
        }
        let mut out = mir::Func::new(name);
        out.align = source.align;
        // Carried rather than worked out here, because where a function was declared is a fact
        // about the source and this is a long way past it. What wants it is the line table.
        out.declared = source.declared;
        out.binding = binding(source.linkage);
        out.visibility = visibility(source.visibility);
        Self {
            source,
            names,
            out,
            regs: vec![None; counts.values],
            written: vec![None; counts.values],
            blocks: vec![None; counts.blocks],
            uses,
            at: None,
            gpr: x86_64::GPR,
            conv,
            elsewhere,
            stack: Stack::default(),
            varargs: None,
            slots: vec![None; counts.values],
            crossing: None,
            control: None,
            answer: None,
            fired: Fired::new(),
        }
    }

    fn run(mut self) -> Result<Lowered, Unsupported> {
        // Every block before any of them is filled, because a block that jumps forward has to
        // name the block it jumps to and a machine IR block is named by a handle rather than by
        // the IR block it came from.
        for block in self.source.blocks() {
            let out = self.out.create_block();
            self.blocks[block.index()] = Some(out);
        }
        for block in self.order() {
            self.block(block)?;
        }
        // And the name each block an image holds the address of was given, which nothing in the
        // walk above would ask for: the `lea` a label address is inside the function needs no
        // symbol, and the one thing that does is a relocation in another section.
        let named: Vec<(Block, Symbol)> = self.source.named_blocks().collect();
        let labels: Vec<(mir::Block, Symbol)> =
            named.into_iter().map(|(block, name)| (self.out_block(block), name)).collect();
        self.out.labels = labels;
        Ok(Lowered { func: self.out, stack: self.stack, fired: self.fired, blocks: self.blocks })
    }

    /// The order the blocks are filled in, which is not the order they are written in.
    ///
    /// Reverse postorder, because a value is written in a block that dominates every block that
    /// reads it and a block in reverse postorder comes before every block it dominates. The order
    /// the blocks are written in does not have that property: a block written early can read a
    /// value a block below it writes, and reading a value with no register yet mints one, so the
    /// register the definition writes later is not the register the read named. Nothing writes the
    /// one the read named, and what comes out is a function that loads a stack slot no store ever
    /// reached. It is the order this walk goes in rather than the order the blocks come out in,
    /// which is what the loop above fixes, so the machine function is still written the way the IR
    /// function was.
    ///
    /// Blocks the entry does not reach come last, in the order they are written in. Nothing runs
    /// them and nothing they name is read by anything that does, but they still have to be filled,
    /// because a machine block with no terminator is not one the passes below can read.
    fn order(&self) -> Vec<Block> {
        let Some(entry) = self.source.entry() else { return self.source.blocks().collect() };
        let count = self.blocks.len();
        let mut succs: Vec<Vec<Block>> = vec![Vec::new(); count];
        for block in self.source.blocks() {
            let Some(term) = self.source.terminator(block) else { continue };
            succs[block.index()] = self.source.successors(term).map(|call| call.block).collect();
        }
        // An explicit stack, because the depth of the walk is the number of blocks and a function
        // built by a generator has as many of those as it likes.
        let mut seen = vec![false; count];
        let mut order = Vec::with_capacity(count);
        let mut stack = vec![(entry, 0usize)];
        seen[entry.index()] = true;
        while let Some((block, at)) = stack.pop() {
            let Some(&next) = succs[block.index()].get(at) else {
                order.push(block);
                continue;
            };
            stack.push((block, at + 1));
            if !seen[next.index()] {
                seen[next.index()] = true;
                stack.push((next, 0));
            }
        }
        order.reverse();
        order.extend(self.source.blocks().filter(|block| !seen[block.index()]));
        order
    }

    /// One block: its parameters, then every instruction in it that is not folded into another.
    fn block(&mut self, block: Block) -> Result<(), Unsupported> {
        let out = self.out_block(block);
        self.at = Some(out);
        if self.source.entry() == Some(block) {
            self.arrive(block, out)?;
        } else {
            let mut arriving = Vec::new();
            for &param in &self.source[block].params {
                // A value with no register to arrive in, which the class would not say, since
                // `class_of` puts one of these in the general purpose file on purpose and what it
                // means by that is that nothing there can hold it. What crosses the edge for one
                // of those is the address of where the value already is, so the parameter is a
                // pointer here and the bytes it points at are copied below.
                let ty = self.source[param].ty;
                let reg = self.out.append_param(out, self.class_of(ty));
                self.regs[param.index()] = Some(reg);
                if on_x87(ty) {
                    arriving.push((param, reg));
                }
            }
            self.settle(block, &arriving)?;
        }

        // What each instruction matched, and which instructions were folded into another. The
        // decision is made for the whole block before any of it is written, and it is made more
        // than once: a value that only some of its readers took has to be put back in a register
        // for all of them, and taking it away from those readers changes what they match.
        let insts: Vec<Inst> = self.source.insts(block).collect();
        let mut refused: HashSet<Value> = HashSet::new();
        let mut decided = self.decide(&insts, &refused);
        while let Some(value) = self.left_alive(&insts, &decided.plans) {
            refused.insert(value);
            decided = self.decide(&insts, &refused);
        }
        let Decided { found, folded, .. } = decided;

        for (&inst, matched) in insts.iter().zip(found) {
            if folded.contains(&inst) || self.writes_nothing(inst) {
                continue;
            }
            // A call is built from the convention rather than matched, which is why it is the one
            // opcode looked at by name here. Through an address it is a different instruction and
            // the same convention, so the two arrive at the same place and differ in one line of
            // it.
            match self.source[inst].opcode {
                Opcode::Call | Opcode::CallIndirect => {
                    self.called(inst)?;
                    continue;
                }
                // Built from the frame rather than matched, for the same shape of reason a call
                // is built from the convention: what a rule replaces a term with is instructions,
                // and what an `alloca` needs first is bytes, which the rule language has no way
                // to ask for.
                Opcode::Alloca => {
                    self.reserve(inst)?;
                    continue;
                }
                // Reading the stack pointer and writing it back, which are the two ends of a scope
                // holding a variable length array. Built here for the reason an `alloca` is: the
                // value is a register the rule language has no way to name, because what it holds
                // is not a value the program computed but where the machine's stack had got to.
                Opcode::StackSave => {
                    self.stack_pointer(inst, false)?;
                    continue;
                }
                Opcode::StackRestore => {
                    self.stack_pointer(inst, true)?;
                    continue;
                }
                // The address of a name, built here for the same reason an `alloca` is: what a
                // rule replaces a term with is instructions over values, and the operand of this
                // one is a symbol, which is a thing the rule language has no way to bind and the
                // solver has no way to say anything about. There is nothing in `lea sym(%rip)` a
                // proof over bitvectors could discharge, because what makes it the right answer
                // is the relocation and what the linker does with it.
                Opcode::GlobalAddr => {
                    self.address_of(inst)?;
                    continue;
                }
                // The address of a label and the branch that reads one, built here for the same
                // reason and for one more. The reason is the same: what the first of them names is
                // a block, which is not a value a rule pattern can bind, and there is nothing in
                // the distance between two places in one function that a proof over bitvectors
                // could discharge. The extra one is that the second is a terminator whose arms are
                // not two and not fixed, and a rule says what an instruction reads rather than
                // where a block goes.
                Opcode::BlockAddr => {
                    self.block_address(inst)?;
                    continue;
                }
                Opcode::IndirectBr => {
                    self.indirect_branch(inst)?;
                    continue;
                }
                // The pair that saves a place in this function and comes back to it. Built here
                // for the reason the address of a label is, and for two more. The reason is the
                // same: the first of them writes down where control comes back to, which is a
                // place in this function and not a value a rule pattern can bind. The extra ones
                // are that each of them is a group of instructions over a buffer the program owns
                // rather than one instruction, and that the first of them leaves the block it was
                // written in and carries on in a new one, which is a thing no rule can do.
                Opcode::SetjmpMarker => {
                    self.saves_place(inst)?;
                    continue;
                }
                Opcode::LongjmpMarker => {
                    self.comes_back(inst)?;
                    continue;
                }
                // Where this thread's own storage starts, built here for a reason of the same
                // shape: what it reads is `%fs`, which is not a register the rule language can
                // bind and not one a proof over bitvectors could say anything about, because what
                // makes the load the right answer is an agreement between the loader and the C
                // library rather than any arithmetic.
                Opcode::ThreadPointer => {
                    self.thread_pointer(inst)?;
                    continue;
                }
                // Where a frame is and what it returns to, built here for the same reason and one
                // more. The reason is the same: what the walk starts from is the frame pointer,
                // which is not a register a rule pattern can bind, and there is nothing in reading
                // the link the prologue saved that a proof over bitvectors could discharge. The
                // extra one is that how long the walk is comes out of a number beside the
                // instruction, so one of these is not one instruction but however many the depth
                // says, and a rule replaces a term with a term.
                Opcode::FrameAddress | Opcode::ReturnAddress => {
                    self.frames(inst)?;
                    continue;
                }
                // Built from the frame for the reason an `alloca` is, and from the convention for
                // the reason a call is: three of the four fields it writes are distances that do
                // not exist until the frame does, and the fourth is where the walk over the
                // argument registers stopped. A function that is not variadic has no such walk to
                // report, so it has nothing here and is refused below, which is the right answer
                // for a `va_start` in one.
                Opcode::VaStart if self.varargs.is_some() => {
                    self.va_start(inst)?;
                    continue;
                }
                // A return of more than one value, which is a structure small enough to come
                // back in a pair of registers. Built from the convention for the reason a call
                // is: which register each half goes in depends on the halves in front of it,
                // because the two register files are walked separately, and a pattern over a term
                // cannot see them. A return of one value is a term with a name and a rule, and it
                // stays one.
                //
                // A return of none in a function whose answer went through memory is here too,
                // and for a different reason: what it gives back is not written in the IR at all.
                // The convention says the address the caller handed over comes back, and only the
                // signature says this function was handed one.
                //
                // And a return of one eighty bit value, for a third reason: what a rule would
                // write is an instruction leaving the value in a register, and this one is left on
                // the x87 stack instead. A rule could not name that stack any more than any other
                // rule about this type could.
                Opcode::Return
                    if self.source[self.source[inst].args].len() > 1
                        || self.sret().is_some()
                        || self.gives_back_x87(inst) =>
                {
                    self.returned(inst)?;
                    continue;
                }
                // A cast between a pointer and an integer of the same width, which on this
                // machine is every one the front end writes. No instruction at all, so no rule
                // could name one.
                Opcode::PtrToInt | Opcode::IntToPtr => {
                    self.rename(inst)?;
                    continue;
                }
                // A barrier, which is one instruction or none depending on the ordering. Written
                // by name because there is nothing about it a rule could be proved against, the
                // way there is nothing to prove about the address of a symbol.
                Opcode::Fence => {
                    self.barrier(inst)?;
                    continue;
                }
                // A hint, written by name for the reason a barrier is and one step further: not
                // only is there no equality for a proof to discharge, there is nothing about the
                // program around it either. Which of the four instructions it is comes out of the
                // number the builtin was given, which is beside the instruction rather than in it.
                Opcode::Prefetch => {
                    self.hint(inst)?;
                    continue;
                }
                // Stopping, written by name for the first half of the barrier's reason: it
                // computes nothing, so there is no term for a rule to replace, and what makes it
                // right is what the operating system does with the fault rather than anything a
                // proof over bitvectors could discharge.
                Opcode::Trap => {
                    self.trap(inst);
                    continue;
                }
                // A compare and exchange, which is written by name because it produces two values
                // and a rule produces one. The replacement of a rule is one term, a term names the
                // value an instruction computes, and there is no way in that language to say that
                // an instruction leaves an answer in one place and a yes or no in another.
                Opcode::Cmpxchg => {
                    self.exchange(inst)?;
                    continue;
                }
                // A read modify write, which is written by name for a different reason: it produces
                // one value, so a rule could name it, and what it does is not in the head a rule
                // matches on. Every one of the thirteen operations is the same opcode at the same
                // type and differs only in what is carried beside it, so one pattern would be all
                // thirteen patterns. Of the thirteen only the three with an instruction reach here,
                // since `crate::retry` turned the rest into loops a long way above this.
                Opcode::AtomicRmw => {
                    self.modify(inst)?;
                    continue;
                }
                // An `asm` statement, whose lowering is its template and there is no term for a
                // string. Written by name for the reason a barrier is, and before the x87 arm
                // below so that an `asm` holding a `long double` is refused as the `asm` it is
                // rather than as an instruction nothing computes.
                Opcode::InlineAsm => {
                    self.assembly(inst)?;
                    continue;
                }
                // Anything at all with an eighty bit float in it, which is the one arm here
                // chosen by a type rather than by an opcode, because what makes these different
                // is not what they do but where the value is. A `long double` has no register,
                // so it has no name in `crate::term` and no rule could bind one: every one of
                // these is a group of instructions over a frame slot, written out below.
                //
                // Last of the arms, so that a call and a return with one of these in them reach
                // the convention first and are refused by it, which is the truer answer: what is
                // wrong there is where the value has to travel and not that nothing can compute
                // it.
                _ if self.touches_x87(inst) => {
                    self.x87(inst)?;
                    continue;
                }
                _ => {}
            }
            let matched = matched.ok_or_else(|| self.unsupported(inst))?;
            self.emit(inst, &matched)?;
            // After it is built rather than when it matched, so that what is recorded is the rules
            // this function was lowered by and not the rules something was tried with.
            self.fired.mark(matched.rule);
        }
        // Whichever block the walk ended in rather than the one it started in. The two are the
        // same block for every function that does not save a place for a `__builtin_longjmp`, and
        // where they differ it is the last of them that the terminator and the arms belong to.
        // See [`Self::saves_place`].
        let last = self.at.expect("a block is being filled");
        self.edges(block, last)
    }

    /// One call, which is built from the convention rather than matched against the table for the
    /// same reason the arguments of the function itself are.
    ///
    /// The arguments are read before the call is built, which is what materializes a constant
    /// argument into a register, since no call passes an immediate.
    ///
    /// A call to a name and a call through an address are both here, and what tells them apart is
    /// the opcode rather than whether a callee was recorded, which is the same thing the verifier
    /// reads. Through an address the first operand is the address and the arguments are the ones
    /// behind it, and everything after that is the same: where each argument goes, where the value
    /// comes back and which registers are gone across it are the convention's answers and the
    /// convention does not ask what is being called.
    fn called(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Call(info) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[info];
        let indirect = data.opcode == Opcode::CallIndirect;

        let values: Vec<Value> = self.source[data.args].to_vec();
        let callee = if indirect {
            let &address = values.first().ok_or_else(|| self.unsupported(inst))?;
            abi::Callee::Through(self.reg_of(address)?)
        } else {
            abi::Callee::Named(info.callee.ok_or_else(|| self.unsupported(inst))?)
        };

        // What the ABI asks of each argument, read out before any of them is, because reading one
        // borrows the function this is a table in. The ones the signature names are the signature's
        // answer and the ones behind them are the call's, which is where a structure passed to a
        // variadic callee by value says that its bytes travel: there is no parameter to say it on.
        let signature = &self.source[info.signature];
        let variadic = signature.variadic;
        let named: Vec<Abi> = signature.params.iter().map(|param| param.abi).collect();
        let beyond: Vec<Abi> = self.source[info.varargs].to_vec();
        // Every value that comes back and not only the first. A structure small enough to travel
        // in registers comes back in up to two of them, and which register each half is in is the
        // convention's answer, which is why the whole list goes to the same place the arguments do
        // rather than to a rule.
        let returns: Vec<Type> = signature.return_types().collect();

        let mut args = Vec::with_capacity(values.len());
        for (index, value) in values.into_iter().skip(usize::from(indirect)).enumerate() {
            let abi = named.get(index).or_else(|| beyond.get(index - named.len()));
            let abi = abi.copied().unwrap_or_default();
            let ty = self.source[value].ty;
            // What travels for an eighty bit value is its bytes, so what the call is handed is
            // where they are rather than a register they are in, and there is no register they
            // could be in. Everything else about it is a sixteen byte object passed by value and
            // is built by the same code.
            let reg =
                if abi::on_the_stack(ty) { self.x87_slot(value) } else { self.reg_of(value)? };
            args.push(abi::Passing { ty, reg, abi });
        }
        let block = self.at.expect("a block is being filled");
        let what = abi::Calling {
            callee,
            args: &args,
            returns: &returns,
            variadic,
            named: named.len(),
            at: self.source.span(inst),
        };
        let made = abi::call(&mut self.out, block, &what, self.conv, self.names)
            .map_err(|refused| Unsupported::Call { inst, refused })?;
        let calls = &mut self.stack.calls;
        *calls = Some(calls.unwrap_or(0).max(made.outgoing));
        // An eighty bit value came back on the x87 stack, and the one thing that has to happen
        // before anything else touches that stack is taking it off. So the `fstp` goes here, in
        // front of everything the block does next, and after it the value is in its slot and is
        // read the way every other one is.
        let results: Vec<Value> = self.source[inst].results().collect();
        if let [result] = results[..] {
            if abi::on_the_stack(self.source[result].ty) {
                let span = self.source.span(inst);
                let into = self.x87_slot(result);
                let into = self.through(into);
                self.x87_at("fstp_t", span, into);
                return Ok(());
            }
        }
        for (result, &reg) in results.into_iter().zip(&made.results) {
            self.regs[result.index()] = Some(reg);
        }
        Ok(())
    }

    /// The pointer a function returning through memory was handed, or nothing in a function that
    /// was not.
    ///
    /// It is the first parameter and the signature is what says so, since in the IR it is an
    /// ordinary pointer and reads like one everywhere in the body. A function with a signature
    /// like that and no entry block has nothing to give back and no body to give it back from.
    fn sret(&self) -> Option<Value> {
        let first = self.source.signature().params.first()?;
        if !matches!(first.abi, Abi::Sret { .. }) {
            return None;
        }
        self.source[self.source.entry()?].params.first().copied()
    }

    /// One `return` the convention has to write, as the place each value has to be in by the end.
    ///
    /// One pseudo per value, each a read constrained to a return register, which is what a return
    /// of one value already is and is the whole of what either does. The `ret` itself comes from
    /// the epilogue for both, long after this, because the frame has to be given back first.
    ///
    /// The two register files are counted separately, so a structure of a `double` and a `long`
    /// leaves the `double` in the first vector register and the `long` in the first integer one
    /// rather than in the second of either. That is the same walk `rucc_codegen::abi` makes on
    /// the other side of the call, which is what makes the two ends agree.
    ///
    /// A function whose answer went through memory gives back the address it was handed, in front
    /// of nothing else, because a signature that returns that way returns nothing else. That the
    /// caller already knows the address is not enough: it is allowed to read the register instead,
    /// and a caller that does gets whatever the allocator last left there. In a leaf function that
    /// is usually the right answer by accident, and one call in the body is enough to make it a
    /// wild pointer, which is why this is written rather than left to luck.
    ///
    /// Where everything goes is worked out before anything is written, so a return this cannot
    /// make leaves no half of one behind.
    /// Whether what a `return` gives back is the one value that goes back on the x87 stack.
    fn gives_back_x87(&self, inst: Inst) -> bool {
        let [value] = self.source[self.source[inst].args] else { return false };
        abi::on_the_stack(self.source[value].ty)
    }

    fn returned(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let values: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let (mut ints, mut floats) = (0usize, 0usize);
        let mut parts = Vec::with_capacity(values.len() + 1);
        // An eighty bit value goes back on the x87 stack, which is where the convention says it is
        // and is the one place a value is left rather than put in a register. So the whole of the
        // return is an `fld` of its slot, and the stack it leaves the value on is not empty at the
        // `ret`, which is the one time in this file that is true and is what the convention asks
        // for. What comes after is the epilogue, which gives the frame back and touches nothing in
        // the unit.
        if let [value] = values[..] {
            let ty = self.source[value].ty;
            if abi::on_the_stack(ty) && self.sret().is_none() {
                let span = self.source.span(inst);
                let from = self.x87_slot(value);
                let from = self.through(from);
                self.x87_at("fld_t", span, from);
                return Ok(());
            }
        }
        for value in self.sret().into_iter().chain(values) {
            let ty = self.source[value].ty;
            let at = if crate::term::in_vector_file(ty) { &mut floats } else { &mut ints };
            // Why it cannot come back, and not only that it cannot. A type that travels nowhere
            // says so itself, and a type that travels perfectly well ran out of registers.
            let missing = abi::refuses(ty).unwrap_or(Missing::NoRoom);
            let name = abi::ret_of(ty, *at).ok_or(Unsupported::Returned { inst, missing })?;
            *at += 1;
            // The register is the target's answer and not one worked out here, the same as it is
            // for a return of one value, so that both halves of a pair and every rule that writes
            // half of one are reading the same table.
            let opcode = name.strip_prefix(PREFIX).expect("a machine instruction of this target");
            let form = x86_64::form(opcode).ok_or_else(|| self.unsupported(inst))?;
            let [desc] = form.operands() else { return Err(self.unsupported(inst)) };
            parts.push((self.names.intern(name), self.reg_of(value)?, *desc));
        }

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        for (opcode, reg, desc) in parts {
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            self.out.build(block, mir::Opcode::new(opcode)).at(span).operand(operand).finish();
        }
        Ok(())
    }

    /// One `alloca`: the bytes it asks for go on the list the frame is laid out from, and the
    /// address of them is one instruction.
    ///
    /// The instruction is a `lea` off the stack pointer, which is the one register that reaches
    /// the frame in every function, and its displacement is left at nothing because there is no
    /// frame yet. Which instruction is waiting for which local is remembered, and
    /// [`crate::finish`] fills the numbers in after [`crate::frame::Frame`] has placed them.
    ///
    /// There is deliberately no rule for `alloca` and no name for one in [`crate::term`], and
    /// that is what stops it being folded into something else. An operand shown as the
    /// instruction that computed it is offered to the matcher by its name, so an `alloca` with no
    /// name is one no pattern can reach past, and the address it computes is always in a register
    /// by the time anything reads it.
    fn reserve(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        // A variable length array carries the size it wants as an operand rather than in the
        // instruction, which is the whole of what tells the two apart here.
        if let Some(&size) = self.source[data.args].first() {
            return self.grow(inst, size);
        }
        let Extra::Mem(mem) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[mem];
        let size = u32::try_from(info.size)
            .map_err(|_| Unsupported::Dynamic { inst, growing: Growing::Huge })?;
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;

        // At least one, because the frame divides by the alignment and an object with no
        // alignment at all is one the front end had nothing to say about rather than one that may
        // go anywhere.
        let index = self.stack.locals.len();
        self.stack.locals.push(Local { size, align: info.align.max(1) });

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let lea = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.lea)));
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let made =
            self.out.build(block, lea).at(span).def(reg, self.gpr).mem(mir::Mem::at(sp)).finish();
        self.stack.addresses.push((made, index));
        Ok(())
    }

    /// The other kind of `alloca`: one whose size the function does not know until it runs, which
    /// is what a variable length array is.
    ///
    /// Nothing about it is a slot the frame laid out, because the frame is laid out once and this
    /// happens as often as control reaches the declaration. The bytes come off the stack pointer
    /// where the declaration stands, which is two instructions:
    ///
    /// ```text
    ///   sub sp, bytes     the stack pointer moves down over the memory, which is what takes it
    ///   lea reg, [sp+n]   where the memory starts, which is above the outgoing argument area
    /// ```
    ///
    /// The displacement is left at nothing for the reason the constant kind leaves its own at
    /// nothing, and for a different number: that area belongs to the arguments of whatever this
    /// function calls, it stays at the bottom of the frame wherever the bottom has moved to, and
    /// how big it is is not known until every call in the function has been seen.
    ///
    /// The bytes are already a multiple of the stack pointer's alignment by the time they arrive,
    /// because [`crate::expand::rounds`] rounded them up in the IR, so nothing here has to mask the
    /// stack pointer afterwards and the stack pointer stays somewhere a call can be made from.
    ///
    /// Two instructions here and not always two in the finished function. On a command line that
    /// asked for the stack to be touched a page at a time, the subtraction becomes a loop that
    /// walks the same distance a page at a time, which [`crate::finish`] writes. That is why the
    /// instruction is written down in [`Stack::grown`] as well as left where it is.
    ///
    /// An array wanting more alignment than the convention leaves the stack pointer with does not
    /// reach here asking for it: [`crate::expand::rounds`] gives it the alignment in extra bytes
    /// and turns the array into a `ptr_add` of the offset that lands inside them, so what arrives
    /// is a block asking for the convention's alignment like any other. The refusal below is what
    /// answers IR that came from somewhere other than that pass, since forcing the alignment here
    /// would be a second rounding of a register the frame already rounded, and after it no
    /// constant reaches the rest of the frame from anywhere. See `Growing` in [`crate::frame`].
    fn grow(&mut self, inst: Inst, size: Value) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Mem(mem) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[mem];
        if info.align > self.conv.stack_align {
            return Err(Unsupported::Dynamic { inst, growing: Growing::Aligned });
        }
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let bytes = self.reg_of(size)?;

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        let grow = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.grow)));
        let took = self
            .out
            .build(block, grow)
            .at(span)
            .operand(mir::Operand::write(stack, self.gpr))
            .operand(mir::Operand::read(stack, self.gpr))
            .operand(mir::Operand::read(bytes, self.gpr))
            .finish();
        self.stack.grown.push(took);

        let reg = self.new_reg(result);
        let lea = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.lea)));
        let sp = mir::Operand::read(stack, self.gpr);
        let made =
            self.out.build(block, lea).at(span).def(reg, self.gpr).mem(mir::Mem::at(sp)).finish();
        self.stack.dynamic.push(made);
        self.stack.grown_at.get_or_insert(inst);
        Ok(())
    }

    /// Where the stack pointer is, kept so that something later can put it back.
    ///
    /// One move out of the stack pointer and one move into it, which is the whole of what the two
    /// halves are. What makes them worth writing is where the front end puts them: a scope holding
    /// a variable length array saves the stack pointer as it opens and puts it back as it closes,
    /// so a loop declaring one takes its bytes once round rather than once per iteration, and a
    /// jump out of the scope gives the bytes back on the way out.
    ///
    /// The value travels in an ordinary register the allocator hands out, so it may be spilled like
    /// any other, and a spill slot in a frame that grows is reached through the frame pointer,
    /// which is exactly the register that still means something after the stack pointer has moved.
    fn stack_pointer(&mut self, inst: Inst, into: bool) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        let mov = x86_64::FRAME.moves(self.gpr).expect("a class the target says how to move").mov;
        let mov = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{mov}")));
        let (write, read) = if into {
            let &saved = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
            (stack, self.reg_of(saved)?)
        } else {
            let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
            (self.new_reg(result), stack)
        };
        self.out
            .build(block, mov)
            .at(span)
            .operand(mir::Operand::write(write, self.gpr))
            .operand(mir::Operand::read(read, self.gpr))
            .finish();
        // Only the write is a move of the stack pointer, and it is the one that makes the frame a
        // growing one. A read of it in a function that never writes it back is a function that
        // asked where the stack was and did nothing with the answer.
        if into {
            self.stack.grown_at.get_or_insert(inst);
        }
        Ok(())
    }

    /// Whether an instruction has an eighty bit float anywhere in it.
    ///
    /// Producing one and reading one are the same question here, because what makes one of these
    /// different from every other instruction is not the operation but where the value is. A
    /// `long double` is on the x87 stack while it is being worked on and in a frame slot the rest
    /// of the time, and neither of those is somewhere the operand of a rule could point.
    fn touches_x87(&self, inst: Inst) -> bool {
        let data = &self.source[inst];
        data.results().any(|value| on_x87(self.source[value].ty))
            || self.source[data.args].iter().any(|&arg| on_x87(self.source[arg].ty))
    }

    /// Everything that happens to an eighty bit float, as the group of instructions it is.
    ///
    /// The first six move one, and every one of those is a load, a store, or a load and a store at
    /// two different formats, because that is the whole of what this machine converts with: the
    /// x87 has no instruction that turns one thing on its stack into another, so a widening is
    /// `fld` of the narrow format and a narrowing is `fstp` of it.
    ///
    /// The rest work on one, and they are here rather than in a rule for the same reason the six
    /// are. An add is a push, a push, the add and a pop, and what passes between those four is the
    /// top of a stack nothing allocates from, so there is no value in the middle of the group for
    /// a pattern to bind or a replacement to name. The comparison is the same shape with its last
    /// two instructions folded into one opcode, which is where the byte it produces comes from.
    ///
    /// Every group leaves the stack as empty as it found it, which is what `spec/10-backend.md`
    /// section 10.8 asks of one and is why nothing in this file has to track a depth: each push
    /// below is answered by a pop a line or two later, so no two groups can ever be looking at
    /// the same eight registers.
    fn x87(&mut self, inst: Inst) -> Result<(), Unsupported> {
        match self.source[inst].opcode {
            Opcode::Load => self.x87_load(inst),
            Opcode::Store => self.x87_store(inst),
            Opcode::FPExt => self.x87_widen(inst),
            Opcode::FPTrunc => self.x87_narrow(inst),
            Opcode::SIToFP => self.x87_from_signed(inst),
            Opcode::FPToSI => self.x87_to_signed(inst),
            Opcode::FAdd => self.x87_arith(inst, "fadd_p"),
            Opcode::FSub => self.x87_arith(inst, "fsubr_p"),
            Opcode::FMul => self.x87_arith(inst, "fmul_p"),
            Opcode::FDiv => self.x87_arith(inst, "fdivr_p"),
            Opcode::FNeg => self.x87_flip(inst),
            Opcode::FCmp => self.x87_compare(inst),
            Opcode::FConst => self.x87_const(inst),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// The eighty bit parameters of a block, copied out of the addresses an edge handed over and
    /// into slots of the block's own.
    ///
    /// What crosses an edge for a value of this type is an address, because the value is sixteen
    /// bytes of the frame and no register holds any of it. The block cannot keep that address: a
    /// second edge into the same block hands over a second one, and a read after the block would
    /// then be a read of whichever edge was taken rather than of one place. So the block has a
    /// slot per parameter and the bytes are copied into it here, which is the move on an edge that
    /// every other type gets from the allocator.
    ///
    /// Every load runs before every store and the stores run backwards, so all of the values are
    /// on the x87 stack at once and nothing reads a slot another one has already written. That
    /// costs nothing in the ordinary case of one parameter and is what makes the back edge of a
    /// loop that swaps two of these work. It is also the reason for the limit: the stack is eight
    /// deep, and a block with more of these than that is refused rather than copied in an order
    /// that could be wrong.
    fn settle(&mut self, block: Block, arriving: &[(Value, mir::Reg)]) -> Result<(), Unsupported> {
        let Some(&(first, _)) = arriving.first() else { return Ok(()) };
        if arriving.len() > X87_DEPTH {
            let ty = self.source[first].ty;
            return Err(Unsupported::Phi { block, count: arriving.len(), ty });
        }
        // A block parameter comes from no instruction, so what this points at is the first thing
        // in the block, which is where a reader looking for the copy would look.
        let first_inst = self.source.insts(block).next();
        let span = first_inst.map_or(Span::DUMMY, |it| self.source.span(it));
        for &(_, reg) in arriving {
            let from = self.through(reg);
            self.x87_at("fld_t", span, from);
        }
        for &(param, _) in arriving.iter().rev() {
            let into = self.x87_slot(param);
            let into = self.through(into);
            self.x87_at("fstp_t", span, into);
        }
        Ok(())
    }

    /// The frame slot an eighty bit value lives in, as its address in a fresh register.
    ///
    /// The slot is the value's for the whole function and is taken the first time somebody asks.
    /// The address is worked out again every time, which is a `lea` per use and is deliberate: one
    /// address kept in a register from the definition to the last use would hold a general purpose
    /// register open across everything in between, and a function with a handful of these in it
    /// would spend its registers on addresses of things rather than on things.
    fn x87_slot(&mut self, value: Value) -> mir::Reg {
        // An argument of the function has a slot already and it is the caller's. The convention
        // puts the bytes in the argument area and hands over where they are, so the address that
        // arrived is the answer and no second copy of the value is made. Nothing ever writes to a
        // value of this type once it exists, so nothing writes to the caller's copy either. A
        // parameter of any other block is not this: what arrived there is an address a predecessor
        // chose, [`Lowering::settle`] has already copied the bytes out of it, and the slot those
        // bytes landed in is the one below.
        let entry = self.source.entry();
        if let (Def::Param { block, .. }, Some(reg)) =
            (self.source[value].def, self.regs[value.index()])
        {
            if entry == Some(block) {
                return reg;
            }
        }
        let index = match self.slots[value.index()] {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: X87_BYTES, align: X87_BYTES });
                self.slots[value.index()] = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// The bytes a value crosses between a register and the x87 stack through, as their address
    /// in a fresh register.
    fn x87_crossing(&mut self) -> mir::Reg {
        let index = match self.crossing {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: X87_CROSSING, align: X87_CROSSING });
                self.crossing = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// The two control words, as the address of the first of them in a fresh register.
    fn x87_control(&mut self) -> mir::Reg {
        let index = match self.control {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: 4, align: 4 });
                self.control = Some(index);
                index
            }
        };
        let block = self.at.expect("a block is being filled");
        self.frame_address(block, index)
    }

    /// An address held in a register, as the addressing mode that reaches it.
    fn through(&self, reg: mir::Reg) -> mir::Mem {
        mir::Mem::at(mir::Operand::read(reg, self.gpr))
    }

    /// One instruction of a group, which names an address and nothing else.
    ///
    /// Every x87 instruction that moves a value is one of these. What it does to the stack is in
    /// the mnemonic rather than in an operand, so there is no register to write down and no
    /// register the allocator gets a say in.
    fn x87_at(&mut self, name: &str, span: Span, at: mir::Mem) {
        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        self.out.build(block, opcode).at(span).mem(at).finish();
    }

    /// The one instruction of a group that reaches the program's own memory.
    ///
    /// A `long double` moves in two instructions with a frame slot at one end of them, and the
    /// other end is the address the program wrote. That end is the access, so it is the one that
    /// carries what the program said about it, and the trip through the slot is this compiler's
    /// own business the way a spill is. See [`Self::carried`].
    fn x87_touching(&mut self, name: &str, inst: Inst, at: mir::Mem) {
        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        self.out.build(block, opcode).at(span).flags(flags).mem(at).finish();
    }

    /// One instruction of a group that names nothing at all.
    ///
    /// The arithmetic is these. Both of an add's operands are already on the stack when it runs
    /// and so is where the answer goes, and the stack is not somewhere an instruction says, so
    /// `faddp` has an argument in the assembler's syntax and nothing here for the argument to come
    /// from. What it works on is which two pushes came before it, which is a fact about the order
    /// of the group and is why the group is written in one place.
    fn x87_only(&mut self, name: &str, span: Span) {
        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        self.out.build(block, opcode).at(span).finish();
    }

    /// A `load` of a `long double`: onto the stack from where it was, and off it into the slot.
    ///
    /// Two instructions rather than the two general purpose moves the same sixteen bytes would
    /// take, because `fld` and `fstp` at this format neither convert nor look: the value goes on
    /// in the format it was already in and comes back off in it, so a signalling NaN stays one
    /// and nothing is raised. Which is what makes this a copy at all.
    fn x87_load(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &address = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.reg_of(address)?;
        let from = self.through(from);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_touching("fld_t", inst, from);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A `store` of a `long double`: the same pair the other way round.
    fn x87_store(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let args = self.source[self.source[inst].args].to_vec();
        let [value, address] = args[..] else { return Err(self.unsupported(inst)) };
        let span = self.source.span(inst);
        let from = self.x87_slot(value);
        let from = self.through(from);
        let into = self.reg_of(address)?;
        let into = self.through(into);
        self.x87_at("fld_t", span, from);
        self.x87_touching("fstp_t", inst, into);
        Ok(())
    }

    /// A `float`, a `double` or an integer becoming a `long double`.
    ///
    /// Through memory, because the x87 reads memory and nothing else: the value is in a register
    /// the machine has and the unit has no way to be handed one, so it is written to the crossing
    /// bytes and loaded back at the format that widens it. Every one of these is exact. Sixty four
    /// bits of significand and fifteen of exponent hold every `float`, every `double` and every
    /// sixty four bit integer outright, so none of the four can round and none can raise.
    fn x87_across(
        &mut self,
        inst: Inst,
        put: &'static str,
        class: RegClass,
        get: &'static str,
    ) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let value = self.reg_of(source)?;
        let across = self.x87_crossing();
        let across = self.through(across);
        let into = self.x87_slot(result);
        let into = self.through(into);

        let block = self.at.expect("a block is being filled");
        let store = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{put}")));
        self.out.build(block, store).at(span).uses(value, class).mem(across).finish();
        self.x87_at(get, span, across);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A `long double` becoming a `float`, a `double` or an integer.
    ///
    /// Through memory for the reason above and in the same three instructions backwards. The two
    /// that go to a float round to nearest, which is what the control word says unless somebody
    /// has changed it and is what C wants. The two that go to an integer do not, which is why they
    /// do not come here.
    fn x87_back(
        &mut self,
        inst: Inst,
        put: &'static str,
        get: &'static str,
        class: RegClass,
    ) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.x87_slot(source);
        let from = self.through(from);
        let across = self.x87_crossing();
        let across = self.through(across);

        self.x87_at("fld_t", span, from);
        self.x87_at(put, span, across);
        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let load = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{get}")));
        self.out.build(block, load).at(span).def(reg, class).mem(across).finish();
        Ok(())
    }

    /// An `fpext` up to a `long double`, which is the only direction this machine has one in.
    fn x87_widen(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let sse = self.conv.sse_class;
        match self.source[self.narrow(inst)?].ty.bits() {
            32 => self.x87_across(inst, "movss_mr", sse, "fld_s"),
            64 => self.x87_across(inst, "movsd_mr", sse, "fld_l"),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// An `fptrunc` down from a `long double`, which is the other direction of the same.
    fn x87_narrow(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let sse = self.conv.sse_class;
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        match self.source[result].ty.bits() {
            32 => self.x87_back(inst, "fstp_s", "movss_rm", sse),
            64 => self.x87_back(inst, "fstp_l", "movsd_rm", sse),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// A `sitofp` up to a `long double`.
    ///
    /// Thirty two bits and sixty four, and nothing narrower, because C widens an integer to `int`
    /// before it converts one and the front end writes that widening down. An unsigned integer is
    /// not here at all: `fild` reads its operand as signed, so a value above the signed range
    /// comes back short by two to the sixty fourth and has to be added back, which is arithmetic
    /// rather than a move and waits with the rest of it.
    fn x87_from_signed(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let gpr = self.gpr;
        match self.source[self.narrow(inst)?].ty.bits() {
            32 => self.x87_across(inst, "mov_mr_32", gpr, "fild_l"),
            64 => self.x87_across(inst, "mov_mr_64", gpr, "fild_ll"),
            _ => Err(self.unsupported(inst)),
        }
    }

    /// An `fptosi` down from a `long double`, which is the one conversion here with no single
    /// instruction behind it.
    ///
    /// C cuts towards zero and the unit rounds the way its control word says, so the store that
    /// takes the value off the stack is wrapped in the control word being saved, changed and put
    /// back. Five instructions around the one that does the work, and three more moving the word
    /// through a register, because this machine has no way to OR a constant into memory at this
    /// width. The unit has a shorter answer in `fisttp`, and `spec/10-backend.md` section 10.8
    /// says why it is not used: it is SSE3, the x86-64 baseline is not, and there is nothing here
    /// that can gate an instruction on a feature yet.
    fn x87_to_signed(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let (put, get) = match self.source[result].ty.bits() {
            32 => ("fistp_l", "mov_rm_32"),
            64 => ("fistp_ll", "mov_rm_64"),
            _ => return Err(self.unsupported(inst)),
        };
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let from = self.x87_slot(source);
        let from = self.through(from);
        let across = self.x87_crossing();
        let across = self.through(across);
        let control = self.x87_control();
        let saved = self.through(control).plus(0);
        let cut = self.through(control).plus(2);

        // The word the unit has now, into the first of the two slots and into a register, with the
        // rounding field turned to truncate on the way to the second.
        self.x87_at("fnstcw", span, saved);
        let block = self.at.expect("a block is being filled");
        let was = self.out.new_vreg(gpr);
        let read = mir::Opcode::new(self.names.intern("x64.mov_rm_16"));
        self.out.build(block, read).at(span).def(was, gpr).mem(saved).finish();
        let now = self.out.new_vreg(gpr);
        let set = mir::Opcode::new(self.names.intern("x64.or_ri_16"));
        // Two address, which is written out here rather than taken from the two shorthands
        // because the shorthands leave an operand unconstrained: this machine ORs into the
        // register it read, so the two have to be the same one and only the constraint says so.
        self.out
            .build(block, set)
            .at(span)
            .operand(mir::Operand::write(now, gpr).with(Constraint::Reuse(1)))
            .operand(mir::Operand::read(was, gpr))
            .imm(X87_TRUNCATE)
            .finish();
        let write = mir::Opcode::new(self.names.intern("x64.mov_mr_16"));
        self.out.build(block, write).at(span).uses(now, gpr).mem(cut).finish();

        // The conversion itself, under the changed word, and then the word the unit had put back
        // before anything else runs.
        self.x87_at("fldcw", span, cut);
        self.x87_at("fld_t", span, from);
        self.x87_at(put, span, across);
        self.x87_at("fldcw", span, saved);

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let load = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{get}")));
        self.out.build(block, load).at(span).def(reg, gpr).mem(across).finish();
        Ok(())
    }

    /// A constant of this type, as the bits of it written into its slot.
    ///
    /// No x87 instruction at all, which is the surprise here. A slot holding an eighty bit value is
    /// the value, so a constant is ten bytes put where the value lives, and the unit never has to
    /// see it: whatever reads it will `fld` it out of the slot the way it reads any other one.
    ///
    /// Ten bytes in two goes, because the machine stores eight at a time and there is no store of
    /// an immediate to memory, so each half is put in a register first. The six bytes above the ten
    /// are left alone, since nothing reads them: they are the padding that makes the type sixteen
    /// wide and they are unspecified in the psABI rather than zero.
    ///
    /// The other way is a constant pool, an `fldt` of a symbol, and a relocation, which is what a
    /// compiler with somewhere to put a literal does. This back end has nowhere to put one yet, and
    /// four instructions in the frame is what that costs until it does.
    fn x87_const(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Imm(imm) = self.source[inst].extra else { return Err(self.unsupported(inst)) };
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let bits = self.source[imm].bits();
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let slot = self.x87_slot(result);
        let low = self.through(slot).plus(0);
        let high = self.through(slot).plus(8);

        let block = self.at.expect("a block is being filled");
        for (bytes, at, into) in
            [(bits as u64 as i64, low, "64"), (((bits >> 64) & 0xffff) as i64, high, "16")]
        {
            let held = self.out.new_vreg(gpr);
            let put = mir::Opcode::new(self.names.intern(&format!("{PREFIX}mov_ri_{into}")));
            self.out.build(block, put).at(span).def(held, gpr).imm(bytes).finish();
            let store = mir::Opcode::new(self.names.intern(&format!("{PREFIX}mov_mr_{into}")));
            self.out.build(block, store).at(span).uses(held, gpr).mem(at).finish();
        }
        Ok(())
    }

    /// One arithmetic instruction on two eighty bit values, as the four it takes.
    ///
    /// The left operand is pushed first and the right one on top of it, so the left ends up
    /// underneath and the answer wanted is the one below against the top in that order. Which of
    /// the two mnemonics computes that is a question about the spelling rather than about the
    /// machine, and the two spellings disagree. Intel's `FSUBP ST(i), ST(0)` is `ST(i) - ST(0)`
    /// and is `DE E8+i`, and AT&T's `fsubp` is `DE E0+i`, which is the other subtraction. This
    /// compiler writes AT&T and encodes what gas encodes, so what it asks for here is `fsubr_p`
    /// and `fdivr_p`, and the `r` is not a reversal of anything the code generator decided.
    ///
    /// An addition and a multiplication have one form each and do not care, which is why a test
    /// that reads the mnemonic back would not have caught this and one that computes a subtraction
    /// and checks the answer does.
    ///
    /// The answer is left where the deeper of the two was and the shallower is gone, which is what
    /// the `p` on the mnemonic means, so one push has already been paid back by the time the
    /// `fstp` runs and the stack is level again after it.
    ///
    /// Nothing here is folded and nothing is reused. Two values that are the same value get two
    /// pushes of the same slot, and an operand that was just computed is read back out of the slot
    /// it was written to rather than left on the stack, which costs a store and a load per
    /// instruction in an expression. Keeping a partial result on the stack across the next
    /// instruction's operands means knowing how deep the stack is at every point in the block, and
    /// that is a different thing from writing a group.
    fn x87_arith(&mut self, inst: Inst, with: &'static str) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let [left, right] = args[..] else { return Err(self.unsupported(inst)) };
        let span = self.source.span(inst);
        let left = self.x87_slot(left);
        let left = self.through(left);
        let right = self.x87_slot(right);
        let right = self.through(right);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_at("fld_t", span, left);
        self.x87_at("fld_t", span, right);
        self.x87_only(with, span);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A negation, which is a push, the sign bit turned over and a pop.
    ///
    /// `fchs` does not read the value as a number, so this is right for a zero, for an infinity
    /// and for a NaN, and it raises nothing on any of them. Which is what C asks of a negation and
    /// is not what subtracting from zero would give: `0.0L - x` is a different answer at a
    /// negative zero and a signalling one at a NaN.
    fn x87_flip(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let (args, result) = self.ends(inst)?;
        let &source = args.first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let from = self.x87_slot(source);
        let from = self.through(from);
        let into = self.x87_slot(result);
        let into = self.through(into);
        self.x87_at("fld_t", span, from);
        self.x87_only("fchs", span);
        self.x87_at("fstp_t", span, into);
        Ok(())
    }

    /// A comparison of two eighty bit values, as the two pushes and the one opcode that reads them.
    ///
    /// The right operand is pushed first and the left one on top of it, which is the other way
    /// round from the arithmetic and is because `fucomip` asks about the top against what is under
    /// it: the comparison this machine can do is the top's, so the value the predicate is about
    /// has to be the top. The pop that gets the loser off the stack and the byte that reads the
    /// flags are both inside the opcode, since what passes between those and the comparison is the
    /// flags and the flags are not something anything here can name.
    ///
    /// Which of the ten opcodes, and which way round, is the same table the vector comparisons
    /// match against in `rules/x86-64.rules`, and it has to stay the same table: a predicate that
    /// picked a different condition here than there would be a `long double` comparison that
    /// disagreed with the `double` comparison of the same two numbers, which is the one thing a
    /// wider format is not allowed to do.
    ///
    /// The always false and the always true are refused rather than folded into a constant,
    /// because a comparison this machine never has to do is one the optimizer should have removed
    /// and an instruction here that quietly agreed with it would hide that it did not.
    fn x87_compare(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::FloatPred(pred) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let (args, result) = self.ends(inst)?;
        let [left, right] = args[..] else { return Err(self.unsupported(inst)) };
        // Two of the fourteen need a second byte and an instruction to put the two together,
        // because they are two conditions at once: an ordered equal is equal and not unordered,
        // and an unordered not equal is either. The opcode carries all of that and says here only
        // that it writes somewhere else as well.
        let (name, reversed, both) = match pred {
            FloatPred::Ogt => ("fucomip_set_a", false, false),
            FloatPred::Oge => ("fucomip_set_ae", false, false),
            FloatPred::Olt => ("fucomip_set_a", true, false),
            FloatPred::Ole => ("fucomip_set_ae", true, false),
            FloatPred::One => ("fucomip_set_ne", false, false),
            FloatPred::Ord => ("fucomip_set_np", false, false),
            FloatPred::Uno => ("fucomip_set_p", false, false),
            FloatPred::Ueq => ("fucomip_set_e", false, false),
            FloatPred::Ult => ("fucomip_set_b", false, false),
            FloatPred::Ule => ("fucomip_set_be", false, false),
            FloatPred::Ugt => ("fucomip_set_b", true, false),
            FloatPred::Uge => ("fucomip_set_be", true, false),
            FloatPred::Oeq => ("fucomip_set_e_and_np", false, true),
            FloatPred::Une => ("fucomip_set_ne_or_p", false, true),
            FloatPred::False | FloatPred::True => return Err(self.unsupported(inst)),
        };
        let (top, under) = if reversed { (right, left) } else { (left, right) };

        let span = self.source.span(inst);
        let gpr = self.gpr;
        let under = self.x87_slot(under);
        let under = self.through(under);
        let top = self.x87_slot(top);
        let top = self.through(top);
        self.x87_at("fld_t", span, under);
        self.x87_at("fld_t", span, top);

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        // Taken before the instruction is started rather than inside it, since both come from the
        // same function being built and only one thing at a time may be adding to it.
        let spare = both.then(|| self.out.new_vreg(gpr));
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        let mut build = self.out.build(block, opcode).at(span).def(reg, gpr);
        if let Some(spare) = spare {
            build = build.def(spare, gpr);
        }
        build.finish();
        Ok(())
    }

    /// The operands and the one result of an instruction that has exactly one.
    fn ends(&self, inst: Inst) -> Result<(&'a [Value], Value), Unsupported> {
        let data = &self.source[inst];
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        Ok((&self.source[data.args], result))
    }

    /// The operand of a conversion, which is the end of it that is not the `long double`.
    fn narrow(&self, inst: Inst) -> Result<Value, Unsupported> {
        let args = &self.source[self.source[inst].args];
        args.first().copied().ok_or_else(|| self.unsupported(inst))
    }

    /// One `va_start`, as the fields of the list it was handed.
    ///
    /// On the four field list, two of them are numbers this already knows, and each costs an
    /// instruction to put in a register before it can be stored, because the machine here has no
    /// store of an immediate to memory. The other two are addresses in the frame, and each is a
    /// `lea` [`crate::finish`] finishes: the save area is one of the function's own stack objects,
    /// and the caller's argument area is where the parameters that had no register came from, which
    /// is the same place and the same fixup a parameter past the sixth already uses.
    ///
    /// On the list that is a pointer it is the second of those four and nothing else, since the
    /// whole of what that list says is where the walk is and the walk starts at the first argument
    /// the signature does not name. One `lea` and one store.
    ///
    /// What is written is exactly the fields [`crate::varargs`] describes, in the order they are
    /// laid out, so that reading this beside that table is the whole of the check.
    fn va_start(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Some(&list) = self.source[self.source[inst].args].first() else {
            return Err(self.unsupported(inst));
        };
        let started = self.varargs.ok_or_else(|| self.unsupported(inst))?;
        let list = self.reg_of(list)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);

        let (save, incoming) = match started {
            Varargs::Pointer { incoming } => (None, incoming),
            Varargs::Fields { save, incoming, integers, floats } => {
                for (at, count) in [(varargs::GP_OFFSET, integers), (varargs::FP_OFFSET, floats)] {
                    let held = self.out.new_vreg(self.gpr);
                    let load = mir::Opcode::new(self.names.intern("x64.mov_ri_32"));
                    let build = self.out.build(block, load).at(span);
                    build.def(held, self.gpr).imm(i64::from(count)).finish();

                    let store = mir::Opcode::new(self.names.intern("x64.mov_mr_32"));
                    let mem = self.field(list, at);
                    self.out.build(block, store).at(span).uses(held, self.gpr).mem(mem).finish();
                }
                (Some(save), incoming)
            }
        };

        // The first argument the signature did not name, which is as far up the caller's argument
        // area as the ones it did name reached. Nothing here knows where that area is, so the
        // distance is recorded the way a parameter read out of it is and finished with it.
        let overflow = self.out.new_vreg(self.gpr);
        let lea = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.lea)));
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let made = self
            .out
            .build(block, lea)
            .at(span)
            .def(overflow, self.gpr)
            .mem(mir::Mem::at(sp))
            .finish();
        self.stack.arguments.push((made, incoming));

        // At the front of the list when that address is the whole of it, and at the field the
        // layout gives it when there are four, with the save area behind it.
        let fields = match save {
            None => vec![(0, overflow)],
            Some(save) => {
                let save = self.frame_address(block, save);
                vec![(varargs::OVERFLOW, overflow), (varargs::SAVE_AREA, save)]
            }
        };
        for (at, held) in fields {
            let store = mir::Opcode::new(self.names.intern("x64.mov_mr_64"));
            let mem = self.field(list, at);
            self.out.build(block, store).at(span).uses(held, self.gpr).mem(mem).finish();
        }
        Ok(())
    }

    /// One field of a list, as the addressing mode that reaches it.
    fn field(&self, list: mir::Reg, at: i64) -> mir::Mem {
        let base = mir::Operand::read(list, self.gpr);
        mir::Mem::at(base).plus(i32::try_from(at).expect("a field of a list is a small offset"))
    }

    /// The address of a name: one `lea` off the instruction pointer, with the name on it.
    ///
    /// The same instruction an `alloca` gets and for a related reason. An address that is not in
    /// the program is a `lea` of an addressing mode that names no register, and the mode carries
    /// the symbol so that [`rucc_asm`] can write it relative to `%rip` and leave the relocation
    /// for the assembler. Both halves of that already existed: the printer writes `sym(%rip)` and
    /// the encoder emits the relocation, because a call to a name the file does not define needed
    /// them first.
    ///
    /// One `mov` and not one `lea` when the name is one [`Elsewhere`] holds, because the distance
    /// the `lea` adds to the instruction pointer is a number only a link that puts the name in
    /// this program can work out, and the address of a function this file merely declares is not
    /// such a number. The load reads the address out of the slot the linker fills in instead. The
    /// linker turns it back into the `lea` when the name turns out to have been here all along,
    /// so this is not slower in the case that was already right.
    ///
    /// There is deliberately no name for this in [`crate::term`], which is what stops the address
    /// being folded into the instruction that reads it. Folding it is the right thing to do and
    /// is what turns a load of a global from two instructions into one, but it is a separate
    /// question about addressing modes and issue #282 is it. Until then the address is in a
    /// register before anything uses it, which is correct and one instruction longer.
    ///
    /// What this does not do is give the name anything to refer to. A module carries its globals
    /// and nothing writes them out, so a file that defines the variable it reads compiles to a
    /// reference the linker cannot resolve. Issue #293 is the other half.
    ///
    /// A thread-local variable is neither of the two above and is [`Self::thread_address`].
    fn address_of(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Symbol(symbol) = data.extra else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        if self.elsewhere.thread(symbol) {
            return self.thread_address(inst, symbol, result);
        }

        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let (mnemonic, mem) = if self.elsewhere.holds(symbol) {
            (GOT_LOAD, mir::Mem::got(symbol))
        } else {
            (x86_64::FRAME.lea, mir::Mem::of(symbol))
        };
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{mnemonic}")));
        self.out.build(block, opcode).at(span).def(reg, self.gpr).mem(mem).finish();
        Ok(())
    }

    /// The address of a thread-local variable, which is this thread's copy of it.
    ///
    /// Neither instruction the ordinary case writes would mean anything here. There is no distance
    /// to the variable for a `lea` to add, because there is no variable: there is one copy of it per
    /// thread and they are at different addresses, so a link asked for the distance to the name
    /// refuses rather than picking one. And there is no address for a table slot to hold either, for
    /// the same reason.
    ///
    /// What is the same in every thread is where the variable sits inside the block of storage a
    /// thread gets, so that offset is what the link writes down, and the address of the running
    /// thread's block is what turns it into an address. x86-64 keeps that address in `%fs`, at the
    /// front of the block, so the whole of this is three instructions:
    ///
    /// ```text
    /// movq  x@gottpoff(%rip), %off   # how far into the block x sits, which the link fills in
    /// movq  %fs:0, %tp               # where this thread's block is, which only the machine knows
    /// addq  %tp, %off                # this thread's copy of x
    /// ```
    ///
    /// That is the initial exec model. It is one instruction longer than what gcc writes at `-O2`
    /// in an executable, which folds the addition into the instruction that uses the address, and
    /// the difference is issue #282 rather than anything about threads: nothing here folds an
    /// address into its reader yet. The link relaxes the first instruction into an immediate when it
    /// is making an executable, since it lays the blocks out and therefore knows the number, so the
    /// table slot costs nothing in the case that is common.
    ///
    /// It is not the most general model. A library loaded by `dlopen` gets its storage after the
    /// program is already running, and the block this reaches was laid out before it started, so
    /// the loader has to find room in that block for the library's variables. glibc keeps a little
    /// spare room for exactly this and a library that fits in it loads and runs; one that does not
    /// fails to load, with a message saying so. The model with no such limit calls `__tls_get_addr`
    /// and is what gcc writes under `-fPIC` by default, and it is issue #1104.
    ///
    /// So this is the model gcc writes under `-ftls-model=initial-exec`: right for an executable,
    /// right for a library the program is linked against, and a load that either works or is
    /// refused out loud for a library something opens later. What it is never is quietly wrong.
    fn thread_address(
        &mut self,
        inst: Inst,
        symbol: Symbol,
        result: Value,
    ) -> Result<(), Unsupported> {
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let gpr = self.gpr;
        let load = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{GOT_LOAD}")));

        let offset = self.out.new_vreg(gpr);
        self.out
            .build(block, load)
            .at(span)
            .def(offset, gpr)
            .mem(mir::Mem::thread(symbol))
            .finish();
        // The front of the block, which is the one thing on this machine that no instruction can
        // work out: `%fs` is not a register a program can read, and what it points at is a word
        // holding its own address, so reading through it at zero is how the address is come by.
        let pointer = self.out.new_vreg(gpr);
        let at = mir::Mem::in_segment(Segment::Fs, 0);
        self.out.build(block, load).at(span).def(pointer, gpr).mem(at).finish();

        // Two address, spelled out for the reason `x87_to_int` gives: this machine adds into the
        // register it read, and only the constraint says the two are the same one.
        let reg = self.new_reg(result);
        let add = mir::Opcode::new(self.names.intern(&format!("{PREFIX}add_rr_64")));
        self.out
            .build(block, add)
            .at(span)
            .operand(mir::Operand::write(reg, gpr).with(Constraint::Reuse(1)))
            .operand(mir::Operand::read(offset, gpr))
            .operand(mir::Operand::read(pointer, gpr))
            .finish();
        Ok(())
    }

    /// `&&label`, GNU's address of a label, which is the same `lea` a global gets against a place
    /// in this same function.
    ///
    /// What the two have in common is the whole of the instruction: an address worked out from
    /// where the instruction is, which is what `(%rip)` means and is the only way this compiler
    /// reaches anything. What they do not have in common is what fills the four bytes in. A
    /// global is a name, so the number is a relocation and the linker writes it. A block is a
    /// place in this function, so both ends are in one section and the number is known as soon as
    /// the blocks have been laid out, which is why `rucc_asm` fills it in the way it fills in a
    /// jump rather than leaving a relocation behind.
    ///
    /// Nothing here says the block is one control can arrive at. That is said by the
    /// [`Opcode::IndirectBr`] that reads the address, which lists every block it can arrive at,
    /// and by nothing else: an address on its own is a number.
    fn block_address(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let Some(call) = self.source.successors(inst).next() else {
            return Err(self.unsupported(inst));
        };
        let block = self.at.expect("a block is being filled");
        let reg = self.new_reg(result);
        let span = self.source.span(inst);
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.lea)));
        let mem = mir::Mem::block(self.out_block(call.block));
        self.out.build(block, opcode).at(span).def(reg, self.gpr).mem(mem).finish();
        Ok(())
    }

    /// `goto *p`, GNU's computed goto, which is a jump through a register.
    ///
    /// Where it goes is not written here and cannot be. Every block it can arrive at is on the
    /// block this ends, the way every other arm is, and which of them the address holds is decided
    /// while the program runs. So this is one instruction with one operand, and the arms are
    /// copied across by [`Self::edges`] like anybody else's.
    fn indirect_branch(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let &address = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let reg = self.reg_of(address)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let name = x86_64::BRANCH.indirect;
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        self.out.build(block, opcode).at(span).operand(mir::Operand::read(reg, self.gpr)).finish();
        Ok(())
    }

    /// `__builtin_setjmp`, which writes down where the function is so that a `__builtin_longjmp`
    /// somewhere else can bring control back here, and answers zero on the way past.
    ///
    /// Four words of the buffer, the three gcc writes and one of this compiler's own, and then the
    /// block ends: everything after the save in the IR block is put into a new machine IR block,
    /// and the address of that block is what went into the buffer. That is the whole reason the
    /// block is split here. An address points at a label, a machine IR block is the only thing in
    /// this representation that has one, and a save is in the middle of a block rather than at the
    /// end of one.
    ///
    /// # How the answer gets back
    ///
    /// Through the frame rather than through a register. The save writes a zero into a word of its
    /// own frame, puts the address of that word in the buffer, and the new block reads the word
    /// back. The restore writes a one through the address it finds in the buffer before it goes.
    /// So one load answers zero on the way past and one on the way back, and neither path has to
    /// agree with the other about a register.
    ///
    /// gcc does it the other way round, with a second block that sets the answer to one and is
    /// what the restore arrives at. That block is one nothing in the function jumps to, and a
    /// machine IR whose blocks are walked from the entry has nowhere to put such a thing: the
    /// allocator lays a function out in the line it is going to be emitted in, and a block no edge
    /// reaches is not in that line. The word in the frame costs eight bytes of stack and one load,
    /// and it needs nothing said anywhere about a block arrived at from outside.
    ///
    /// # What the allocator is told
    ///
    /// That every register it hands out is gone at the end of the first block. That is what makes
    /// the rest of the function right on the way back: control arrives from a `__builtin_longjmp`
    /// in some other function, and the only two registers that puts back are the stack pointer and
    /// the frame pointer, so anything this function still wants has to be in the frame those two
    /// reach. It is said with a write of every one of those registers, which is the same thing a
    /// call says about the registers a callee may destroy, on an instruction with nothing else on
    /// it so that the stores above are not caught up in it.
    fn saves_place(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let &buffer = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let buf = self.reg_of(buffer)?;
        let at = self.at.expect("a block is being filled");
        let gpr = self.gpr;
        let moves = x86_64::FRAME.moves(gpr).expect("a class the target says how to move");
        let store = self.named(moves.store);
        let load = self.named(moves.load);
        let lea = self.named(x86_64::FRAME.lea);
        let put = self.named(x86_64::FRAME.imm);
        let nothing = x86_64::FRAME.pad.expect("a target with an instruction that does nothing");
        let nothing = self.named(nothing);
        self.stack.saves_place = true;
        let answer = self.answer_slot();
        let back = self.out.create_block();

        // The zero this answers with, into the word a restore writes a one into.
        let zero = self.out.new_vreg(gpr);
        self.out.build(at, put).at(span).def(zero, gpr).imm(0).finish();
        let mem = self.frame_mem();
        let made = self.out.build(at, store).at(span).uses(zero, gpr).mem(mem).finish();
        self.stack.addresses.push((made, answer));

        // The four words: where that word is, where control comes back to, and the two registers
        // the restore puts back.
        let found = self.frame_address(at, answer);
        self.write_word(at, span, store, found, buf, JUMP_ANSWER);
        let pc = self.out.new_vreg(gpr);
        self.out.build(at, lea).at(span).def(pc, gpr).mem(mir::Mem::block(back)).finish();
        self.write_word(at, span, store, pc, buf, JUMP_PC);
        let frame = mir::Reg::physical(self.conv.frame_pointer);
        self.write_word(at, span, store, frame, buf, JUMP_FRAME);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        self.write_word(at, span, store, stack, buf, JUMP_STACK);

        // Nothing is in a register past this point, which is what the rest of the function is
        // allowed to assume about the way back in.
        let gone = self.across_jump();
        let mut build = self.out.build(at, nothing).at(span);
        for (reg, class) in gone {
            build = build.operand(mir::Operand::write(reg, class));
        }
        build.finish();

        // And the rest of the block, which is the block the address above was of.
        *self.out.succs_mut(at) = vec![mir::BlockCall::to(back)];
        self.at = Some(back);
        let reg = self.new_reg(result);
        let mem = self.frame_mem();
        let made = self.out.build(back, load).at(span).def(reg, gpr).mem(mem).finish();
        self.stack.addresses.push((made, answer));
        Ok(())
    }

    /// `__builtin_longjmp`, which reads a buffer a `__builtin_setjmp` filled in and goes there.
    ///
    /// Everything comes out of the buffer before anything is put back, and the four registers it
    /// comes out into are physical ones rather than values the allocator places. Both of those are
    /// about the same moment. The stack pointer is one of the things being put back, a value the
    /// allocator sent to the stack is reached through the stack pointer, and between the
    /// instruction that moves it and the jump there is no stack this function owns any more. A
    /// register named outright is a register nothing reloads into and nothing else is in, which is
    /// the only way to hold something across that moment.
    ///
    /// Four of them because that is how many things are in the air at once: where to go, the frame
    /// pointer to put back, the one the matching save is to answer with, and one register used
    /// twice, first for the address that one is written through and then for the stack pointer.
    ///
    /// Nothing after this in the block is reached. The marker is not a terminator, for the reason
    /// `spec/08-ir.md` gives, so the block goes on and whatever the front end wrote after it is
    /// written out and never run.
    fn comes_back(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let &buffer = self.source[data.args].first().ok_or_else(|| self.unsupported(inst))?;
        let span = self.source.span(inst);
        let buf = self.reg_of(buffer)?;
        let at = self.at.expect("a block is being filled");
        let gpr = self.gpr;
        let moves = x86_64::FRAME.moves(gpr).expect("a class the target says how to move");
        let load = self.named(moves.load);
        let store = self.named(moves.store);
        let mov = self.named(moves.mov);
        let put = self.named(x86_64::FRAME.imm);
        let jump = self.named(x86_64::BRANCH.indirect);

        let held = self.jump_regs();
        if held.len() < JUMP_REGS {
            return Err(self.unsupported(inst));
        }
        let pc = mir::Reg::physical(held[0]);
        let frame = mir::Reg::physical(held[1]);
        let spare = mir::Reg::physical(held[2]);
        let one = mir::Reg::physical(held[3]);

        self.read_word(at, span, load, pc, buf, JUMP_PC);
        self.read_word(at, span, load, frame, buf, JUMP_FRAME);
        self.read_word(at, span, load, spare, buf, JUMP_ANSWER);

        // What the matching save answers with, written through the address that came out of the
        // buffer, because the word it goes in is in the other function's frame and this one has no
        // way of knowing where that is.
        self.out.build(at, put).at(span).def(one, gpr).imm(1).finish();
        let mem = mir::Mem::at(mir::Operand::read(spare, gpr));
        self.out.build(at, store).at(span).uses(one, gpr).mem(mem).finish();

        // The stack last of the four, so that the register the buffer is reached through is done
        // with before the stack it may have been spilled to stops being this function's.
        self.read_word(at, span, load, spare, buf, JUMP_STACK);
        let stack = mir::Reg::physical(self.conv.stack_pointer);
        self.copy(at, span, mov, stack, spare);
        let base = mir::Reg::physical(self.conv.frame_pointer);
        self.copy(at, span, mov, base, frame);

        // And the jump, which reads the two registers just put back as well as the address it
        // goes through. Neither of those is printed, because the target's spelling of an indirect
        // jump has one argument and it is the first one read. They are there because the code
        // control arrives at reaches its frame through them, and because without them the two
        // instructions above write registers nothing reads: a scheduler is then free to put the
        // jump in front of them, and at `-O2` it does.
        self.out
            .build(at, jump)
            .at(span)
            .operand(mir::Operand::read(pc, gpr))
            .operand(mir::Operand::read(stack, gpr))
            .operand(mir::Operand::read(base, gpr))
            .finish();
        Ok(())
    }

    /// One word of the buffer of a `__builtin_setjmp`, written from a register.
    fn write_word(
        &mut self,
        at: mir::Block,
        span: Span,
        store: mir::Opcode,
        from: mir::Reg,
        buf: mir::Reg,
        word: i32,
    ) {
        let mem = mir::Mem::at(mir::Operand::read(buf, self.gpr)).plus(word);
        self.out.build(at, store).at(span).uses(from, self.gpr).mem(mem).finish();
    }

    /// One word of that buffer, read back into a register.
    fn read_word(
        &mut self,
        at: mir::Block,
        span: Span,
        load: mir::Opcode,
        into: mir::Reg,
        buf: mir::Reg,
        word: i32,
    ) {
        let mem = mir::Mem::at(mir::Operand::read(buf, self.gpr)).plus(word);
        self.out.build(at, load).at(span).def(into, self.gpr).mem(mem).finish();
    }

    /// One register into another, which is the one shape of instruction the builder has no word
    /// for because neither operand is a definition of a value or a read of memory.
    fn copy(
        &mut self,
        at: mir::Block,
        span: Span,
        mov: mir::Opcode,
        into: mir::Reg,
        from: mir::Reg,
    ) {
        self.out
            .build(at, mov)
            .at(span)
            .operand(mir::Operand::write(into, self.gpr))
            .operand(mir::Operand::read(from, self.gpr))
            .finish();
    }

    /// The word a `__builtin_setjmp` in this function answers with, asked for once and kept.
    fn answer_slot(&mut self) -> usize {
        match self.answer {
            Some(index) => index,
            None => {
                let index = self.stack.locals.len();
                self.stack.locals.push(Local { size: JUMP_WORD, align: JUMP_WORD });
                self.answer = Some(index);
                index
            }
        }
    }

    /// An address in this function's frame with nothing in its displacement, which is what an
    /// instruction reaching one of its stack objects is written with until [`crate::finish`] knows
    /// where the object is.
    fn frame_mem(&self) -> mir::Mem {
        mir::Mem::at(mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr))
    }

    /// Every register the allocator hands out, which is what a `__builtin_setjmp` destroys.
    ///
    /// Both files, since a `double` live across a save has the same problem an integer does. The
    /// two registers a frame is reached through are not here: the restore puts both of them back,
    /// which is the whole of what it puts back, and a function whose frame pointer was destroyed
    /// by its own save would have nothing left to find its caller with.
    fn across_jump(&self) -> Vec<(mir::Reg, RegClass)> {
        let mut gone = Vec::new();
        for &reg in self.conv.int_order {
            if reg == self.conv.stack_pointer || reg == self.conv.frame_pointer {
                continue;
            }
            gone.push((mir::Reg::physical(reg), self.gpr));
        }
        for &reg in self.conv.sse_order {
            gone.push((mir::Reg::physical(reg), self.conv.sse_class));
        }
        gone
    }

    /// The registers a `__builtin_longjmp` may hold things in while it puts a frame back.
    ///
    /// The ones the allocator hands out, less the two a frame is reached through. The scratch
    /// registers are not among them on purpose: the rewriter writes a reload into one of those
    /// wherever it likes, and one of these has to survive from the load that fills it to the
    /// instruction that reads it however many instructions apart those are.
    fn jump_regs(&self) -> Vec<PhysReg> {
        self.conv
            .int_order
            .iter()
            .copied()
            .filter(|&reg| {
                reg != self.conv.stack_pointer
                    && reg != self.conv.frame_pointer
                    && !crate::pipeline::SCRATCH.contains(&reg)
            })
            .collect()
    }

    /// A machine opcode of this target from the name the target gives it.
    fn named(&mut self, name: &str) -> mir::Opcode {
        mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")))
    }

    /// `__builtin_frame_address` and `__builtin_return_address`, which are a walk up the chain of
    /// saved frame pointers and then one thing read at the end of it.
    ///
    /// Every frame that kept a frame pointer holds the caller's at the address the register points
    /// at, and the address that frame returns to one word above that, which is where the call
    /// instruction put it and where the prologue's push left it. So the walk is a load through the
    /// register for each link, the frame address is wherever the walk stopped, and the return
    /// address is one more load from a word above it. gcc 16.2.0 writes exactly this, measured on
    /// x86-64 at `-O2` for depths zero to three of both builtins.
    ///
    /// The function is given a frame pointer because of this, which is what [`Stack::walks_frames`]
    /// carries out to the layout. A depth of zero needs it as the answer and every depth above zero
    /// needs it as the start, so there is no case here where it is not wanted.
    ///
    /// How far the chain actually reaches is the program's business and not this one's. A caller
    /// compiled without a frame pointer has no link in it for the walk to follow, so a depth above
    /// zero is a promise about how the whole program was built. That is why gcc documents a nonzero
    /// depth as unsafe rather than as an answer, and why the depth is refused above a limit in
    /// `check/builtin/frame.rs` rather than walked as far as it says.
    fn frames(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Depth(depth) = data.extra else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        let returning = data.opcode == Opcode::ReturnAddress;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let moves = x86_64::FRAME.moves(self.gpr).expect("a class the target says how to move");
        let load = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", moves.load)));
        self.stack.walks_frames = true;

        // Where the walk is up to. The frame pointer to begin with, and the register the last load
        // wrote after that.
        let reg = self.new_reg(result);
        let mut base = mir::Reg::physical(self.conv.frame_pointer);
        for link in 0..depth {
            // The last load of a walk that is looking for a frame writes the answer itself, which
            // is what keeps a walk of so many links that many instructions and not one more.
            let ends_here = link + 1 == depth && !returning;
            let next = if ends_here { reg } else { self.out.new_vreg(self.gpr) };
            let at = mir::Mem::at(mir::Operand::read(base, self.gpr));
            self.out.build(block, load).at(span).def(next, self.gpr).mem(at).finish();
            base = next;
        }

        if returning {
            let up = i32::try_from(self.conv.return_address).expect("a word above the frame");
            let at = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(up);
            self.out.build(block, load).at(span).def(reg, self.gpr).mem(at).finish();
        } else if depth == 0 {
            // The one case with no load in it at all: the frame this function is running in is the
            // register itself, and a physical register is not one the allocator hands out, so the
            // answer is a copy of it.
            let mov = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", moves.mov)));
            self.out
                .build(block, mov)
                .at(span)
                .operand(mir::Operand::write(reg, self.gpr))
                .operand(mir::Operand::read(base, self.gpr))
                .finish();
        }
        Ok(())
    }

    /// `__builtin_thread_pointer`, which is the front of the block [`Self::thread_address`] adds
    /// an offset to.
    ///
    /// The same one instruction, on its own this time and with nothing to add to it. A program
    /// writes this when what it wants is a number that is different in every thread and cheap to
    /// come by, rather than a variable of its own in the block, so there is no relocation here and
    /// no name for the link to resolve.
    fn thread_pointer(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let reg = self.new_reg(result);
        let load = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{GOT_LOAD}")));
        let at = mir::Mem::in_segment(Segment::Fs, 0);
        self.out.build(block, load).at(span).def(reg, self.gpr).mem(at).finish();
        Ok(())
    }

    /// A conversion that converts nothing: the result is the operand under another type.
    ///
    /// `ptrtoint` and `inttoptr` at one width are the whole of this. An address on this machine is
    /// an integer as wide as the machine addresses, so a cast between the two changes what the
    /// type system calls the value and changes nothing about the value, and the register holding
    /// it is the register that already held it. The front end never writes either of them at any
    /// other width, because it widens or narrows around the cast rather than through it, so the
    /// two widths disagreeing here means the IR came from somewhere else and is refused rather
    /// than guessed at.
    ///
    /// Reading the operand first is what materializes it when it is a constant, which is the case
    /// that matters: a null pointer is an `inttoptr` of zero, and that zero has to reach a
    /// register before anything can call it an address.
    fn rename(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let [arg] = self.source[data.args] else { return Err(self.unsupported(inst)) };
        let result = data.first_result.ok_or_else(|| self.unsupported(inst))?;
        if !self.is_address_width(self.source[arg].ty)
            || !self.is_address_width(self.source[result].ty)
        {
            return Err(self.unsupported(inst));
        }
        let reg = self.reg_of(arg)?;
        self.regs[result.index()] = Some(reg);
        Ok(())
    }

    /// One barrier, which on this machine is one instruction at the strongest ordering and no
    /// instruction at all at every other one.
    ///
    /// x86-64 is total store order, so the only reordering the machine does is a store followed by
    /// a load of a different address, and the only ordering that forbids that is sequential
    /// consistency. An acquire, a release and an acquire release fence are therefore already true
    /// of every program running here, and what a program wanted from writing one is that the
    /// compiler not move memory accesses across it. The optimizer has finished by the time this
    /// runs and nothing below reorders one access past another, so the constraint is already
    /// discharged and there is nothing to write.
    ///
    /// The strongest one is `mfence`, which is what gcc 16.2.0 writes for
    /// `__atomic_thread_fence(__ATOMIC_SEQ_CST)` and for `__sync_synchronize`. A locked instruction
    /// on the stack is faster on most parts and is what some compilers write instead; it is also a
    /// write to memory the program did not ask for, and the plain barrier is the one that says what
    /// it means.
    ///
    /// Written here by name rather than by a rule, for the same reason a `lea` of a symbol is:
    /// there is nothing in a barrier that a proof over bitvectors could discharge. It computes
    /// nothing, so there is no equality to state, and what makes it the right answer is the memory
    /// model, which the rule language cannot talk about.
    fn barrier(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Order(order) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        if order != MemOrder::SeqCst {
            return Ok(());
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let fence = mir::Opcode::new(self.names.intern("x64.mfence"));
        self.out.build(block, fence).at(span).finish();
        Ok(())
    }

    /// The instruction a program stops on, which is one byte pair and no operands.
    ///
    /// `ud2` is an opcode the manual promises will never be given a meaning, so a processor that
    /// reaches it raises the fault for an instruction it does not know, and on Linux that arrives
    /// at the program as `SIGILL`. That is what `__builtin_trap` is for: a stop that cannot be
    /// caught by anything the program installed for an ordinary error, cannot be returned from,
    /// and leaves the address of the fault in the core file.
    ///
    /// Why not a call to `abort`. It is two bytes against a call and a relocation, it needs no
    /// library, and it works in the places this one is written most, which are a kernel and a
    /// freestanding program that has no `abort` to call. gcc 16.2.0 writes `ud2` here too.
    fn trap(&mut self, inst: Inst) {
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let stop = mir::Opcode::new(self.names.intern("x64.ud2"));
        self.out.build(block, stop).at(span).finish();
    }

    /// One hint that an address is about to be used, which is one instruction and no promise.
    ///
    /// Four instructions on this machine and the locality picks between them, which is what the
    /// number means: how much of the data will still be wanted after the access. None of it wanted
    /// is `prefetchnta`, which brings the line in without keeping it, and all of it wanted is
    /// `prefetcht0`, which brings it as close as the machine can. The two in between are the levels
    /// between those. Measured against gcc 16.2.0 on x86-64 rather than read off the manual: zero
    /// gives `prefetchnta`, one `prefetcht2`, two `prefetcht1` and three `prefetcht0`.
    ///
    /// Whether the access will write is not read here, and that is this machine rather than an
    /// omission. The write hint is `prefetchw`, which is not in the base instruction set, and gcc
    /// writes it only when the command line said the part has it. So a prefetch for a write is the
    /// same instruction as a prefetch for a read, which is what gcc 16.2.0 writes without
    /// `-mprfchw`, and the difference is carried in the IR for a target that can use it.
    ///
    /// The address goes in the addressing mode rather than in an operand, the way a store's does.
    /// It is built here as the plainest one there is, a register and nothing else, because what
    /// arrives is a value and folding an addition into the mode is a rule's job and no rule reaches
    /// this instruction. An address the program computed is therefore one `lea` or one add in front
    /// of this, which is what it would have been for the load the hint is about anyway.
    fn hint(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Prefetch(hint) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let [address] = args[..] else { return Err(self.unsupported(inst)) };
        let name = match hint.locality {
            0 => "prefetch_nta",
            1 => "prefetch_t2",
            2 => "prefetch_t1",
            PrefetchHint::MOST => "prefetch_t0",
            // Nothing else exists. The checker reads a locality outside the range as zero and the
            // verifier refuses one that got here another way, so this is a hint that was built
            // rather than checked, and the safe answer for a hint is to write no instruction.
            _ => return Err(self.unsupported(inst)),
        };
        let base = self.reg_of(address)?;
        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        self.out
            .build(block, opcode)
            .at(self.source.span(inst))
            .mem(mir::Mem::at(mir::Operand::read(base, self.gpr)))
            .finish();
        Ok(())
    }

    /// One compare and exchange, which is the instruction every other atomic on this machine is
    /// built out of.
    ///
    /// What the IR asks for is: read what is at an address, compare it against a value the program
    /// expected, put a second value there if the two were equal, and say both what was read and
    /// whether the exchange happened. The machine has exactly that instruction, and the `lock` in
    /// front of it is what makes the whole of it one step as far as every other processor is
    /// concerned.
    ///
    /// The ordering is not read here, and that is the memory model rather than an omission. A
    /// locked instruction on x86-64 is a full barrier whatever the program asked for, so a relaxed
    /// compare and exchange and a sequentially consistent one are the same instruction, and there
    /// is nothing weaker to emit for the weaker orderings. The failure ordering is not read for the
    /// same reason.
    ///
    /// The two values it produces are why this is written by name. The one the program compares
    /// against and the one it gets back are both `rax`, which the instruction reads and writes
    /// without being told, and the table says so with a fixed constraint at each end rather than
    /// leaving the allocator to find out. The second value is the byte behind it, which is the zero
    /// flag read out by a `sete`, and it is a definition of the same instruction so that the
    /// allocator knows the two are live together and never gives the byte the register the answer
    /// is in.
    fn exchange(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let results: Vec<Value> = self.source[inst].results().collect();
        let [addr, expected, desired] = args[..] else { return Err(self.unsupported(inst)) };
        let [old, exchanged] = results[..] else { return Err(self.unsupported(inst)) };

        // A value the machine can compare in one instruction, which is an integer or an address at
        // one of the four widths it has a compare and exchange for. Anything else is a type this
        // has no instruction for rather than a program that is wrong, and the front end refuses it
        // before ever getting here.
        let ty = self.source[old].ty;
        let bits = if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() };
        if (!ty.is_int() && !ty.is_ptr()) || !matches!(bits, 8 | 16 | 32 | 64) {
            return Err(self.unsupported(inst));
        }

        let base = self.reg_of(addr)?;
        let want = self.reg_of(expected)?;
        let put = self.reg_of(desired)?;
        let got = self.new_reg(old);
        let flag = self.new_reg(exchanged);

        let name = format!("cmpxchg_{bits}");
        let form = x86_64::form(&name).ok_or_else(|| self.unsupported(inst))?;
        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in form.operands().iter().zip([got, flag, want, put]) {
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            build = build.operand(operand);
        }
        build.mem(mir::Mem::at(mir::Operand::read(base, self.gpr))).finish();
        Ok(())
    }

    /// One read modify write, for the three operations this machine does in a single instruction.
    ///
    /// What the IR asks for is: read what is at an address, do something to it, put the answer back,
    /// say what was there before, and let nothing get between the three steps. The machine has
    /// `xchg` for putting a value there and `lock xadd` for adding one, and both leave what they
    /// found in the register the operand arrived in, which is why the value that comes back and the
    /// value that went in are one register here.
    ///
    /// A subtraction is the add over the negated operand, which is right at every width because the
    /// machine's arithmetic wraps and negating then adding is subtracting in two's complement
    /// whatever the operands were. The negate is a separate instruction in front, over a register of
    /// its own, so that the value the program handed over is not the one written on: an operand may
    /// be live after this and a program that read it again would read the negation.
    ///
    /// The ordering is not read, for the reason the compare and exchange beside this does not read
    /// it. `xchg` with memory locks the bus whether it is asked to or not and `lock xadd` is asked
    /// to, so both are full barriers on this machine and there is nothing weaker to fall to.
    ///
    /// Eight of the other ten never arrive, because `crate::retry` turned each of them into a loop
    /// around a compare and exchange before anything here saw it. The two that do arrive are the
    /// ones on floating values, and they are refused: a compare and exchange of a float wants the
    /// value carried through an integer of the same width, and an eighty bit float has no such
    /// width. Neither family of builtins can write one yet either, so a program that reaches this
    /// refusal is a program that reached an unimplemented builtin first.
    fn modify(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let Extra::Rmw(op, _) = self.source[inst].extra else {
            return Err(self.unsupported(inst));
        };
        let args: Vec<Value> = self.source[self.source[inst].args].to_vec();
        let [addr, operand] = args[..] else { return Err(self.unsupported(inst)) };
        let old = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;

        // A value the machine can exchange in one instruction, which is an integer at one of the
        // four widths it has these for. A pointer arrives as an address, so it is an integer by the
        // time it is here, and anything else is a type this has no instruction for.
        let ty = self.source[old].ty;
        if !ty.is_int() || !matches!(ty.bits(), 8 | 16 | 32 | 64) {
            return Err(self.unsupported(inst));
        }
        let name = match op {
            RmwOp::Xchg => format!("xchg_{}", ty.bits()),
            RmwOp::Add | RmwOp::Sub => format!("xadd_{}", ty.bits()),
            _ => return Err(self.unsupported(inst)),
        };

        let base = self.reg_of(addr)?;
        let mut put = self.reg_of(operand)?;
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        if op == RmwOp::Sub {
            let negated = self.out.new_vreg(self.gpr);
            let negate =
                mir::Opcode::new(self.names.intern(&format!("{PREFIX}neg_r_{}", ty.bits())));
            let form = x86_64::form(&format!("neg_r_{}", ty.bits()))
                .ok_or_else(|| self.unsupported(inst))?;
            let mut build = self.out.build(block, negate).at(span);
            for (desc, reg) in form.operands().iter().zip([negated, put]) {
                build = build.operand(mir::Operand {
                    reg,
                    class: desc.class,
                    role: desc.role,
                    constraint: desc.constraint,
                });
            }
            build.finish();
            put = negated;
        }

        let got = self.new_reg(old);
        let form = x86_64::form(&name).ok_or_else(|| self.unsupported(inst))?;
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{name}")));
        let flags = self.carried(inst);
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in form.operands().iter().zip([got, put]) {
            build = build.operand(mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            });
        }
        build.mem(mir::Mem::at(mir::Operand::read(base, self.gpr))).finish();
        Ok(())
    }

    /// One `asm` statement.
    ///
    /// An empty template is most of the inline assembly in a test suite, and it is not a corner
    /// case somebody wrote by accident. A program that wants a value computed where it stands, or a
    /// loop the optimizer must not touch, writes `asm volatile ("" : : : "memory")`, and forty
    /// years of bug reports about optimizers are full of them. What such a statement asks for is
    /// the barrier and the operand places, and no instructions at all.
    ///
    /// So the operands are the half that is always real: a constraint says where a value has to be,
    /// and where it has to be is still true when the template between them is empty.
    ///
    /// What the constraints ask for, on an empty template, is only ever that two operands share a
    /// place. Nothing reads a register no text names, so `"r"` on its own asks for a register and
    /// no particular one, and any register at all answers it. A matching constraint is different,
    /// because it says the output the assembly leaves is the place the input arrived in, and with
    /// no instructions between them that is the input unchanged. So it is a rename and not a move:
    /// the value is already in a register and the result is that register.
    ///
    /// An output nothing is tied to and no instruction writes is whatever the assembly left there,
    /// which for a template that writes nothing is whatever was in the register. That is a value
    /// the program is not entitled to, and this writes a zero rather than reading one, because the
    /// allocator has to be given a definition before a use whatever the program is entitled to.
    ///
    /// # A template with instructions in it
    ///
    /// [`x86_64::read`] turns the text into the opcodes this backend already has, which is what
    /// `spec/11-asm-objects-debug.md` section 11.1 asks for: the machine is described once, and an
    /// instruction a program wrote is looked up in that description rather than copied through to
    /// an assembler that has one of its own. So nothing here assembles anything. What it does is
    /// put the statement's operands where the opcode holds them, and from there an `asm` statement
    /// is ordinary machine code: the allocator picks the registers, the listing and the object file
    /// are written from the same table as every other instruction, and a spill around one works
    /// because there is nothing left about it for a spill to get wrong.
    ///
    /// Three things are refused, all for one reason, which is that placing them by a guess gives a
    /// program that assembles into something other than what it says.
    ///
    /// A register the template named itself. The registers an instruction here names are the ones
    /// the allocator handed out, and a name in the text is a claim on a register nobody told the
    /// allocator about. A register a constraint letter names is a different thing and is placed,
    /// which the paragraph below is about: there the statement said which of its own operands is
    /// in the register, and a name in the middle of a template says no such thing.
    ///
    /// An output the template writes more than once, which is one place with two definitions in it,
    /// and the machine IR between here and the allocator has one definition per register by
    /// construction. An output tied to an input and written once is not that: it is two registers
    /// the description ties together, which is what [`Place`] is about.
    ///
    /// An operand read where the opcode writes, or written where it reads. An output that has not
    /// been written yet is not a value, and an input the assembly writes over is a value something
    /// else may still be using.
    ///
    /// # A register the instruction uses without being told
    ///
    /// An instruction may reach a register its text does not name, and `cpuid` is all of them at
    /// once: the leaf goes in `eax`, the subleaf in `ecx`, and the answer comes back in all four
    /// registers. The description holds every bit of that already, so what is left is to say which
    /// of the statement's operands is in each of those registers, and the constraint letter is the
    /// one thing in an assembly statement that says it. `"=a"` is an output in `rax` and `"c"` is
    /// an input in `rcx`, which is why a program writing `cpuid` writes its constraints that way
    /// and has no choice about it.
    ///
    /// A register no letter named is one the statement put nothing in, and that is the usual case
    /// rather than an unusual one, since an instruction that answers four questions is written by
    /// programs that asked one. A write of one is the register being destroyed and gets a register
    /// of its own, which is what tells the allocator to keep everything else out of it. A read of
    /// one is a register the instruction looks at and the program never filled, which gets a zero
    /// for the reason [`Self::undefined`] gives.
    ///
    /// # The clobber list
    ///
    /// Read now, as the registers it names being written by every instruction of the template. By
    /// every one rather than by one of them, because the list says the assembly as a whole leaves
    /// them ruined and nothing here knows which line did it. Every entry has to be a register this
    /// machine has a name for or the statement is refused, since a name nobody read is a register
    /// nobody is keeping out of.
    ///
    /// `memory` and `cc` are the two entries that are not registers and both are skipped. `memory`
    /// says the assembly touches storage, which is already true of every `asm` this writes and is
    /// nothing a register list could hold. `cc` says it ruins the condition flags, and the flag
    /// tracking already has that from the instructions the template was read into, since it takes
    /// every instruction it does not recognize as writing them and every instruction here is one
    /// this machine describes.
    ///
    /// A clobber the instruction already writes is left off it. `cpuid` writes all four registers
    /// by description, and a statement listing three of them as clobbers as well is saying the
    /// same thing twice, which the allocator would read as one register with two definitions.
    ///
    /// On a template with nothing in it the list is ignored, as it was before, since a template
    /// with no instructions ruins nothing whatever it said about what it ruins.
    fn assembly(&mut self, inst: Inst) -> Result<(), Unsupported> {
        let data = &self.source[inst];
        let Extra::Asm(asm) = data.extra else { return Err(self.unsupported(inst)) };
        let info = self.source[asm];
        if !self.source[info.targets].is_empty() {
            return Err(Unsupported::Assembly { inst, refused: Written::Goto });
        }
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };

        let constraints = self.names.resolve(info.constraints).to_string();
        let results: Vec<Value> = data.results().collect();
        let operands = AsmOperands::read(&constraints, &results, &self.source[data.args])
            .ok_or_else(refused)?;
        let list: Vec<AsmOperand> = operands.iter().copied().collect();

        // Read after the constraints and not before them, because a mnemonic whose suffix the
        // program left off is read at the width of the operands it names, and the operands are
        // what the constraints are a list of.
        let widths: Vec<Option<x86_64::Width>> = list
            .iter()
            .map(|operand| {
                let ty = self.source[operand.result.or(operand.value)?].ty;
                if !ty.is_scalar() {
                    return None;
                }
                x86_64::Width::of_bits(if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() })
            })
            .collect();
        let template = self.names.resolve(info.template).to_string();
        let steps = if template.trim().is_empty() {
            Vec::new()
        } else {
            x86_64::read(&template, &widths)
                .ok_or(Unsupported::Assembly { inst, refused: Written::Template })?
        };

        // Which operands the template writes, counted before anything is placed, because the answer
        // decides where each of the three below comes from and one instruction may name an operand
        // that a later one writes. Which of them any instruction puts in a register at all is
        // counted in the same walk, since an operand no instruction reaches that way is one nothing
        // has to put anywhere: a constant a template names only as the distance into an address is
        // written into the instruction, and a register holding a copy of it would be one nobody
        // reads. An operand the address is counted from is reached that way and is counted here for
        // that reason, because the walk below it is over the opcode's operands and an address is
        // not one of those.
        let mut writes = vec![0usize; list.len()];
        let mut reads = vec![false; list.len()];
        let mut held = vec![false; list.len()];
        for step in &steps {
            let x86_64::Step::Line(line) = step else { continue };
            if let Some(x86_64::Piece::Operand { index, .. }) = line.at.and_then(|at| at.base) {
                *held.get_mut(index).ok_or_else(refused)? = true;
            }
            let form = x86_64::form(line.opcode).ok_or_else(refused)?;
            // Which registers the instruction reaches, asked the same way it is asked again when
            // the instruction is written. See [`Self::lettered`] for the one opcode whose answer
            // comes from the constraint letters rather than from the description.
            let lettered = (line.opcode == x86_64::LITERAL).then(|| self.lettered(&list));
            let (described, pieces) = match &lettered {
                Some((described, pieces)) => (described.as_slice(), pieces.as_slice()),
                None => (form.operands(), line.operands.as_slice()),
            };
            for (desc, piece) in described.iter().zip(pieces) {
                // An operand the instruction reaches without its text saying so is the statement's
                // only when a constraint letter put something there. One that is nobody's writes
                // nothing of the program's, so it is counted nowhere and is dealt with where it is
                // placed.
                let index = match *piece {
                    x86_64::Piece::Operand { index, .. } => index,
                    x86_64::Piece::Implicit { reg } => match bound(&list, reg, desc.role) {
                        Some(index) => index,
                        None => continue,
                    },
                    x86_64::Piece::Reg { .. } => continue,
                };
                *held.get_mut(index).ok_or_else(refused)? = true;
                if matches!(desc.role, Role::Def | Role::EarlyDef) {
                    *writes.get_mut(index).ok_or_else(refused)? += 1;
                } else {
                    *reads.get_mut(index).ok_or_else(refused)? = true;
                }
            }
        }

        // Where every operand is. Worked out in full before the first instruction is written, since
        // reading a value may be what puts it in a register in the first place, and that has to
        // happen in front of the assembly rather than in the middle of it.
        let mut places: Vec<Place> = vec![Place::default(); list.len()];
        for (index, operand) in list.iter().copied().enumerate() {
            let Some(result) = operand.result else {
                // An input, or an output the assembly was handed the address of, and both are a
                // value that arrives in a register and is read out of it, unless no instruction of
                // the template reads it out of one.
                let value = operand.value.ok_or_else(refused)?;
                if held[index] {
                    places[index].read = Some(self.reg_of(value)?);
                }
                continue;
            };
            let ty = self.source[result].ty;
            if on_x87(ty) || writes[index] > 1 {
                return Err(refused());
            }
            let tied = operands.tied_to(index);
            if let Some(from) = tied {
                if self.class_of(self.source[from].ty) != self.class_of(ty) {
                    return Err(refused());
                }
                places[index].read = Some(self.reg_of(from)?);
            }
            if writes[index] == 1 {
                places[index].write = Some(self.new_reg(result));
                continue;
            }
            match tied {
                // The place the input arrived in, which the assembly wrote nothing over. One
                // register, so this is a rename rather than a move.
                Some(_) => {
                    let reg = places[index].read.ok_or_else(refused)?;
                    self.regs[result.index()] = Some(reg);
                    places[index].write = Some(reg);
                }
                None => {
                    self.undefined(inst, result)?;
                    places[index].write = self.regs[result.index()];
                }
            }
        }

        // An output an instruction of the template also reads, which the statement said nothing
        // about because an output is what a statement says the other thing about. What it holds
        // there is undefined, and a program writing one means it: `sbb %0, %0` in libgmp's
        // `add_mssaaaa` subtracts a register from itself and is asking for the borrow bit rather
        // than for the number, so whatever the register held, the answer is the same. Undefined is
        // not the same as absent though, since the allocator is owed a definition in front of every
        // use, so it gets the zero an output nothing wrote gets and for the same reason.
        for index in 0..list.len() {
            if !reads[index] || places[index].read.is_some() || places[index].write.is_none() {
                continue;
            }
            places[index].read = Some(self.seeded(inst, list[index])?);
        }

        // Worked out once for the whole template, since the list is one list and every instruction
        // of the template gets it. Not worked out at all for a template with no instructions, which
        // is where there is nothing for it to go on.
        let clobbers = self.names.resolve(info.clobbers).to_string();
        let clobbered =
            if steps.is_empty() { Vec::new() } else { Self::clobbered(inst, &clobbers)? };

        // A template with a label in it is not one run of instructions, and what it is instead is
        // in [`Self::woven`]. Every other template is what it has always been, which is every
        // instruction of it written into the block the statement stands in.
        if steps.iter().any(|step| !matches!(step, x86_64::Step::Line(_))) {
            return self.woven(inst, &steps, &mut places, &list, &clobbered, &writes);
        }
        for step in &steps {
            let x86_64::Step::Line(line) = step else { continue };
            self.instruction(inst, line, &places, &list, &clobbered)?;
        }
        Ok(())
    }

    /// A register holding a zero, for an operand of a template that is read before anything filled
    /// it.
    ///
    /// Two things ask for this and they are the same thing twice. An output the template reads has
    /// nothing to be read out of until the instruction that writes it has run, and a loop carries
    /// an operand into a block before the instruction that fills it, so both are a use in front of
    /// every definition. What the program is owed there is nothing, since the value is undefined
    /// either way, and what the allocator is owed is a register something wrote.
    fn seeded(&mut self, inst: Inst, operand: AsmOperand) -> Result<mir::Reg, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let value = operand.result.or(operand.value).ok_or_else(refused)?;
        let class = self.class_of(self.source[value].ty);
        if class != self.gpr {
            return Err(refused());
        }
        let block = self.at.expect("a block is being filled");
        let reg = self.out.new_vreg(class);
        let put = mir::Opcode::new(self.names.intern(&format!("{PREFIX}mov_ri_64")));
        self.out.build(block, put).at(self.source.span(inst)).def(reg, class).imm(0).finish();
        Ok(reg)
    }

    /// A template with labels in it, as the blocks its jumps leave and arrive at.
    ///
    /// A statement is an instruction of the IR and stands inside one block, so a template that
    /// jumps has to stop being one thing. Each label becomes a block, each jump ends the block it
    /// stands in and gives it two arms, and whatever follows the statement goes into whichever
    /// block the walk finished in, which is what [`Self::block`] already reads off `self.at` and
    /// what [`Self::saves_place`] already does for the same reason.
    ///
    /// # What is carried between them
    ///
    /// The machine IR here is in the form where a register is written once, so an operand written
    /// inside a loop and read again at the top of it cannot be one register. What arrives at the
    /// top is a parameter of that block, and every jump to it carries whichever register held the
    /// operand where the jump stands. That is the whole of the bookkeeping: every block a label
    /// made takes one parameter for each operand that is in a register at all, in one order, so an
    /// arm's arguments and a block's parameters are the same list read twice.
    ///
    /// Which register an operand is in at each point is kept in the read half of its place, since
    /// that is what the instructions below read it out of. An instruction that writes an operand
    /// leaves it in the register it wrote, and a jump below carries that one. The block an
    /// untaken jump falls into is arrived at one way only and so takes no parameters, and nothing
    /// about where the operands are changes there.
    ///
    /// An operand written by the template and filled by nothing is written as a zero first, for
    /// the reason [`Self::undefined`] gives and one more: a jump may carry it before the
    /// instruction that fills it has run, and an argument has to be a register something wrote.
    ///
    /// # The condition state
    ///
    /// Nothing carries it and nothing has to. The instruction that sets it and the jump that reads
    /// it are both written here, next to each other in one block, and what the allocator may put
    /// between them is a move, which on this machine leaves the condition state alone. The edge
    /// into a block a loop goes back to is a critical edge and `crate::split` gives it a block of
    /// its own, so the moves an arm turns into land behind the jump rather than in front of it.
    fn woven(
        &mut self,
        inst: Inst,
        steps: &[x86_64::Step],
        places: &mut [Place],
        list: &[AsmOperand],
        clobbered: &[PhysReg],
        writes: &[usize],
    ) -> Result<(), Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let span = self.source.span(inst);

        // Which operands are carried, which is every one that is in a register at all. An operand
        // the template never puts in one, such as a constant it names only as the distance into an
        // address, is in the instruction and has nowhere to be carried from.
        let mut carried: Vec<(usize, RegClass)> = Vec::new();
        for (index, operand) in list.iter().enumerate() {
            if places[index].read.is_none() && places[index].write.is_none() {
                continue;
            }
            let value = operand.result.or(operand.value).ok_or_else(refused)?;
            let ty = self.source[value].ty;
            if on_x87(ty) {
                return Err(refused());
            }
            carried.push((index, self.class_of(ty)));
        }

        // What each of them holds where the template starts.
        for &(index, _) in &carried {
            if places[index].read.is_some() {
                continue;
            }
            if writes[index] == 0 {
                places[index].read = places[index].write;
                continue;
            }
            places[index].read = Some(self.seeded(inst, list[index])?);
        }

        // The blocks, made before the walk because a jump forwards names a label the walk has not
        // reached yet.
        let mut labels: Vec<(&str, mir::Block, Vec<mir::Reg>)> = Vec::new();
        for step in steps {
            let x86_64::Step::Label(name) = step else { continue };
            let block = self.out.create_block();
            let mut params = Vec::with_capacity(carried.len());
            for &(_, class) in &carried {
                params.push(self.out.append_param(block, class));
            }
            labels.push((name.as_str(), block, params));
        }

        for step in steps {
            match step {
                x86_64::Step::Label(name) => {
                    let (block, params) = Self::went(&labels, name).ok_or_else(refused)?;
                    let from = self.at.expect("a block is being filled");
                    let args = Self::held(places, &carried).ok_or_else(refused)?;
                    *self.out.succs_mut(from) = vec![mir::BlockCall::with(block, args)];
                    self.at = Some(block);
                    for (at, &(index, _)) in carried.iter().enumerate() {
                        places[index].read = params.get(at).copied();
                    }
                }
                x86_64::Step::Jump { opcode, to } => {
                    let (block, _) = Self::went(&labels, to).ok_or_else(refused)?;
                    let from = self.at.expect("a block is being filled");
                    let args = Self::held(places, &carried).ok_or_else(refused)?;
                    let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{opcode}")));
                    self.out.build(from, opcode).at(span).finish();
                    let next = self.out.create_block();
                    *self.out.succs_mut(from) =
                        vec![mir::BlockCall::with(block, args), mir::BlockCall::to(next)];
                    self.at = Some(next);
                }
                x86_64::Step::Line(line) => {
                    self.instruction(inst, line, places, list, clobbered)?;
                    let form = x86_64::form(line.opcode).ok_or_else(refused)?;
                    for (desc, piece) in form.operands().iter().zip(&line.operands) {
                        if !desc.role.is_def() {
                            continue;
                        }
                        let index = match *piece {
                            x86_64::Piece::Operand { index, .. } => index,
                            x86_64::Piece::Implicit { reg } => match bound(list, reg, desc.role) {
                                Some(index) => index,
                                None => continue,
                            },
                            x86_64::Piece::Reg { .. } => continue,
                        };
                        let place = places.get_mut(index).ok_or_else(refused)?;
                        if place.write.is_some() {
                            place.read = place.write;
                        }
                    }
                }
            }
        }

        // Where the walk left each output, which is the parameter of the block a label made when
        // the template ends in one and the register an instruction wrote when it does not.
        for (index, operand) in list.iter().enumerate() {
            let Some(result) = operand.result else { continue };
            if let Some(reg) = places[index].read {
                self.regs[result.index()] = Some(reg);
            }
        }
        Ok(())
    }

    /// The block one of the template's labels made, and the parameters it takes.
    fn went<'b>(
        labels: &'b [(&str, mir::Block, Vec<mir::Reg>)],
        name: &str,
    ) -> Option<(mir::Block, &'b [mir::Reg])> {
        labels
            .iter()
            .find(|(had, ..)| *had == name)
            .map(|(_, block, params)| (*block, params.as_slice()))
    }

    /// The register each carried operand is in, which is what an arm to a label carries.
    fn held(places: &[Place], carried: &[(usize, RegClass)]) -> Option<Vec<mir::Reg>> {
        carried.iter().map(|&(index, _)| places.get(index)?.read).collect()
    }

    /// The registers a clobber list names, in the order it named them.
    ///
    /// Nothing is dropped. A name this has no register for is refused, because the list is the
    /// program telling the compiler which registers it may not leave anything in, and an entry
    /// nobody read is a register something may still be left in. See [`Self::assembly`] for the
    /// two entries that are not registers and for why they are skipped rather than refused.
    fn clobbered(inst: Inst, clobbers: &str) -> Result<Vec<PhysReg>, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Clobber };
        let mut named = Vec::new();
        for entry in clobbers.split(',') {
            let entry = entry.trim().trim_matches('"');
            // The sigil is optional in a clobber list and means nothing when it is there, unlike
            // in a template, where it is what tells a register from an operand.
            let entry = entry.strip_prefix('%').unwrap_or(entry);
            if entry.is_empty() || entry == "memory" || entry == "cc" {
                continue;
            }
            let (reg, _) = x86_64::gpr_named(entry).ok_or_else(refused)?;
            if !named.contains(&reg) {
                named.push(reg);
            }
        }
        Ok(named)
    }

    /// One instruction of a template, as the machine instruction it was read back into.
    fn instruction(
        &mut self,
        inst: Inst,
        line: &x86_64::Line,
        places: &[Place],
        list: &[AsmOperand],
        clobbered: &[PhysReg],
    ) -> Result<(), Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let form = x86_64::form(line.opcode).ok_or_else(refused)?;
        // What the instruction reaches and what is in each of them. The description answers the
        // first for every opcode but one, and the pieces the template was read into answer the
        // second. Bytes a program wrote out itself are the one, since nothing in a number is a
        // register anybody could read, so the constraint letters answer both. See
        // [`Self::lettered`].
        let lettered = (line.opcode == x86_64::LITERAL).then(|| self.lettered(list));
        let (described, pieces) = match &lettered {
            Some((described, pieces)) => (described.as_slice(), pieces.as_slice()),
            None => (form.operands(), line.operands.as_slice()),
        };
        let mut built = Vec::with_capacity(pieces.len() + clobbered.len());
        for (desc, piece) in described.iter().zip(pieces) {
            built.push(self.placed(inst, *desc, *piece, places, list)?);
        }
        // The clobbers go in among the definitions rather than behind the reads, because an operand
        // vector in the machine IR is every definition and then every use and what counts them
        // reads that order rather than each operand's role.
        let defs = built.iter().take_while(|operand| operand.role.is_def()).count();
        let mut added = 0usize;
        for &reg in clobbered {
            if described.iter().any(|desc| desc.constraint == Constraint::Fixed(reg)) {
                continue;
            }
            built.insert(defs, mir::Operand::write(mir::Reg::physical(reg), self.gpr));
            added += 1;
        }
        // A constraint tying one operand to another names it by its place in this vector, and the
        // clobbers were put in the middle of the vector, so everything behind them moved. The
        // description is written against an instruction with no clobbers in it and cannot know
        // that, which makes this the one place the two numberings have to be reconciled.
        for operand in &mut built {
            if let Constraint::Reuse(at) = operand.constraint {
                if usize::from(at) >= defs {
                    let moved = usize::from(at) + added;
                    operand.constraint =
                        Constraint::Reuse(u8::try_from(moved).map_err(|_| refused())?);
                }
            }
        }
        let at = match line.at {
            Some(at) => Some(self.addressed(inst, at, places, list)?),
            None => None,
        };

        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let opcode = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", line.opcode)));
        let mut build = self.out.build(block, opcode).at(span);
        for operand in built {
            build = build.operand(operand);
        }
        if let Some(value) = line.imm {
            build = build.imm(value);
        }
        if let Some(mem) = at {
            build = build.mem(mem);
        }
        build.finish();
        Ok(())
    }

    /// The registers a run of bytes reaches, taken from the constraint letters rather than from the
    /// description of an opcode.
    ///
    /// Every other instruction of a template has a description saying which registers it reaches
    /// without naming them, and [`Self::assembly`] matches the letters against that. Bytes a program
    /// wrote out itself have no such description and could not have one: what the instruction is, is
    /// a number, and nothing in a number is a register anything could read. So the letters are the
    /// whole of what is known, and they are enough, because a program writing an instruction this
    /// way has to say where its operands go for exactly the reason a program writing `cpuid` does.
    ///
    /// Each register named by a letter gets one entry for the write and one for the read, the same
    /// two `cpuid` has, and only the half the statement asked for: a register no output names is not
    /// written here and one no input names is not read. The writes come first because that is the
    /// order an operand vector in the machine IR is counted in. A register named by nothing is left
    /// out rather than given a spare one, which is the difference from `cpuid` and is right for the
    /// same reason: `cpuid` writes four registers whatever the program said, and what these bytes
    /// touch is known only from what the program said.
    fn lettered(&self, list: &[AsmOperand]) -> (Vec<OperandDesc>, Vec<x86_64::Piece>) {
        let mut named: Vec<PhysReg> = Vec::new();
        for operand in list {
            if let Some(reg) = operand.fixed.and_then(x86_64::gpr_letter) {
                if !named.contains(&reg) {
                    named.push(reg);
                }
            }
        }
        let mut described = Vec::with_capacity(named.len() * 2);
        let mut pieces = Vec::with_capacity(named.len() * 2);
        for role in [Role::Def, Role::Use] {
            for &reg in &named {
                if bound(list, reg, role).is_none() {
                    continue;
                }
                let desc = if role.is_def() {
                    OperandDesc::write(self.gpr)
                } else {
                    OperandDesc::read(self.gpr)
                };
                described.push(desc.with(Constraint::Fixed(reg)));
                pieces.push(x86_64::Piece::Implicit { reg });
            }
        }
        (described, pieces)
    }

    /// One operand of one instruction of a template, in the register the statement put it in.
    fn placed(
        &mut self,
        inst: Inst,
        desc: OperandDesc,
        piece: x86_64::Piece,
        places: &[Place],
        list: &[AsmOperand],
    ) -> Result<mir::Operand, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        // A register the instruction reaches without its text naming it belongs to whichever of the
        // statement's operands a constraint letter put there, and to nobody when no letter did.
        // There is no width to check in that case: the operand is the register the letter named and
        // the instruction does what it does to it, which is what a program writing `"=a"` asked for.
        let (index, spelled) = match piece {
            x86_64::Piece::Operand { index, width, stated } => (index, Some((width, stated))),
            x86_64::Piece::Implicit { reg } => match bound(list, reg, desc.role) {
                Some(index) => (index, None),
                None => return self.spare(inst, desc),
            },
            x86_64::Piece::Reg { .. } => return Err(refused()),
        };
        let operand = list.get(index).copied().ok_or_else(refused)?;
        // The two halves of an operand written `+`, which arrives in one register and leaves in
        // another with the allocator told to make them the same one. Everything else has one of
        // the two and asking for the other is the refusal below.
        let place = places.get(index).copied().ok_or_else(refused)?;
        let reg = match desc.role {
            Role::Use => place.read,
            Role::Def | Role::EarlyDef => place.write,
        }
        .ok_or_else(refused)?;

        // Read where the opcode reads and written where it writes, which is what the first half of
        // this asks. An output has a result and an input has a value, an output written `+` has
        // both because it is read before it is written, and an output a matching constraint names
        // is read as the input that named it. See [`read_as`].
        // An output with neither is read as well, and what it holds there is undefined, which
        // [`Self::assembly`] says why and puts a zero in a register for.
        let placeable = match desc.role {
            Role::Use => read_as(list, index).is_some() || operand.result.is_some(),
            Role::Def | Role::EarlyDef => operand.result.is_some(),
        };
        let ty = match (operand.result, operand.value) {
            (Some(result), _) => self.source[result].ty,
            (None, Some(value)) => self.source[value].ty,
            (None, None) => return Err(refused()),
        };
        let bits = if ty.is_ptr() { ADDRESS_BITS } else { ty.bits() };
        if !placeable || self.class_of(ty) != desc.class {
            return Err(refused());
        }
        if let Some((width, stated)) = spelled {
            // An operand the template wrote a width on may be written by an instruction that fills
            // more of the register than the object in it does, and the object is then the low part
            // of what was written. That is what gmp asks for when it counts the low zero bits of a
            // limb into an `unsigned` and spells the count `%q0`: one quadword instruction writes
            // the whole register and the `unsigned` is the bottom of it, which is every bit of an
            // answer that cannot exceed sixty four anyway.
            //
            // Only written, and only wider. A read of more of a register than its type fills is a
            // program handing an instruction bits nothing ever put there. A write of less of one
            // leaves the top of the object holding whatever the register held before, which is the
            // same thing one instruction later. Both are refused, and an operand the template left
            // plain is refused either way, because what gets spelled for that one is the register
            // at the width of its type and no other instruction is the one written down.
            let widened = stated && desc.role.is_def() && width.bits() > bits;
            if bits != width.bits() && !widened {
                return Err(refused());
            }
        }
        Ok(mir::Operand { reg, class: desc.class, role: desc.role, constraint: desc.constraint })
    }

    /// A register an instruction of a template uses and the statement put nothing in.
    ///
    /// A write of one is the register being destroyed, which is what a clobber list is usually
    /// written to say and what an instruction with more answers than the program asked for does
    /// anyway: `cpuid` writes all four registers whether or not the statement wanted all four. A
    /// register of its own is the whole of what that needs, since a value nothing reads is one the
    /// allocator may put anywhere and is told about so that nothing else is put there.
    ///
    /// A read of one is a register the instruction looks at and the program never filled, which
    /// gcc leaves as whatever happened to be there. A zero is written instead, for the reason
    /// [`Self::undefined`] gives: the allocator has to be given a definition before a use, and a
    /// zero is the one answer that reads the same on every run.
    fn spare(&mut self, inst: Inst, desc: OperandDesc) -> Result<mir::Operand, Unsupported> {
        let refused = Unsupported::Assembly { inst, refused: Written::Operand };
        if desc.class != self.gpr {
            return Err(refused);
        }
        let reg = self.out.new_vreg(desc.class);
        if !desc.role.is_def() {
            let block = self.at.expect("a block is being filled");
            let span = self.source.span(inst);
            let put = mir::Opcode::new(self.names.intern(&format!("{PREFIX}mov_ri_64")));
            self.out.build(block, put).at(span).def(reg, desc.class).imm(0).finish();
        }
        Ok(mir::Operand { reg, class: desc.class, role: desc.role, constraint: desc.constraint })
    }

    /// The address one instruction of a template reads or writes.
    fn addressed(
        &mut self,
        inst: Inst,
        at: x86_64::At,
        places: &[Place],
        list: &[AsmOperand],
    ) -> Result<mir::Mem, Unsupported> {
        let refused = || Unsupported::Assembly { inst, refused: Written::Operand };
        let base = match at.base {
            None => None,
            Some(x86_64::Piece::Operand { index, .. }) => {
                // The register an address is counted from is read and never written, whatever the
                // instruction does to what it finds there.
                let reg = places.get(index).and_then(|place| place.read).ok_or_else(refused)?;
                Some(mir::Operand::read(reg, self.gpr))
            }
            // An address counted from a register the instruction reaches without being told is
            // not something this machine has: every addressing mode is written out in the text it
            // is part of, so a base that got here another way is a base nothing wrote down.
            Some(x86_64::Piece::Reg { .. } | x86_64::Piece::Implicit { .. }) => {
                return Err(refused());
            }
        };
        // A distance the template wrote, or the one in an operand the template pointed at, which is
        // the same distance said by something that knows how big a thing is. It has to be a number
        // the compiler can read at translation time, since it goes in the instruction rather than
        // in a register, and an operand holding anything else is refused rather than put somewhere.
        let disp = match at.disp {
            x86_64::Disp::Number(disp) => disp,
            x86_64::Disp::Operand(index) => {
                let value =
                    list.get(index).and_then(|operand| operand.value).ok_or_else(refused)?;
                let number = self.number(value).ok_or_else(refused)?;
                i32::try_from(number).map_err(|_| refused())?
            }
        };
        Ok(mir::Mem { base, scale: 1, disp, segment: at.segment, ..mir::Mem::default() })
    }

    /// The number in that value, for one an `iconst` defined, read at the width of its own type.
    ///
    /// Signed, because the two things a template asks this for are a distance into an address and
    /// the number on an instruction, and both of those are signed wherever they land. A constant
    /// whose type is unsigned and whose top bit is set therefore reads as a negative number here,
    /// which is the same number and is the reading that fits in the thirty two bits an addressing
    /// mode has room for.
    fn number(&self, value: Value) -> Option<i128> {
        let Def::Result { inst, .. } = self.source[value].def else { return None };
        if self.source[inst].opcode != Opcode::IConst {
            return None;
        }
        let Extra::Imm(imm) = self.source[inst].extra else { return None };
        let bits = self.source[imm].bits();
        let width = self.source[value].ty.bits();
        if width == 0 || width > 128 {
            return None;
        }
        let spare = 128 - width;
        Some(((bits << spare) as i128) >> spare)
    }

    /// A register holding a value the program has no claim on, written as a zero.
    ///
    /// Every other way of saying it costs the same instruction or needs a word the machine IR does
    /// not have, and a zero is the one that reads the same on every run.
    fn undefined(&mut self, inst: Inst, result: Value) -> Result<(), Unsupported> {
        let ty = self.source[result].ty;
        let refused = Unsupported::Assembly { inst, refused: Written::Operand };
        if self.class_of(ty) != self.gpr || !matches!(ty.bits(), 8 | 16 | 32 | 64) {
            return Err(refused);
        }
        let block = self.at.expect("a block is being filled");
        let span = self.source.span(inst);
        let reg = self.new_reg(result);
        let put = mir::Opcode::new(self.names.intern(&format!("{PREFIX}mov_ri_{}", ty.bits())));
        self.out.build(block, put).at(span).def(reg, self.gpr).imm(0).finish();
        Ok(())
    }

    /// Whether a type is the width an address is, which is what makes a cast to or from one free.
    fn is_address_width(&self, ty: Type) -> bool {
        ty.is_ptr() || (ty.is_int() && ty.bits() == ADDRESS_BITS)
    }

    /// Where a block goes, which in machine IR is on the block rather than on its terminator.
    ///
    /// That is why no rule ever names a block: a branch is selected for what it reads and the
    /// edges are copied across here, arguments and all. The arguments are read last, after every
    /// instruction of the block is written, because an argument that is a constant is
    /// materialized where it is first wanted and the end of the block is where an edge wants it.
    ///
    /// Which is not quite the end. A block that leaves two ways has the branch as its last
    /// instruction, and a block that leaves through a register has the indirect jump as its last,
    /// and anything appended after either is something it has already jumped past, so a constant
    /// materialized here would be a register the block below reads and nothing ever writes. The
    /// one that was there is put back on the end when that happened, which is the only reordering
    /// anything in this crate does and is why it is remembered before a single argument is read.
    fn edges(&mut self, block: Block, out: mir::Block) -> Result<(), Unsupported> {
        let Some(term) = self.source.terminator(block) else { return Ok(()) };
        let leaves = matches!(self.source[term].opcode, Opcode::BrIf | Opcode::IndirectBr);
        let branch = if leaves { self.out.terminator(out) } else { None };

        let calls: Vec<rucc_ir::BlockCall> = self.source.successors(term).collect();
        let mut succs = Vec::with_capacity(calls.len());
        for call in calls {
            let args: Vec<Value> = self.source[call.args].to_vec();
            let mut regs = Vec::with_capacity(args.len());
            for value in args {
                // The address of where the value is rather than the value, for the one type a
                // register holds none of. The block on the other side copies the bytes out of it
                // into a slot of its own, which is what makes a second edge into the same block
                // safe.
                let reg = if on_x87(self.source[value].ty) {
                    self.x87_slot(value)
                } else {
                    self.reg_of(value)?
                };
                regs.push(reg);
            }
            succs.push(mir::BlockCall::with(self.out_block(call.block), regs));
        }
        if let Some(branch) = branch {
            if self.out.terminator(out) != Some(branch) {
                self.out.remove_inst(branch);
                self.out.append_inst(out, branch);
            }
        }
        *self.out.succs_mut(out) = succs;
        Ok(())
    }

    /// The machine IR block an IR block became.
    fn out_block(&self, block: Block) -> mir::Block {
        self.blocks[block.index()].expect("every block was created before any was filled")
    }

    /// The parameters of the entry block, which are the function's arguments.
    ///
    /// They are not block parameters in the machine IR and they cannot be. A block parameter is
    /// given its value by a move on the edge into the block, and there is no edge into an entry
    /// block, so what arrives in a function is the convention's to say. [`crate::abi`] is what
    /// says it.
    ///
    /// The ones past the last register arrived in the caller's memory and are read out of it, and
    /// the loads that read them come back here so that the frame can finish them the way it
    /// finishes an `alloca`.
    fn arrive(&mut self, block: Block, out: mir::Block) -> Result<(), Unsupported> {
        let params = self.source[block].params.clone();
        // The type of each is the block's answer and what the ABI asks of it is the signature's,
        // and the two lists are the same list: a parameter the classification turned into a
        // pointer is a pointer in the block too. A block with more parameters than the signature
        // names is not one the front end writes, and each of those is taken as a plain value.
        let asked: Vec<Abi> = self.source.signature().params.iter().map(|it| it.abi).collect();
        let types: Vec<Param> = params
            .iter()
            .enumerate()
            .map(|(index, &value)| {
                let abi = asked.get(index).copied().unwrap_or_default();
                Param { ty: self.source[value].ty, abi }
            })
            .collect();
        // A save area for a function that takes arguments its signature does not name, which is a
        // block of this function's frame on one convention and the shadow space the caller already
        // reserved on the other. Which of the two it is is [`varargs::Area::of`]'s answer and
        // [`Self::save_area`] is where the difference is spent.
        let variadic = self.source.signature().variadic;
        let area = variadic.then(|| varargs::Area::of(self.conv));
        let arrived = abi::entry(&mut self.out, out, &types, self.conv, self.names, area)
            .map_err(|(index, missing)| Unsupported::Argument { index, missing })?;
        for (&param, reg) in params.iter().zip(&arrived.regs) {
            self.regs[param.index()] = Some(*reg);
        }
        if let Some(area) = area {
            self.save_area(out, &arrived, area);
        }
        self.stack.arguments.extend(arrived.stack);
        Ok(())
    }

    /// The prologue of a variadic function, which is every argument register it was handed written
    /// into the frame.
    ///
    /// Every one the signature did not name, that is. Which of those hold anything is a thing only
    /// the caller knew and there is nothing here to ask, so all of them are written, and the ones a
    /// named parameter took are not, because `va_start` sets the two offsets past them and nothing
    /// ever reads their slots.
    ///
    /// What that costs is up to fourteen stores in the prologue of a function that may read none of
    /// them, and the convention's answer to that is the count of vector registers in `%al`, which
    /// lets a callee skip the eight vector stores when the call passed no floats. Skipping them is a
    /// branch in a prologue, and a prologue is written long after this by [`crate::finish`], which
    /// has no blocks to branch between. So they are all written every time, which is correct and is
    /// what `-O0` costs. Issue #323 is the branch.
    ///
    /// A vector register is written all sixteen bytes at a time, because a `_Float128` fills one and
    /// a `va_arg` of a quad reads the slot back whole. gcc writes the same sixteen with the same
    /// instruction, which is what [`crate::varargs`] says a list has to be built out of.
    ///
    /// The address is computed once into a register rather than written as a displacement off the
    /// stack pointer, because a displacement into a frame is not known until after allocation and
    /// one `lea` costs less than a fixup list for a dozen stores. It is the same `lea` an `alloca`
    /// gets and [`crate::finish`] fills it in the same way.
    ///
    /// A convention that homes its register arguments has none of that. Its area is the shadow
    /// space the caller reserved above the return address, so there is no object to make and no
    /// address to work out: each store reaches into the caller's argument area the way the load of
    /// a parameter the registers ran out before does, which is the same waiting list and the same
    /// fixup. There are at most four of them and none is a vector register, since a float the
    /// signature does not name arrived in a general purpose register too and that is the copy the
    /// walk reads.
    fn save_area(&mut self, out: mir::Block, arrived: &abi::Arrived, area: varargs::Area) {
        if self.conv.shared_positions {
            self.varargs = Some(Varargs::Pointer { incoming: arrived.beyond });
            let store = mir::Opcode::new(self.names.intern("x64.mov_mr_64"));
            for &(reg, class, at) in &arrived.spare {
                let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
                let made =
                    self.out.build(out, store).uses(reg, class).mem(mir::Mem::at(sp)).finish();
                self.stack.arguments.push((made, at));
            }
            return;
        }

        let save = self.stack.locals.len();
        self.stack.locals.push(Local { size: area.size, align: varargs::VECTOR_SLOT });
        self.varargs = Some(Varargs::Fields {
            save,
            incoming: arrived.beyond,
            integers: u32::try_from(arrived.took.0).unwrap_or(0) * area.stride(false),
            floats: area.starts_at(true)
                + u32::try_from(arrived.took.1).unwrap_or(0) * area.stride(true),
        });

        let base = self.frame_address(out, save);
        for &(reg, class, at) in &arrived.spare {
            let name = if class == self.gpr { "x64.mov_mr_64" } else { "x64.movaps_mr" };
            let store = mir::Opcode::new(self.names.intern(name));
            let up = i32::try_from(at).expect("a register save area under two gigabytes");
            let mem = mir::Mem::at(mir::Operand::read(base, self.gpr)).plus(up);
            self.out.build(out, store).uses(reg, class).mem(mem).finish();
        }
    }

    /// The address of one of the function's stack objects, in a fresh register.
    ///
    /// Written with nothing in its displacement, because where an object is in a frame is not known
    /// until after allocation, and given to [`crate::finish`] to fill in the way an `alloca` is.
    fn frame_address(&mut self, out: mir::Block, local: usize) -> mir::Reg {
        let reg = self.out.new_vreg(self.gpr);
        let lea = mir::Opcode::new(self.names.intern(&format!("{PREFIX}{}", x86_64::FRAME.lea)));
        let sp = mir::Operand::read(mir::Reg::physical(self.conv.stack_pointer), self.gpr);
        let made = self.out.build(out, lea).def(reg, self.gpr).mem(mir::Mem::at(sp)).finish();
        self.stack.addresses.push((made, local));
        reg
    }

    /// Whether an instruction is one no machine instruction is written for where it stands.
    ///
    /// Four of them, and none is a lowering decision, which is why none is a rule. A constant is
    /// written where a register for it is first wanted rather than where the IR put it, and every
    /// reader of one may have folded it into an immediate, in which case nowhere is the right
    /// place. A return of nothing has nothing to put anywhere: the epilogue gives the frame back
    /// and leaves, and it is appended to every block with no successors long after this has
    /// finished, so a return with a value is one instruction here and a return without one is
    /// none. Unless the value went back through memory, in which case there is something to put
    /// somewhere after all and the IR does not carry it: the address the caller handed over has
    /// to be in `rax` on the way out, and [`Lowering::returned`] is what writes that.
    ///
    /// An unconditional jump is the third, and there is even less of it: the edge is on the
    /// block, and whether the block it goes to is the next one and needs no jump at all is the
    /// block layout's answer rather than this one's.
    ///
    /// The fourth is a point control does not arrive at, in both of the forms the IR has for it:
    /// the `unreachable` terminator the front end puts at the end of a function whose body can run
    /// off the bottom, and the `unreachable_hint` a call to `__builtin_unreachable` becomes. What
    /// to write for a place nothing reaches is a question with no wrong answer, and nothing is the
    /// smallest one and the one gcc 16.2.0 gives at `-O0`. The terminator leaves the block with no
    /// successors, so the epilogue lands at the end of it the way it does on any other block that
    /// goes nowhere, and the function cannot fall out of its own last instruction into whatever
    /// the assembler puts next.
    fn writes_nothing(&self, inst: Inst) -> bool {
        let data = &self.source[inst];
        match data.opcode {
            Opcode::IConst | Opcode::Jump | Opcode::Unreachable | Opcode::UnreachableHint => true,
            Opcode::Return => self.source[data.args].is_empty() && self.sret().is_none(),
            _ => false,
        }
    }

    /// What every instruction in one block matched, with a set of values nobody may take.
    ///
    /// Backwards, because an instruction that has been folded into a later one does not get to
    /// fold anything into itself: the rule that took it only reached one level down, so what is
    /// under it is not in the term the matcher saw and cannot be replaced.
    fn decide(&self, insts: &[Inst], refused: &HashSet<Value>) -> Decided {
        let mut found: Vec<Option<Match<Term>>> = (0..insts.len()).map(|_| None).collect();
        let mut plans: Vec<Option<Plan>> = vec![None; insts.len()];
        let mut folded: Vec<Inst> = Vec::new();
        for (index, &inst) in insts.iter().enumerate().rev() {
            if folded.contains(&inst) {
                continue;
            }
            if let Some((plan, matched)) = self.select(inst, refused) {
                folded.extend(self.folds(inst, plan));
                found[index] = Some(matched);
                plans[index] = Some(plan);
            }
        }
        Decided { found, plans, folded }
    }

    /// A value some of its readers took and some of them did not, which is the one case folding
    /// buys nothing.
    ///
    /// Folding does not delete the instruction that computed a value for anybody else, so a
    /// reader that did not take it still needs it in a register and the instruction stays. The
    /// reader that did take it now does that work again. Either all of them take it, in which
    /// case nothing is left to read it and the instruction goes, or none of them do.
    ///
    /// The count is over the whole function rather than over the block, since a value read from
    /// another block is read from a register there whatever this block decides. An instruction
    /// built by name rather than matched, a call being the one that matters, has no plan and so
    /// takes nothing, which is the right answer for it as well.
    fn left_alive(&self, insts: &[Inst], plans: &[Option<Plan>]) -> Option<Value> {
        let mut taken = vec![0u32; self.uses.len()];
        for (&inst, plan) in insts.iter().zip(plans) {
            let Some(plan) = plan else { continue };
            let args = &self.source[self.source[inst].args];
            for (index, &arg) in args.iter().take(MAX_ARGS).enumerate() {
                if plan[index] == Shown::Expand {
                    taken[arg.index()] += 1;
                }
            }
        }
        for (&inst, plan) in insts.iter().zip(plans) {
            let Some(plan) = plan else { continue };
            let args = &self.source[self.source[inst].args];
            for (index, &arg) in args.iter().take(MAX_ARGS).enumerate() {
                if plan[index] == Shown::Expand && taken[arg.index()] < self.uses[arg.index()] {
                    return Some(arg);
                }
            }
        }
        None
    }

    /// The rule that fires on an instruction, and what it bound.
    ///
    /// The plans are tried in order and the first that matches wins, which is the maximal munch
    /// `spec/10-backend.md` asks for: a plan that offers more to the matcher is tried before one
    /// that offers less.
    fn select(&self, inst: Inst, refused: &HashSet<Value>) -> Option<(Plan, Match<Term>)> {
        for plan in self.plans(inst, refused) {
            let terms = Terms::new(self.source, inst, plan);
            if let Some(matched) = TABLE.find(&terms, Term::Root) {
                return Some((plan, matched));
            }
        }
        None
    }

    /// Every way this instruction can be shown to the matcher, most offered first.
    fn plans(&self, inst: Inst, refused: &HashSet<Value>) -> Vec<Plan> {
        let args = &self.source[self.source[inst].args];
        let mut plans = vec![PLAIN];
        for (index, &arg) in args.iter().enumerate().take(MAX_ARGS) {
            let mut ways = Vec::new();
            if self.foldable(inst, arg, refused) {
                ways.push(Shown::Expand);
            }
            if Terms::new(self.source, inst, PLAIN).constant(arg).is_some() {
                ways.push(Shown::Const);
            }
            ways.push(Shown::Reg);
            plans = plans
                .into_iter()
                .flat_map(|plan| {
                    ways.iter().map(move |&way| {
                        let mut next = plan;
                        next[index] = way;
                        next
                    })
                })
                .collect();
        }
        plans
    }

    /// Whether an operand may be shown as the instruction that computed it.
    ///
    /// It has to be in the same block, because a rule that folds one instruction into another
    /// moves the work to where the second one is. It has to be something rather than a block
    /// parameter, and not a constant, which is shown as a constant instead. And it has to be a
    /// value [`Lowering::left_alive`] has not put back, which is how the one reader at a time
    /// question is asked here: this says yes to a value with any number of readers, and a value
    /// only some of them could take is refused after the fact and asked again.
    ///
    /// A value with several readers used to be refused outright, on the reasoning that folding
    /// does not delete the instruction for anybody else. That reasoning is about the set of
    /// readers and was being applied to one reader at a time, which is stricter than it needs to
    /// be: when every reader takes it there is nobody left to read it and the instruction goes.
    /// An address a store and a load share is the shape that matters, since a memory operand has
    /// room for the whole of it and both readers have a memory operand.
    fn foldable(&self, into: Inst, value: Value, refused: &HashSet<Value>) -> bool {
        let Def::Result { inst, .. } = self.source[value].def else { return false };
        if self.source[inst].opcode == Opcode::IConst || refused.contains(&value) {
            return false;
        }
        self.source.block_of(inst).is_some()
            && self.source.block_of(inst) == self.source.block_of(into)
    }

    /// The instructions a match folded into the one it matched.
    ///
    /// The plan is what says this, not the bindings: a binding is a register or a number either
    /// way, and an operand shown as the instruction that computed it is one no rule could have
    /// matched without taking that instruction, because the plan offered the matcher nothing
    /// else to call it.
    fn folds(&self, inst: Inst, plan: Plan) -> Vec<Inst> {
        let args = &self.source[self.source[inst].args];
        args.iter()
            .take(MAX_ARGS)
            .enumerate()
            .filter(|&(index, _)| plan[index] == Shown::Expand)
            .filter_map(|(_, &arg)| match self.source[arg].def {
                Def::Result { inst, .. } => Some(inst),
                Def::Param { .. } => None,
            })
            .collect()
    }

    /// What the IR instruction said about itself that the machine instruction has to keep saying.
    ///
    /// One flag today. `volatile` says the access happens exactly once and is never moved or
    /// merged, and nothing below here can work that out again: a `volatile` load and an ordinary
    /// one are the same instruction over the same address, so a pass that puts two accesses
    /// together would put these together too. Carried rather than checked here, because the pass
    /// that has to refuse is a long way down and this is the last place the answer is known.
    ///
    /// The instructions this compiler writes for itself get nothing, which is the right answer
    /// for all of them: a prologue, a spill and the moves around a call were asked for by the
    /// machine rather than by the program.
    ///
    /// Every access the flag is legal on carries it: the loads and the stores a rule matched,
    /// the two ends of a `long double` copy that are the program's own memory, and the compare
    /// and exchange and the read modify write. An `asm` statement does not, and it is the one
    /// exception on purpose. What the flag says there is that the statement stays even when
    /// nothing reads what it wrote, which is a different sentence about a different thing, and
    /// every `asm` is already fixed where it stands whether the word was written or not.
    fn carried(&self, inst: Inst) -> mir::Flags {
        if self.source[inst].flags.contains(Flags::VOLATILE) {
            mir::Flags::VOLATILE
        } else {
            mir::Flags::NONE
        }
    }

    /// Build the machine instruction a match calls for.
    fn emit(&mut self, inst: Inst, matched: &Match<Term>) -> Result<(), Unsupported> {
        let rule: &Rule = TABLE.rule(matched);
        let pieces = rule.replacement;
        let Some(Piece::App { head, arity }) = pieces.first() else {
            return Err(self.unsupported(inst));
        };
        let opcode = head.strip_prefix(PREFIX).ok_or_else(|| self.unsupported(inst))?;
        let form = x86_64::form(opcode).ok_or_else(|| self.unsupported(inst))?;

        let mut read = Read::default();
        let mut at = 1;
        for _ in 0..*arity {
            at = self.read(inst, pieces, at, &matched.bindings, &mut read)?;
        }

        let descs = form.operands();
        let writes = descs.iter().take_while(|desc| desc.role.is_def()).count();
        if descs.len() - writes != read.regs.len() {
            return Err(self.unsupported(inst));
        }

        // The first thing the instruction writes is what it computes, and any others are
        // registers the machine destroys on the way, which are fresh because nothing else is in
        // them and nothing reads them. An instruction that writes nothing at all is one whose
        // whole purpose is its effect, which is what a store is, and there is no result to put
        // anywhere.
        let mut regs = Vec::new();
        if writes > 0 {
            let result = self.source[inst].first_result.ok_or_else(|| self.unsupported(inst))?;
            regs.push(self.new_reg(result));
            // The rest are the registers the machine destroys on the way, and the class each is in
            // is the one the instruction's description gives it rather than a guess, so that an
            // instruction that wrecks a register in the other file says so.
            regs.extend(descs[1..writes].iter().map(|desc| self.out.new_vreg(desc.class)));
        } else if self.source[inst].first_result.is_some() {
            // A rule that throws away a value the IR gave a name to would leave every reader of
            // that name with nothing to read, so it is a rule this and the target disagree about.
            return Err(self.unsupported(inst));
        }
        regs.extend(read.regs.iter().copied());

        let block = self.at.expect("a block is being filled");
        let opcode = mir::Opcode::new(self.names.intern(head));
        let (span, flags) = (self.source.span(inst), self.carried(inst));
        let mut build = self.out.build(block, opcode).at(span).flags(flags);
        for (desc, reg) in descs.iter().zip(regs) {
            let operand = mir::Operand {
                reg,
                class: desc.class,
                role: desc.role,
                constraint: desc.constraint,
            };
            build = build.operand(operand);
        }
        if let Some(mem) = read.mem {
            build = build.mem(mem);
        }
        if let Some(imm) = read.imm {
            build = build.imm(imm);
        }
        build.finish();
        Ok(())
    }

    /// Read one argument of a replacement, which is a register, a number or an address.
    ///
    /// Gives back the position after it, because a replacement is flat and an address takes
    /// arguments of its own.
    fn read(
        &mut self,
        inst: Inst,
        pieces: &'static [Piece],
        at: usize,
        bindings: &[Term],
        out: &mut Read,
    ) -> Result<usize, Unsupported> {
        match pieces.get(at) {
            Some(Piece::Int(value)) => {
                out.imm = i64::try_from(*value).ok();
                Ok(at + 1)
            }
            // A number the rule worked out of the ones it matched rather than one it wrote down,
            // which is an immediate once it has been worked out and is read here as one. It gives
            // nothing back when a binding it reads is a register, and a replacement that cannot be
            // built is a rule this file and the matcher disagree about, which is what `unsupported`
            // is for.
            Some(Piece::Computed { work, .. }) => {
                let matched: Vec<Option<i128>> = bindings
                    .iter()
                    .map(|term| match *term {
                        Term::Num(value) => Some(value),
                        _ => None,
                    })
                    .collect();
                let number = work(&matched).ok_or_else(|| self.unsupported(inst))?;
                out.imm = i64::try_from(number).ok();
                Ok(at + 1)
            }
            Some(Piece::Var { index, .. }) => {
                match bindings.get(*index) {
                    Some(&Term::Reg(value)) => {
                        let reg = self.reg_of(value)?;
                        out.regs.push(reg);
                    }
                    Some(&Term::Num(value)) => out.imm = i64::try_from(value).ok(),
                    // A pattern binds a register or a number and nothing else, so this is a
                    // rule the matcher and this file disagree about.
                    _ => return Err(self.unsupported(inst)),
                }
                Ok(at + 1)
            }
            Some(Piece::App { head, arity }) => {
                let kind = x86_64::address(head).ok_or_else(|| self.unsupported(inst))?;
                let mut inner = Read::default();
                let mut next = at + 1;
                for _ in 0..*arity {
                    next = self.read(inst, pieces, next, bindings, &mut inner)?;
                }
                let mem = address(kind, &inner, self.gpr).ok_or_else(|| self.unsupported(inst))?;
                out.mem = Some(mem);
                Ok(next)
            }
            None => Err(self.unsupported(inst)),
        }
    }

    /// The register a value is in, materializing it if it is a constant that has not been put in
    /// one yet.
    ///
    /// A constant is written where it is wanted rather than where the IR defined it, and where it
    /// is wanted is a block that need not be the one the IR defined it in. So the register holding
    /// one is only good inside the block it was written into, and a second block that wants the
    /// same constant gets its own. Anything else is a register read where nothing wrote it: the
    /// IR guarantees a definition dominates its uses, and this moved the definition.
    ///
    /// Writing the number again is also the right answer and not merely the safe one. It is one
    /// instruction that reads nothing, which is cheaper than holding a register live across a
    /// branch for it, and it is what a rematerializing allocator would do with the value anyway.
    fn reg_of(&mut self, value: Value) -> Result<mir::Reg, Unsupported> {
        let constant = match self.source[value].def {
            Def::Result { inst, .. } => {
                (self.source[inst].opcode == Opcode::IConst).then_some(inst)
            }
            Def::Param { .. } => None,
        };
        let here = self.at.expect("a block is being filled");
        if let Some(reg) = self.regs[value.index()] {
            if constant.is_none() || self.written[value.index()] == Some(here) {
                return Ok(reg);
            }
        }
        if let Some(inst) = constant {
            // Cleared so that the register the constant is written into is a new one rather than
            // the one the block above wrote, which is still being read up there.
            self.regs[value.index()] = None;
            // Nothing is refused here. A constant is written on its own, out of the loop over the
            // block, and the operands of the rule that writes one are the number and nothing else.
            let matched = self
                .select(inst, &HashSet::new())
                .map(|(_, matched)| matched)
                .ok_or_else(|| self.unsupported(inst))?;
            self.emit(inst, &matched)?;
            // The same mark the loop over the instructions makes, and it has to be made here as
            // well because this is the only place a constant is ever selected: the loop skips one
            // where the IR wrote it, so a rule that lowers a constant fires from nowhere else and
            // would be reported as a rule nothing reaches.
            self.fired.mark(matched.rule);
            self.written[value.index()] = Some(here);
            return Ok(self.regs[value.index()].expect("a constant is written into a register"));
        }
        Ok(self.new_reg(value))
    }

    /// Which register file a value of that type lives in.
    ///
    /// The vector one for the two float widths the machine has scalar instructions for and for the
    /// one it only moves, and the general purpose one for everything else. An eighty bit `long
    /// double` is in neither, and it is here rather than in the vector class on purpose: it would
    /// be put in a register that cannot hold it, and there is no rule that names one, so the
    /// instruction computing it is reported. The wrong class would make that a wrong program
    /// instead of a refused one.
    ///
    /// A hundred and twenty eight bit float is in the vector class and fits it exactly, which is
    /// the difference. Nothing computes in it, so every arithmetic on one is still reported, and
    /// what the class buys is the moves: a register that holds the whole value is a register a
    /// spill, a reload and a copy are each one instruction for.
    fn class_of(&self, ty: Type) -> RegClass {
        if crate::term::in_vector_file(ty) { self.conv.sse_class } else { self.gpr }
    }

    /// A fresh register for a value, which is what the instruction computing it writes.
    fn new_reg(&mut self, value: Value) -> mir::Reg {
        if let Some(reg) = self.regs[value.index()] {
            return reg;
        }
        let reg = self.out.new_vreg(self.class_of(self.source[value].ty));
        self.regs[value.index()] = Some(reg);
        reg
    }

    fn unsupported(&self, inst: Inst) -> Unsupported {
        let data = &self.source[inst];
        Unsupported::Inst {
            inst,
            term: Terms::new(self.source, inst, PLAIN).name(inst),
            opcode: data.opcode,
            ty: data.first_result.map(|result| self.source[result].ty),
        }
    }
}

/// What the arguments of one replacement came to.
#[derive(Debug, Default)]
struct Read {
    regs: Vec<mir::Reg>,
    imm: Option<i64>,
    mem: Option<mir::Mem>,
}

/// The addressing mode an address constructor's arguments make.
///
/// One arm per constructor rather than a question asked of the kind, because what the arguments
/// mean is the whole of what tells the four apart: the same register is a base in one and an
/// index in another, and the same constant is a scale in one and a displacement in another.
fn address(kind: x86_64::Address, read: &Read, gpr: RegClass) -> Option<mir::Mem> {
    let mut regs = read.regs.iter().copied().map(|reg| mir::Operand::read(reg, gpr));
    match kind {
        x86_64::Address::BaseIndexScale => {
            let base = regs.next()?;
            let index = regs.next()?;
            Some(mir::Mem::at(base).indexed(index, u8::try_from(read.imm?).ok()?))
        }
        x86_64::Address::IndexScale => Some(mir::Mem {
            base: None,
            index: Some(regs.next()?),
            scale: u8::try_from(read.imm?).ok()?,
            disp: 0,
            symbol: None,
            block: None,
            reach: mir::Reach::Itself,
            segment: None,
        }),
        x86_64::Address::Base => Some(mir::Mem::at(regs.next()?)),
        // The rule that writes this has a guard saying the constant fits, so a displacement that
        // does not is a rule and a target that disagree rather than a program this cannot compile.
        x86_64::Address::BaseOffset => {
            Some(mir::Mem { disp: i32::try_from(read.imm?).ok()?, ..mir::Mem::at(regs.next()?) })
        }
    }
}

/// The table this selector matches with.
///
/// One target for now, because one target has a rule file. Which table to use becomes a question
/// the moment a second one does, and the answer will be the target the session was given rather
/// than a constant here.
static TABLE: &Table = &crate::select::x86_64::TABLE;

#[cfg(test)]
mod tests {
    use rucc_ir::{
        AsmInfo, Builder, CallInfo, Flags, InstData, MemInfo, MemOrder, Restrict, Signature, Type,
    };
    use rucc_regalloc::assign::Env;
    use rucc_target::x86_64::{FRAME, REGS, SYSV};

    use super::*;
    use crate::finish::{Convention, finish};
    use crate::frame::{Frame, Incoming, Layout};

    /// A function of as many 64 bit parameters as the test wants, and the block they are in.
    fn blank(params: &[Type]) -> (Interner, Func, Block, Vec<Value>) {
        let mut names = Interner::new();
        let mut func = Func::new(names.intern("f"), Signature::new());
        let block = func.create_block();
        let values = params.iter().map(|&ty| func.append_param(block, ty)).collect();
        (names, func, block, values)
    }

    /// An ordinary access: not atomic, and aligned enough that nothing here has an opinion.
    /// Neither field reaches selection, which is the point of saying it once here.
    fn plain() -> MemInfo {
        MemInfo {
            size: 0,
            align: 1,
            order: MemOrder::NotAtomic,
            tbaa: None,
            owns: 0,
            restrict: Restrict::NONE,
        }
    }

    /// What the allocator is given: every integer register the convention offers except two, held
    /// back so that a move on an edge has somewhere to break a cycle and a spilled value has
    /// somewhere to be read into. Which two does not matter, and holding back the last two the
    /// convention would reach for leaves every expectation below unchanged.
    fn env() -> Env {
        const SCRATCH: [PhysReg; 2] = [x86_64::R10, x86_64::R11];
        let order: Vec<PhysReg> =
            SYSV.int_order.iter().copied().filter(|reg| !SCRATCH.contains(reg)).collect();
        Env::new().with(x86_64::GPR, &order, &SCRATCH)
    }

    /// The machine IR text a function lowers to.
    fn lower(names: &mut Interner, source: &Func) -> String {
        let out = func(source, names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        mir::print_func(&out.func, names, &REGS)
    }

    #[test]
    fn an_addition_of_two_registers_is_one_instruction() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::Add, args[0], args[1], Flags::default());

        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr(reuse 1) = x64.add_rr_32 %0, %1\n}\n"
        );
    }

    #[test]
    fn a_constant_operand_becomes_an_immediate() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32]);
        let mut build = Builder::new(&mut func, block);
        let seven = build.iconst(i32, 7);
        build.binary(Opcode::Add, args[0], seven, Flags::default());

        // The constant is in the instruction and nothing was written to hold it, which is what
        // materializing one where a register for it is wanted buys.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr(reuse 1) = x64.add_ri_32 %0, 7\n}\n"
        );
    }

    #[test]
    fn a_constant_too_wide_for_an_immediate_goes_into_a_register() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let big = build.iconst(i64, i128::from(i32::MAX) + 1);
        build.binary(Opcode::Add, args[0], big, Flags::default());

        // Nobody wrote this fallback down. The rule that takes an immediate has a guard that
        // turns a number this wide down, so it does not fire, and the next way of showing the
        // operand puts it in a register.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_ri_64 2147483648\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n}\n"
        );
    }

    #[test]
    fn an_index_calculation_folds_into_an_address() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        build.binary(Opcode::Add, args[0], scaled, Flags::default());

        // Three IR instructions and one machine instruction. The multiply is gone because the
        // rule that matched reached down and took it.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.lea_64 [%0 + %1*4]\n}\n"
        );
    }

    #[test]
    fn an_instruction_every_reader_can_take_is_folded_into_all_of_them() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        let first = build.binary(Opcode::Add, args[0], scaled, Flags::default());
        build.binary(Opcode::Add, first, scaled, Flags::default());

        // Both readers have room for a scaled index, so both of them take it and nothing is left
        // to read the multiply. Three IR instructions become two machine ones, where refusing to
        // fold into either reader would have left three.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.lea_64 [%0 + %1*4]\n    \
             %3:gpr = x64.lea_64 [%2 + %1*4]\n}\n"
        );
    }

    #[test]
    fn an_instruction_one_of_its_readers_cannot_take_is_folded_into_none_of_them() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let four = build.iconst(i64, 4);
        let scaled = build.binary(Opcode::Mul, args[1], four, Flags::default());
        build.binary(Opcode::Add, args[0], scaled, Flags::default());
        build.store(scaled, args[0], plain(), Flags::default());

        // The addition has room for the multiply and the store does not: what a store writes is
        // a register, and no rule reaches through it. Folding into the addition alone would
        // leave the multiply where it is for the store to read and do the work twice, so the
        // multiply is put back and both readers read the register it wrote.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.lea_64 [%1*4]"), "{text}");
        assert!(text.contains("x64.add_rr_64"), "{text}");
    }

    #[test]
    fn a_shift_by_a_register_asks_for_it_in_cl() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::Shl, args[0], args[1], Flags::default());

        // The fixed register is not in the rule. It is what the target says the instruction does
        // with its operands, and the allocator is what will act on it.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.shl_rcl_32 %0, %1($rcx)"), "{text}");
    }

    #[test]
    fn a_division_names_the_registers_and_the_register_it_destroys() {
        let i32 = Type::int(32);
        let (mut names, mut func, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut func, block);
        build.binary(Opcode::SDiv, args[0], args[1], Flags::default());

        // Two definitions, because a division writes the remainder whether anybody wanted it or
        // not, and the second one is early because it is destroyed before the operands are read.
        let text = lower(&mut names, &func);
        assert!(
            text.contains("%2:gpr($rax), early %3:gpr($rdx) = x64.idiv_quo_32 %0($rax), %1"),
            "{text}"
        );
    }

    #[test]
    fn a_load_reads_through_the_register_the_address_is_in() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        build.load(Type::int(32), args[0], plain(), Flags::default());

        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_rm_32 [%0]\n}\n"
        );
    }

    #[test]
    fn a_store_writes_no_register_and_the_value_it_writes_is_the_one_the_ir_gave_it() {
        let (mut names, mut func, block, args) = blank(&[Type::int(32), Type::int(64)]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[0], args[1], plain(), Flags::default());

        // The value is the first parameter and the address is the second, and the instruction
        // takes them the other way round. Getting that backwards would compile to a store of the
        // address into the value, which is a program that runs and does the wrong thing.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    x64.mov_mr_32 %0, [%1]\n}\n"
        );
    }

    #[test]
    fn an_address_with_a_constant_added_folds_into_the_access() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let twelve = build.iconst(i64, 12);
        let field = build.binary(Opcode::Add, args[0], twelve, Flags::default());
        build.load(Type::int(64), field, plain(), Flags::default());

        // Two IR instructions and one machine instruction, which is what every read of a field
        // of a structure comes to.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr = x64.mov_rm_64 [%0 + 12]\n}\n"
        );
    }

    #[test]
    fn a_displacement_too_wide_to_encode_leaves_the_addition_where_it_is() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut func, block);
        let big = build.iconst(i64, i128::from(i32::MAX) + 1);
        let far = build.binary(Opcode::Add, args[0], big, Flags::default());
        build.load(Type::int(32), far, plain(), Flags::default());

        // A displacement is signed and 32 bits. The rule that folds one has a guard that turns
        // this down, so the addition stays and the load reads through what it produced. Nobody
        // wrote that fallback: it is the next way of showing the operand.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.mov_rm_32 [%2]"), "{text}");
        assert!(text.contains("x64.add_rr_64"), "{text}");
    }

    #[test]
    fn a_store_of_a_value_that_was_loaded_is_two_instructions_and_no_arithmetic() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        let got = build.load(Type::int(8), args[0], plain(), Flags::default());
        build.store(got, args[1], plain(), Flags::default());

        // A load feeding a store is the one place folding would be wrong: an x86-64 `mov` has at
        // most one memory operand, and there is no rule that takes two, so the load is left where
        // it is and the store reads the register it wrote.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr = x64.mov_rm_8 [%0]\n    \
             x64.mov_mr_8 %2, [%1]\n}\n"
        );
    }

    #[test]
    fn an_access_at_a_width_no_rule_is_written_at_is_reported() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64]);
        let mut build = Builder::new(&mut source, block);
        build.load(Type::int(128), args[0], plain(), Flags::default());

        // The width is the whole of what is wrong here, so the width is in the message: `load`
        // on its own is written about at every other width and would send a reader looking in
        // the wrong place.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("nothing loads 128 bits");
        assert_eq!(failed.to_string(), "no rule lowers a `load` producing a `i128`");
    }

    #[test]
    fn a_return_asks_for_the_value_in_the_register_the_caller_reads() {
        let (mut names, mut func, block, args) = blank(&[Type::int(32)]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0]]);

        // The register is not in the rule, the same way `cl` is not in the rule for a shift. It
        // is what the target says the instruction does with its operand, and the allocator is
        // what will act on it. There is no `ret` here, because giving the frame back has to
        // happen between this and leaving and the frame is not worked out yet.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_return_of_two_values_asks_for_the_second_register_as_well() {
        let i64 = Type::int(64);
        let (mut names, mut func, block, args) = blank(&[i64, i64]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { long a, b; } f(long a, long b)`, after the front end has classified it. Both
        // halves are integers, so the second is in the second integer return register, and both
        // pseudos say so the same way the one for a single value does.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    x64.ret_val_64 %0($rax)\n    \
             x64.ret_val2_64 %1($rdx)\n}\n"
        );
    }

    #[test]
    fn two_values_back_in_different_files_are_both_the_first_of_their_own() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut func, block, args) = blank(&[f64, Type::int(64)]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { double a; long b; } f(double a, long b)`. The two files are counted apart, so
        // neither half is the second of anything and the `double` is in `xmm0` rather than in the
        // register a second `double` would have been in. Getting this wrong is not a crash: the
        // caller reads a register nobody wrote, and this is where that is ruled out.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:gpr($rdi) = x64.arg_val_64\n    x64.ret_val_f64 %0($xmm0)\n    \
             x64.ret_val_64 %1($rax)\n}\n"
        );
    }

    #[test]
    fn two_of_the_same_file_back_take_the_first_two_of_it() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut func, block, args) = blank(&[f64, f64]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[args[0], args[1]]);

        // `struct { double x, y; } f(double x, double y)`, which is the vector half of the pair
        // above and counts in its own file the same way.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:xmm($xmm1) = x64.arg_val_f64\n    x64.ret_val_f64 %0($xmm0)\n    \
             x64.ret_val2_f64 %1($xmm1)\n}\n"
        );
    }

    /// A function whose answer goes back through memory, with the pointer to the space for it in
    /// front of whatever else it takes. Only the signature says it is one.
    fn returning_through_memory(params: &[Type]) -> (Interner, Func, Block, Vec<Value>) {
        let mut names = Interner::new();
        let sret = Abi::Sret { size: 32, align: 8 };
        let mut signature = Signature::new().and_param(Param::with_abi(Type::PTR, sret));
        signature.params.extend(params.iter().copied().map(Param::new));
        let mut func = Func::new(names.intern("f"), signature);
        let block = func.create_block();
        let space = func.append_param(block, Type::PTR);
        let values = std::iter::once(space)
            .chain(params.iter().map(|&ty| func.append_param(block, ty)))
            .collect();
        (names, func, block, values)
    }

    #[test]
    fn the_space_a_return_through_memory_was_given_goes_back_in_the_first_return_register() {
        let (mut names, mut func, block, _) = returning_through_memory(&[]);
        Builder::new(&mut func, block).ret(&[]);

        // `struct big f(void)`, where `big` is too large to come back in registers. The `return`
        // carries nothing, because the value went into the space the caller handed over, and the
        // document still says that address comes back in `rax`. Nothing in the IR says it, so the
        // convention says it, and the pseudo is the one any other pointer return would use.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn what_the_function_did_in_between_does_not_take_the_register_off_it() {
        let (mut names, mut func, block, args) = returning_through_memory(&[Type::int(32)]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[1], args[0], plain(), Flags::default());
        build.ret(&[]);

        // The register is a read at the end and not a move at the start, so it is live across
        // everything between the two and the allocator has to keep it somewhere. In a function
        // with a call in it that somewhere is a callee saved register, and the address comes back
        // into `rax` here rather than whatever the last instruction happened to leave there. That
        // is issue #333, and a store is enough to show the value outlives the entry block.
        let text = lower(&mut names, &func);
        assert!(text.contains("x64.mov_mr_32 %1, [%0]"), "{text}");
        assert!(text.ends_with("    x64.ret_val_64 %0($rax)\n}\n"), "{text}");
    }

    #[test]
    fn a_pointer_that_is_only_a_pointer_is_not_given_back() {
        let (mut names, mut func, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut func, block);
        build.store(args[0], args[0], plain(), Flags::default());
        build.ret(&[]);

        // `void f(void **p)`. It takes a pointer first and returns nothing, which is the shape of
        // the one above and none of its meaning, and what tells them apart is the signature. A
        // `void` function leaves `rax` alone.
        assert!(!lower(&mut names, &func).contains("ret_val"));
    }

    #[test]
    fn a_return_of_a_constant_puts_it_in_a_register_first() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        // No rule returns an immediate, so the plan that offers one is turned down and the next
        // one materializes it. That is `int main(void) { return 0; }` in full, once the epilogue
        // is appended to it.
        assert_eq!(
            lower(&mut names, &func),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_rule_that_writes_a_constant_down_is_recorded_as_a_rule_that_fired() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        // The loop over the instructions passes a constant by, because a constant is written where
        // a register for it is first wanted rather than where the IR put it. So the only place a
        // rule about one is ever selected is the materialization, and a mark made in the loop
        // alone would report every rule about a constant as a rule nothing reaches.
        let out = super::func(&func, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let rules = &crate::select::x86_64::TABLE.rules;
        let fired: Vec<&str> = rules
            .iter()
            .enumerate()
            .filter(|(index, _)| out.fired.has(*index))
            .map(|(_, rule)| rule.pattern)
            .collect();
        assert!(fired.contains(&"(iconst.i32 k)"), "{fired:?}");
    }

    #[test]
    fn a_return_of_nothing_is_no_instruction_at_all() {
        let (mut names, mut func, block, _) = blank(&[]);
        let mut build = Builder::new(&mut func, block);
        build.ret(&[]);

        // Every part of leaving a function that returns nothing is the epilogue's, and the
        // epilogue goes in after allocation. A block with nothing in it is the right answer here
        // rather than a function that could not be lowered.
        assert_eq!(lower(&mut names, &func), "mfunc @f {\nblock0:\n}\n");
    }

    #[test]
    fn the_allocator_is_what_moves_the_answer_into_the_return_register() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let zero = build.iconst(Type::int(32), 0);
        build.ret(&[zero]);

        let mut out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // `int main(void) { return 0; }` end to end. Nothing here asked for `rax`: the rule said
        // the value goes back, the target said where, and the allocator is what made it true. The
        // epilogue is what leaves, and this function needs no frame, so it is the return alone.
        //
        // Two instructions and no copy, which is what a hint buys. The return insists on `rax`,
        // so `rax` is the register the allocator tries first for the value the return reads, and
        // the constant is written straight into it.
        assert_eq!(
            mir::print_func(&out, &names, &REGS),
            "mfunc @f {\nblock0:\n    $rax = x64.mov_ri_32 0\n    \
             x64.ret_val_32 $rax($rax)\n    x64.ret\n}\n"
        );
    }

    #[test]
    fn a_function_of_two_arguments_is_a_whole_function_now() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, args[0], args[1], Flags::default());
        build.ret(&[sum]);

        let mut out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // `int f(int a, int b) { return a + b; }` end to end, and this is the test the argument
        // side exists for. Before it there was no way to write one: the allocator refuses a
        // function whose entry block takes parameters, because there is no edge into an entry
        // block for the moves that give a block parameter its value to go on.
        //
        // One move, and it is the one the machine's addition needs rather than one the allocator
        // owes anybody. Each argument stays in the register it arrived in, because the pseudo
        // that defines it insists on that register and the allocator now tries it first, and the
        // sum stays in the register the addition wrote it to until the return reads it out. The
        // copy in front of a two address instruction is what makes its destination one of the
        // registers it reads, and the source operand keeps its own name because the destination
        // is what the encoder writes.
        assert_eq!(
            mir::print_func(&out, &names, &REGS),
            "mfunc @f {\nblock0:\n    $rdi($rdi) = x64.arg_val_32\n    \
             $rsi($rsi) = x64.arg_val_32\n    \
             $rdi(reuse 1) = x64.add_rr_32 $rdi, $rsi\n    $rax = x64.mov_rr_64 $rdi\n    \
             x64.ret_val_32 $rax($rax)\n    x64.ret\n}\n"
        );
    }

    #[test]
    fn an_argument_with_no_register_left_for_it_is_read_out_of_the_caller_s_stack() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 7]);
        let mut build = Builder::new(&mut source, block);
        build.ret(&[args[6]]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("the seventh is read from memory");

        // SysV passes six integers in registers and the seventh in the caller's memory, so six of
        // these are pseudos that encode to nothing and the seventh is a load that encodes to real
        // bytes. Its displacement is nothing here for the reason a local's is: there is no frame
        // yet. What the walk hands on is which instruction is waiting, and for how far up the
        // caller's argument area, which is the bottom of it because it is the first one there.
        assert_eq!(lowered.stack.arguments.len(), 1);
        assert_eq!(lowered.stack.arguments[0].1, 0);
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("%6:gpr = x64.mov_rm_64 [$rsp]"), "{text}");
        assert_eq!(text.matches("x64.arg_val_64").count(), 6, "{text}");
    }

    #[test]
    fn the_frame_is_what_says_how_far_up_the_caller_s_stack_an_argument_is() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 8]);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, args[6], args[7], Flags::default());
        build.ret(&[sum]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("both are read from memory");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // A leaf that takes no frame, so the stack pointer never moves and the only thing between
        // it and the caller's arguments is the return address the call pushed. The seventh
        // parameter is at the bottom of the caller's argument area and the eighth is one word
        // further up, which is the eight bytes between the two offsets.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(frame.size(), 0);
        assert_eq!(frame.incoming(), Incoming::from_stack(8));
        assert!(text.contains("x64.mov_rm_64 [$rsp + 8]"), "{text}");
        assert!(text.contains("x64.mov_rm_64 [$rsp + 16]"), "{text}");
    }

    #[test]
    fn a_realigned_frame_reaches_the_caller_s_arguments_through_the_frame_pointer() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64; 7]);
        let wide = slot(&mut source, block, 64, 32);
        let mut build = Builder::new(&mut source, block);
        build.store(args[6], wide, plain(), Flags::default());
        build.ret(&[args[6]]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // A local wanting thirty two byte alignment makes the prologue force the stack pointer,
        // which throws away how far the caller's stack was. So the load the lowering wrote off the
        // stack pointer is rewritten to read through the frame pointer, at the one distance that
        // survives: the word the prologue pushed the frame pointer into, and the return address
        // above it.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(frame.realign(), Some(32));
        assert_eq!(frame.incoming(), Incoming::from_frame(16));
        assert!(text.contains("x64.mov_rm_64 [$rbp + 16]"), "{text}");
        assert!(!text.contains("x64.mov_rm_64 [$rsp"), "{text}");
    }

    #[test]
    fn a_jump_is_the_edge_and_nothing_else() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32]);
        let next = source.create_block();
        let got = source.append_param(next, i32);
        Builder::new(&mut source, entry).jump(next, &[args[0]]);
        Builder::new(&mut source, next).ret(&[got]);

        // Two blocks and two instructions, and the jump is neither of them. What it was is the
        // arm on the first block, and what the arm carries is the argument it was called with.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32 block1(%0)\n\n\
             block1(%1:gpr):\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    /// A block that reads what a block below it writes is filled after it, not before it.
    ///
    /// The blocks are written entry, `early`, `late`, `exit`, and the entry jumps straight past
    /// `early` to `late`, so `late` dominates `early` while sitting below it in the function.
    /// Filling them in the order they are written reaches the read in `early` first, and reading
    /// a value with no register yet mints one. The cast in `late` is no instruction at all, so
    /// what it does is give its answer the register its operand is already in, and that is not
    /// the register the read minted. Nothing writes the register the read minted. The printer
    /// says `%?` for a register nothing defines, which is what this looks for, and what came out
    /// of the real bug was SQLite loading a stack slot no store ever reached.
    #[test]
    fn a_block_that_reads_what_a_block_below_it_writes_is_filled_after_it() {
        let i64 = Type::int(64);
        let (mut names, mut source, entry, args) = blank(&[i64, i64]);
        let early = source.create_block();
        let late = source.create_block();
        let exit = source.create_block();

        Builder::new(&mut source, entry).jump(late, &[]);
        let ptr = cast(&mut source, late, Opcode::IntToPtr, args[0], Type::PTR);
        Builder::new(&mut source, early).ret(&[ptr]);
        let mut build = Builder::new(&mut source, late);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, early, &[], exit, &[]);
        Builder::new(&mut source, exit).ret(&[args[1]]);

        let text = lower(&mut names, &source);
        assert!(!text.contains("%?"), "every register has something that writes it: {text}");
    }

    /// A constant is written where it is wanted rather than where the IR defined it, and two
    /// blocks wanting the same one is two places. Writing it once and reading it in both is a
    /// register read where nothing wrote it, unless the block it was written in happens to
    /// dominate the other, which nothing here checks and which the second arm of a branch never
    /// does. Each block gets its own copy of the number instead.
    #[test]
    fn a_constant_two_blocks_want_is_written_in_both_of_them() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);

        let mut build = Builder::new(&mut source, entry);
        let seven = build.iconst(i32, 7);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        // Both arms want the seven in a register, because a block argument is never an immediate,
        // and neither arm dominates the other.
        Builder::new(&mut source, then).jump(join, &[seven]);
        Builder::new(&mut source, other).jump(join, &[seven]);
        Builder::new(&mut source, join).ret(&[got]);

        let text = lower(&mut names, &source);
        assert_eq!(text.matches("x64.mov_ri_32 7").count(), 2, "one seven per block: {text}");
    }

    /// An argument on an edge out of a block that leaves two ways is read after every instruction
    /// of the block is written, and reading one can write an instruction, which would land after
    /// the branch that has already jumped past it. The branch goes back on the end.
    #[test]
    fn a_constant_an_edge_wants_is_written_before_the_branch_and_not_after_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);

        let mut build = Builder::new(&mut source, entry);
        let nine = build.iconst(i32, 9);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], join, &[nine]);
        Builder::new(&mut source, then).jump(join, &[args[0]]);
        Builder::new(&mut source, join).ret(&[got]);

        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        let entry = out.entry().expect("an entry block");
        let last = out.terminator(entry).expect("a block that leaves two ways has a branch");
        let branch = names.intern("x64.br_cond_8");
        assert_eq!(
            out[last].opcode,
            mir::Opcode::new(branch),
            "the branch is last: {}",
            mir::print_func(&out, &names, &REGS)
        );
    }

    #[test]
    fn a_conditional_branch_is_lowered_to_the_condition_and_nothing_about_where_it_goes() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        Builder::new(&mut source, then).ret(&[args[0]]);
        Builder::new(&mut source, other).ret(&[args[1]]);

        // The comparison writes a byte and the branch reads it, and neither says a block. Both
        // arms are on the entry block, in the order the branch took them, so the arm that runs
        // when the condition holds is the first.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr = x64.cmp_set_l_32 %0, %1\n    \
             x64.br_cond_8 %2, block1, block2\n\n\
             block1:\n    x64.ret_val_32 %0($rax)\n\n\
             block2:\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    /// A choice between two values, which is one instruction and no blocks at all.
    ///
    /// The arms come out the other way round from the IR, because a conditional move overwrites its
    /// destination and the destination is the arm taken when the condition does not hold. The
    /// condition arrives last for the same reason: it is read by the test in front of the move
    /// rather than by the move.
    #[test]
    fn a_select_is_lowered_to_a_test_and_a_conditional_move() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        let picked = build.select(cond, args[0], args[1]);
        build.ret(&[picked]);

        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             %1:gpr($rsi) = x64.arg_val_32\n    %2:gpr = x64.cmp_set_l_32 %0, %1\n    \
             %3:gpr(reuse 1) = x64.test_cmov_ne_32 %1, %0, %2\n    \
             x64.ret_val_32 %3($rax)\n}\n"
        );
    }

    #[test]
    fn a_branch_over_a_block_is_a_whole_function_now() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let other = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], other, &[]);
        let mut build = Builder::new(&mut source, then);
        let sum = build.binary(Opcode::Add, args[0], args[1], Flags::default());
        build.jump(join, &[sum]);
        Builder::new(&mut source, other).jump(join, &[args[1]]);
        Builder::new(&mut source, join).ret(&[got]);

        // `int f(int a, int b) { if (a < b) return a + b; else return b; }` end to end, written
        // the way a front end writes it: both arms of the branch are blocks of their own and the
        // return is the block they meet at. No edge here is critical, because the two arms out of
        // the entry carry nothing and the two arms into the join each leave a block that goes
        // nowhere else, so each has its own end to put its move at.
        let mut out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        assert_eq!(crate::split::critical(&mut out), 0, "no edge here is critical");
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // One epilogue, on the join, which is the one block the function leaves from, and the
        // moves that give the join its parameter are at the end of each arm. Every register is
        // physical and the branch is still a branch on a register, because turning it into a
        // `test` and a `jcc` is the block layout's and there is no block layout yet.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(text.matches("x64.ret\n").count(), 1, "{text}");
        assert!(text.contains("x64.br_cond_8"), "{text}");
        assert!(text.contains("x64.add_rr_32"), "{text}");
        assert!(!text.contains('%'), "{text}");
    }

    #[test]
    fn a_critical_edge_is_split_before_the_allocator_ever_sees_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, entry, args) = blank(&[i32, i32]);
        let then = source.create_block();
        let join = source.create_block();
        let got = source.append_param(join, i32);
        let mut build = Builder::new(&mut source, entry);
        let cond = build.icmp(rucc_ir::IntPred::Slt, args[0], args[1]);
        build.br_if(cond, then, &[], join, &[args[1]]);
        Builder::new(&mut source, then).jump(join, &[args[0]]);
        let mut build = Builder::new(&mut source, join);
        let twice = build.binary(Opcode::Add, got, got, Flags::default());
        build.ret(&[twice]);

        // The else arm is critical: the entry block leaves two ways and the join is arrived at
        // two ways, and the arm carries a value. Without splitting it the allocator asserts,
        // because the move that gives the join its parameter would have to run at the end of a
        // block that also goes to the other arm.
        let mut out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule")
            .func;
        assert_eq!(crate::split::critical(&mut out), 1);
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &Layout::new(&SYSV, REGS));
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // The block the split added is where the move went, and it is the whole of that block.
        let text = mir::print_func(&out, &names, &REGS);
        assert_eq!(out.block_count(), 4, "{text}");
        assert_eq!(text.matches("x64.ret\n").count(), 1, "{text}");
    }

    #[test]
    fn a_call_passes_what_the_convention_says_and_takes_back_what_it_says() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32, i32]);
        let sig =
            source.add_signature(Signature::new().with_params(&[i32, i32]).with_returns(&[i32]));
        let callee = names.intern("g");
        let call = Builder::new(&mut source, block).call(callee, sig, &[args[0], args[1]]);
        let got = source[call].first_result.expect("an integer comes back");
        Builder::new(&mut source, block).ret(&[got]);

        // `int f(int a, int b) { return g(a, b); }`. The arguments arrived where the call wants
        // them, so what the call reads is what arrived, and the whole of the convention is in the
        // constraints rather than in a move.
        let text = lower(&mut names, &source);
        assert!(text.contains("= x64.call %0($rdi), %1($rsi), @g"), "{text}");
        assert!(text.contains("x64.ret_val_32 %2($rax)"), "{text}");
        // What the call writes is the value that comes back and then every register the callee is
        // free to destroy, in both classes, which is the whole of what stops the allocator from
        // leaving something in one of them.
        assert!(text.contains("%2:gpr($rax), $rcx, $rdx, $r8, $r9, $r10, $r11, $xmm0,"), "{text}");
        assert!(text.contains("$xmm15 = x64.call"), "{text}");
    }

    #[test]
    fn what_the_frame_owes_a_call_comes_back_with_the_function() {
        let i32 = Type::int(32);
        let sig = |source: &mut Func| source.add_signature(Signature::new().with_params(&[i32]));

        let (mut names, mut source, block, args) = blank(&[i32]);
        let sig = sig(&mut source);
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[args[0]]);
        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // Nothing on the stack, so nothing owed, but not a leaf either: a function that calls
        // owes the callee an aligned stack pointer and may not use the red zone.
        assert_eq!(out.stack.calls, Some(0));
        let layout = out.stack.layout(Layout::new(&SYSV, REGS));
        assert!(!layout.leaf);
        assert_eq!(layout.outgoing, 0);

        // The same call under the other convention owes thirty two bytes for the callee to spill
        // its register arguments into, which is a fact about the convention and not about the call.
        let out = func(&source, &mut names, &x86_64::WIN64, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(out.stack.calls, Some(32));

        // And a function that calls nothing is a leaf, which is what says it may use the red zone.
        let (mut names, mut source, block, args) = blank(&[i32]);
        Builder::new(&mut source, block).ret(&[args[0]]);
        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(out.stack.calls, None);
        assert!(out.stack.layout(Layout::new(&SYSV, REGS)).leaf);
    }

    /// A Windows variadic prologue writes the argument registers the signature did not name into
    /// the shadow space the caller already reserved, which makes every argument one run of words up
    /// there and a `va_start` the address of the first of them. One `lea` and one store, and no
    /// counts, because a list that is a pointer has nowhere to put one and nothing that reads one.
    #[test]
    fn a_windows_variadic_function_homes_its_spare_registers_in_the_callers_area() {
        let mut names = Interner::new();
        let params = [Type::int(32), Type::PTR];
        let signature = Signature::new().with_params(&params).variadic();
        let mut source = Func::new(names.intern("f"), signature);
        let block = source.create_block();
        let values: Vec<Value> = params.iter().map(|&ty| source.append_param(block, ty)).collect();
        let mut build = Builder::new(&mut source, block);
        let args = build.func().push_values(&values[1..]);
        build.inst(InstData { args, ..InstData::new(Opcode::VaStart) }, &[]);
        build.ret(&[]);

        let out = func(&source, &mut names, &x86_64::WIN64, &Elsewhere::default())
            .expect("every instruction has a rule");
        let text = mir::print_func(&out.func, &names, &REGS);

        // Two named parameters, so the registers at the next two positions hold arguments nobody
        // named and both are written up into the caller's area. The displacement is empty here and
        // `finish` fills it in, the same way it does for a parameter the registers ran out before.
        assert!(text.contains("($r8) = x64.arg_val_64"), "{text}");
        assert!(text.contains("($r9) = x64.arg_val_64"), "{text}");
        assert_eq!(text.matches("x64.mov_mr_64").count(), 3, "two homed and one stored: {text}");
        assert!(!text.contains("x64.mov_ri_32"), "and no field holds a count: {text}");

        // All three waiting on the same fixup, and the last of them is the `lea` the list is given,
        // sixteen bytes up, which is where the two arguments the signature does name stopped.
        assert_eq!(out.stack.arguments.len(), 3);
        assert_eq!(out.stack.arguments[2].1, 16);
    }

    #[test]
    fn a_value_that_outlives_a_call_is_not_left_where_the_call_destroys_it() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let sig = source.add_signature(Signature::new().with_params(&[i32]).with_returns(&[i32]));
        let callee = names.intern("g");
        let call = Builder::new(&mut source, block).call(callee, sig, &[args[0]]);
        let got = source[call].first_result.expect("an integer comes back");
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::Add, got, args[0], Flags::default());
        build.ret(&[sum]);

        // `int f(int a) { return g(a) + a; }`, which is the smallest program that asks the
        // question: `a` is read after the call and `rdi` is a register the call destroys.
        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let layout = lowered.stack.layout(Layout::new(&SYSV, REGS));
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let frame = Frame::of(&out, &allocation, &layout);
        finish(
            &mut out,
            &allocation,
            &frame,
            &Stack::default(),
            Convention::new(&SYSV, &FRAME),
            &mut names,
        );

        // It went to a register the callee has to put back, and the prologue and epilogue are what
        // put it back, which is the whole bargain the two halves of a convention make.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(text.contains("$rbx"), "{text}");
        assert!(!text.contains('%'), "{text}");
        assert_eq!(text.matches("x64.call").count(), 1, "{text}");
    }

    #[test]
    fn a_call_with_more_arguments_than_registers_writes_the_rest_into_the_outgoing_area() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64]);
        let seven = vec![i64; 7];
        let sig = source.add_signature(Signature::new().with_params(&seven));
        let callee = names.intern("g");
        let passed = vec![args[0]; 7];
        Builder::new(&mut source, block).call(callee, sig, &passed);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("the seventh goes to memory");
        // The bytes the call needs are on the layout the frame is worked out from, so that the
        // frame reserves as many as the widest call in the function asked for.
        assert_eq!(lowered.stack.calls, Some(8));
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("x64.mov_mr_64 %0, [$rsp]\n"), "{text}");
    }

    #[test]
    fn a_call_this_cannot_make_is_reported_rather_than_made() {
        let (mut names, mut source, block, _) = blank(&[]);
        let returns = [Type::float(rucc_ir::Float::F80), Type::int(64)];
        let sig = source.add_signature(Signature::new().with_returns(&returns));
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[]);
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("a long double is on the x87");
        assert_eq!(failed.to_string(), "what this call gives back is on the x87 stack");
    }

    /// A `long double` on its own is a different answer, because on its own it comes back on the
    /// x87 stack rather than in a register, which is somewhere the call cannot be said to write.
    ///
    /// So the call gives back nothing at all and the value is taken off the stack by the `fstp`
    /// straight after it. That instruction has to be straight after it: the stack is one place and
    /// anything else that touched it before this ran would be looking at the value still on it.
    #[test]
    fn a_call_that_gives_back_a_long_double_takes_it_off_the_stack_at_once() {
        let (mut names, mut source, block, _) = blank(&[]);
        let long_double = Type::float(rucc_ir::Float::F80);
        let sig = source.add_signature(Signature::new().with_returns(&[long_double]));
        let callee = names.intern("g");
        Builder::new(&mut source, block).call(callee, sig, &[]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("the value comes back in st0");
        let text = mir::print_func(&lowered.func, &names, &REGS);
        let after: Vec<&str> =
            text.lines().skip_while(|line| !line.contains("x64.call")).skip(1).collect();
        assert_eq!(after[0].trim(), "%0:gpr = x64.lea_64 [$rsp]", "{text}");
        assert_eq!(after[1].trim(), "x64.fstp_t [%0]", "{text}");
        // And the slot it went into is the sixteen bytes the type takes, like every other one.
        assert_eq!(lowered.stack.locals.len(), 1, "{text}");
        assert_eq!(lowered.stack.locals[0].size, X87_BYTES);
    }

    #[test]
    fn a_call_through_an_address_goes_through_the_register_the_address_is_in() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[Type::PTR, i32]);
        let sig = source.add_signature(Signature::new().with_params(&[i32]).with_returns(&[i32]));
        let varargs = source.push_abis(&[]);
        let info = source.add_call(CallInfo { callee: None, signature: sig, varargs });
        let mut build = Builder::new(&mut source, block);
        let inst = InstData {
            args: build.func().push_values(&[args[0], args[1]]),
            extra: Extra::Call(info),
            ..InstData::new(Opcode::CallIndirect)
        };
        let called = build.inst(inst, &[i32]);
        let got = source[called].first_result.expect("an integer comes back");
        Builder::new(&mut source, block).ret(&[got]);

        // `int f(int (*g)(int), int a) { return g(a); }`. The first operand is the address and
        // the arguments are the ones behind it, and everything else about the call is what a call
        // to a name would have been.
        let text = lower(&mut names, &source);
        assert!(text.contains("= x64.call_reg %0, %1($rdi)"), "{text}");
        assert!(text.contains("x64.ret_val_32 %2($rax)"), "{text}");
        assert!(!text.contains("@g"), "a call through an address names nobody: {text}");
    }

    #[test]
    fn an_instruction_no_rule_covers_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let operands = build.func().push_values(&[args[0]]);
        build.inst(InstData { args: operands, ..InstData::new(Opcode::MetaBegin) }, &[]);

        // The mark that an object has come into being, which nothing writes an instruction for
        // yet: what it needs is a write over a range of the lifetime plane, and that is
        // `tamnd/rucc#856`. Nothing about it is a width or a register, so there is nothing for the
        // message to add beyond the name.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("no rule writes the beginning of a lifetime");
        assert_eq!(failed.to_string(), "no rule lowers a `meta_begin`");

        // It produces nothing, so there is no type in the message and nothing invents one, and the
        // instruction comes back so a caller can ask the function where it was.
        let inst = failed.inst().expect("the instruction it is about");
        assert_eq!(source[inst].opcode, Opcode::MetaBegin);
    }

    /// A barrier is written by name here, and what it is depends on the ordering and on nothing
    /// else. `crate::expand` is where the reasoning about this machine's memory model lives.
    #[test]
    fn a_barrier_is_one_instruction_at_the_strongest_ordering_and_none_below_it() {
        for order in MemOrder::all().filter(|&order| order != MemOrder::NotAtomic) {
            let (mut names, mut source, block, _) = blank(&[]);
            let mut build = Builder::new(&mut source, block);
            build
                .inst(InstData { extra: Extra::Order(order), ..InstData::new(Opcode::Fence) }, &[]);

            let text = lower(&mut names, &source);
            assert_eq!(text.contains("x64.mfence"), order == MemOrder::SeqCst, "{order:?}: {text}");
        }
    }

    /// A compare and exchange is written by name too, and at the width of the value rather than at
    /// the width of the address, which is the mistake worth pinning: everything here is a pointer
    /// and only the value says how many bytes the instruction touches.
    #[test]
    fn a_compare_and_exchange_is_one_instruction_at_the_width_of_the_value() {
        for bits in [8, 16, 32, 64] {
            let ty = Type::int(bits);
            let (mut names, mut source, block, args) = blank(&[Type::PTR, ty, ty]);
            let mut build = Builder::new(&mut source, block);
            let mem = build.func().add_mem(MemInfo {
                size: u64::from(bits / 8),
                align: bits / 8,
                order: MemOrder::SeqCst,
                ..plain()
            });
            let operands = build.func().push_values(&[args[0], args[1], args[2]]);
            build.inst(
                InstData {
                    args: operands,
                    extra: Extra::Mem(mem),
                    ..InstData::new(Opcode::Cmpxchg)
                },
                &[ty, Type::I1],
            );

            // Two values out of one instruction, the first of them in the register the machine
            // reads the expected value out of, the second free for the allocator to place. The
            // address is the memory operand and neither of the two values is.
            let text = lower(&mut names, &source);
            let written = format!("%3:gpr($rax), %4:gpr = x64.cmpxchg_{bits} %1($rax), %2, [%0]");
            assert!(text.contains(&written), "{bits}: {text}");
        }
    }

    #[test]
    fn more_values_back_than_the_convention_has_registers_for_is_reported() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, args) = blank(&[i64, i64, i64]);
        let mut build = Builder::new(&mut source, block);
        build.ret(&[args[0], args[1], args[2]]);

        // Two integers come back in `rax` and `rdx` and a third has nowhere to go, which is not a
        // gap in the rules but the convention saying no. The front end classifies before it gets
        // here, so this is the shape that would mean the classification went wrong.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("only two come back");
        assert_eq!(
            failed.to_string(),
            "what this function gives back takes more registers than this convention has for it"
        );

        let inst = failed.inst().expect("the instruction it is about");
        assert_eq!(source[inst].opcode, Opcode::Return);
    }

    /// A refusal about a signature has no instruction, which is what makes it the one arm apart.
    ///
    /// Everything else is about something written somewhere in the body and hands it back so a
    /// caller can ask the function where it came from. A parameter arrives before the first
    /// instruction runs, so there is nothing in the body to point at and the message is about
    /// the function.
    #[test]
    fn a_refusal_about_a_parameter_has_no_instruction_to_point_at() {
        let missing = Unsupported::Argument { index: 0, missing: Missing::OnX87 };
        assert_eq!(missing.inst(), None);
    }

    /// An `alloca` of a fixed size, which is what every local whose address is taken becomes.
    fn slot(source: &mut Func, block: Block, size: u64, align: u32) -> Value {
        let info = MemInfo { size, align, ..plain() };
        let mut build = Builder::new(source, block);
        let mem = build.func().add_mem(info);
        build.value(InstData { extra: Extra::Mem(mem), ..InstData::new(Opcode::Alloca) }, Type::PTR)
    }

    #[test]
    fn a_local_is_memory_in_the_frame_and_one_instruction_that_says_where() {
        let (mut names, mut source, block, _) = blank(&[]);
        let slot = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, slot, plain(), Flags::default());
        let loaded = build.load(Type::int(32), slot, plain(), Flags::default());
        build.ret(&[loaded]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // Four bytes on the list the frame is laid out from, and the one instruction that reads
        // where they went. Its displacement is nothing here because there is no frame yet, and
        // which instruction is waiting for which local is what `finish` is handed.
        assert_eq!(lowered.stack.locals, vec![Local { size: 4, align: 4 }]);
        assert_eq!(lowered.stack.addresses.len(), 1);
        assert_eq!(lowered.stack.addresses[0].1, 0);
        assert_eq!(
            mir::print_func(&lowered.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.lea_64 [$rsp]\n    \
             %1:gpr = x64.mov_ri_32 9\n    x64.mov_mr_32 %1, [%0]\n    \
             %2:gpr = x64.mov_rm_32 [%0]\n    x64.ret_val_32 %2($rax)\n}\n"
        );
    }

    #[test]
    fn the_frame_is_what_fills_the_address_of_a_local_in() {
        let (mut names, mut source, block, _) = blank(&[]);
        let slot = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, slot, plain(), Flags::default());
        let loaded = build.load(Type::int(32), slot, plain(), Flags::default());
        build.ret(&[loaded]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // `int f(void) { int x; x = 9; return x; }` with the address of `x` taken, end to end.
        // A leaf small enough to live in the red zone takes no frame at all, so the stack pointer
        // never moves and the four bytes are below it, which is what the negative offset is. The
        // instruction the lowering left with nothing in its displacement now has the answer in it.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(text.contains("$rax = x64.lea_64 [$rsp - 8]"), "{text}");
        assert!(!text.contains("x64.sub_ri_64"), "{text}");
        assert_eq!(frame.size(), 0);
        assert_eq!(frame.local(0), Some(-8));
    }

    /// An `alloca` whose size is an operand, which is a variable length array.
    fn growing(source: &mut Func, block: Block, size: Value, align: u32) -> Value {
        let info = MemInfo { size: 0, align, ..plain() };
        let mut build = Builder::new(source, block);
        let mem = build.func().add_mem(info);
        let args = build.func().push_values(&[size]);
        build.value(
            InstData { args, extra: Extra::Mem(mem), ..InstData::new(Opcode::Alloca) },
            Type::PTR,
        )
    }

    #[test]
    fn a_stack_slot_whose_size_is_not_known_until_it_runs_takes_the_bytes_off_the_stack_pointer() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let slot = growing(&mut source, block, args[0], 16);
        Builder::new(&mut source, block).ret(&[slot]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");

        // The bytes come off the stack pointer where the declaration stands and the address is
        // where the stack pointer then is, which is one subtraction and one `lea` rather than a
        // slot the frame laid out. Nothing is on the list of locals, because there is nothing
        // about this the frame could place.
        let text = mir::print_func(&lowered.func, &names, &REGS);
        assert!(text.contains("$rsp = x64.sub_rr_64 $rsp, %0"), "{text}");
        assert!(text.contains("x64.lea_64 [$rsp]"), "{text}");
        assert!(lowered.stack.locals.is_empty(), "{text}");
        assert_eq!(lowered.stack.dynamic.len(), 1);
        assert!(lowered.stack.grown_at.is_some());
    }

    #[test]
    fn a_growing_slot_wanting_more_alignment_than_the_stack_pointer_has_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let slot = growing(&mut source, block, args[0], 32);
        Builder::new(&mut source, block).ret(&[slot]);

        // Thirty two is more than a call leaves the stack pointer on, so giving it what it asked
        // for means masking the stack pointer after moving it, and after that no constant reaches
        // the rest of the frame from the frame pointer either. A second pointer held for the
        // purpose is what fixes it and there is not one yet.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("nothing realigns a frame that grows");
        assert_eq!(
            failed.to_string(),
            "this local wants more alignment than the stack pointer is left on, which needs a \
             base register nothing here keeps"
        );
    }

    #[test]
    fn a_frame_that_grows_reaches_its_own_locals_through_the_frame_pointer() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let fixed = slot(&mut source, block, 4, 4);
        let mut build = Builder::new(&mut source, block);
        let nine = build.iconst(Type::int(32), 9);
        build.store(nine, fixed, plain(), Flags::default());
        let grown = growing(&mut source, block, args[0], 16);
        Builder::new(&mut source, block).ret(&[grown]);

        let lowered = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        let stack = lowered.stack;
        let mut out = lowered.func;
        let env = env();
        let allocation = rucc_regalloc::run(&mut out, &env, "test", true);
        let layout = stack.layout(Layout::new(&SYSV, REGS));
        let frame = Frame::of(&out, &allocation, &layout);
        finish(&mut out, &allocation, &frame, &stack, Convention::new(&SYSV, &FRAME), &mut names);

        // The stack pointer moves in the middle of the function, so the four bytes of the fixed
        // local are not a constant away from it any more and the frame pointer is what reaches
        // them. The frame keeps one whatever the flags asked for, takes its bytes rather than
        // living in the red zone, and the address of the growing slot is off the stack pointer as
        // it stands after the subtraction rather than off anything the prologue left.
        let text = mir::print_func(&out, &names, &REGS);
        assert!(frame.grows());
        assert!(frame.frame_pointer());
        assert!(frame.size() > 0, "{text}");
        assert!(text.contains("x64.lea_64 [$rbp"), "{text}");
        assert!(text.contains("$rsp = x64.sub_rr_64 $rsp"), "{text}");
        assert!(text.contains("x64.lea_64 [$rsp]"), "{text}");
    }

    #[test]
    fn an_address_is_read_written_and_added_to_like_the_integer_it_is() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR, Type::int(64)]);
        let mut build = Builder::new(&mut source, block);
        let stepped = build.func().push_values(&[args[0], args[1]]);
        let next =
            build.value(InstData { args: stepped, ..InstData::new(Opcode::PtrAdd) }, Type::PTR);
        let loaded = build.load(Type::int(32), next, plain(), Flags::default());
        build.ret(&[loaded]);

        // `int f(int *p, long i) { return *(int *)((char *)p + i); }`. Nothing about this is new
        // in the rule set, which is the point: the two addresses arrive in registers because an
        // address is an integer as wide as one, and the arithmetic on them is the add it always
        // was, so every rule written about an add reaches it.
        //
        // The add stays its own instruction rather than folding into the address the load reads
        // from. Two registers with no scale on either is the one addressing mode the rules have no
        // load through, because the folds that exist are the displacement one and the scaled ones,
        // and this is neither. That is a peephole worth having and not a thing this changes.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             %1:gpr($rsi) = x64.arg_val_64\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n    \
             %3:gpr = x64.mov_rm_32 [%2]\n    x64.ret_val_32 %3($rax)\n}\n"
        );
    }

    /// The address of a file scope name, which is what every use of a global and every string
    /// literal starts from.
    fn address_of(source: &mut Func, block: Block, names: &mut Interner, name: &str) -> Value {
        let symbol = names.intern(name);
        let mut build = Builder::new(source, block);
        build.value(
            InstData { extra: Extra::Symbol(symbol), ..InstData::new(Opcode::GlobalAddr) },
            Type::PTR,
        )
    }

    #[test]
    fn the_address_of_a_name_is_one_instruction_carrying_the_name() {
        let (mut names, mut source, block, _) = blank(&[]);
        let counter = address_of(&mut source, block, &mut names, "counter");
        let mut build = Builder::new(&mut source, block);
        let loaded = build.load(Type::int(32), counter, plain(), Flags::default());
        build.ret(&[loaded]);

        // `extern int counter; int f(void) { return counter; }`. The address is an addressing mode
        // that names no register and carries the symbol, which is what the assembler writes
        // relative to `%rip` and what the object writer leaves a relocation for.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.lea_64 [@counter]\n    \
             %1:gpr = x64.mov_rm_32 [%0]\n    x64.ret_val_32 %1($rax)\n}\n"
        );
    }

    #[test]
    fn the_address_of_a_name_outside_the_file_is_read_out_of_the_offset_table() {
        let (mut names, mut source, block, _) = blank(&[]);
        let away = address_of(&mut source, block, &mut names, "away");
        Builder::new(&mut source, block).ret(&[away]);
        let elsewhere: Elsewhere = [names.intern("away")].into_iter().collect();

        // `extern void away(void); void *f(void) { return away; }`. A load and not an address
        // computation, because the distance from here to a name a shared library may be the one
        // that defines is not a number any link can work out, and the slot the linker fills in is
        // in this program and so is a distance it has.
        let out =
            func(&source, &mut names, &SYSV, &elsewhere).expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [got @away]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_address_of_a_thread_local_is_an_offset_out_of_the_table_plus_where_this_thread_starts() {
        let (mut names, mut source, block, _) = blank(&[]);
        let own = address_of(&mut source, block, &mut names, "own");
        Builder::new(&mut source, block).ret(&[own]);
        let elsewhere = Elsewhere::default().with_threads([names.intern("own")]);

        // `extern _Thread_local int own; void *f(void) { return &own; }`. Three instructions where
        // the two cases above are one, because there is no address to load or to work out: the
        // slot holds how far into a thread's block the variable sits, `%fs:0` is where this
        // thread's block starts, and the sum of the two is this thread's copy.
        let out =
            func(&source, &mut names, &SYSV, &elsewhere).expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [thread @own]\n    \
             %1:gpr = x64.mov_rm_64 [fs:0]\n    %2:gpr(reuse 1) = x64.add_rr_64 %0, %1\n    \
             x64.ret_val_64 %2($rax)\n}\n"
        );
    }

    /// The same load with nothing added to it, which is the whole of `__builtin_thread_pointer`.
    #[test]
    fn the_start_of_this_thread_s_own_storage_is_the_one_load_and_no_arithmetic() {
        let (mut names, mut source, block, _) = blank(&[]);
        let here =
            Builder::new(&mut source, block).value(InstData::new(Opcode::ThreadPointer), Type::PTR);
        Builder::new(&mut source, block).ret(&[here]);

        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction has a rule");
        assert_eq!(
            mir::print_func(&out.func, &names, &REGS),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [fs:0]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    /// One `asm` statement, with its template and its constraint list written as a program does.
    fn assembly(
        source: &mut Func,
        block: Block,
        names: &mut Interner,
        template: &str,
        constraints: &str,
        args: &[Value],
        results: &[Type],
    ) -> Inst {
        clobbering(source, block, names, template, constraints, "memory", args, results)
    }

    /// The same with a clobber list of its own, for the statements that are about one.
    #[allow(clippy::too_many_arguments)]
    fn clobbering(
        source: &mut Func,
        block: Block,
        names: &mut Interner,
        template: &str,
        constraints: &str,
        clobbers: &str,
        args: &[Value],
        results: &[Type],
    ) -> Inst {
        let info = AsmInfo {
            template: names.intern(template),
            constraints: names.intern(constraints),
            clobbers: names.intern(clobbers),
            targets: rucc_ir::BlockCallList::EMPTY,
        };
        Builder::new(source, block).inline_asm(info, args, results, Flags::VOLATILE)
    }

    /// What a program asking the processor what it can do writes, which is the instruction whose
    /// every operand is a register its text does not name.
    #[test]
    fn a_template_whose_registers_are_named_by_the_constraints_places_them_from_the_letters() {
        let u32 = Type::int(32);
        let (mut names, mut source, block, _) = blank(&[]);
        let zero = Builder::new(&mut source, block).iconst(u32, 0);
        let out = clobbering(
            &mut source,
            block,
            &mut names,
            "cpuid",
            "=a,a",
            "ebx,ecx,edx",
            &[zero],
            &[u32],
        );
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("cpuid" : "=a" (n) : "a" (0) : "ebx", "ecx", "edx")`, which is the first thing
        // every program that has a faster path on some machines writes. Four registers written and
        // two read, none of them in the template, all of them out of the description, and the two
        // that the letters named are the statement's own. The subleaf is a zero because the
        // instruction reads `ecx` and the program said nothing about what is in it. The three
        // clobbers are gone because `cpuid` writes those three anyway, and saying it twice is one
        // register with two definitions.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    \
             %1:gpr = x64.mov_ri_64 0\n    \
             %2:gpr($rax), %3:gpr($rbx), %4:gpr($rcx), %5:gpr($rdx) = x64.cpuid %0($rax), \
             %1($rcx)\n    x64.ret_val_32 %2($rax)\n}\n"
        );
    }

    /// A clobber the instruction does not write itself, which is the case the list is there for.
    /// It goes on as a definition of the register, in among the other definitions, because that is
    /// the whole of how a machine function says a register is not worth anything after this.
    #[test]
    fn a_clobber_the_instruction_does_not_write_itself_is_a_definition_of_that_register() {
        let (mut names, mut source, block, _) = blank(&[]);
        clobbering(&mut source, block, &mut names, "pause", "", "rsi,cc,memory", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n    $rsi = x64.pause\n}\n");
    }

    /// A clobber naming something this has no register for. Refused rather than dropped, since the
    /// list is the program saying which registers it may not leave anything in, and an entry
    /// nobody read is a register something may still be left in.
    #[test]
    fn a_clobber_this_has_no_register_for_is_refused() {
        let (mut names, mut source, block, _) = blank(&[]);
        clobbering(&mut source, block, &mut names, "pause", "", "zmm0", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("there is no such register here");
        assert_eq!(
            failed.to_string(),
            "this `asm` says it destroys a register this has no name for"
        );
    }

    #[test]
    fn an_asm_with_an_empty_template_and_no_operands_is_no_instructions() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        // `asm volatile ("" : : : "memory")`, which is a barrier and nothing else. The barrier was
        // spent on the optimizer, which has finished by now, so what is left is nothing.
        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n}\n");
    }

    #[test]
    fn an_output_an_input_is_tied_to_is_the_register_that_input_arrived_in() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let out = assembly(&mut source, block, &mut names, "", "=r,0", &args, &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "=r" (x) : "0" (x))`, which is how a program stops the optimizer following a
        // value without changing it. The two share a place and the template writes nothing over
        // it, so the value comes back out of the register it went in.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn an_output_written_plus_is_the_same_rename() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        let out = assembly(&mut source, block, &mut names, "", "+r", &args, &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "+r" (x))`, which says the same thing in one operand instead of two.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_32\n    \
             x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn an_output_nothing_is_tied_to_is_a_zero() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "", "=r", &[], &[i32]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("" : "=r" (y))`, whose answer is whatever the assembly left in the register, and
        // an empty template leaves nothing. A definite value rather than a register nothing wrote,
        // because the allocator is owed a definition before the use however little the program is.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_32 0\n    x64.ret_val_32 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_template_that_is_one_instruction_becomes_that_instruction() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "pause", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        // `asm volatile ("pause")`, which is what every spin lock in every allocator writes. One
        // instruction, no operands, and nothing between the template and the machine but the table
        // that already says what a `pause` is.
        assert_eq!(lower(&mut names, &source), "mfunc @f {\nblock0:\n    x64.pause\n}\n");
    }

    #[test]
    fn a_template_that_reads_a_segment_becomes_the_load_it_already_was() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "movq %%fs:0, %0", "=r", &[], &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        // `asm ("movq %%fs:0, %0" : "=r" (tid))`, which is how a program finds the block its own
        // thread owns. The same instruction `crate::lower` already writes for a thread-local
        // variable, reached this time because a program wrote it out by hand.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_rm_64 [fs:0]\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_template_naming_an_instruction_this_machine_has_not_got_is_refused() {
        let (mut names, mut source, block, _) = blank(&[]);
        assembly(&mut source, block, &mut names, "hcf", "", &[], &[]);
        Builder::new(&mut source, block).ret(&[]);

        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("there is no such instruction");
        assert_eq!(
            failed.to_string(),
            "this `asm` has instructions in its template, which nothing here assembles"
        );
    }

    /// A register the template named is a claim on a register nobody told the allocator about.
    /// Refused rather than placed, because a register two things believe they own is a wrong
    /// program that nothing reports. A register a constraint letter names is a different thing and
    /// is placed, which the test above is about: there the statement said which of its own operands
    /// is in the register, and a name in the middle of a template says no such thing.
    #[test]
    fn a_template_naming_a_register_the_allocator_did_not_hand_out_is_refused() {
        let i64 = Type::int(64);
        let (mut names, mut source, block, _) = blank(&[]);
        let out = assembly(&mut source, block, &mut names, "movq %%rax, %0", "=r", &[], &[i64]);
        let produced = source[out].results().next().expect("one result");
        Builder::new(&mut source, block).ret(&[produced]);

        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("the template named a register");
        assert_eq!(failed.to_string(), "this `asm` has an operand this cannot place");
    }

    #[test]
    fn a_constraint_list_that_does_not_describe_the_operands_is_refused() {
        let i32 = Type::int(32);
        let (mut names, mut source, block, args) = blank(&[i32]);
        assembly(&mut source, block, &mut names, "", "=r", &args, &[]);
        Builder::new(&mut source, block).ret(&[]);

        // An output with no result to be, which is what the front end never writes and what a
        // hand written module can. Refused rather than placed by a guess.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("the list and the instruction disagree");
        assert_eq!(failed.to_string(), "this `asm` has an operand this cannot place");
    }

    /// A cast between a pointer and an integer, at whatever width the result is asked for.
    fn cast(source: &mut Func, block: Block, opcode: Opcode, from: Value, to: Type) -> Value {
        let mut build = Builder::new(source, block);
        let args = build.func().push_values(&[from]);
        build.value(InstData { args, ..InstData::new(opcode) }, to)
    }

    #[test]
    fn a_cast_between_a_pointer_and_an_integer_as_wide_is_no_instruction_at_all() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let number = cast(&mut source, block, Opcode::PtrToInt, args[0], Type::int(64));
        Builder::new(&mut source, block).ret(&[number]);

        // `long f(void *p) { return (long)p; }`. An address on this machine is an integer as wide
        // as the machine addresses, so the cast changes what the type system calls the value and
        // changes nothing about the value, and the register holding it is the one that held it.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr($rdi) = x64.arg_val_64\n    \
             x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn a_null_pointer_is_a_constant_that_reaches_a_register_before_anything_reads_it() {
        let (mut names, mut source, block, _) = blank(&[]);
        let mut build = Builder::new(&mut source, block);
        let zero = build.iconst(Type::int(64), 0);
        let null = cast(&mut source, block, Opcode::IntToPtr, zero, Type::PTR);
        Builder::new(&mut source, block).ret(&[null]);

        // `void *f(void) { return 0; }`. The cast is nothing, and reading its operand is what
        // writes the zero down: a constant is materialized where it is wanted rather than where
        // the IR defined it, and without the read there would be no instruction at all.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    %0:gpr = x64.mov_ri_64 0\n    x64.ret_val_64 %0($rax)\n}\n"
        );
    }

    #[test]
    fn the_five_linkages_the_ir_has_narrow_to_the_three_an_object_file_can_say() {
        let readings = [
            (Linkage::External, mir::Binding::Global),
            (Linkage::Common, mir::Binding::Global),
            (Linkage::Internal, mir::Binding::Local),
            (Linkage::Weak, mir::Binding::Weak),
            (Linkage::LinkOnce, mir::Binding::Weak),
        ];
        for (linkage, wanted) in readings {
            let (mut names, mut source, block, _) = blank(&[]);
            source.linkage = linkage;
            Builder::new(&mut source, block).ret(&[]);
            let out = func(&source, &mut names, &SYSV, &Elsewhere::default()).expect("a return");
            // The narrowing is done here rather than where the object is written, because a
            // machine function is all the assembler and the writer are ever handed.
            assert_eq!(out.func.binding, wanted, "{linkage:?}");
        }
    }

    /// The visibility makes the same trip and is not narrowed on the way, because ELF says all
    /// three of them.
    ///
    /// Here for the reason the linkage above is here. A machine function is the whole of what the
    /// assembler and the object writer are handed, so a fact about the symbol that does not get
    /// onto one is a fact that is gone by the time anything could write it down, and the way that
    /// shows up is a shared library exporting the wrong set of names with nothing said anywhere.
    #[test]
    fn the_visibility_survives_the_trip_from_the_ir_to_a_machine_function() {
        let readings = [
            (Visibility::Default, mir::Visibility::Default),
            (Visibility::Hidden, mir::Visibility::Hidden),
            (Visibility::Protected, mir::Visibility::Protected),
        ];
        for (visibility, wanted) in readings {
            let (mut names, mut source, block, _) = blank(&[]);
            source.visibility = visibility;
            Builder::new(&mut source, block).ret(&[]);
            let out = func(&source, &mut names, &SYSV, &Elsewhere::default()).expect("a return");
            assert_eq!(out.func.visibility, wanted, "{visibility:?}");
        }
    }

    #[test]
    fn a_cast_between_a_pointer_and_a_narrower_integer_is_reported() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let number = cast(&mut source, block, Opcode::PtrToInt, args[0], Type::int(32));
        Builder::new(&mut source, block).ret(&[number]);

        // The front end never writes one: it casts at the address width and truncates or extends
        // around it, so both of those are the rules they always were. IR from somewhere else that
        // does write one is refused rather than compiled to a move that keeps the high half.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("no rule narrows an address");
        assert_eq!(failed.to_string(), "no rule lowers a `ptrtoint` producing a `i32`");
    }

    /// The type this machine has no register for.
    fn long_double() -> Type {
        Type::float(rucc_ir::Float::F80)
    }

    #[test]
    fn a_double_widened_and_narrowed_again_goes_out_through_the_frame_and_back() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let back = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(double d) { long double x = d; return x; }`. The x87 reads memory and nothing
        // else, so the value is written to the crossing slot, loaded at the format that widens it
        // and put in the slot the eighty bit value lives in. Coming back is the same three the
        // other way. Both slots are addressed by a `lea` with nothing in it yet, which is what
        // every address in a frame looks like here until `finish` has the numbers.
        assert_eq!(
            lower(&mut names, &source),
            "mfunc @f {\nblock0:\n    \
             %0:xmm($xmm0) = x64.arg_val_f64\n    \
             %1:gpr = x64.lea_64 [$rsp]\n    \
             %2:gpr = x64.lea_64 [$rsp]\n    \
             x64.movsd_mr %0, [%1]\n    \
             x64.fld_l [%1]\n    \
             x64.fstp_t [%2]\n    \
             %3:gpr = x64.lea_64 [$rsp]\n    \
             %4:gpr = x64.lea_64 [$rsp]\n    \
             x64.fld_t [%3]\n    \
             x64.fstp_l [%4]\n    \
             %5:xmm = x64.movsd_rm [%4]\n    \
             x64.ret_val_f64 %5($xmm0)\n}\n"
        );
    }

    #[test]
    fn a_long_double_has_sixteen_bytes_of_its_own_and_keeps_them() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let once = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        let twice = cast(&mut source, block, Opcode::FPTrunc, wide, f64);
        let mut build = Builder::new(&mut source, block);
        let sum = build.binary(Opcode::FAdd, once, twice, Flags::default());
        build.ret(&[sum]);

        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");

        // Two slots and not four: sixteen bytes for the one eighty bit value, which is what the
        // psABI says one takes and is aligned to, and eight for the crossing, which every group
        // in the function shares because nothing is ever left in it. The value's slot is its own
        // for the whole function, so reading it twice reads the same sixteen bytes.
        assert_eq!(
            out.stack.locals,
            vec![Local { size: 8, align: 8 }, Local { size: 16, align: 16 }]
        );
    }

    #[test]
    fn an_integer_becomes_a_long_double_by_being_loaded_as_one() {
        let (mut names, mut source, block, args) = blank(&[Type::int(64)]);
        let wide = cast(&mut source, block, Opcode::SIToFP, args[0], long_double());
        let back =
            cast(&mut source, block, Opcode::FPTrunc, wide, Type::float(rucc_ir::Float::F64));
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(long n) { long double x = n; return x; }`. `fild` is the same push at another
        // format, so the conversion is the load and there is no instruction that converts.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_mr_64 %0, [%1]"), "{text}");
        assert!(text.contains("x64.fild_ll [%1]"), "{text}");
    }

    #[test]
    fn a_long_double_becoming_an_integer_cuts_towards_zero_with_the_control_word() {
        let (mut names, mut source, block, args) = blank(&[Type::float(rucc_ir::Float::F64)]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let whole = cast(&mut source, block, Opcode::FPToSI, wide, Type::int(32));
        Builder::new(&mut source, block).ret(&[whole]);

        // The one conversion here with no single instruction behind it. C cuts towards zero and
        // the unit rounds the way its control word says, so the word is saved, ORed with the two
        // bits that mean truncate, loaded, used and put back. Nine instructions for what `fisttp`
        // does in one, and `spec/10-backend.md` section 10.8 says why that one is not used.
        let text = lower(&mut names, &source);
        let group: Vec<&str> = text
            .lines()
            .map(str::trim)
            .filter(|line| line.starts_with("x64.f") || line.contains("_16"))
            .collect();
        assert_eq!(
            group,
            [
                "x64.fld_l [%1]",
                "x64.fstp_t [%2]",
                "x64.fnstcw [%5]",
                "%6:gpr = x64.mov_rm_16 [%5]",
                "%7:gpr(reuse 1) = x64.or_ri_16 %6, 3072",
                "x64.mov_mr_16 %7, [%5 + 2]",
                "x64.fldcw [%5 + 2]",
                "x64.fld_t [%3]",
                "x64.fistp_l [%4]",
                "x64.fldcw [%5]",
            ],
            "{text}"
        );
    }

    #[test]
    fn a_long_double_is_read_and_written_as_the_bits_it_already_is() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR, Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let value = build.load(long_double(), args[0], plain(), Flags::default());
        build.store(value, args[1], plain(), Flags::default());
        build.ret(&[]);

        // `void f(long double *a, long double *b) { *b = *a; }`. A copy is a push and a pop at the
        // format the value is already in, which neither converts nor looks: a signalling NaN stays
        // one and nothing is raised, which is the whole of what makes it a copy.
        let text = lower(&mut names, &source);
        let group: Vec<&str> =
            text.lines().map(str::trim).filter(|line| line.starts_with("x64.f")).collect();
        assert_eq!(
            group,
            ["x64.fld_t [%0]", "x64.fstp_t [%2]", "x64.fld_t [%3]", "x64.fstp_t [%1]"],
            "{text}"
        );
    }

    /// Two `long double` values, from two `double` parameters, and the instructions that made
    /// them, which every test below this one throws away.
    fn two_long_doubles(source: &mut Func, block: Block, args: &[Value]) -> (Value, Value) {
        let left = cast(source, block, Opcode::FPExt, args[0], long_double());
        let right = cast(source, block, Opcode::FPExt, args[1], long_double());
        (left, right)
    }

    /// The x87 instructions of a function, in order, with everything else dropped.
    fn stack_only(text: &str) -> Vec<&str> {
        text.lines().map(str::trim).filter(|line| line.contains("x64.f")).collect()
    }

    /// The two frame slots the last two addresses of a function were taken of, which in a
    /// comparison are the two operands in the order they go on the stack.
    fn pushed(out: &Lowered) -> Vec<usize> {
        let taken: Vec<usize> = out.stack.addresses.iter().map(|&(_, local)| local).collect();
        taken[taken.len() - 2..].to_vec()
    }

    #[test]
    fn adding_two_long_doubles_pushes_both_and_leaves_the_answer_in_a_slot() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let sum =
            Builder::new(&mut source, block).binary(Opcode::FAdd, left, right, Flags::default());
        let back = cast(&mut source, block, Opcode::FPTrunc, sum, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `double f(double a, double b) { return (long double) a + (long double) b; }`. The last
        // four lines are the add: both operands pushed, the instruction that names neither of
        // them because they are the top two of a stack, and the answer taken off into its slot.
        let text = lower(&mut names, &source);
        assert_eq!(
            stack_only(&text),
            [
                "x64.fld_l [%2]",
                "x64.fstp_t [%3]",
                "x64.fld_l [%4]",
                "x64.fstp_t [%5]",
                "x64.fld_t [%6]",
                "x64.fld_t [%7]",
                "x64.fadd_p",
                "x64.fstp_t [%8]",
                "x64.fld_t [%9]",
                "x64.fstp_l [%10]",
            ],
            "{text}"
        );
    }

    #[test]
    fn a_subtraction_pushes_the_left_operand_first_and_asks_for_the_att_spelling() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let less =
            Builder::new(&mut source, block).binary(Opcode::FSub, left, right, Flags::default());
        let back = cast(&mut source, block, Opcode::FPTrunc, less, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // The left one goes on first, so it ends up under the right one, and the answer wanted is
        // the one below minus the top. In AT&T that is `fsubrp`, since `fsubp` there is `DE E0+i`
        // and computes the other one. The `r` says which spelling this is and not which order the
        // pushes were in. `crates/rucc/tests/x87.rs` is what says the answer is right, because a
        // name is what got this wrong the first time.
        let text = lower(&mut names, &source);
        assert_eq!(
            &stack_only(&text)[4..8],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "x64.fsubr_p", "x64.fstp_t [%8]"],
            "{text}"
        );
    }

    #[test]
    fn negating_a_long_double_turns_the_sign_over_and_reads_nothing() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let flipped = Builder::new(&mut source, block).unary(Opcode::FNeg, wide, long_double());
        let back = cast(&mut source, block, Opcode::FPTrunc, flipped, f64);
        Builder::new(&mut source, block).ret(&[back]);

        // `fchs` and not a subtraction from zero, which would give a different answer at a negative
        // zero and would signal at a NaN. It does not read the value as a number at all.
        let text = lower(&mut names, &source);
        assert_eq!(
            &stack_only(&text)[2..5],
            ["x64.fld_t [%3]", "x64.fchs", "x64.fstp_t [%4]"],
            "{text}"
        );
    }

    #[test]
    fn comparing_two_long_doubles_puts_the_left_one_on_top() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Ogt, left, right, Flags::default());
        build.ret(&[]);

        // `a > b`. `fucomip` asks about the top of the stack against what is under it, so the
        // operand the predicate is about has to go on last, which is the other way round from the
        // arithmetic above. The pop that clears the loser and the byte that reads the flags are
        // both inside the one opcode.
        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");
        let slots = pushed(&out);
        assert_eq!(slots, [2, 1], "the right operand goes on first and the left one on top");
        let text = mir::print_func(&out.func, &names, &REGS);
        assert_eq!(
            &stack_only(&text)[4..],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "%8:gpr = x64.fucomip_set_a"],
            "{text}"
        );
    }

    #[test]
    fn a_comparison_that_the_machine_has_backwards_swaps_the_two_pushes() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Olt, left, right, Flags::default());
        build.ret(&[]);

        // `a < b` is `b > a` and this machine has the one condition, so the same opcode runs with
        // the operands the other way round. The same trade the vector rules make, and it has to
        // be the same one: a `long double` comparison that picked a different condition from the
        // `double` comparison of the same two numbers would be wrong at exactly the unordered
        // cases the two conditions differ on.
        //
        // Which slot each push names is the whole of the difference from the test above, and the
        // text does not show it, since an address in a frame is a `lea` with nothing in it until
        // `finish` has the numbers. So the slots are what is read here.
        let out = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect("every instruction is written");
        let slots = pushed(&out);
        assert_eq!(slots, [1, 2], "the left operand goes on first and the right one on top");
        let text = mir::print_func(&out.func, &names, &REGS);
        assert_eq!(
            &stack_only(&text)[4..],
            ["x64.fld_t [%6]", "x64.fld_t [%7]", "%8:gpr = x64.fucomip_set_a"],
            "{text}"
        );
    }

    #[test]
    fn an_ordered_equal_needs_a_second_byte_to_put_the_two_conditions_together() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::Oeq, left, right, Flags::default());
        build.ret(&[]);

        // Equal and ordered are two conditions and the flags carry both, so the opcode writes a
        // second register as well as the one the value is in and ANDs them together. Said here by
        // handing it a spare, since an instruction that wrote a register nothing knew about would
        // be an instruction the allocator could put a live value in the way of.
        let text = lower(&mut names, &source);
        assert!(text.contains("%8:gpr, %9:gpr = x64.fucomip_set_e_and_np"), "{text}");
    }

    #[test]
    fn a_comparison_that_is_never_asked_is_reported() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64, f64]);
        let (left, right) = two_long_doubles(&mut source, block, &args);
        let mut build = Builder::new(&mut source, block);
        build.fcmp(FloatPred::False, left, right, Flags::default());
        build.ret(&[]);

        // Always false is a constant and not a comparison, so there is no condition to pick and
        // nothing here folds it into one: an instruction that quietly agreed with it would hide
        // that the optimizer left a comparison in that it should have taken out.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("no condition is always false");
        assert_eq!(failed.to_string(), "no rule lowers a `fcmp` producing a `i1`");
    }

    #[test]
    fn a_long_double_constant_is_the_bits_of_it_put_where_the_value_lives() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        // `1.5L`, which is the leading bit and one more of significand, and an exponent of zero.
        let one_and_a_half = build.fconst(long_double(), 0x3fff_c000_0000_0000_0000);
        build.store(one_and_a_half, args[0], plain(), Flags::default());
        build.ret(&[]);

        // No x87 instruction at all. A slot holding one of these is the value, so a constant is
        // its ten bytes written where the value lives, and whatever reads it does the `fld`.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_ri_64 -4611686018427387904"), "{text}");
        assert!(text.contains("x64.mov_ri_16 16383"), "{text}");
        assert!(text.contains("x64.mov_mr_16 %3, [%1 + 8]"), "{text}");
        // The six bytes above the ten are the padding that makes the type sixteen wide, and they
        // are unspecified rather than zero, so nothing writes them.
        assert_eq!(text.matches("x64.mov_mr").count(), 2, "{text}");
    }

    #[test]
    fn a_negative_long_double_constant_keeps_the_bit_above_its_exponent() {
        let (mut names, mut source, block, args) = blank(&[Type::PTR]);
        let mut build = Builder::new(&mut source, block);
        let minus = build.fconst(long_double(), 0xbfff_c000_0000_0000_0000);
        build.store(minus, args[0], plain(), Flags::default());
        build.ret(&[]);

        // `-1.5L`. The sign is the top bit of the two byte half, so the immediate that half is put
        // in a register with is above the signed range of sixteen bits and has to stay there: read
        // as a number it would be negative, and it is not a number, it is two bytes.
        let text = lower(&mut names, &source);
        assert!(text.contains("x64.mov_ri_16 49151"), "{text}");
    }

    #[test]
    fn a_long_double_crosses_an_edge_as_an_address_and_is_copied_where_it_lands() {
        let (mut names, mut source, block, args) = blank(&[Type::float(rucc_ir::Float::F64)]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let next = source.create_block();
        let param = source.append_param(next, long_double());
        Builder::new(&mut source, block).jump(next, &[wide]);
        Builder::new(&mut source, next).ret(&[param]);

        // What the edge carries is the address of the slot the value is already in, which is an
        // ordinary register the allocator has an opinion about. The block on the other side copies
        // the sixteen bytes into a slot of its own before anything reads them, so a second edge
        // handing over a second address would still leave one place for a reader to look.
        let text = lower(&mut names, &source);
        let second: Vec<&str> = text
            .lines()
            .skip_while(|line| !line.starts_with("block1"))
            .skip(1)
            .take(3)
            .map(str::trim)
            .collect();
        assert_eq!(
            second,
            ["x64.fld_t [%4]", "%5:gpr = x64.lea_64 [$rsp]", "x64.fstp_t [%5]"],
            "{text}"
        );
    }

    #[test]
    fn more_long_doubles_at_a_block_than_the_stack_is_deep_are_reported() {
        let f64 = Type::float(rucc_ir::Float::F64);
        let (mut names, mut source, block, args) = blank(&[f64]);
        let wide = cast(&mut source, block, Opcode::FPExt, args[0], long_double());
        let next = source.create_block();
        let params: Vec<Value> =
            (0..=X87_DEPTH).map(|_| source.append_param(next, long_double())).collect();
        let carried: Vec<Value> = params.iter().map(|_| wide).collect();
        Builder::new(&mut source, block).jump(next, &carried);
        Builder::new(&mut source, next).ret(&[params[0]]);

        // The copies go through the x87 stack so that every one of them is read before any of them
        // is written, which is what makes a block that swaps two of these right. Nine of them do
        // not fit on the stack, and copying the ninth before or after the rest is the order that
        // could be wrong, so it is refused instead.
        let failed = func(&source, &mut names, &SYSV, &Elsewhere::default())
            .expect_err("nine do not fit on the stack");
        assert_eq!(
            failed.to_string(),
            "block1 takes 9 parameters of type `f80` and only 8 can cross an edge at once"
        );
        assert_eq!(failed.inst(), None);
    }
}