Skip to main content

rucc_opt/
split.rs

1//! Splits a loop into a run of iterations that needs no checks and the rest of it, which keeps them.
2//!
3//! Design: `spec/safe-memory/07-check-elimination.md` section 7.4, which names this and says what it
4//! is for: "Loop splitting is the general form. The checked part and the unchecked part are divided
5//! at `min(n, extent / sizeof(T))`."
6//!
7//! [`crate::hoist`] is the pass next door and it answers a different question. It puts one check in
8//! front of a loop that covers every access the loop makes, which needs the loop to make every one
9//! of them: an exact count, one way out, and a check every iteration reaches. Most loops in real code
10//! are not like that. The census on tamnd/rucc#782 says that of the roughly fifteen hundred checks
11//! SQLite still carries at `-O2`, six hundred and ninety two are in loops with a second way out and
12//! sixty four are checks an iteration can finish without reaching. Neither is a loop hoisting can say
13//! anything about, and both are loops this one can, because it never has to claim the loop reaches
14//! the end of what it might read. It only has to know a prefix that is safe.
15//!
16//! # What the two halves are
17//!
18//! The loop is copied. The original becomes the fast half and loses its checks, the copy becomes the
19//! slow half and keeps them, and a new block in front of the original decides which one runs. That
20//! block counts iterations of the fast half and hands over to the slow half once the count reaches a
21//! limit worked out in the preheader.
22//!
23//! The counter is a new one rather than the loop's own, and the test is against a limit rather than
24//! against anything the loop compares. That is what makes this work on a loop with several ways out:
25//! the fast half keeps every exit the loop had, so leaving early still leaves early, and the extra
26//! test is only ever the reason the fast half stops early and never the reason it runs longer.
27//!
28//! A loop where no address moves gets neither the block nor the counter. Which half runs is settled
29//! by an answer that does not change while the loop runs, so the way into the loop is where the two
30//! halves are chosen between and there is nothing to count. Half the loops this takes on SQLite are
31//! that shape.
32//!
33//! # Where the limit comes from
34//!
35//! For a check whose address is `first + i * step` reading `reach` bytes each time, every iteration
36//! with `i * step + reach <= extent` is one the check cannot fail on, where `extent` is how many
37//! bytes from `first` on belong to whatever owns `first`. So the limit is
38//! `(extent - reach) / step + 1`, or zero when `extent` is smaller than `reach`, and where a loop has
39//! several such checks in it the limit is the smallest of theirs.
40//!
41//! The extent is the half of that a compiler cannot work out, so it is asked at run time, through the
42//! `cap_extent` query that tamnd/rucc#792 added. The query takes a limit on how far to look and the
43//! answer is never more than that, which is why this pass still wants a trip count: what it asks for
44//! is how many bytes the loop was going to read anyway, so the walk in the runtime is bounded by work
45//! the loop is already doing. A count that is too small costs iterations in the slow half and a count
46//! that is too large costs a slightly longer walk, and neither is a wrong answer, which is why the
47//! count is read from any exit that offers one rather than from an exit that runs every time.
48//!
49//! An address that does not move is the same expression with a step of zero, which is a division
50//! that does not have to happen and a question with a shorter answer. Such a check fits on the first
51//! iteration or on none of them, so what there is to work out is which of the two, and how far the
52//! runtime is asked to look is just the bytes the access reads. Hoisting would rather have these,
53//! and it takes the ones in loops it is willing to touch. What is left over is the ones in loops it
54//! refused for one of its own reasons, a second way out or a call inside, and those come back here.
55//!
56//! # Why the fast half may drop a check
57//!
58//! `check_bounds` asks whether the bytes an access names lie inside one object. Every address in
59//! `[first, first + extent)` is inside the object that owns `first`, by what the query answers, and
60//! the limit is exactly the iterations whose access stays in that window. So no check in the fast
61//! half could have failed.
62//!
63//! `check_live` asks whether anything owns the address right now, and the query answered that too,
64//! since a byte belonging to the owner of `first` is a byte with an owner. Right now is the catch,
65//! and it is why nothing that could free may be in the loop. A call in the body could free the object
66//! between the question and the iteration that reads it, and then the fast half would read freed
67//! storage with nothing to say so.
68//!
69//! That is a question about the callee rather than about calling, and [`crate::nofree`] answers it
70//! before the pipeline starts, so a call carrying [`rucc_ir::Flags::NOFREE`] is one the loop may
71//! keep. Hoisting refuses every call whatever it does, and the reason is not this one: it needs the
72//! loop to reach the end of what its count says, and a call that does not come back leaves it short.
73//! Splitting never claims the loop reaches the end, so a call that might not come back costs it
74//! nothing.
75//!
76//! Two answers of the query carry the weight and both are argued where the query is implemented. An
77//! address no watched region covers gets the whole limit back, so a loop over a local or a global
78//! splits into a fast half that runs the whole way, which is right because no check on such an
79//! address ever fires under this milestone. An address whose granule nobody owns gets zero, so the
80//! limit is zero, the fast half runs no iterations, and the check inside the slow half is what reports
81//! the dangling pointer, at the access rather than at the loop.
82//!
83//! # Which loops
84//!
85//! Innermost, one latch, a preheader, nothing in it that could free, and no value defined inside it
86//! that anything outside reads. Not a count, unlike hoisting, because the count is not something
87//! this rests on: it is spent on how far to ask the runtime to look, and the runtime answers with a
88//! true count of the bytes that belong to the object whatever it was asked for. A loop nobody
89//! counted gets the same guess everything else that has to guess about a loop gets, ten, which is
90//! GCC's `avg-loop-niter` and the number [`crate::scev::Estimate`] already hands out. The last is
91//! loop closed form, which [`crate::canon`] establishes, and it is checked rather than assumed
92//! because the copy would otherwise leave a reader outside the loop seeing whichever half happened
93//! to define the value.
94//!
95//! Canonicalization runs a long way in front of this, and `simplify-cfg` between the two undoes some
96//! of what it did, so on SQLite the closed form condition is what refuses 351 of the checks this
97//! would otherwise have taken out. Running canonicalization again in front of this gets 156 of them
98//! back and costs 17672 bytes of `.text`, which is a bad trade for eleven more checks, so the answer
99//! is for this to repair the exits of the one loop it is splitting rather than for the pipeline to
100//! repair every loop in the function. That is its own piece of work.
101//!
102//! Not every check in the loop has to be one this can size. A check whose address the analysis cannot
103//! follow simply stays in both halves, and the fast half is then a loop with fewer checks in it rather
104//! than none. That is worth having on its own and it is worth having because it is what a real loop
105//! looks like: one sweep the analysis reads and one index that came out of a table.
106//!
107//! # Which level
108//!
109//! `-O2` and `-O3`, alongside `crate::unroll` and for the same reason. The loop body is copied, so
110//! the function grows by about the size of the loop, and buying speed with code is what those levels
111//! are for and what `-Os` and `-Oz` are for declining.
112
113use std::collections::{HashMap, HashSet};
114
115use rucc_cost::heuristics;
116use rucc_ir::{
117    Block, BlockCall, Builder, Extra, Flags, Func, Inst, InstData, IntPred, Opcode, Type, Value,
118};
119
120use crate::canon;
121use crate::cfg::Cfg;
122use crate::copy;
123use crate::discharge::operand_of;
124use crate::dom::Dominators;
125use crate::loops::{LoopId, Loops};
126use crate::scev::{Evolution, Scev};
127use crate::trip::{Around, counted, covered, inst_of};
128use crate::{Analyses, Fuel, Pass, Preserved, Stats};
129
130/// What is reported when a loop is split.
131const SPLIT: &str = "loop split, the iterations in front of the first one that could fail a check \
132                     run without them";
133
134/// What is reported when a loop had to be put back into closed form before it could be split.
135const CLOSED_HERE: &str = "loop put back into closed form, a value it defines is read after it and both halves define one";
136
137/// What is reported when the pass ran out of fuel with a loop it was about to split.
138const NO_FUEL: &str = "loop left alone, the pass ran out of fuel";
139
140/// What is reported for a loop with nowhere to work the limit out.
141const NO_PREHEADER: &str = "loop left alone, it has no block in front of it to put a check in";
142
143/// What is reported for a loop with a loop inside it.
144const A_LOOP_INSIDE: &str = "loop left alone, it has another loop inside it";
145
146/// What is reported for a loop with more than one way round.
147const MANY_LATCHES: &str = "loop left alone, it goes back to its header from more than one place";
148
149/// What is reported for a loop with a call in it that could free.
150const A_CALL_INSIDE: &str = "loop left alone, a call in it might free what the loop is reading";
151
152/// What is reported for a loop holding something that ends a lifetime outright.
153const ENDS_A_LIFETIME: &str = "loop left alone, something in it ends a lifetime";
154
155/// What is reported for a loop holding something the copier cannot copy.
156const NOT_COPYABLE: &str = "loop left alone, something in it carries a side table this cannot copy";
157
158/// What is reported for a loop whose values are read after it without going through a parameter.
159const ESCAPES: &str = "loop left alone, a value it defines is read outside it";
160
161/// What is reported for a loop whose two halves would be too much code.
162const TOO_BIG: &str = "loop left alone, the two halves would be more code than the limit allows";
163
164/// What is reported for a check whose address does not walk the loop.
165const NOT_A_SWEEP: &str = "check kept in both halves, its address does not walk the loop by a \
166                           constant";
167
168/// What is reported for a check whose address the analysis has nothing to say about.
169const NOT_FOLLOWED: &str = "check kept in both halves, what its address does round the loop is not \
170                            something the analysis follows";
171
172/// What is reported for a check whose address walks backwards.
173const BACKWARDS: &str = "check kept in both halves, its address walks the loop from high to low";
174
175/// What is reported for a check whose step does not keep its alignment.
176const MISALIGNED: &str =
177    "check kept in both halves, its step is not a whole number of its alignment";
178
179/// What is reported for a check that already covers a range the program worked out.
180const ALREADY_COMPUTED: &str =
181    "check kept in both halves, how many bytes it covers is a number only the program has";
182
183/// The pass.
184#[derive(Debug)]
185pub struct Split;
186
187impl Pass for Split {
188    fn name(&self) -> &'static str {
189        "split"
190    }
191
192    fn describe(&self) -> &'static str {
193        "a loop becomes a run of iterations with no checks in it and the rest of the loop with them"
194    }
195
196    fn preserves(&self) -> Preserved {
197        // Blocks appear and edges move, so nothing built on the graph stands.
198        Preserved::NONE
199    }
200
201    fn run(&self, func: &mut Func, an: &mut Analyses, fuel: &mut Fuel) -> Stats {
202        let mut stats = Stats::new();
203        if func.entry().is_none() {
204            return stats;
205        }
206        let cfg = an.cfg(func).clone();
207        let loops = an.loops(func).clone();
208        if loops.count() == 0 {
209            return stats;
210        }
211
212        // Worked out first and applied afterwards, because scalar evolution reads the function and
213        // the transformation writes it. Every plan is about an innermost loop and no two innermost
214        // loops share a block, so applying one leaves every other one's blocks where they were.
215        let mut plans = planned(func, &cfg, &loops, &mut stats);
216
217        // Closed form put back where it is missing, before anything is copied. The repair adds a
218        // block parameter and rewrites uses, so it moves no edge and creates no block, which is why
219        // the graph and the loop forest above are both still good after it. What it does move is
220        // which value a use inside another loop names, and a plan is a list of values, so a repair
221        // means the plans are worked out again rather than trusted. The stats go with them, or the
222        // first round's reasons would be counted twice.
223        let dom = an.dominators(func).clone();
224        let repairs = repaired(func, &cfg, &dom, &loops, &plans, fuel);
225        if repairs.made > 0 {
226            stats = Stats::new();
227            plans = planned(func, &cfg, &loops, &mut stats);
228            for _ in 0..repairs.worked {
229                stats.optimized(CLOSED_HERE);
230            }
231        }
232        plans.retain(|plan| {
233            if leaving(func, plan) {
234                stats.missed(ESCAPES);
235                return false;
236            }
237            true
238        });
239
240        let mut changed = false;
241        for plan in plans {
242            if !fuel.take() {
243                stats.missed(NO_FUEL);
244                continue;
245            }
246            apply(func, &plan);
247            stats.optimized(SPLIT);
248            changed = true;
249        }
250        if changed {
251            an.clear();
252        }
253        stats
254    }
255}
256
257/// One check the fast half will not need, and the walk that says so.
258#[derive(Debug)]
259struct Sweep {
260    /// The check itself, which is removed from the fast half and kept in the copy.
261    check: Inst,
262    /// Where the first iteration's address is computed from.
263    base: Value,
264    /// How far past that value the first iteration reads.
265    offset: i128,
266    /// How far the address moves each time round, which is a number of bytes and never negative.
267    /// Zero is an address that does not move, which is allowed and puts no limit on the loop.
268    step: i128,
269    /// How many bytes one access covers.
270    reach: i128,
271}
272
273/// One loop to split, worked out before anything is written.
274#[derive(Debug)]
275struct Plan {
276    /// The loop itself, which is read again when its closed form has to be repaired.
277    id: LoopId,
278    /// Where the limit is worked out.
279    preheader: Block,
280    /// The block the guard takes over from.
281    header: Block,
282    /// The block the back edge leaves from, which is where the iteration count goes up.
283    latch: Block,
284    /// Everything that is copied, which is the whole loop.
285    body: Vec<Block>,
286    /// How many times the loop goes round, which is what the runtime is asked to look no further
287    /// than.
288    around: Around,
289    /// The checks the fast half will not need, which is never empty in a plan.
290    sweeps: Vec<Sweep>,
291}
292
293/// Plans a loop, or counts what stopped it.
294///
295/// Nothing is reported for a loop with no check in it, because a loop that does no memory access is
296/// not a missed opportunity and a report for every one of them would bury the loops that are.
297fn sweep(
298    func: &Func,
299    cfg: &Cfg,
300    loops: &Loops,
301    scev: &mut Scev<'_>,
302    id: LoopId,
303    plans: &mut Vec<Plan>,
304    stats: &mut Stats,
305) {
306    let body = loops.blocks(id).to_vec();
307    let checks: Vec<Inst> = body
308        .iter()
309        .flat_map(|&block| func.insts(block).collect::<Vec<Inst>>())
310        .filter(|&inst| matches!(func[inst].opcode, Opcode::CheckBounds | Opcode::CheckLive))
311        .collect();
312    if checks.is_empty() {
313        return;
314    }
315
316    let (preheader, latch) = match shaped(func, cfg, loops, id, &body) {
317        Ok(shape) => shape,
318        Err(why) => {
319            stats.missed(why);
320            return;
321        }
322    };
323    // Not a refusal when there is no count, unlike in hoisting, because the count is not something
324    // this rests on. It is spent on how far to ask the runtime to look, and the runtime answers with
325    // a true count of the bytes that belong to the object whatever it was asked for. A guess that is
326    // too small costs iterations in the half that keeps its checks and a guess that is too large
327    // costs a slightly longer walk, so a loop nobody counted gets the same guess everything else
328    // that has to guess about a loop gets.
329    let around =
330        counted(scev, id).unwrap_or(Around::Number(i128::from(crate::scev::ASSUMED_ITERATIONS)));
331
332    let mut sweeps = Vec::new();
333    for check in checks {
334        match walked(func, scev, id, check) {
335            Ok(sweep) => sweeps.push(sweep),
336            Err(why) => stats.missed(why),
337        }
338    }
339    if sweeps.is_empty() {
340        return;
341    }
342    plans.push(Plan { id, preheader, header: loops.header(id), latch, body, around, sweeps });
343}
344
345/// The preheader and the latch of a loop this pass may copy, or why there is not one.
346///
347/// The conditions are the module comment's. The one worth restating is freeing, because it is the
348/// only one that is about what the fast half is allowed to leave out rather than about whether the
349/// copy can be made at all: the extent is asked once before the loop and believed for the whole of
350/// the fast half, so anything that could hand the storage back in the middle would make the answer
351/// stale, and the fast half has nothing left in it to notice.
352///
353/// Which is a question about the callee and not about calling, so it is asked of the callee.
354/// [`crate::nofree`] settles it before the pipeline starts and writes the answer onto the call site,
355/// and a call carrying it reaches nothing that ends a lifetime. Note that this is a weaker
356/// requirement than [`crate::hoist`]'s, which refuses every call whatever it does, because hoisting
357/// needs the loop to reach the end of what its count says and a call that does not come back leaves
358/// it short. Splitting never claims that, so coming back is not something it needs.
359fn shaped(
360    func: &Func,
361    cfg: &Cfg,
362    loops: &Loops,
363    id: LoopId,
364    body: &[Block],
365) -> Result<(Block, Block), &'static str> {
366    let Some(preheader) = loops.preheader(cfg, id) else {
367        return Err(NO_PREHEADER);
368    };
369    let [latch] = loops.latches(id) else {
370        return Err(MANY_LATCHES);
371    };
372    for &block in body {
373        if loops.innermost(block) != Some(id) {
374            return Err(A_LOOP_INSIDE);
375        }
376        for inst in func.insts(block) {
377            match func[inst].opcode {
378                Opcode::Call | Opcode::CallIndirect | Opcode::TailCall
379                    if !func[inst].flags.contains(Flags::NOFREE) =>
380                {
381                    return Err(A_CALL_INSIDE);
382                }
383                // Assembly could do anything and the two meta instructions end a lifetime by
384                // definition, which is the same answer `crate::nofree` gives for all three.
385                Opcode::InlineAsm | Opcode::MetaEnd | Opcode::MetaTransfer => {
386                    return Err(ENDS_A_LIFETIME);
387                }
388                _ => {}
389            }
390            if !copy::copyable(func, inst) {
391                return Err(NOT_COPYABLE);
392            }
393        }
394    }
395    let size = body.iter().map(|&block| func.insts(block).count()).sum::<usize>();
396    if size > heuristics::SPLIT_MAX_INSNS as usize {
397        return Err(TOO_BIG);
398    }
399    Ok((preheader, *latch))
400}
401
402/// Every loop in the function that is worth copying, and why each of the others is not.
403fn planned(func: &Func, cfg: &Cfg, loops: &Loops, stats: &mut Stats) -> Vec<Plan> {
404    let mut plans = Vec::new();
405    let mut scev = Scev::new(func, cfg, loops);
406    for id in loops.all() {
407        sweep(func, cfg, loops, &mut scev, id, &mut plans, stats);
408    }
409    plans
410}
411
412/// How many loops the closed form repair touched, and how many of those it finished.
413///
414/// Two numbers rather than one because they answer different questions. Anything touched at all is
415/// why the plans have to be worked out again, and only the ones it finished are loops that can now
416/// be copied and so are what gets reported.
417struct Repairs {
418    /// Loops the repair wrote something into.
419    made: usize,
420    /// Loops that are in closed form afterwards.
421    worked: usize,
422}
423
424/// Puts the loops that need it back into closed form, before anything is copied.
425///
426/// [`crate::canon`] establishes closed form a long way in front of this pass and `simplify-cfg`
427/// between the two undoes some of what it did. Running the whole of canonicalization again was
428/// measured and it costs 17672 bytes of `.text` on the SQLite amalgamation, because it repairs every
429/// loop in the function rather than the ones about to be copied. This repairs those, which costs
430/// nothing on a function with no loop to split.
431///
432/// Not every loop can be repaired this way. A value read past a join that no single exit dominates
433/// needs a parameter at the join as well as at each exit, and the repair adds one at the exits only,
434/// so the count of what worked is a second look rather than an assumption that the first one did.
435fn repaired(
436    func: &mut Func,
437    cfg: &Cfg,
438    dom: &Dominators,
439    loops: &Loops,
440    plans: &[Plan],
441    fuel: &mut Fuel,
442) -> Repairs {
443    let mut repairs = Repairs { made: 0, worked: 0 };
444    for plan in plans {
445        if !leaving(func, plan) {
446            continue;
447        }
448        let mut wrote = false;
449        while let Some(job) = canon::leaked(func, cfg, dom, loops, plan.id) {
450            if !fuel.take() {
451                break;
452            }
453            canon::close(func, &job);
454            wrote = true;
455        }
456        if !wrote {
457            continue;
458        }
459        repairs.made += 1;
460        if !leaving(func, plan) {
461            repairs.worked += 1;
462        }
463    }
464    repairs
465}
466
467/// Whether anything after this loop reads a value its body defines.
468fn leaving(func: &Func, plan: &Plan) -> bool {
469    let inside: HashSet<Block> = plan.body.iter().copied().collect();
470    escapes(func, &plan.body, &inside)
471}
472
473/// Whether anything outside the loop reads a value defined inside it.
474///
475/// Where there is one, the two halves would leave it reading whichever of them happened to define
476/// it. Closed form is what makes it not one: the use names a parameter of the block the loop leaves
477/// to, and each half fills that parameter in on its own way out.
478fn escapes(func: &Func, body: &[Block], inside: &HashSet<Block>) -> bool {
479    let mut defined: HashSet<Value> = HashSet::new();
480    for &block in body {
481        defined.extend(func[block].params.iter().copied());
482        for inst in func.insts(block) {
483            defined.extend(func[inst].results());
484        }
485    }
486    for block in func.blocks() {
487        if inside.contains(&block) {
488            continue;
489        }
490        for inst in func.insts(block) {
491            if func[func[inst].args].iter().any(|value| defined.contains(value)) {
492                return true;
493            }
494            for call in func.successors(inst) {
495                if func[call.args].iter().any(|value| defined.contains(value)) {
496                    return true;
497                }
498            }
499        }
500    }
501    false
502}
503
504/// What one check's address does round the loop, or why the pass cannot say.
505fn walked(
506    func: &Func,
507    scev: &mut Scev<'_>,
508    id: LoopId,
509    check: Inst,
510) -> Result<Sweep, &'static str> {
511    let args = &func[func[check].args];
512    // A check that already carries its own extent is one hoisting put somewhere, and how many bytes
513    // it covers is not a number this pass can divide by a step.
514    if args.len() > 2 {
515        return Err(ALREADY_COMPUTED);
516    }
517    let (Some(&capability), Some(&pointer)) = (args.first(), args.get(1)) else {
518        return Err(NOT_A_SWEEP);
519    };
520    if operand_of(func, capability, Opcode::CapOf, 0) != Some(pointer) {
521        return Err(NOT_A_SWEEP);
522    }
523    // A liveness check reads no bytes, so the window it needs is the one byte its address is in.
524    // A bounds check carries how many it reads in its payload.
525    let (reach, align) = match func[check].extra {
526        Extra::Mem(held) => (i128::from(func[held].size), i128::from(func[held].align)),
527        _ => (1, 1),
528    };
529
530    // An address that does not move is a sweep with a step of zero, and the arithmetic below takes
531    // it without a special case anywhere except the division. Hoisting would rather have these, but
532    // hoisting only gets the ones in loops it is willing to touch at all, and a loop it refused for
533    // one of its own reasons leaves the check where it is. Splitting is willing to touch more loops,
534    // so the same check comes back here and there is no reason to hand it back.
535    let (start, step) = match scev.evolution(id, pointer) {
536        Evolution::Affine(chrec) => {
537            let Some(step) = chrec.step.as_number() else {
538                return Err(NOT_A_SWEEP);
539            };
540            (chrec.base, step)
541        }
542        Evolution::Invariant(base) => (base, 0),
543        _ => return Err(NOT_FOLLOWED),
544    };
545    if step < 0 {
546        return Err(BACKWARDS);
547    }
548    // Scale one because the base is an address. Anything else is a multiple of a pointer, which is
549    // not a thing the loop computed, so it is a shape this reads rather than a case to handle.
550    let (Some(base), 1) = (start.value, start.scale) else {
551        return Err(NOT_A_SWEEP);
552    };
553    if step != 0 && step % align != 0 {
554        return Err(MISALIGNED);
555    }
556    Ok(Sweep { check, base, offset: start.offset, step, reach })
557}
558
559/// Makes the two halves and the block that chooses between them.
560///
561/// The order matters in two places. The copy is made before anything is rewired, so the copy's back
562/// edge is remapped to the copy's own header rather than to a guard that did not exist yet. The
563/// checks come out of the fast half last, so the copy still has them.
564fn apply(func: &mut Func, plan: &Plan) {
565    // The slow half, which is the loop as it stands, under a substitution that renames everything it
566    // defines. Nothing is seeded, so its header gets parameters of its own, which is what a copy
567    // reached from a block that also reaches the original needs.
568    let mut renamed: HashMap<Value, Value> = HashMap::new();
569    let copies = copy::blocks(func, &plan.body, &mut renamed);
570    let slow = copies[&plan.header];
571
572    let choice = limited(func, plan);
573    let (limit, start) = match choice {
574        // Nothing in the loop moves, so which half runs is settled in the preheader and settled for
575        // good. There is no guard block and no counter: the way into the loop becomes the choice.
576        Choice::Once(fits) => {
577            let term = func.terminator(plan.preheader).expect("a preheader ends in a jump");
578            let args = copy::edge_args(func, term, plan.header);
579            func.remove_inst(term);
580            Builder::new(func, plan.preheader).br_if(fits, plan.header, &args, slow, &args);
581            take(func, plan);
582            return;
583        }
584        Choice::Counted(limit, start) => (limit, start),
585    };
586
587    // The guard, which takes over the header's place: the preheader arrives here, the back edge
588    // comes back to here, and the header is reached from here and nowhere else. Its first parameter
589    // is a counter of its own, because the loop's counter is not something this pass has to find and
590    // a loop with several ways out may not have one.
591    let word = Type::int(64);
592    let types: Vec<Type> = func[plan.header].params.iter().map(|&param| func[param].ty).collect();
593    let guard = func.create_block();
594    let round = func.append_param(guard, word);
595    let carried: Vec<Value> = types.iter().map(|&ty| func.append_param(guard, ty)).collect();
596
597    let mut build = Builder::new(func, guard);
598    let inside = build.icmp(IntPred::Slt, round, limit);
599    build.br_if(inside, plan.header, &carried, slow, &carried);
600
601    // The way in, which now hands the guard a count of no iterations so far.
602    let term = func.terminator(plan.preheader).expect("a preheader ends in a jump to the header");
603    route(func, term, plan.header, guard, start);
604
605    // The way round, which counts one more. The counter is the guard's parameter and the guard
606    // dominates every block in the fast half, so the latch may read it.
607    let term = func.terminator(plan.latch).expect("a latch ends in a branch back to the header");
608    let mut build = Builder::new(func, plan.latch);
609    let one = build.iconst(word, 1);
610    let next = build.binary(Opcode::Add, round, one, Flags::NSW);
611    for value in [one, next] {
612        let inst = inst_of(func, value);
613        func.remove_inst(inst);
614        func.insert_before(inst, term);
615    }
616    route(func, term, plan.header, guard, next);
617    take(func, plan);
618}
619
620/// Takes the checks the fast half does not need out of it.
621///
622/// The `cap_of` each one was reading is left where it is, for `dce` after this pass to take away,
623/// which is the arrangement [`crate::hoist`] and [`crate::discharge`] are both in.
624fn take(func: &mut Func, plan: &Plan) {
625    for sweep in &plan.sweeps {
626        func.remove_inst(sweep.check);
627    }
628}
629
630/// Sends every edge this terminator has to `from` to `to` instead, with one more argument in front.
631fn route(func: &mut Func, term: Inst, from: Block, to: Block, first: Value) {
632    for at in func.target_list(term).iter() {
633        let call = func[at];
634        if call.block != from {
635            continue;
636        }
637        let mut args = vec![first];
638        args.extend_from_slice(&func[call.args]);
639        let args = func.push_values(&args);
640        func.set_block_call(at, BlockCall { block: to, args });
641    }
642}
643
644/// How the two halves are chosen between, which depends on whether any address in the loop moves.
645enum Choice {
646    /// No address moves, so the answer is a yes or a no and it is the same on every iteration. The
647    /// value is that answer, and a loop like this needs no guard block and no counter.
648    Once(Value),
649    /// Something moves, so the fast half runs a bounded number of iterations. The values are the
650    /// limit and the zero the guard's counter starts at.
651    Counted(Value, Value),
652}
653
654/// Builds what the preheader has to work out before either half can run.
655///
656/// One `cap_extent` per check and the smallest of what they allow, all of it in the preheader in
657/// front of the jump into the loop. A builder appends to the end of a block, which in a block that
658/// already has its terminator is after it, so everything is built first and then moved in front of
659/// the terminator in the order it was built.
660fn limited(func: &mut Func, plan: &Plan) -> Choice {
661    let term = func.terminator(plan.preheader).expect("a preheader ends in a jump to the header");
662    let still = plan.sweeps.iter().all(|sweep| sweep.step == 0);
663    let mut made = Vec::new();
664    let mut build = Builder::new(func, plan.preheader);
665    let start = build.iconst(Type::int(64), 0);
666    made.push(start);
667
668    let mut settled: Option<Value> = None;
669    for sweep in &plan.sweeps {
670        settled = Some(match (settled, still) {
671            // Every check has to fit for the fast half to be the one that runs, and each of them
672            // fits or does not on its own, so what the halves are chosen on is all of them together.
673            (None, true) => fits(&mut build, &mut made, sweep),
674            (Some(so_far), true) => {
675                let also = fits(&mut build, &mut made, sweep);
676                let both = build.binary(Opcode::And, so_far, also, Flags::NONE);
677                made.push(both);
678                both
679            }
680            // Every check has to fit for an iteration to be one the fast half may run, so the number
681            // of iterations it may run is the smallest of what they allow.
682            (None, false) => reachable(&mut build, &mut made, sweep, plan.around),
683            (Some(so_far), false) => {
684                let allows = reachable(&mut build, &mut made, sweep, plan.around);
685                let smaller = build.icmp(IntPred::Slt, allows, so_far);
686                made.push(smaller);
687                let least = build.select(smaller, allows, so_far);
688                made.push(least);
689                least
690            }
691        });
692    }
693    let settled = settled.expect("a plan holds at least one check");
694
695    for value in made {
696        let inst = inst_of(func, value);
697        func.remove_inst(inst);
698        func.insert_before(inst, term);
699    }
700    if still { Choice::Once(settled) } else { Choice::Counted(settled, start) }
701}
702
703/// Whether one check on an address that does not move fits, which is the same answer every time.
704///
705/// No iterations, because how far the runtime is asked to look is how many bytes the loop reads from
706/// this address on, and an address that does not move reads the same bytes however many times the
707/// loop goes round. So the count the loop was going to run does not come into it.
708fn fits(build: &mut Builder<'_>, made: &mut Vec<Value>, sweep: &Sweep) -> Value {
709    let (left, zero) = spare(build, made, sweep, Around::Number(0));
710    let fits = build.icmp(IntPred::Sge, left, zero);
711    made.push(fits);
712    fits
713}
714
715/// How many iterations one check allows, which is `(extent - reach) / step + 1` and never negative.
716///
717/// The subtraction and the addition carry `nsw` and the division does not need it. Everything here
718/// is worked out from the extent, which the runtime answers with a count of bytes it walked and so
719/// is never negative and never larger than the object, so none of this can leave sixty four bits
720/// whatever the limit it was asked for turned out to be.
721///
722/// Only ever asked about a check whose address moves. One that does not goes through [`fits`], which
723/// asks the shorter question and gets a yes or a no rather than a count.
724fn reachable(
725    build: &mut Builder<'_>,
726    made: &mut Vec<Value>,
727    sweep: &Sweep,
728    around: Around,
729) -> Value {
730    let word = Type::int(64);
731    let (left, zero) = spare(build, made, sweep, around);
732    // Not even one access fits, which is a dangling pointer or an object smaller than the thing being
733    // read out of it. The fast half runs no iterations and the check in the slow half reports it, at
734    // the access rather than at the loop.
735    let short = build.icmp(IntPred::Slt, left, zero);
736    made.push(short);
737
738    let mut steps = left;
739    if sweep.step != 1 {
740        let by = build.iconst(word, sweep.step);
741        made.push(by);
742        steps = build.binary(Opcode::SDiv, left, by, Flags::NONE);
743        made.push(steps);
744    }
745    let one = build.iconst(word, 1);
746    made.push(one);
747    let allows = build.binary(Opcode::Add, steps, one, Flags::NSW);
748    made.push(allows);
749    let clamped = build.select(short, zero, allows);
750    made.push(clamped);
751    clamped
752}
753
754/// How many bytes past the first access belong to whatever owns it, and a zero to compare that with.
755///
756/// The question both callers rest on. `extent - reach` is negative when the first access does not
757/// fit at all, zero when exactly one fits, and how much room there is for further ones otherwise.
758fn spare(
759    build: &mut Builder<'_>,
760    made: &mut Vec<Value>,
761    sweep: &Sweep,
762    around: Around,
763) -> (Value, Value) {
764    let word = Type::int(64);
765    let first = if sweep.offset == 0 {
766        sweep.base
767    } else {
768        let by = build.iconst(word, sweep.offset);
769        made.push(by);
770        let args = build.func().push_values(&[sweep.base, by]);
771        let sum = build.value(InstData { args, ..InstData::new(Opcode::PtrAdd) }, Type::PTR);
772        made.push(sum);
773        sum
774    };
775    // How many bytes the loop was going to read, which is how far the runtime is asked to look and
776    // nothing more. An answer short of the truth costs iterations in the slow half and is never
777    // wrong, so a count that saturates rather than one that refuses is the right thing here.
778    let want = match around {
779        Around::Number(times) => {
780            let far = times.saturating_mul(sweep.step).saturating_add(sweep.reach);
781            let far = i64::try_from(far).unwrap_or(i64::MAX);
782            let bytes = build.iconst(word, i128::from(far));
783            made.push(bytes);
784            bytes
785        }
786        Around::Computed(count, reading) => {
787            covered(build, made, count, sweep.step, sweep.reach, reading, Flags::NONE)
788        }
789    };
790
791    let args = build.func().push_values(&[first]);
792    let capability = build.value(InstData { args, ..InstData::new(Opcode::CapOf) }, Type::CAP);
793    made.push(capability);
794    let args = build.func().push_values(&[capability, first, want]);
795    let extent = build.value(InstData { args, ..InstData::new(Opcode::CapExtent) }, word);
796    made.push(extent);
797
798    let reach = build.iconst(word, sweep.reach);
799    made.push(reach);
800    let left = build.binary(Opcode::Sub, extent, reach, Flags::NSW);
801    made.push(left);
802    let zero = build.iconst(word, 0);
803    made.push(zero);
804    (left, zero)
805}
806
807#[cfg(test)]
808mod tests {
809    use rucc_base::Interner;
810    use rucc_ir::{
811        Block, Builder, Extra, Flags, Func, Inst, InstData, IntPred, MemInfo, MemOrder, Module,
812        Opcode, Restrict, Signature, Type, Value, verify_func,
813    };
814    use rucc_target::{TargetInfo, Triple};
815
816    use super::{SPLIT, Split};
817    use crate::canon::Canon;
818    use crate::stats::Kind;
819    use crate::{Fuel, Pass, Stats};
820
821    /// How many times the loop goes round, and how wide each element of the walk is.
822    const TRIPS: i128 = 16;
823    const WIDTH: i128 = 4;
824
825    /// A counted loop that reads one element each time round and can stop on what it read.
826    ///
827    /// ```text
828    /// entry(a): jump head(0)
829    /// head(i):  p = a + i*4; check_bounds cap_of(p), p; v = load p
830    ///           br v == 0 -> done, more
831    /// more:     next = i + 1; br next < 16 -> head(next), done
832    /// done:     ret
833    /// ```
834    ///
835    /// The second way out is the point. Hoisting refuses this loop, because a loop that can stop in
836    /// the middle reads fewer bytes than its count says and one check in front of it for all of them
837    /// would refuse a program that was right. Splitting does not care, because the count it reads is
838    /// only ever an upper limit on how far to look.
839    fn leaving() -> (Interner, Func, Vec<Block>) {
840        walking(Some(TRIPS), Flags::NSW)
841    }
842
843    /// The same loop, with how many times it goes round handed in rather than written down.
844    ///
845    /// What this reaches is the other half of [`crate::trip::covered`], the one that builds the
846    /// count out of something the loop does not change. It is worth its own test because that
847    /// arithmetic promises not to wrap for hoisting and promises nothing for this pass, and the two
848    /// callers now ask for different things from the same code.
849    fn counting() -> (Interner, Func, Vec<Block>) {
850        walking(None, Flags::NSW)
851    }
852
853    /// The same loop again, with an increment that promises nothing, so nobody counts it.
854    ///
855    /// What `-fwrapv` produces, and the shape a great deal of real code is in. Hoisting refuses it,
856    /// because a count that rests on the counter not wrapping is not a count it may size a check
857    /// with. This pass does not size anything with it, so it guesses.
858    fn uncounted() -> (Interner, Func, Vec<Block>) {
859        walking(Some(TRIPS), Flags::NONE)
860    }
861
862    /// Builds the loop, with the exit test against a number or against a second parameter.
863    fn walking(times: Option<i128>, flags: Flags) -> (Interner, Func, Vec<Block>) {
864        let mut names = Interner::new();
865        let mut params = vec![Type::PTR];
866        params.extend(times.is_none().then_some(Type::int(64)));
867        let mut func = Func::new(names.intern("f"), Signature::new().with_params(&params));
868        let entry = func.create_block();
869        let head = func.create_block();
870        let more = func.create_block();
871        let done = func.create_block();
872        let array = func.append_param(entry, Type::PTR);
873        let handed = times.is_none().then(|| func.append_param(entry, Type::int(64)));
874        let counter = func.append_param(head, Type::int(64));
875
876        let zero = Builder::new(&mut func, entry).iconst(Type::int(64), 0);
877        Builder::new(&mut func, entry).jump(head, &[zero]);
878
879        let mut build = Builder::new(&mut func, head);
880        let by = build.iconst(Type::int(64), WIDTH);
881        let scaled = build.binary(Opcode::Mul, counter, by, Flags::NSW);
882        let args = build.func().push_values(&[array, scaled]);
883        let pointer = build.value(InstData { args, ..InstData::new(Opcode::PtrAdd) }, Type::PTR);
884        check(&mut build, pointer);
885        let read = build.load(Type::int(32), pointer, mem(), Flags::NONE);
886        let nothing = build.iconst(Type::int(32), 0);
887        let stop = build.icmp(IntPred::Eq, read, nothing);
888        build.br_if(stop, done, &[], more, &[]);
889
890        let mut build = Builder::new(&mut func, more);
891        let one = build.iconst(Type::int(64), 1);
892        let next = build.binary(Opcode::Add, counter, one, flags);
893        let limit = match (times, handed) {
894            (Some(times), _) => build.iconst(Type::int(64), times),
895            (None, handed) => handed.expect("a loop with no number for a limit was handed one"),
896        };
897        let again = build.icmp(IntPred::Slt, next, limit);
898        build.br_if(again, head, &[next], done, &[]);
899        Builder::new(&mut func, done).ret(&[]);
900        (names, func, vec![entry, head, more, done])
901    }
902
903    /// What one access in the loop covers.
904    fn mem() -> MemInfo {
905        MemInfo {
906            size: WIDTH as u64,
907            align: WIDTH as u32,
908            order: MemOrder::NotAtomic,
909            tbaa: None,
910            restrict: Restrict::NONE,
911        }
912    }
913
914    /// Puts `cap_of` and a `check_bounds` at `pointer` into a block.
915    ///
916    /// The shape `rucc-safety` emits, written out here rather than reached for, because `rucc-opt`
917    /// is rank 9 alongside `rucc-safety` and cannot depend on it.
918    fn check(build: &mut Builder<'_>, pointer: Value) {
919        let args = build.func().push_values(&[pointer]);
920        let capability = build.value(InstData { args, ..InstData::new(Opcode::CapOf) }, Type::CAP);
921        let args = build.func().push_values(&[capability, pointer]);
922        let extra = Extra::Mem(build.func().add_mem(mem()));
923        build.inst(InstData { args, extra, ..InstData::new(Opcode::CheckBounds) }, &[]);
924    }
925
926    /// Canonicalizes and then splits, with as much fuel as both want.
927    ///
928    /// Both, because the pass is written against the shape [`Canon`] leaves, and it is
929    /// canonicalization that gives the loop the preheader the limit is worked out in.
930    fn split_up(func: &mut Func) -> Stats {
931        let mut an = crate::machine::fixtures::analyses();
932        Canon.run(func, &mut an, &mut Fuel::unlimited());
933        Split.run(func, &mut an, &mut Fuel::unlimited())
934    }
935
936    #[test]
937    fn a_loop_whose_result_is_read_after_it_is_put_back_into_closed_form_first() {
938        // Canonicalization runs a long way in front of this pass and `simplify-cfg` between the two
939        // undoes some of what it did, which is why the loop here is canonicalized and then broken.
940        // Both halves would define the value the code after the loop reads, so the pass repairs the
941        // one loop it is about to copy rather than refusing it or running canonicalization again.
942        let (mut names, mut func, blocks) = leaving();
943        let mut an = crate::machine::fixtures::analyses();
944        Canon.run(&mut func, &mut an, &mut Fuel::unlimited());
945
946        let (head, done) = (blocks[1], blocks[3]);
947        let read = func
948            .insts(head)
949            .find(|&inst| func[inst].opcode == Opcode::Load)
950            .and_then(|inst| func[inst].results().next())
951            .expect("the loop loads what it walks over");
952        let term = func.terminator(done).expect("the block after the loop returns");
953        let sum = Builder::new(&mut func, done).binary(Opcode::Add, read, read, Flags::NONE);
954        let inst = super::inst_of(&func, sum);
955        func.remove_inst(inst);
956        func.insert_before(inst, term);
957        an.clear();
958
959        let stats = Split.run(&mut func, &mut an, &mut Fuel::unlimited());
960        assert_eq!(stats.count(Kind::Optimized, super::CLOSED_HERE), 1);
961        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
962        assert_eq!(func[done].params.len(), 1, "the block after the loop took the value in");
963        assert_eq!(all(&func, Opcode::CheckBounds).len(), 1, "the fast half lost its check");
964        sound(&func, &mut names);
965    }
966
967    /// Every instruction in the function with this opcode, and the block it is in.
968    fn all(func: &Func, opcode: Opcode) -> Vec<(Block, Inst)> {
969        func.blocks()
970            .flat_map(|block| func.insts(block).map(move |inst| (block, inst)).collect::<Vec<_>>())
971            .filter(|&(_, inst)| func[inst].opcode == opcode)
972            .collect()
973    }
974
975    /// Insists the function is one the rest of the compiler may believe.
976    ///
977    /// This is what the tests here rest on. The pass makes a second copy of a loop, gives a new
978    /// block parameters that stand for the old header's, and moves a preheader's worth of
979    /// arithmetic in front of a terminator that was already there, so whether every value is in
980    /// scope where it is read is not something reading the code settles.
981    fn sound(func: &Func, names: &mut Interner) {
982        let target = TargetInfo::new("x86_64-unknown-linux-gnu".parse::<Triple>().unwrap());
983        let module = Module::new(names.intern("t.c"), &target);
984        if let Err(errors) = verify_func(&module, func, names) {
985            panic!("{errors:#?}");
986        }
987    }
988
989    #[test]
990    fn a_loop_that_can_stop_early_is_split_even_though_hoisting_will_not_touch_it() {
991        // The census row this pass was written for. Of the checks SQLite still carries at -O2, the
992        // largest group by far is in loops with a second way out, which is exactly the loop here.
993        let (mut names, mut func, _) = leaving();
994        let mut an = crate::machine::fixtures::analyses();
995        Canon.run(&mut func, &mut an, &mut Fuel::unlimited());
996        let refused = crate::hoist::Hoist.run(&mut func, &mut an, &mut Fuel::unlimited());
997        assert!(!refused.changed(), "hoisting has nothing to say about this loop");
998
999        let stats = Split.run(&mut func, &mut an, &mut Fuel::unlimited());
1000        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1001        sound(&func, &mut names);
1002    }
1003
1004    #[test]
1005    fn the_half_the_loop_runs_first_has_no_check_in_it_and_the_other_one_keeps_it() {
1006        // One check went in and one check came out, and the one that came out is in the copy. That
1007        // is the whole transformation: the same work, with the checking half reached only once the
1008        // guard says the run of safe iterations is over.
1009        let (mut names, mut func, blocks) = leaving();
1010        let head = blocks[1];
1011        split_up(&mut func);
1012
1013        let left = all(&func, Opcode::CheckBounds);
1014        assert_eq!(left.len(), 1, "one check, and it is the one the slow half kept");
1015        assert_ne!(left[0].0, head, "and it is not in the block the loop started in");
1016        sound(&func, &mut names);
1017    }
1018
1019    #[test]
1020    fn how_far_the_runtime_is_asked_to_look_is_settled_in_front_of_the_loop() {
1021        // The one thing a compiler cannot work out here is how many bytes belong to the object, so
1022        // it is asked, once, before the loop starts. Once is what makes this worth doing: a query
1023        // per loop in place of a check per iteration.
1024        let (mut names, mut func, _) = leaving();
1025        split_up(&mut func);
1026
1027        let asked = all(&func, Opcode::CapExtent);
1028        assert_eq!(asked.len(), 1, "one question for the one check that was sized");
1029        let cfg = crate::Cfg::new(&func);
1030        let doms = crate::Dominators::new(&cfg);
1031        let loops = crate::Loops::new(&cfg, &doms);
1032        assert!(
1033            loops.all().all(|id| !loops.contains(id, asked[0].0)),
1034            "and it is outside the loop"
1035        );
1036        sound(&func, &mut names);
1037    }
1038
1039    #[test]
1040    fn a_loop_with_a_call_in_it_that_might_free_is_left_alone() {
1041        // The extent is asked once and believed for the whole of the fast half, so anything that
1042        // could hand the storage back in the middle makes the answer stale and the fast half has
1043        // nothing left in it to notice.
1044        let (_, mut func, _) = calling(Flags::NONE);
1045        let stats = split_up(&mut func);
1046        assert!(!stats.changed());
1047        assert_eq!(stats.count(Kind::Missed, super::A_CALL_INSIDE), 1);
1048    }
1049
1050    #[test]
1051    fn a_loop_with_a_call_in_it_that_cannot_free_is_split() {
1052        // Whether the storage can be handed back is a question about the callee, and `crate::nofree`
1053        // answers it before the pipeline starts. This is the largest row of the census by a long way,
1054        // and it is also the row where this pass and hoisting come apart the furthest: hoisting
1055        // refuses a call whatever it does, because it needs the loop to reach the end of what its
1056        // count says, and this never claims that.
1057        let (mut names, mut func, _) = calling(Flags::NOFREE);
1058        let stats = split_up(&mut func);
1059        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1060        assert_eq!(all(&func, Opcode::CheckBounds).len(), 1, "the fast half lost its check");
1061        assert_eq!(all(&func, Opcode::Call).len(), 2, "and both halves kept the call");
1062        sound(&func, &mut names);
1063    }
1064
1065    /// The loop with a call added to its latch, carrying whatever the caller says about it.
1066    fn calling(flags: Flags) -> (Interner, Func, Vec<Block>) {
1067        let (mut names, mut func, blocks) = leaving();
1068        let more = blocks[2];
1069        let term = func.terminator(more).expect("the latch branches");
1070        let callee = names.intern("somewhere");
1071        let signature = func.add_signature(Signature::new());
1072        let call = Builder::new(&mut func, more).call(callee, signature, &[]);
1073        func[call].flags |= flags;
1074        func.remove_inst(call);
1075        func.insert_before(call, term);
1076        (names, func, blocks)
1077    }
1078
1079    #[test]
1080    fn a_check_whose_address_does_not_move_is_taken_too() {
1081        // One check on the array itself, every time round, alongside the one that walks. Hoisting
1082        // would rather have the still one, but this loop has a second way out, so hoisting will not
1083        // touch it and the check is still here to be taken. A step of zero is what carries it: the
1084        // access fits on the first iteration or on none of them, so it puts no limit on the loop.
1085        let (mut names, mut func, _) = standing(false);
1086        let stats = split_up(&mut func);
1087        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1088        assert_eq!(all(&func, Opcode::CapExtent).len(), 2, "both addresses were sized in front");
1089        assert_eq!(
1090            all(&func, Opcode::CheckBounds).len(),
1091            2,
1092            "the fast half lost both checks and the slow half kept both"
1093        );
1094        sound(&func, &mut names);
1095    }
1096
1097    #[test]
1098    fn a_loop_where_nothing_moves_picks_its_half_once_and_counts_nothing() {
1099        // Half the loops this takes on SQLite are like this, and they need none of the machinery the
1100        // rest of them do. Which half runs is decided by the answer to a question asked in the
1101        // preheader, the answer does not change while the loop runs, so the way into the loop is
1102        // where the two halves are chosen between and there is no counter and no guard block.
1103        let (mut names, mut func, blocks) = standing(true);
1104        let stats = split_up(&mut func);
1105        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1106        assert_eq!(all(&func, Opcode::CapExtent).len(), 1);
1107        assert_eq!(all(&func, Opcode::CheckBounds).len(), 1, "the fast half lost its check");
1108
1109        let (entry, head) = (blocks[0], blocks[1]);
1110        let term = func.terminator(entry).expect("the preheader still ends in something");
1111        assert_eq!(func[term].opcode, Opcode::BrIf, "the way in is the choice");
1112        assert_eq!(func[head].params.len(), 1, "and the header took on no counter");
1113        sound(&func, &mut names);
1114    }
1115
1116    /// The loop with a check on the array itself added to its header, every time round.
1117    ///
1118    /// Hoisting would rather have that check, and it takes the ones in loops it is willing to touch.
1119    /// This loop has a second way out, so hoisting will not touch it and the check is still here.
1120    /// `alone` takes the walking check away, which leaves a loop where nothing moves at all.
1121    fn standing(alone: bool) -> (Interner, Func, Vec<Block>) {
1122        let (names, mut func, blocks) = leaving();
1123        let (entry, head) = (blocks[0], blocks[1]);
1124        let array = func[entry].params[0];
1125        let walking = all(&func, Opcode::CheckBounds);
1126        let term = func.terminator(head).expect("the header branches");
1127        let mut build = Builder::new(&mut func, head);
1128        check(&mut build, array);
1129        let made: Vec<Inst> = func.insts(head).skip_while(|&inst| inst != term).skip(1).collect();
1130        for inst in made {
1131            func.remove_inst(inst);
1132            func.insert_before(inst, term);
1133        }
1134        if alone {
1135            for (_, inst) in walking {
1136                func.remove_inst(inst);
1137            }
1138        }
1139        (names, func, blocks)
1140    }
1141
1142    #[test]
1143    fn a_loop_whose_count_is_an_expression_is_split_on_what_that_expression_says() {
1144        // How far to look is worked out in the preheader rather than written down, out of a value
1145        // the loop does not change. Nothing here promises the arithmetic stays inside sixty four
1146        // bits, and it does not have to: a limit that wrapped is still answered with a true count
1147        // of the bytes that belong to the object.
1148        let (mut names, mut func, _) = counting();
1149        let stats = split_up(&mut func);
1150        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1151        assert_eq!(all(&func, Opcode::CapExtent).len(), 1);
1152        assert_eq!(all(&func, Opcode::CheckBounds).len(), 1, "the fast half lost its check");
1153        sound(&func, &mut names);
1154    }
1155
1156    #[test]
1157    fn a_loop_nobody_counted_is_split_on_a_guess() {
1158        // The difference from hoisting in one test. Hoisting refuses this loop, because the count
1159        // is what it sizes the check it writes with and a count nobody settled is not one it may
1160        // write a check from. Nothing here rests on the count: it is spent on how far to ask the
1161        // runtime to look, and the runtime answers with a true count of the bytes that belong to the
1162        // object whatever it was asked for, so a guess is as safe as a proof and only less useful.
1163        let (mut names, mut func, _) = uncounted();
1164        let mut an = crate::machine::fixtures::analyses();
1165        Canon.run(&mut func, &mut an, &mut Fuel::unlimited());
1166        let refused = crate::hoist::Hoist.run(&mut func, &mut an, &mut Fuel::unlimited());
1167        assert!(!refused.changed(), "hoisting will not size a check from a count nobody settled");
1168
1169        let stats = Split.run(&mut func, &mut an, &mut Fuel::unlimited());
1170        assert_eq!(stats.count(Kind::Optimized, SPLIT), 1);
1171        assert_eq!(all(&func, Opcode::CheckBounds).len(), 1, "the fast half lost its check");
1172        sound(&func, &mut names);
1173    }
1174
1175    #[test]
1176    fn the_pass_stops_when_the_fuel_runs_out() {
1177        // What `-fopt-fuel` is for, and the reason every transformation here goes through the
1178        // counter rather than round it.
1179        let (_, mut func, _) = leaving();
1180        let mut an = crate::machine::fixtures::analyses();
1181        Canon.run(&mut func, &mut an, &mut Fuel::unlimited());
1182        let stats = Split.run(&mut func, &mut an, &mut Fuel::of(0));
1183        assert!(!stats.changed());
1184        assert_eq!(stats.count(Kind::Missed, super::NO_FUEL), 1);
1185    }
1186}