1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
//! Texlang variables API
//!
//! # Texlang variables API
//!
//! The TeX language includes typed variables.
//! There are a few possible types, which are [listed below](#tex-variable-types).
//! Texlang currently supports only a subset of these types,
//! but has been designed so that it will be easy to support the other types in the future.
//!
//! This documentation is targeted at users of Texlang
//! who want to add a new variable to their TeX engine,
//! or consume variables that have already been defined.
//!
//! ## Historical note on variables in the TeXBook
//!
//! This section is not needed to understand how to use Texlang's variables API
//! and can be freely skipped.
//!
//! The TeXBook talks a lot about the different variables that are available in
//! the original TeX engine, like `\year` and `\count`.
//! The impression one sometimes gets from the TeXBook is that
//! these variables are heterogeneous in terms of how they are handled by the interpreter.
//! For example, on page 271 when describing the grammar of the TeX language,
//! we find the following rule:
//!
//! ```text
//! <internal integer> -> <integer parameter> | <special integer> | \lastpenalty
//! <countdef token> | \count<8-bit number>
//! ...
//! ```
//!
//! This makes it seems that an "integer parameter"
//! behaves differently to a "special integer", or to a register accessed via `\count`,
//! even though all of these variables have the same concrete type (a 32 bit integer).
//! To the best of our current knowledge, this is not the case at all!
//! It appears that there is a uniform way to handle all variables of the same concrete type in TeX,
//! and this is what Texlang does.
//! The benefit of this approach is that it makes for a much simpler API,
//! both for adding new variables to a TeX engine or for
//! consuming variables.
//!
//! ## The design of the Texlang variables API
//!
//! The TeXBook makes one distinction among variables which is
//! important and actually reflective of the TeX grammar.
//! This is the distinction between singleton variables (like `\year`)
//! and array variables (like `\count N`, where N is the index of the variable in the registers array).
//! Both of these cases are handled in the same way in the variables API here.
//!
//! In Texlang, the control sequences `\year` and `\count` are not considered variables themselves.
//! This is because without reading the `N` after `\count`, we don't actually know which memory is being referred to.
//! Instead, `\year` and `\count` are _variable commands_ (of type [Command]).
//! A variable command is _resolved_ to obtain a variable (of type [Variable]).
//! A variable is an object that points to a specific piece of memory like an `i32` in the state.
//!
//! For singleton variables, resolving a command essentially does nothing.
//! The command itself has enough information to identify the memory being pointed at.
//!
//! For array variables, resolving a command involves determining the index of the variable within the array.
//! The index has type [Index].
//! The index is determined using the command's index resolver, which has enum type [IndexResolver].
//! There are a few different ways the index can be figured out, corresponding to different
//! variants in the enum type.
//!
//! The next sections in this document move out of theory land,
//! and demonstrate with code how to implement singleton and array variables.
//!
//! ## Implementing a singleton variable
//!
//! Variables require some state in which to store the associated value.
//! We assume that the [component pattern](crate::vm::HasComponent) is being used.
//! In this case, the variable command is associated with a state struct which will be
//! included in the VM state as a component.
//! The value of a variable is just a Rust field of the correct type in the component:
//! ```
//! pub struct MyComponent {
//! my_variable_value: i32
//! }
//! ```
//!
//! To make a Texlang variable out of this `i32` we need to provide an immutable getter
//! and a mutable getter.
//! These getters have the signature [RefFn] and [MutRefFn] respectively.
//! Both getters accept a reference to the state and an index, and return a reference to the variable.
//! (The getters also contain an index argument, which is ignored for singleton variables.)
//!
//! For the component and variable above, our getters look like this:
//!
//! ```
//! # pub struct MyComponent {
//! # my_variable_value: i32
//! # }
//! use texlang::vm::HasComponent;
//! use texlang::variable;
//!
//! fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
//! &state.component().my_variable_value
//! }
//! fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
//! &mut state.component_mut().my_variable_value
//! }
//! ```
//! Once we have the getters, we can create the TeX variable command:
//!
//! ```
//! # pub struct MyComponent {
//! # my_variable_value: i32
//! # }
//! # use texlang::vm::HasComponent;
//! # fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
//! # &state.component().my_variable_value
//! # }
//! # fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
//! # &mut state.component_mut().my_variable_value
//! # }
//! use texlang::variable;
//! use texlang::command;
//!
//! pub fn my_variable<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
//! return variable::Command::new_singleton(
//! getter,
//! mut_getter,
//! ).into()
//! }
//! ```
//!
//! The function [Command::new_singleton] creates a new variable command associated to a singleton variable.
//! We cast the variable command into a generic command using the `into` method.
//! This command can now be included in the VM's command map and the value can be accessed in TeX scripts!
//!
//! As usual with the component pattern, the code we write works for _any_ TeX engine
//! whose state contains our component.
//!
//! Finally, as a matter of style, consider implementing the getters inline as closures.
//! This makes the code a little more compact and readable.
//! With this style, the full code listing is as follows:
//!
//! ```
//! use texlang::vm::HasComponent;
//! use texlang::variable;
//! use texlang::command;
//!
//! pub struct MyComponent {
//! my_variable_value: i32
//! }
//!
//! pub fn my_variable<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
//! return variable::Command::new_singleton(
//! |state: &S, index: variable::Index| -> &i32 {
//! &state.component().my_variable_value
//! },
//! |state: &mut S, index: variable::Index| -> &mut i32 {
//! &mut state.component_mut().my_variable_value
//! },
//! ).into()
//! }
//! ```
//!
//!
//! ## Implementing an array variable
//!
//! The main difference between singleton and array variables is that we
//! need to use the index arguments that were ignored above.
//!
//! In this section we will implement an array variable with 10 entries.
//! In the component, we replace the `i32` with an array of `i32`s:
//! ```
//! pub struct MyComponent {
//! my_array_values: [i32; 10]
//! }
//! ```
//!
//! The getter functions use the provided index argument to determine the index to use for the array:
//! ```
//! # pub struct MyComponent {
//! # my_array_values: [i32; 10]
//! # }
//! # use texlang::vm::HasComponent;
//! # use texlang::variable;
//! fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
//! &state.component().my_array_values[index.0 as usize]
//! }
//! ```
//!
//! The above listing raises an important question: what if the array access is out of bounds?
//! The Rust code here will panic, and in Texlang this is the correct behavior.
//! Texlang always assumes that variable getters are infallible.
//! This is the same as assuming that an instantiated [Variable] type points to a valid piece of memory
//! and is not (say) dangling.
//!
//! Next, we construct the command.
//! Unlike the singleton command, this command will need to figure out the index of the variable.
//! As with `\count`, our command will do this by reading the index from the input token stream.
//! In the variables API, we implement this by providing the following type of function:
//!
//! ```
//! use texlang::*;
//! use texlang::traits::*;
//!
//! fn index<S: TexlangState>(token: token::Token, input: &mut vm::ExpandedStream<S>) -> command::Result<variable::Index> {
//! let index = usize::parse(input)?;
//! if index >= 10 {
//! // for simplicity we panic, but in real code we should return an error
//! panic!["out of bounds"]
//! }
//! return Ok(index.into())
//! }
//! ```
//!
//! Finally we create the command.
//! This is the same as the singleton case, except we pass the index function above as an index resolver
//! with the `Dynamic` variant:
//!
//! ```
//! # use texlang::*;
//! # fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
//! # panic![""]
//! # }
//! # fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
//! # panic![""]
//! # }
//! # fn index_resolver<S>(token: token::Token, input: &mut vm::ExpandedStream<S>) -> command::Result<variable::Index> {
//! # panic![""]
//! # }
//! # pub struct MyComponent {
//! # my_array_values: [i32; 10]
//! # }
//! # use texlang::vm::HasComponent;
//! pub fn my_array<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
//! return variable::Command::new_array(
//! getter,
//! mut_getter,
//! variable::IndexResolver::Dynamic(index_resolver),
//! ).into()
//! }
//! ```
//!
//! ## Implementing a `\countdef` type command
//!
//! In Knuth's TeX, the `\countdef` command is an execution command with the following semantics.
//! After executing the TeX code `\countdef \A 1`,
//! the control sequence `\A` will be a variable command pointing to the same
//! memory as `\count 1`.
//! One way of thinking about it is that `\A` aliases `\count 1`.
//!
//! Using the Texlang variables API it is possible to implement the analogous command
//! for the `\myArray` command implemented above.
//! The implementation is in 3 steps:
//!
//! 1. The target (e.g. `\A`) is read using [crate::parse::Command::parse].
//!
//! 1. The index (e.g. `1`) is read using [usize::parse], just like in the previous section.
//!
//! 1. A new variable command is then created and added to the commands map.
//! This command is created using [Command::new_array] just as above, except in the index
//! resolver we use the [IndexResolver::Static] variant with the index calculated in part 2.
//!
//! For a full example where this is all worked out, consult the implementation of `\countdef`
//! in the Texlang standard library.
//!
//! ## TeX variable types
//!
//! | Type | Rust type | Register accessor command | Texcraft status
//! |-----------|----------------|--------------------------|----------------
//! | Integer | [i32] | `\count` | Implemented
//! | Dimension | TBD | `\dimen` | Not implemented
//! | Glue | TBD | `\skip` | Not implemented
//! | Muglue | TBD | `\muskip` | Not implemented
//! | Box | TBD | `\box` and `\setbox` | Not implemented
//! | Category code | [CatCode](super::token::CatCode) | `\catcode` | Implemented
//! | Math code | TBD | `\mathcode` | Not implemented
//! | Delimiter code | TBD | `\delcode` | Not implemented
//! | Space factor code | TBD | `\sfcode` | Not implemented
//! | Token list | TBD, but presumably a [Vec] of [Tokens](super::token::Token) | `\toks` | Not implemented
//!
use crate error;
use crate OptionalEquals;
use crate*;
use crate vm;
use crate::;
use HashMap;
use Debug;
use ;
use groupingmap;
/// Function signature for a variable's immutable getter.
///
/// In Texcraft all [Variable]s are built from an immutable getter and a mutable getter.
/// This type alias just defines the signature of immutable getters.
/// The first argument is the state, and the second argument is the index of the variable.
pub type RefFn<S, T> = fn ;
/// Function signature for a variable's mutable getters.
///
/// In Texcraft all [Variable]s are built from an immutable getter and a mutable getter.
/// This type alias just defines the signature of mutable getters.
/// The first argument is the state, and the second argument is the index of the variable.
pub type MutRefFn<S, T> = fn ;
/// Index of a variable within an array.
;
/// Specification for how the index of an array variable is determined.
///
/// Obtaining a variable from a command involves determining the variable's [Index].
/// This index is ultimately passed into the variable's getters to get a reference or mutable reference
/// to the underlying Rust value.
/// A TeX variable command.
///
/// Variable commands are _resolved_ to obtain a [Variable].
///
/// A command consists of three parts.
///
/// The first two parts are an immutable getter (of type [RefFn])
/// and a mutable getter (of type [MutRefFn]).
/// Both of these getters accept a reference to the state (where the variable's value lives)
/// and an [Index].
/// The index can be used by the getters to return different values in memory.
/// For example, if the getters read from an array, the index may be just the index in the array.
/// This mechanism allows users of Texlang to write one pair of getters that can be used in many variables.
///
/// The third part of a command is an [IndexResolver] that is used to determine the aforementioned index
/// of a variable at runtime.
/// The process of resolving a command involves determining the [Index] and returning a [Variable] type,
/// which under the hood is just the two getters and this index.
/// Immutable reference to the value of a variable.
/// TeX variable of any type.
///
/// A variable uniquely identifies a Rust value in the state, like an `i32`.
/// Operations on this value (like reading or setting the value) can be done in two ways:
///
/// 1. (Easy, less flexible) Use the methods directly on this type like [Variable::value]
/// to read the value.
/// These methods are really ergonomic.
/// The problem with the value method specifically is that the result
/// is a reference which keeps the borrow of the state alive.
/// Thus, while holding onto the result of the value, you can't do anything this the
/// input stream like reading an argument.
/// This is especially a problem when you need to perform a different action depending on the concrete type of the variable.
///
/// 2. (Trickier, more flexible) Match on the type's enum variants to determine the
/// concrete type of the variable.
/// The [TypedVariable] value obtained in this way can be used to perform operations on the value.
/// The main benefit of this approach is that after matching on the type, you can still use the input
/// stream to do things because there is not borrow alive.
///
pub
pub ;
/// A TeX variable of a specific Rust type `T`.
;
/// Trait satisfied by all Rust types that can be used as TeX variables.
///
/// It exists to make the variables API more ergonomic.
/// For example, it is used to provide a uniform constructor [Command::new_array] for commands.
/// The trait cannot be implemented for new types.
/// This function is used to implement the [SupportedType::update_save_stack] method.
///
/// It's a bit janky to implement the logic as a free function, rather than, say, as
/// a default trait implementation.
/// The main problem is that the implementation calls into the save stack, which is an
/// internal VM data structure.
/// Any way that does this on the trait directly requires making some of the save stack
/// type public, because [SupportedType] is public.
/// Internal VM data structure used to implement TeX's grouping semantics.
pub
/// Internal VM data structure used to implement TeX's grouping semantics.
pub ;