Colin: Hey Raph.
Raph: Hi, Colin, how's it going?
Colin: It's going okay public I've had like a slow holiday, but you seem like you've been a little busy.
Raph: I have!
Raph: Of I've gotten my fingers into even more pies which is which is saying something if you've been doing any pie-counting of my projects.
Colin: So is this been mostly like your Spiro stuff I was seeing drift by, or?
Raph: Well it's interesting. I certainly got very excited about a spline idea which is a continuation of the spiral work that I did for my PhD but.
Raph: That was kind of December and then around the holidays is I switched it kind of slowly from spline to curves to just 2D graphics in general, and so I have this this really intense focus right now on making a 2D graphics abstraction which is something I blogged about you know back in October, and then there wasn't really a lot of follow up, and I'm kind of like "okay now. is the time. Let's do it."
Raph: And so for the last three weeks - I haven't counted like there was one day in there that I wrote thirteen hundred lines of code and I've just been cranking on that, and it's been it's been starting to take shape.
Colin: So why don't we back up a little bit. What is a two D. graphics abstraction.
Raph: So a it's a graphics API.
Raph: But the difference between just an API and an abstraction is to say that you can have multiple backends on this, so that you could be using your platform graphics. If you're on Windows you want to be using direct 2D because Direct2D is very performant very mature. And if you're on Linux today, you might be using Cairo. There might be a lot of different back ends that you want to use. So it's kind of like a language, it's kind of a way of speaking to do graphic that multiple different pieces of software can understand and interpret so you're back and can take the the graphics commands really.
Colin: And convert them into pixels.
Raph: And then if you're writing an app then this is the way that you're gonna express that so if you want to say I want to draw these rectangles, these shapes these text overlaid on top it centered cetera it's really a language or a way of expressing that so that then you can mix and match with you know with different packets.
Colin: Is there a lot of diversity between back-ends on different platforms so to me a sort of think of everything is being kind of like PDF or like.
Colin: Or a major something.
Raph: Yeah so the the answer is yes there is a huge amount of diversity of back ends that you might look that you might have on your platform so that will range anything from a pure software rendering through two things that use the GPU and a very primitive way of just kind of compositing things together but doing a lot of the computation on the CPU, to doing things where you're trying to get all of the rendering to happen on the GPU. And I think that's the way we're moving, because if you do the rendering on the GPU it's gonna be hugely more performant than doing it on the CPU. But it's not even that. The next thing is are you rendering pixels that are gonna be immediately shown on the screen are you recording into like a SVG that will be served on the web are you going to make a PDF out of it this concept of two D. graphics is something that can exist in a lot of different contexts and that's.
Colin: Also one of the goals.
Raph: Of this of this abstraction.
Colin: Okay.
Colin: I'm curious what extent like the diversity of approaches in various like rendering pipelines or come back and I guess the contents of this.
Colin: Conversation is just a product of like their play or like the point in history that they originated.
Raph: Yeah, I think a lot of it is you know that software enders are going to come from the era where you couldn't really expect the GPU to exist on your machine and then you know like are you looking at a small platform with a small screen to big modern computer those are you know those are gonna be different have different constraints but generally today you can expect even on a pocket phone you can you can expect a fairly powerful GPU and you really want to do as much of the rendering work there as possible. And our software hasn't caught up. If you're on a mac if you're doing CoreGraphics on a mac then that's still doing unfortunately a lot of the hearing on the US if you so you're leaving performance on the table.
Colin: So with this project, do you have a sense of like a specific set of use cases or like is this something I'd be able to use to create a desktop environment or is it limited more to like simple drying or like the SVG.
Raph: I think that's going to be an interesting thing to explore this is motivated mostly right now by trying to make GUI apps. But when when you look at the way that I'm breaking it down into individual crates - and I think we're gonna spend a lot more time talking about that - these can be used to address different parts of the problem so like say that you're making a chart do you want to serve on the web you might have software that just draws the lines in the axes on the labels of the chart into this context being the name of the library of course that I'm doing and then there could be a back and that converts that to SVG and then you serve that over the web to the browser and your web browser shows SVG in the screen so that's one context is very different than just writing
---
Raph: Then that's still doing unfortunately a lot of the hearing on the US if you so you're leaving performance on the table.
Colin: So with this project like you a sense of.
Colin: Do you have a sense of like a specific set of use cases or like is this something I'd be able to use to create a desktop environment or is it limited more to like simple drying or like the final things I guess Fiji.
Raph: I think that's going to be an interesting thing to explore this is motivated mostly right now by trying to.
Raph: Apps.
Colin: But.
Raph: When when you look at the way that I'm breaking it down into individual crates and I think we're gonna spend a lot more time talking about that.
Colin: That.
Raph: These can be used to address different parts of the problem so like say that you're making a chart do you want to serve on the web you might have software that just draws the lines in the axes on the labels of the chart into this context being the name of the library of course that I'm doing and then there could be a back and.
Colin: That.
Raph: Converts that to SEG and then you of that over the web to the browser and your web browser shows yes Fiji in the screen so that's one context is very different than just writing a do we act and I think as we get deeper into this like you know the different creates some of them are very focused and like solve one particular problem really really well like Curbelo which is for curves it's like if you're doing anything that involves sophisticated manipulation of curves than I think kurbo is a really good way to you know good good library bring in and then if you're looking at the integration between Piet and then druid which is the graphics tool kit, that is kind of the umbrella project for a lot of this you know that's going to be yeah you're writing a do we op and this is the engine that makes that do we run on.
Colin: Different.
Raph: Platforms and you know kind of is the glue between their we framework and the rendering pipeline.
Colin: Yeah I feel like I'm a little behind on the crate. So why don't we back up and walk down that sort of hierarchy.
Raph: Absolutely.
Colin: So we have we started out awhile ago I'm only paying tagential attention but the corner of my eye I see alerts go by and various chat groups that I am a member of but maybe not a participating member of and awhile ago you start this project druid which came out of an attempts to build a xi front-end for windows?
Raph: That's exactly right.
Raph: So druid started life as xi-win Anzani win was really an experiment of.
Raph: What would happen if you try to write.
Raph: A Zakai editor front end.
Raph: In windows natively using the windows capabilities of the platform.
Raph: Of course one of the capabilities of the windows platform is it has a very rich @D graphics subsystem call Direct2D and then that's attached to a very rich very capable text manipulation library called DirectWrite and so is like what if I just use these in his native way and kind of get the maximum capability out of the platform, and write in Rust. and I think if we want to back up even more.
Raph: There's this question like the rich.
Colin: All's I editor.
Raph: Idea is rust is a great language for doing a lot of things.
Colin: But you wouldn't want to rate you're going.
Raph: So let's make this kind of connection where you've got a JSON- RPC protocol so you've got a core which is doing kind of heavy lifting of text objects but it's not doing the whites not drawing the menus and accepting scroll wheel inputs and those kinds of things. And then you'll use whatever technology is the most appropriate to write your front end. Aad that is kind of the original xi architecture and xi-win is kind of questioning that.
Raph: It saying.
Colin: Well what if you wrote the editor.
Raph: Front end in rust so what if you actually built the gooey in rust on the problem is there is no we.
Colin: Rest framework.
Raph: That's ready for primetime yet there's a bunch that are in the works and I I absolutely.
Colin: Lake I think we're.
Raph: In a really interesting stage right now where there's conrod there's azul, there's OrbTk
Raph: There's things that are kind of more team oriented like ggez and I've left out two or three.
Colin: That.
Raph: I apologize to the authors of those crates, they're all very interesting directions and I kinda want to let a hundred flowers bloom and see which one of these turns out to be the most compelling. And then I believe that the approach I'm taking has a lot to offer and you know definitely you know be be talking about that but when I started I win none of these things were even kind of plausible if you had to you know kind of really do things from scratch yourself and given the capabilities of the platform where you have this two D. graphics library you have this text library that's really strong on that's really solid you know how far can you go how how well can you do?
Raph: And then the other thing is well what is the particular niche for a text editor these days and I think that having fast start up like really awesome performance is compelling that that's a niche that we don't.
Raph: Really have completely.
Colin: Filled.
Raph: At this point and so was I win was really an attempt to say you know if we go low level and you as the platform have this very light weight app, so you're not pulling in these huge heavy weight expensive RAM-hungry frameworks but you're saying all right let's pop up window and start drawing the text into it and accept the the mouse events and just applying those in this kind of most direct way.
Raph: And it was pretty it.
Colin: The.
Raph: Experiment was I mean obviously I didn't make a fully.
Colin: Functional text editor but it.
Raph: Sure popped up the window fast and you know kind of look good.
Colin: Did you do measurements on that and did you do have numbers?
Raph: Unfortunately so that's a good question and unfortunately I don't have anything remotely resembling hard numbers. Doing empirical evaluation is hard work.
Colin: Yeah you need like a high speed camera or?
Raph: Yeah.
Colin: I guess there's probably ways.
Colin: You can but yeah there are ways to do it but.
Raph: You know I was really just doing a qualitative thing of, does this does this feel super responsive is this.
Raph: You know is this is this is this fast you can also do things like what is the what is the size of the binary and.
Colin: Yeah I know these are all listening I'm thing a lot about and and I I'm looking Rust 2019 a little bit yeah and I keep on like considering and failing to write down my thoughts but one of things I've been really wanting to do and the one thing I may be interested in doing if we do xi in GSoC this year, in Google Summer of code is.
Colin: Tools for tracking aggressions a bunch EC across a bunch of metrics I want continuous integration tool that just tracks binary size.
Raph: Right absolutely.
Colin: And similar things as well for sets of benchmarks across the road running on like hardware pool.
Raph: Yeah.
Colin: Binary size for sure start up time for sure these are all important to measure.
Raph: They they absolutely are a ram ram consumption while the app is running there's a lot of metrics there actually you know correlate very much with with user happiness that you know you don't necessarily see unless you start looking at closely.
Colin: So to just sort of like, I'm going by memory, sometime around mid September October.
Colin: There was his eye one project even kinda like fiddling on in your evening yes and it was sort of like Raph's meditative activity for awhile yeah like through the summer but then sometime around October there's some new contribution and then.
Colin: Some part i xi-win became this new crate or project druid.
Raph: Absolutely so they're kind of couple things that happen there and one of them is.
Colin: That.
Raph: in xi-win I started exploring this state oriented you why design and there's this talk that I gave at the San Francisco Bay Area press meet up I forget when that was July August, some time a few months ago.
Colin: We we will find a link okay.
Raph: Great.
Colin: And.
Raph: And I want to explore this kind of data-oriented approach because when you do things in an object oriented approach which is the traditional way of building you know you go a tool kit that's what Smalltalk kind of was built on if you try and write object oriented code in rust it's pretty painful and you end up having to deal with interior mutability you have kind of all these problems and it's just not a very good experience. And so I wanted to try this data oriented approach. Wwe don't have to going to a lot of detail because there's there's the talk, but it's kind of using ideas from the gaming world and it turns those ideas translate really nicely interest and I wanted to explore doing that and I kind of started sketching out like how would you design a widget trait, and how would you design layout that works in this very native way. how do you plumb events from one to another so you can compose a larger system out of smaller says and it just felt like it was coming together in this almost magical way that you're expressing yourself in this very clear concise way of dealing with the problem of actually what GUI am I trying to build rather than fighting against the language fighting against the structure. And I think that like to this question earlier of do you want to build a rest is rust a suitable language that I'm kind of getting to the answer yes.
Colin: But, tou have to do it in a rest native way if you try to just write traditional patterns it's not going to go well and so that was kind of experiment and it was bearing fruit was looking really good.
Speaker 0: The.
Raph: Other thing that was happening around the same time.
Speaker 0: Is that I got.
Raph: Very excited about building a music synthesizer.
Speaker 0: And.
Raph: I wanted that to be a good week.
Speaker 0: And I actually.
Raph: My first plan for that was to build it on the web technology because.
Speaker 0: I wanted.
Raph: It to be deployed cross platform.
And I started, I have a prototype. If you go to the to the repo you'll see a little web based music player that has this.
Speaker 0: Rust.
Raph: Synthesizer engine which compiles allows them but the UI of it is built in web technology.
Speaker 0: And I was like yeah.
Raph: I can do this.
Speaker 0: And then I kind of thought well.
Raph: Which is going to be more fun writing it in rust, or writing it using web technology and that that that deliberation didn't take too long.
Raph: It's just like okay I have techno.
Speaker 0: G. that.
Raph: Came from as I wind that looks kind of exciting.
Speaker 0: And.
Raph: Has the ability to support what the goals of the synthesizers having visual feedback. I have this oscilloscope display in it and I have other ideas for visualisations involving spectrums and frequency responses.
Raph: And I really wanted there to be this very.
Speaker 0: Immediate feedback this kind of low latency approach which you know that.
Raph: Needs performance that needs.
Speaker 0: That need.
Raph: The kind of stuff that I was actually working on in the context of xi-win.
Raph: And so.
Speaker 0:
Raph: At that point is like.
Speaker 0: Okay.
Raph: Let's take this thing which is just xi-in, and let's.
Raph: Split it out into a crate.
Raph: Generically.
Speaker 0: A gooey tool kit and so that gives us drew and that gives us and then I I close my eyes for like a week yeah and now there is yet and there is a kurbo.
Raph: Yes. That's right so what's the relationship between these right now.
Raph: When druid happened, it had one very serious problem.
Raph: It was when the.
Raph: Because it was essentially like abstracting a lot of the problems of lake house and we did lay out and stuff like that but when it came time to draw the appearance of your widgets it's lake here's a two D. rendering here's a direct two D. rendering text draw your stuff is that using this.
Speaker 0: To the API.
Speaker 0: Okay so this is where like the abstraction becomes important you need to be able to say draw this into something that might be direct2D, or might be WebGL or yeah.
Raph: That's exactly right so the idea is what's keeping this windows on.
Speaker 0: Me.
Raph: And the main thing is.
Speaker 0: You thing a concrete.
Raph: Graphics API.
Speaker 0: Like direct two D..
Raph: Which for the most part only exists on windows. It is funny because you have these WINE, Proton projects that get that running on different platforms
Raph: Forms but you know then you're you're dealing with translation you're not you're not you're not doing in need of thing.
Speaker 0: And so.
Raph: Is that is an interesting question can you take those adapter technologies and say well this is a way to deploy but I looked at it and I said now I can do better than this because you don't when he's an adapter.
Speaker 0: And I think more importantly.
Raph: You want express these concepts in Everett.
Speaker 0: Rust need away.
Raph: Rather than saying all I have to build these com objects and have this manipulation of these com objects before I can get anything on the screen.
Speaker 0: Okay so we've got an abstraction and then we need some primitives what's the sort of what is the taxonomy of the project what components are you working on working ones do you need to work on.
Raph: Exactly so.
Raph: So the primitive that you need to be.
Speaker 0: Drawing are dealing.
Raph: In a TD abstraction.
Speaker 0: Are kind.
Raph: Of just as a rough enumeration you've got shapes you've got text you've got images.
Speaker 0: And then you've got.
Raph: Your stuff like gradients and clips and stuff like that that are kind of a graphical effects or.
Speaker 0: Or.
Raph: Things like that so each of those is interesting, obviously. Each of those is kind of a a world unto itself and so kurbo.
Raph: Is about shapes.
Speaker 0: And.
Speaker 0: there are many different kinds of shapes in the world and kind of the most interesting to deal with.
Speaker 0: Our curves.
Raph: And curves are very related to my PhD work so that kind of is the natural center of gravity for that crate.
Raph: But it turns out that curves in curved paths I'm not that's actually not most of what you.
Speaker 0:
Raph: Most of what you draw is rectangles are sometimes rounded rectangle.
Speaker 0: How you draw.
Raph: Some lines you you might draw some days has some curves and that's certainly an important partly You can't have a 2 graphics API that doesn't have you know pads as one of one of its primitive.
Speaker 0: But that's really kurbo. Kurbo is the representation and.
Speaker 0: The.
Raph: Computation of.
Speaker 0: Shapes.
Raph: That are the kind of shape that you would use in 2D graphics whether you're doing.
Speaker 0: List.
Raph: Isshin.
Speaker 0: Or.
Raph: Font design work just as the back end of it of you I in which case most of it is going to rectangles.
Speaker 0: Which.
Raph: Interest only enough is the.
Speaker 0: Naming.
Raph: Scheme of Piet it's named after the artist Puet Mondrian who's very well known for compositions with lots of rectangles.
Colin: So Piet is the library that I or the great that I know the least about.
Raph: right.
Colin: is this specifically about that it is a library for drying rectangle the.
Raph: No it's a library for representing you know pretty much any.
Speaker 0: Too deep.
Raph: Affects.
Raph: Like any image that you would drop.
Speaker 0: Using to your.
Raph: TV graphics.
Speaker 0: Like a vector image any vector image or is that is that fair characterization or is that missing some important detail?
Raph: How that's definitely one way to look at it I guess the thing is that when you think about an SVG, you know of a vector image.
Raph: You don't really tend to think of that as like a composition of a lot of different elements like a GUI.
Speaker 0: Agree but it certainly can be.
Raph: And I think that ultimately that that's what Piet is for and Piet has like the structure that lets you have a lot of different individual elements that get compose together in groups and with clipping and on and so forth so you know it's it's the kind of image that is like a what look out on the screen when you're looking at a GUI app.
Speaker 0: Agree out
Colin: So people can like include like some bitmap that is in some position.
Raph: So that is what the so Pete is probably halfway done to the first cut, and
Raph: Bitmaps are kind of later this week.
Speaker 0: Okay the but that's that's in the yeah it really is like a full two D. graphics whatever anything that's on the screen.
Speaker 0: It can represent yes.
Raph: Exactly that.
Speaker 0: Okay.
Raph: And it in it's important because you know we don't have bit maps in xi-editor yet.
Speaker 0: We might.
Raph: At some point.
Speaker 0: But we do.
Raph: We do have them in the synthesizer we have this oscilloscope so we need to be drawing that.
Speaker 0: That's on the.
Raph: Bill passed to lake kind of taking all this code which is now windows only and.
Raph: You know actually.
Raph: Getting it deployed cross platform.
Speaker 0: So what sort of stage things that now? what we Are there any like major missing components or is it more just like you have a skeleton and now you're slowly flushing it out.
Raph: I have a skeleton and rapidly pushing it out. [chuckle]
Raph: At least it leaves that's where things are right now.
Speaker 0: So.
Raph: Like the current state of Piet.
Speaker 0: Is that it draws shapes and text.
Raph: And there are P. R. in flight to do clip paths and affine transformations.
Speaker 0: And up.
Raph: You know as I say I expect to get in images and gradients.
Speaker 0: Fully.
Raph: Of one not to miss exactly when but that's kind of next you know in the in the path.
Speaker 0: And this is working with a back end, with a variety of back ends?
in so that.
Raph: The current Pete creates I've decided.
Speaker 0: That.
Raph: I am going to focus on three back ends for the original version.
Raph: Because I feel like having.
Speaker 0: I mean the difference between one and.
Raph: And more than one is huge but I have like three keeps the design honest if you can make three beckons happened then it's very unlikely.
Speaker 0: That you've.
Raph: Backed yourself into a corner and can't cannot more and those three back answer direct two D. which is the windows one, Cairo.
Raph: Which is cross platform but it's kind of most associated with Linux, and
Raph: Web canvas so one of the other thing is that it's kind of exciting and motivating.
Raph: Is this realization that a lot of the future of Russell lot of the excitement and potential of rest.
Speaker 0: Is you can write your resco compiler to wow awesome.
Raph: And deployed in a web browser and so a lot of kind of the thinking behind Pete is well web browser has a perfectly good - it has two perfectly good.
Raph: 2D graphics rendering back ends in fact. It has web canvas and it has SVG and ultimately I think that both back and you're gonna existing Pete.
Speaker 0: I'm.
Raph: Doing the canvas one first.
Speaker 0: So.
Raph: That is you know kind of proving out this concept.
Speaker 0: Of you know how portable can it be can you really do it.
Raph: These other on these other platforms for real. The funny thing is - this surprised me.
Speaker 0: That the actual programming work in getting it and getting it to work.
Raph: On the web canvas there's less difference in the code between the Cairo back end and the web canvas then there is between either one of those in the windows one.
Raph: And that so there's a there's a catch phrase that I think I'm going to be using to pitch this which is that Pete is to.
---
Raph: Then there is between either one of those in the windows one.
Raph: And that so there's a there's a catch phrase that I think I'm gonna be using to pitch this which is that Pete is 2D. graphics.
Colin: As.
Raph: serde is to serialization.
Colin: Do you want to elaborate on that?
Colin: Yeah.
Raph: So first of all it's a very conscious aspect to the way the traits are designed if you look at day you know as people think of Saturday as this just pretty amazing J. son poser but they're a lot of JSON parsers in the world and that's not what makes it special. What makes certain a special is that has these treats that.
Colin: Are.
Raph: Really direct inspirations for the kind of thing I'm trying to eat so you've got a serialized trait serialize or you've got a a bunch of traits and the interact in this way that you just you don't have to do a lot of work there's very little busy work is in a meeting like small amount of busy work the just implement a couple of trades on whatever it is you're trying to do and then magically it serializes it to JSON.
Raph: And then if you want to switch from JSON to MsgPack then you just switch whichever back end you're using and boom you're thing is serialized to MsgPack, and you didn't have to do any work.
Colin: Okay as is mostly kind of architecture.
Raph: So I think it's an aspiration rate that.
Colin: Lake.
Raph: This is what I would like to be in the rest ecosystem globally that.
Colin: Are.
Raph: You know and if you look kind of at that at that.
Colin: Now G..
Raph: Sir day if you wanna do serialization certi is the tool you reach for it doesn't do.
Colin: Everything.
Raph: Like if you're trying to do proto buffs then Sir day might not be the best tool you might want to do a more specific serialization for just ProtoBufs but if you just you know how this problem you know I have this thing and I want to write it to describe one of send it over network connection then you say all right how many user day.
Colin: And that'll.
Raph: That'll do it for me.
Colin: You know has.
Raph: Kind of flexibility and then if you're writing a new serializer, if you're writing a new format you're like all I wanna make something that's more binary efficient then obviously you write a serde back-end for it because if you do that then you have huge amounts of software that just works, and so that's exactly what I'm thinking of.
Colin: With.
Raph: With Pete.
Colin: That you know if you say.
Raph: I just have this graphics and it's vector graphics and I want either draw it.
Colin: Or you know.
Raph: Make it into an SVG or do something with it then use reach for piet and piet is the way that you express at that then.
Raph: Is flexible unless you do all those other things with it.
Raph: And then similarly if you're saying I have a great idea for a new.
Colin: Rendering.
Raph: Back and then I want to use compute on Vulkan.
Colin: Right.
Raph: So that I use deep modern use in the most efficient way than well what API do I throw on that and the answer is you know what I'm hoping is you'll say.
Colin: I'll have a P. A. P. I. and then if you've got all the software.
Raph: That is actually written using Pete and whether that's druid or not this is one of the real reasons why these are separate crates.
Colin: Then.
Raph: All of a sudden your fancy new Vulkan compute renderer would just support that software.
Colin: Compute render will.
Raph: Will just.
Colin: Work that suffer okay and so things like accessibility would live in druid exactly because that's like a B. Y..
Colin: Problem that's right where as Pete can deal speak poorly with trying obstructions that that's very much so.
Colin: So, Raph.
Colin: How does text work?
Raph: Well text is one of the big.
Colin: Obviously as you know that text is a very hard problem and.
Raph: You know and also one of the areas unfortunately.
Colin: In the.
Raph: Rest.
Colin: Eco system where the existing our existing libraries existing infrastructure is pretty thin.
Raph: Is pretty weak there are things they do text but there's no kind of end to end solution.
Colin: And if you.
Raph: If you say you know here's some text in here some attributes I want this word to be italic.
Raph: Put it into something that I can use.
Colin: That that doesn't really exist in the rest to go system.
Raph: So when you talk about the traits in Piet.
Colin: Example.
Raph: Shape is one of the fundamental.
Colin: Motives for you wanna draw and.
Raph: So in the case of shapes that's a separate create that's kurbo because there's a lot of times you want to deal with shapes that are not necessarily.
Colin: Drawing them into the.
Raph: Our but that's kind of the answer if you have a shape you're representing carbo, it implements the shape trait anything that implements the Shape trait.
Raph: In kurbo can draw.
Raph: Your get.
Raph: Text I think is probably going to be in the peak crate because I think text into the graphics are more closely tied.
Colin: Although.
Raph: That is a question like that you know like we can they would not be surprising if this iterates.
Raph: Like you see the distinction for example Direct2D and DirectWrite are two different packages and even though there.
Colin: There is.
Raph: Methods like when you say draw text layout.
Colin: In direct two D..
Raph: You know that takes a direct right you actually can run direct right without a TV.
Colin: Or DirectTV context so.
Raph: This is this is still still to be worked out but somewhere in this ecosystem there's gonna be a text trait and there is a very primitive simple text trait in
Colin: In.
Raph: This actually a couple others font and text.
Colin: A TextLayout that are.
Raph: There in Pete.
Colin: And.
Raph: The idea is that seem kind of trait architecture that you're not tying yourself to a specific text layout, specific text rendering back and and in the case of windows I think.
Colin: I'm.
Raph: I'm really as it builds out like right now the you can measure the width of texting you can specify the font and size so actually that's enough to get started.
Colin: Building agree and that's all direct right or is that using something like like Cairo panga so on windows.
Raph: So the there's if that's a back independent thing so this is you know this is why it's kind of a P. trait because there's this piet RenderContext trait which which abstracts the entire.
Colin: Rendering back and.
Raph: And then there's these associated types.
Colin: In there which are.
Raph: Also treats.
Raph: Some kind of really getting deep into this like rust.
Colin: You know how do you design you know.
Raph: Types.
Colin: In.
Raph: Complex systems of types and rest.
Raph: And so some of those are text types and so when you have a piet back and.
Raph: Your text types are socially the tapes and so then.
Colin: You you.
Raph: You basically just ask you know like creative.
Colin: Onto you know with this name.
Raph: Created layout with this text in these parameters.
Colin: So you have like a font manager traits that will be implemented by, like on windows you actually an imitation of that was DirectWrite under the under the covers yeah and on mac OS you would have an invitation of this that's CoreText under the covers and that does like you give it a name for a font it resolves that to an actual phone yeah.
Raph: So just the simple like if you're only talking about the really simple thing where you you resolve unnamed font that exists now I mean there's gonna be more stuff because you want to be able to set because I'm finding this you know this is a huge task.
Raph: But.
Raph: You know I'm just getting started with the basics and not exist right now and exactly that on windows.
Colin: Its direct right.
Raph: On Cairo right now it's using what's called the Cairo toy text API, which does not do shaping, but that's good meat that's like one of the first things that needs to get developed more carefully and then on what on the web you have you know it's really CSS.
Colin: Okay and is there is it does that seem to you like one of the major points of work or is it going to be easy enough to adapt existing work.
Raph: Yes it's gonna be major work.
Colin: And.
Raph: One of the ways in which I'm a little bit confident is that I think that we're going to be using DirectWrite as the kind of main guidance for how this thing is structured how applications access it you know how the API is presented because direct.
Colin: Right.
Raph: Is kind of the most recent spin on this that's really principled and really trying to be general and really trying to be sophisticated.
Colin: How does that sophistication manifest? sort of other any like examples of the way direct right approach is the problem compared to other API's that might be illuminative?
Raph: Absolutely so one of the biggest problems that you have.
Colin: When you're dealing with text in agree.
Raph: Is measurement.
Colin: And.
Raph: I know that you've been running into that.
Colin: And.
Raph: You need to know.
Colin: How wide a particular string of text is but you also need to know more fine grained stuff like if you're putting your cursor in the middle of the word.
Colin: You know where do you put that.
Raph: Sir if you click in the middle of the word, what string offset does that cursor click correspond to.
Colin: Slug is even resolving a grapheme cluster and then finding like visual offset of that cluster of any given contacts actually yeah and that might be different depending on the subsequent graphemes.
Raph: Yes.
Colin: Or preceding graphemes
Raph: That's exactly correct because this this depends.
Colin: Are.
Raph: You know it depends on the script that you've got that if you're doing English it's usually fairly simple although there are slight complications like "fi" is often.
Raph: That's two graphemes but it's represented by one cliff if you've got a ligature in your font so you you get into it a little bit but then when you get into scripts like Devanagari for Hindi or Arabic script or any of these complex scripts than those combinations things can get reordered thing.
Colin: Can get.
Raph: Positioned.
Colin: You've got.
Raph: Accent marks the get position.
Colin: Over and under the line.
Raph: So these questions actually get really complicated so you know there are a lot of different APIC do that their score tax there's pain go which is kind of a.
Colin: The.
Raph: Sushi with Cairo, a Linux thing and these all work I don't.
Colin: Know.
Raph: Take away from any of them but if you look at direct right it's kind of a very complete solution and it's also presented in a way that's pretty easy for programmer some of these things have a very low level feel where you have to connect you to iterate through your graphing classed as you know one by one individually and direct right I think does a really nice job of set.
Colin: Of kind of you know.
Raph: Like here's my what do I do now.
Colin: When.
Raph: You know like you're some tax that has attributes to different places you know how to express that.
Colin: You know it in a kind of a clean way I'm curious about undertaking a Pango of the other day.
---
Raph: Like here's my what do I do now when you know like you're some tax that has attributes to different places you know how to express that you know it in a kind of a clean way.
Colin: I'm curious about other digging into paying the other day I was playing around with measure and stuff and one thing I noticed is that it looks like it sort of itself contains a similar approach what you're talking about and yet we're like you can basically you can swap and like font information providers, I'm not sure what the terminology is.
Colin: And I wonder if they use DirectWrite on windows or.
Raph: I believe they do I'm not a hundred percent positive that I I I think that I mean this is it.
Colin: Of.
Raph: A future of Cairo that it has some of that nature of being an abstraction and a lot of the nature of being its own you know rendering pipeline and kind of finding more specifically so it's sort of happy you can think of it as sort of halfway in the middle.
Colin: Okay and so if I'm rendering I'm rendering a page of text using piet on mac OS I'm gonna get a visual artifact it'd be the same as I would get if I were rendering through CoreText?
Raph: So.
Colin: Okay.
Raph: So.
Raph: So I think that that's a really interesting question and.
Colin: What your what they're.
Raph: Kind of underlying question is is it you've got to check the back end.
Colin: Like you've got this piet abstraction and piet abstraction as okay this how you express it but it doesn't nail down what back and to actually use to draw this for the user.
Raph: And that choice has a lot of different trade off and this is one of the reasons why I think it needs to be an abstraction.
Raph: If your goal is that it needs look native then you better use the native stuff.
Raph: And there are people that try, there's a lot of things out there.
Raph: That are like we'll do our own pipeline and when what comes out of the and its pipeline will end up looking just like the need of stuff. And some of it is not terrible but it's always that's a very tricky place to be it's very difficult to really really look exactly like that than the native stuff.
Raph: So if you're using a piet backend that uses core text and uses.
Colin: You know.
Raph: Actually you can separately can look at xi-mac, and xi-mac uses CoreText but it does not use core graphics to paint that text because its own OpenGL he based render so that that that's a mix and match if you're if you want your stuff to look native on mac then you better be using.
Colin: Core text to do the text.
Raph: Formatting.
Raph: And then you know the other side of the street off his performance and.
Colin: You know.
Raph: You may find that if you use like for example there's.
Colin: Really interesting work.
Raph: Such as pathfinder from Patrick Walton, and pathfinder is doing the font rendering on the GPU, and as such it is a potential to be way faster than doing the font rendering on the CPU this is no choice you might say well I can get a lot higher performance in if you're on a high DP I screamed and matching the exact subpixel stuff might not be as necessary so I think that the goal of the project is to enable all of the choices and then you're gonna have to make those choices you know if you're on a mac you're gonna have to say do I want to buy in fully to CoreGraphics, CoreText, or do I want to take pieces of those and I think this is also something is going to evolve over time because like today those those pieces either don't exist or are not very mature.
Colin: But I.
Raph: Have this kind of vision and then you know I think it's gonna happen that you're going to see.
Raph: Sophisticated text layout you're going to see sophisticated GPU based rendering that is going to be way faster than the typical 2D rendering pipeline you see today.
Raph: This is also I think one of the motivating questions about this entire project is like well is this just a retread of existing GUI graphics ideas and what I'm seeing when I go deeper into it is, no, we actually have to rethink some of the stuff we have to think about it in terms of what works well on the GPU.
Colin: What happens when you're programming this stuff rust native what happens when you're designing these traits so that you can iterate through a shape without havigng toallocate anything. Things are different! If a few years down the line we have these things and we have this kind of GPU rust need of rent.
Colin: That's lake.
Raph: You know I'm not gonna say how many times faster but like you know really noticeably faster.
Raph: Then that's gonna be compelling and you might make a different trade off and say well it doesn't have to look pixel for pixel.
Colin: But the role.
Raph: Is to you know make these things possible.
Colin: So that we have kind of like a loose story for text.
Colin: Is there any other like is that what's on deck right now or do you have another sense of what your immediate next steps are?
Raph: So so I think that there are really only a few major categories of primitive that need to be in like the first cut of P. none I see you know some some refinement and like a really good list to work from is what do you need to render SVG and.
Colin: Author of resvg has a list out there - we'll link to this in the show notes and I'm kind of going from that and there's not a whole lot more when you talk about shapes images.
Colin: Text.
Raph: And a couple of other things like gradients and blend modes that's pretty much it in terms of what do you need to render SVG and so that's kind of the goal that I feel like I'm racing towards is you know how do we get this into shape where you can express most of SVG, not every single SVG but most of SVG with a speech and then I'm really hoping to work with the author of resvg of getting those those things bounce the you you can start using Mr to render your SP Jeez that's kind of an immediate goal that I think is is is coming up pretty fast.
Raph:
Raph: V. E. R. there's other piece of work lake the.
Colin: You know.
Raph: Druid of course today's using direct two D. so there's this kind of very clean thing to do of take the direct to dis dependencies out and replace them with piet dependencies and kind of mostly have your existing out a lot of suffer written but there's the xi-win prototype is the synthesizer but it's enough that you know.
Raph: That that certainly needs to get done and that'll also.
Raph: The date like if we get these things converted over then that gives you a good confidence that that this thing is going to work for at least simple simple GUI's. Now obviously text is big thing and you know when we go a little bit further out then we have the situation where the direct to DVD the windows back and is gonna have a pretty sophisticated pretty fully realized text story and the other for backends much less so.
Raph: But I'm pretty confident here because I feel like it doesn't all have to be me and I think one of the things that's really interesting is the names in a lot of collaboration come forward just in the last week.
Raph: And I'm really gratified to see that you know I love working with collaborators and I want to make this a project that's collaborator friendly and that can welcome people and say okay look they're all these little things that need to be done, and big things. And one of those bigger things is like getting the text story on non windows platforms kind of up to the the standard that window sets. And then, just in general.
Colin: There.
Raph: Are probably like we're starting with three back ends in and at least three others you know have popped up you know made themselves visible so one that does not currently exist is core graphics mac native. another one that popped up is Qt Qt has a campus and the currently the back end of choice of resvg and of our so that one makes a lot of sense to wire up. another one that makes a lot of sense to wire up his web under our the combination of Webber.
Colin:
Raph: And pathfinder.
Raph: Of course a hugely interesting rest native you know render GPU based rendering pipeline I think that'll be really interesting to wire up. And then there there's like three or four others that I'm thinking of that I'm not going in right now but when you have that structure when you have that trait and you you know it's once is kind of proved out, then if someone says I have this idea I want to make a new back and hopefully that meet that makes it a very clear path of you do you just say these are the methods of the trade you have to implement and then you also have a basis for comparing like do you think your back and is going to be faster? Do you think it's going to use GPU more effectively? Well, bench mark it, you know.
Colin: Yeah.
Raph: Yeah.
Colin: Do you have a like a ballpark how how many lines of code is involved in friends the direct.
Raph: To DVD back and right now.
Colin: To get an idea of like what the task looks like.
Raph: So it's not a huge amount of code I mean obviously I can line count it but it's under a thousand I think.
Colin: So like a a vigorous weeks work?
Raph: yeah.
Raph: Much and you know it's gonna be as I say that shapes and text and it's going to be more than that but it's not gonna be an order of magnitude more than that yeah anytime soon.
Colin: Okay, I feel like that's a pretty good overview for me. I feel like I'm slightly more caught up in what you're doing here if people are interested in following along are there any particular resources you would recommend?
Raph: Sure so I I think the main thing to do is we have is a Zulip chat and that's really where most of the discussion happens and that's certainly true for site and that's also what I'm using for the druid and Pete work and so there's a #druid stream which is where most of the stuff is being discussed, All you need to sign up is a Github account and people are definitely invited even just lurk but a certainly a happy to answer questions and it feels like a community is forming around this stack, kind of rapidly.
Colin: Battling but that's at design XII dot suit chat dot com yes.
Colin: All right and.
Colin: I'll see you on there later today and probably tomorrow and probably they after that.
RaphL Probably, I'm usually around.
Colin: No I okay if I'm clearly fading.
Raph: Yeah.
Colin: But thank you for taking the time and we'll try this again in a couple yeah look forward to it.
Raph: Appreciate it and I'll talk to you again soon.