1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661
// Copyright (c) 2016 The vulkano developers
// Licensed under the Apache License, Version 2.0
// <LICENSE-APACHE or
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT
// license <LICENSE-MIT or https://opensource.org/licenses/MIT>,
// at your option. All files in the project carrying such
// notice may not be copied, modified, or distributed except
// according to those terms.
//! In Vulkan, suballocation of [`DeviceMemory`] is left to the application, because every
//! application has slightly different needs and one can not incorporate an allocator into the
//! driver that would perform well in all cases. Vulkano stays true to this sentiment, but aims to
//! reduce the burden on the user as much as possible. You have a toolbox of configurable
//! [suballocators] to choose from that cover all allocation algorithms, which you can compose into
//! any kind of [hierarchy] you wish. This way you have maximum flexibility while still only using
//! a few `DeviceMemory` blocks and not writing any of the very error-prone code.
//!
//! If you just want to allocate memory and don't have any special needs, look no further than the
//! [`StandardMemoryAllocator`].
//!
//! # Why not just allocate `DeviceMemory`?
//!
//! But the driver has an allocator! Otherwise you wouldn't be able to allocate `DeviceMemory`,
//! right? Indeed, but that allocation is very expensive. Not only that, there is also a pretty low
//! limit on the number of allocations by the drivers. See, everything in Vulkan tries to keep you
//! away from allocating `DeviceMemory` too much. These limits are used by the implementation to
//! optimize on its end, while the application optimizes on the other end.
//!
//! # Alignment
//!
//! At the end of the day, memory needs to be backed by hardware somehow. A *memory cell* stores a
//! single *bit*, bits are grouped into *bytes* and bytes are grouped into *words*. Intuitively, it
//! should make sense that accessing single bits at a time would be very inefficient. That is why
//! computers always access a whole word of memory at once, at least. That means that if you tried
//! to do an unaligned access, you would need to access twice the number of memory locations.
//!
//! Example aligned access, performing bitwise NOT on the (64-bit) word at offset 0x08:
//!
//! ```plain
//! | 08 | 10 | 18
//! ----+-------------------------+-------------------------+----
//! ••• | 35 35 35 35 35 35 35 35 | 01 23 45 67 89 ab cd ef | •••
//! ----+-------------------------+-------------------------+----
//! , | ,
//! +------------|------------+
//! ' v '
//! ----+-------------------------+-------------------------+----
//! ••• | ca ca ca ca ca ca ca ca | 01 23 45 67 89 ab cd ef | •••
//! ----+-------------------------+-------------------------+----
//! ```
//!
//! Same example as above, but this time unaligned with a word at offset 0x0a:
//!
//! ```plain
//! | 08 0a | 10 | 18
//! ----+-------------------------+-------------------------+----
//! ••• | cd ef 35 35 35 35 35 35 | 35 35 01 23 45 67 89 ab | •••
//! ----+-------------------------+-------------------------+----
//! , | ,
//! +------------|------------+
//! ' v '
//! ----+-------------------------+-------------------------+----
//! ••• | cd ef ca ca ca ca ca ca | ca ca 01 23 45 67 89 ab | •••
//! ----+-------------------------+-------------------------+----
//! ```
//!
//! As you can see, in the unaligned case the hardware would need to read both the word at offset
//! 0x08 and the word at the offset 0x10 and then shift the bits from one register into the other.
//! Safe to say it should to be avoided, and this is why we need alignment. This example also goes
//! to show how inefficient unaligned writes are. Say you pieced together your word as described,
//! and now you want to perform the bitwise NOT and write the result back. Difficult, isn't it?
//! That's due to the fact that even though the chunks occupy different ranges in memory, they are
//! still said to *alias* each other, because if you try to write to one memory location, you would
//! be overwriting 2 or more different chunks of data.
//!
//! ## Pages
//!
//! It doesn't stop at the word, though. Words are further grouped into *pages*. These are
//! typically power-of-two multiples of the word size, much like words are typically powers of two
//! themselves. You can easily extend the concepts from the previous examples to pages if you think
//! of the examples as having a page size of 1 word. Two resources are said to alias if they share
//! a page, and therefore should be aligned to the page size. What the page size is depends on the
//! context, and a computer might have multiple different ones for different parts of hardware.
//!
//! ## Memory requirements
//!
//! A Vulkan device might have any number of reasons it would want certain alignments for certain
//! resources. For example, the device might have different caches for different types of
//! resources, which have different page sizes. Maybe the device wants to store images in some
//! other cache compared to buffers which needs different alignment. Or maybe images of different
//! layouts require different alignment, or buffers with different usage/mapping do. The specifics
//! don't matter in the end, this just goes to illustrate the point. This is why memory
//! requirements in Vulkan vary not only with the Vulkan implementation, but also with the type of
//! resource.
//!
//! ## Buffer-image granularity
//!
//! This unfortunately named granularity is the page size which a linear resource neighboring a
//! non-linear resource must be aligned to in order for them not to alias. The difference between
//! the memory requirements of the individual resources and the [buffer-image granularity] is that
//! the memory requirements only apply to the resource they are for, while the buffer-image
//! granularity applies to two neighboring resources. For example, you might create two buffers,
//! which might have two different memory requirements, but as long as those are satisfied, you can
//! put these buffers cheek to cheek. On the other hand, if one of them is an (optimal layout)
//! image, then they must not share any page, whose size is given by this granularity. The Vulkan
//! implementation can use this for additional optimizations if it needs to, or report a
//! granularity of 1.
//!
//! # Fragmentation
//!
//! Memory fragmentation refers to the wastage of memory that results from alignment requirements
//! and/or dynamic memory allocation. As such, some level of fragmentation is always going to be
//! inevitable. Different allocation algorithms each have their own characteristics and trade-offs
//! in relation to fragmentation.
//!
//! ## Internal Fragmentation
//!
//! This type of fragmentation arises from alignment requirements. These might be imposed by the
//! Vulkan implementation or the application itself.
//!
//! Say for example your allocations need to be aligned to 64B, then any allocation whose size is
//! not a multiple of the alignment will need padding at the end:
//!
//! ```plain
//! | 0x040 | 0x080 | 0x0c0 | 0x100
//! ----+------------------+------------------+------------------+--------
//! | ############ | ################ | ######## | #######
//! ••• | ### 48 B ### | ##### 64 B ##### | # 32 B # | ### •••
//! | ############ | ################ | ######## | #######
//! ----+------------------+------------------+------------------+--------
//! ```
//!
//! If this alignment is imposed by the Vulkan implementation, then there's nothing one can do
//! about this. Simply put, that space is unusable. One also shouldn't want to do anything about
//! it, since these requirements have very good reasons, as described in further detail in previous
//! sections. They prevent resources from aliasing so that performance is optimal.
//!
//! It might seem strange that the application would want to cause internal fragmentation itself,
//! but this is often a good trade-off to reduce or even completely eliminate external
//! fragmentation. Internal fragmentation is very predictable, which makes it easier to deal with.
//!
//! ## External fragmentation
//!
//! With external fragmentation, what happens is that while the allocations might be using their
//! own memory totally efficiently, the way they are arranged in relation to each other would
//! prevent a new contiguous chunk of memory to be allocated even though there is enough free space
//! left. That is why this fragmentation is said to be external to the allocations. Also, the
//! allocations together with the fragments in-between add overhead both in terms of space and time
//! to the allocator, because it needs to keep track of more things overall.
//!
//! As an example, take these 4 allocations within some block, with the rest of the block assumed
//! to be full:
//!
//! ```plain
//! +-----+-------------------+-------+-----------+-- - - --+
//! | | | | | |
//! | A | B | C | D | ••• |
//! | | | | | |
//! +-----+-------------------+-------+-----------+-- - - --+
//! ```
//!
//! The allocations were all done in order, and naturally there is no fragmentation at this point.
//! Now if we free B and D, since these are done out of order, we will be left with holes between
//! the other allocations, and we won't be able to fit allocation E anywhere:
//!
//! ```plain
//! +-----+-------------------+-------+-----------+-- - - --+ +-------------------------+
//! | | | | | | ? | |
//! | A | | C | | ••• | <== | E |
//! | | | | | | | |
//! +-----+-------------------+-------+-----------+-- - - --+ +-------------------------+
//! ```
//!
//! So fine, we use a different block for E, and just use this block for allocations that fit:
//!
//! ```plain
//! +-----+---+-----+---------+-------+-----+-----+-- - - --+
//! | | | | | | | | |
//! | A | H | I | J | C | F | G | ••• |
//! | | | | | | | | |
//! +-----+---+-----+---------+-------+-----+-----+-- - - --+
//! ```
//!
//! Sure, now let's free some shall we? And voilà, the problem just became much worse:
//!
//! ```plain
//! +-----+---+-----+---------+-------+-----+-----+-- - - --+
//! | | | | | | | | |
//! | A | | I | J | | F | | ••• |
//! | | | | | | | | |
//! +-----+---+-----+---------+-------+-----+-----+-- - - --+
//! ```
//!
//! # Leakage
//!
//! Memory leaks happen when allocations are kept alive past their shelf life. This most often
//! occurs because of [cyclic references]. If you have structures that have cycles, then make sure
//! you read the documentation for [`Arc`]/[`Rc`] carefully to avoid memory leaks. You can also
//! introduce memory leaks willingly by using [`mem::forget`] or [`Box::leak`] to name a few. In
//! all of these examples the memory can never be reclaimed, but that doesn't have to be the case
//! for something to be considered a leak. Say for example you have a [region] which you
//! suballocate, and at some point you drop all the suballocations. When that happens, the region
//! can be returned (freed) to the next level up the hierarchy, or it can be reused by another
//! suballocator. But if you happen to keep alive just one suballocation for the duration of the
//! program for instance, then the whole region is also kept as it is for that time (and keep in
//! mind this bubbles up the hierarchy). Therefore, for the program, that memory might be a leak
//! depending on the allocator, because some allocators wouldn't be able to reuse the entire rest
//! of the region. You must always consider the lifetime of your resources when choosing the
//! appropriate allocator.
//!
//! [suballocators]: Suballocator
//! [hierarchy]: Suballocator#memory-hierarchies
//! [buffer-image granularity]: crate::device::Properties::buffer_image_granularity
//! [cyclic references]: Arc#breaking-cycles-with-weak
//! [`Rc`]: std::rc::Rc
//! [`mem::forget`]: std::mem::forget
//! [region]: Suballocator#regions
pub mod suballocator;
use self::array_vec::ArrayVec;
pub use self::suballocator::{
AllocationType, BuddyAllocator, BumpAllocator, FreeListAllocator, MemoryAlloc, PoolAllocator,
SuballocationCreateInfo, SuballocationCreationError, Suballocator,
};
use super::{
DedicatedAllocation, DeviceMemory, ExternalMemoryHandleTypes, MemoryAllocateFlags,
MemoryAllocateInfo, MemoryProperties, MemoryPropertyFlags, MemoryRequirements, MemoryType,
};
use crate::{
device::{Device, DeviceOwned},
DeviceSize, RequirementNotMet, RequiresOneOf, Version, VulkanError,
};
use ash::vk::{MAX_MEMORY_HEAPS, MAX_MEMORY_TYPES};
use parking_lot::RwLock;
use std::{
error::Error,
fmt::{Display, Error as FmtError, Formatter},
sync::Arc,
};
const B: DeviceSize = 1;
const K: DeviceSize = 1024 * B;
const M: DeviceSize = 1024 * K;
const G: DeviceSize = 1024 * M;
/// General-purpose memory allocators which allocate from any memory type dynamically as needed.
pub unsafe trait MemoryAllocator: DeviceOwned {
/// Finds the most suitable memory type index in `memory_type_bits` using a filter. Returns
/// [`None`] if the requirements are too strict and no memory type is able to satisfy them.
fn find_memory_type_index(
&self,
memory_type_bits: u32,
filter: MemoryTypeFilter,
) -> Option<u32>;
/// Allocates memory from a specific memory type.
fn allocate_from_type(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
) -> Result<MemoryAlloc, AllocationCreationError>;
/// Allocates memory from a specific memory type without checking the parameters.
///
/// # Safety
///
/// - If `memory_type_index` refers to a memory type with the [`protected`] flag set, then the
/// [`protected_memory`] feature must be enabled on the device.
/// - If `memory_type_index` refers to a memory type with the [`device_coherent`] flag set,
/// then the [`device_coherent_memory`] feature must be enabled on the device.
/// - `create_info.size` must not be zero.
/// - `create_info.size` must not exceed the size of the heap that the memory type
/// corresponding to `memory_type_index` resides in.
/// - `create_info.alignment` must not be zero.
/// - `create_info.alignment` must be a power of two.
///
/// [`protected`]: MemoryPropertyFlags::protected
/// [`protected_memory`]: crate::device::Features::protected_memory
/// [`device_coherent`]: MemoryPropertyFlags::device_coherent
/// [`device_coherent_memory`]: crate::device::Features::device_coherent_memory
#[cfg_attr(not(feature = "document_unchecked"), doc(hidden))]
unsafe fn allocate_from_type_unchecked(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
never_allocate: bool,
) -> Result<MemoryAlloc, AllocationCreationError>;
/// Allocates memory according to requirements.
fn allocate(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError>;
/// Allocates memory according to requirements without checking the parameters.
///
/// # Safety
///
/// - `create_info.requirements.size` must not be zero.
/// - `create_info.requirements.alignment` must not be zero.
/// - `create_info.requirements.alignment` must be a power of two.
/// - If `create_info.dedicated_allocation` is `Some` then `create_info.requirements.size` must
/// match the memory requirements of the resource.
/// - If `create_info.dedicated_allocation` is `Some` then the device the resource was created
/// with must match the device the allocator was created with.
#[cfg_attr(not(feature = "document_unchecked"), doc(hidden))]
unsafe fn allocate_unchecked(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError>;
/// Creates a root allocation/dedicated allocation without checking the parameters.
///
/// # Safety
///
/// - `allocation_size` must not exceed the size of the heap that the memory type corresponding
/// to `memory_type_index` resides in.
/// - The handle types in `export_handle_types` must be supported and compatible, as reported by
/// [`ExternalBufferProperties`] or [`ImageFormatProperties`].
/// - If any of the handle types in `export_handle_types` require a dedicated allocation, as
/// reported by [`ExternalBufferProperties::external_memory_properties`] or
/// [`ImageFormatProperties::external_memory_properties`], then `dedicated_allocation` must
/// not be `None`.
///
/// [`ExternalBufferProperties`]: crate::buffer::ExternalBufferProperties
/// [`ImageFormatProperties`]: crate::image::ImageFormatProperties
/// [`ExternalBufferProperties::external_memory_properties`]: crate::buffer::ExternalBufferProperties
/// [`ImageFormatProperties::external_memory_properties`]: crate::image::ImageFormatProperties::external_memory_properties
#[cfg_attr(not(feature = "document_unchecked"), doc(hidden))]
unsafe fn allocate_dedicated_unchecked(
&self,
memory_type_index: u32,
allocation_size: DeviceSize,
dedicated_allocation: Option<DedicatedAllocation<'_>>,
export_handle_types: ExternalMemoryHandleTypes,
) -> Result<MemoryAlloc, AllocationCreationError>;
}
/// Describes what memory property flags are required, preferred and not preferred when picking a
/// memory type index.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct MemoryTypeFilter {
pub required_flags: MemoryPropertyFlags,
pub preferred_flags: MemoryPropertyFlags,
pub not_preferred_flags: MemoryPropertyFlags,
}
impl From<MemoryUsage> for MemoryTypeFilter {
#[inline]
fn from(usage: MemoryUsage) -> Self {
let mut filter = Self::default();
match usage {
MemoryUsage::GpuOnly => {
filter.preferred_flags.device_local = true;
filter.not_preferred_flags.host_visible = true;
}
MemoryUsage::Upload => {
filter.required_flags.host_visible = true;
filter.preferred_flags.device_local = true;
filter.not_preferred_flags.host_cached = true;
}
MemoryUsage::Download => {
filter.required_flags.host_visible = true;
filter.preferred_flags.host_cached = true;
}
}
filter
}
}
/// Parameters to create a new [allocation] using a [memory allocator].
///
/// [allocation]: MemoryAlloc
/// [memory allocator]: MemoryAllocator
#[derive(Clone, Debug)]
pub struct AllocationCreateInfo<'d> {
/// Requirements of the resource you want to allocate memory for.
///
/// If you plan to bind this memory directly to a non-sparse resource, then this must
/// correspond to the value returned by either [`RawBuffer::memory_requirements`] or
/// [`RawImage::memory_requirements`] for the respective buffer or image.
///
/// All of the fields must be non-zero, [`alignment`] must be a power of two, and
/// [`memory_type_bits`] must be below 2<sup>*n*</sup> where *n* is the number of available
/// memory types.
///
/// The default is all zeros, which must be overridden.
///
/// [`alignment`]: MemoryRequirements::alignment
/// [`memory_type_bits`]: MemoryRequirements::memory_type_bits
/// [`RawBuffer::memory_requirements`]: crate::buffer::sys::RawBuffer::memory_requirements
/// [`RawImage::memory_requirements`]: crate::image::sys::RawImage::memory_requirements
pub requirements: MemoryRequirements,
/// What type of resource this allocation will be used for.
///
/// This should be [`Linear`] for buffers and linear images, and [`NonLinear`] for optimal
/// images. You can not bind memory allocated with the [`Linear`] type to optimal images or
/// bind memory allocated with the [`NonLinear`] type to buffers and linear images. You should
/// never use the [`Unknown`] type unless you have to, as that can be less memory efficient.
///
/// The default value is [`AllocationType::Unknown`].
///
/// [`Linear`]: AllocationType::Linear
/// [`NonLinear`]: AllocationType::NonLinear
/// [`Unknown`]: AllocationType::Unknown
pub allocation_type: AllocationType,
/// The intended usage for the allocation.
///
/// The default value is [`MemoryUsage::GpuOnly`].
pub usage: MemoryUsage,
/// How eager the allocator should be to allocate [`DeviceMemory`].
///
/// The default value is [`MemoryAllocatePreference::Unknown`].
pub allocate_preference: MemoryAllocatePreference,
/// Allows a dedicated allocation to be created.
///
/// You should always fill this field in if you are allocating memory for a non-sparse
/// resource, otherwise the allocator won't be able to create a dedicated allocation if one is
/// recommended.
///
/// This option is silently ignored (treated as `None`) if the device API version is below 1.1
/// and the [`khr_dedicated_allocation`] extension is not enabled on the device.
///
/// The default value is [`None`].
///
/// [`khr_dedicated_allocation`]: crate::device::DeviceExtensions::khr_dedicated_allocation
pub dedicated_allocation: Option<DedicatedAllocation<'d>>,
pub _ne: crate::NonExhaustive,
}
impl Default for AllocationCreateInfo<'_> {
#[inline]
fn default() -> Self {
AllocationCreateInfo {
requirements: MemoryRequirements {
size: 0,
alignment: 0,
memory_type_bits: 0,
prefers_dedicated_allocation: false,
requires_dedicated_allocation: false,
},
allocation_type: AllocationType::Unknown,
usage: MemoryUsage::GpuOnly,
allocate_preference: MemoryAllocatePreference::Unknown,
dedicated_allocation: None,
_ne: crate::NonExhaustive(()),
}
}
}
/// Describes how a memory allocation is going to be used.
///
/// This is mostly an optimization, except for `MemoryUsage::GpuOnly` which will pick a memory type
/// that is not CPU-accessible if such a type exists.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum MemoryUsage {
/// The memory is intended to only be used by the GPU.
///
/// Prefers picking a memory type with the [`device_local`] flag and without the
/// [`host_visible`] flag.
///
/// This option is what you will always want to use unless the memory needs to be accessed by
/// the CPU, because a memory type that can only be accessed by the GPU is going to give the
/// best performance. Example use cases would be textures and other maps which are written to
/// once and then never again, or resources that are only written and read by the GPU, like
/// render targets and intermediary buffers.
///
/// [`device_local`]: MemoryPropertyFlags::device_local
/// [`host_visible`]: MemoryPropertyFlags::host_visible
GpuOnly,
/// The memory is intended for upload to the GPU.
///
/// Guarantees picking a memory type with the [`host_visible`] flag. Prefers picking one
/// without the [`host_cached`] flag and with the [`device_local`] flag.
///
/// This option is best suited for resources that need to be constantly updated by the CPU,
/// like vertex and index buffers for example. It is also neccessary for *staging buffers*,
/// whose only purpose in life it is to get data into `device_local` memory or texels into an
/// optimal image.
///
/// [`host_visible`]: MemoryPropertyFlags::host_visible
/// [`host_cached`]: MemoryPropertyFlags::host_cached
/// [`device_local`]: MemoryPropertyFlags::device_local
Upload,
/// The memory is intended for download from the GPU.
///
/// Guarantees picking a memory type with the [`host_visible`] flag. Prefers picking one with
/// the [`host_cached`] flag and without the [`device_local`] flag.
///
/// This option is best suited if you're using the GPU for things other than rendering and you
/// need to get the results back to the CPU. That might be compute shading, or image or video
/// manipulation, or screenshotting for example.
///
/// [`host_visible`]: MemoryPropertyFlags::host_visible
/// [`host_cached`]: MemoryPropertyFlags::host_cached
/// [`device_local`]: MemoryPropertyFlags::device_local
Download,
}
/// Describes whether allocating [`DeviceMemory`] is desired.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum MemoryAllocatePreference {
/// There is no known preference, let the allocator decide.
Unknown,
/// The allocator should never allocate `DeviceMemory` and should instead only suballocate from
/// existing blocks.
///
/// This option is best suited if you can not afford the overhead of allocating `DeviceMemory`.
NeverAllocate,
/// The allocator should always allocate `DeviceMemory`.
///
/// This option is best suited if you are allocating a long-lived resource that you know could
/// benefit from having a dedicated allocation.
AlwaysAllocate,
}
/// Error that can be returned when creating an [allocation] using a [memory allocator].
///
/// [allocation]: MemoryAlloc
/// [memory allocator]: MemoryAllocator
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum AllocationCreationError {
VulkanError(VulkanError),
/// There is not enough memory in the pool.
///
/// This is returned when using [`MemoryAllocatePreference::NeverAllocate`] and there is not
/// enough memory in the pool.
OutOfPoolMemory,
/// A dedicated allocation is required but was explicitly forbidden.
///
/// This is returned when using [`MemoryAllocatePreference::NeverAllocate`] and the
/// implementation requires a dedicated allocation.
DedicatedAllocationRequired,
/// The block size for the allocator was exceeded.
///
/// This is returned when using [`MemoryAllocatePreference::NeverAllocate`] and the allocation
/// size exceeded the block size for all heaps of suitable memory types.
BlockSizeExceeded,
/// The block size for the suballocator was exceeded.
///
/// This is returned when using [`GenericMemoryAllocator<Arc<PoolAllocator<BLOCK_SIZE>>>`] if
/// the allocation size exceeded `BLOCK_SIZE`.
SuballocatorBlockSizeExceeded,
}
impl Error for AllocationCreationError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
Self::VulkanError(err) => Some(err),
_ => None,
}
}
}
impl Display for AllocationCreationError {
fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), FmtError> {
match self {
Self::VulkanError(_) => write!(f, "a runtime error occurred"),
Self::OutOfPoolMemory => write!(f, "the pool doesn't have enough free space"),
Self::DedicatedAllocationRequired => write!(
f,
"a dedicated allocation is required but was explicitly forbidden",
),
Self::BlockSizeExceeded => write!(
f,
"the allocation size was greater than the block size for all heaps of suitable \
memory types and dedicated allocations were explicitly forbidden",
),
Self::SuballocatorBlockSizeExceeded => write!(
f,
"the allocation size was greater than the suballocator's block size",
),
}
}
}
impl From<VulkanError> for AllocationCreationError {
fn from(err: VulkanError) -> Self {
AllocationCreationError::VulkanError(err)
}
}
/// Standard memory allocator intended as a global and general-purpose allocator.
///
/// This type of allocator should work well in most cases, it is however **not** to be used when
/// allocations need to be made very frequently (say, once or more per frame). For that purpose,
/// use [`FastMemoryAllocator`].
///
/// See [`FreeListAllocator`] for details about the allocation algorithm and example usage.
pub type StandardMemoryAllocator = GenericMemoryAllocator<Arc<FreeListAllocator>>;
impl StandardMemoryAllocator {
/// Creates a new `StandardMemoryAllocator` with default configuration.
pub fn new_default(device: Arc<Device>) -> Self {
#[allow(clippy::erasing_op, clippy::identity_op)]
let create_info = GenericMemoryAllocatorCreateInfo {
#[rustfmt::skip]
block_sizes: &[
(0 * B, 64 * M),
(1 * G, 256 * M),
],
..Default::default()
};
unsafe { Self::new_unchecked(device, create_info) }
}
}
/// Fast memory allocator intended as a local and special-purpose allocator.
///
/// This type of allocator is only useful when you need to allocate a lot, for example once or more
/// per frame. It is **not** to be used when allocations are long-lived. For that purpose use
/// [`StandardMemoryAllocator`].
///
/// See [`BumpAllocator`] for details about the allocation algorithm.
pub type FastMemoryAllocator = GenericMemoryAllocator<Arc<BumpAllocator>>;
impl FastMemoryAllocator {
/// Creates a new `FastMemoryAllocator` with default configuration.
pub fn new_default(device: Arc<Device>) -> Self {
#[allow(clippy::erasing_op, clippy::identity_op)]
let create_info = GenericMemoryAllocatorCreateInfo {
#[rustfmt::skip]
block_sizes: &[
( 0 * B, 16 * M),
(512 * M, 32 * M),
( 1 * G, 64 * M),
],
..Default::default()
};
unsafe { Self::new_unchecked(device, create_info) }
}
}
/// A generic implementation of a [memory allocator].
///
/// The allocator keeps a pool of [`DeviceMemory`] blocks for each memory type and uses the type
/// parameter `S` to [suballocate] these blocks. You can also configure the sizes of these blocks.
/// This means that you can have as many `GenericMemoryAllocator`s as you you want for different
/// needs, or for performance reasons, as long as the block sizes are configured properly so that
/// too much memory isn't wasted.
///
/// See also [the `MemoryAllocator` implementation].
///
/// # `DeviceMemory` allocation
///
/// If an allocation is created with the [`MemoryAllocatePreference::Unknown`] option, and the
/// allocator deems the allocation too big for suballocation (larger than half the block size), or
/// the implementation prefers or requires a dedicated allocation, then that allocation is made a
/// dedicated allocation. Using [`MemoryAllocatePreference::NeverAllocate`], a dedicated allocation
/// is never created, even if the allocation is larger than the block size or a dedicated
/// allocation is required. In such a case an error is returned instead. Using
/// [`MemoryAllocatePreference::AlwaysAllocate`], a dedicated allocation is always created.
///
/// In all other cases, `DeviceMemory` is only allocated if a pool runs out of memory and needs
/// another block. No `DeviceMemory` is allocated when the allocator is created, the blocks are
/// only allocated once they are needed.
///
/// # Locking behavior
///
/// The allocator never needs to lock while suballocating unless `S` needs to lock. The only time
/// when a pool must be locked is when a new `DeviceMemory` block is allocated for the pool. This
/// means that the allocator is suited to both locking and lock-free (sub)allocation algorithms.
///
/// [memory allocator]: MemoryAllocator
/// [suballocate]: Suballocator
/// [the `MemoryAllocator` implementation]: Self#impl-MemoryAllocator-for-GenericMemoryAllocator<S>
#[derive(Debug)]
pub struct GenericMemoryAllocator<S: Suballocator> {
device: Arc<Device>,
// Each memory type has a pool of `DeviceMemory` blocks.
pools: ArrayVec<Pool<S>, MAX_MEMORY_TYPES>,
// Each memory heap has its own block size.
block_sizes: ArrayVec<DeviceSize, MAX_MEMORY_HEAPS>,
allocation_type: AllocationType,
dedicated_allocation: bool,
export_handle_types: ArrayVec<ExternalMemoryHandleTypes, MAX_MEMORY_TYPES>,
flags: MemoryAllocateFlags,
// Global mask of memory types.
memory_type_bits: u32,
// How many `DeviceMemory` allocations should be allowed before restricting them.
max_allocations: u32,
}
#[derive(Debug)]
struct Pool<S> {
blocks: RwLock<Vec<S>>,
// This is cached here for faster access, so we don't need to hop through 3 pointers.
memory_type: ash::vk::MemoryType,
}
impl<S: Suballocator> GenericMemoryAllocator<S> {
// This is a false-positive, we only use this const for static initialization.
#[allow(clippy::declare_interior_mutable_const)]
const EMPTY_POOL: Pool<S> = Pool {
blocks: RwLock::new(Vec::new()),
memory_type: ash::vk::MemoryType {
property_flags: ash::vk::MemoryPropertyFlags::empty(),
heap_index: 0,
},
};
/// Creates a new `GenericMemoryAllocator<S>` using the provided suballocator `S` for
/// suballocation of [`DeviceMemory`] blocks.
///
/// # Panics
///
/// - Panics if `create_info.block_sizes` is not sorted by threshold.
/// - Panics if `create_info.block_sizes` contains duplicate thresholds.
/// - Panics if `create_info.block_sizes` does not contain a baseline threshold of `0`.
/// - Panics if the block size for a heap exceeds the size of the heap.
pub fn new(
device: Arc<Device>,
create_info: GenericMemoryAllocatorCreateInfo<'_, '_>,
) -> Result<Self, GenericMemoryAllocatorCreationError> {
Self::validate_new(&device, &create_info)?;
Ok(unsafe { Self::new_unchecked(device, create_info) })
}
fn validate_new(
device: &Device,
create_info: &GenericMemoryAllocatorCreateInfo<'_, '_>,
) -> Result<(), GenericMemoryAllocatorCreationError> {
let &GenericMemoryAllocatorCreateInfo {
block_sizes,
allocation_type: _,
dedicated_allocation: _,
export_handle_types,
device_address: _,
_ne: _,
} = create_info;
assert!(
block_sizes.windows(2).all(|win| win[0].0 < win[1].0),
"`create_info.block_sizes` must be sorted by threshold without duplicates",
);
assert!(
matches!(block_sizes.first(), Some((0, _))),
"`create_info.block_sizes` must contain a baseline threshold `0`",
);
if !export_handle_types.is_empty() {
if !(device.api_version() >= Version::V1_1
&& device.enabled_extensions().khr_external_memory)
{
return Err(GenericMemoryAllocatorCreationError::RequirementNotMet {
required_for: "`create_info.export_handle_types` was not empty",
requires_one_of: RequiresOneOf {
api_version: Some(Version::V1_1),
device_extensions: &["khr_external_memory"],
..Default::default()
},
});
}
assert!(
export_handle_types.len()
== device
.physical_device()
.memory_properties()
.memory_types
.len(),
"`create_info.export_handle_types` must contain as many elements as the number of \
memory types if not empty",
);
for export_handle_types in export_handle_types {
// VUID-VkExportMemoryAllocateInfo-handleTypes-parameter
export_handle_types.validate_device(device)?;
}
}
Ok(())
}
#[cfg_attr(not(feature = "document_unchecked"), doc(hidden))]
pub unsafe fn new_unchecked(
device: Arc<Device>,
create_info: GenericMemoryAllocatorCreateInfo<'_, '_>,
) -> Self {
let GenericMemoryAllocatorCreateInfo {
block_sizes,
allocation_type,
dedicated_allocation,
export_handle_types,
mut device_address,
_ne: _,
} = create_info;
let MemoryProperties {
memory_types,
memory_heaps,
} = device.physical_device().memory_properties();
let mut pools = ArrayVec::new(memory_types.len(), [Self::EMPTY_POOL; MAX_MEMORY_TYPES]);
for (i, memory_type) in memory_types.iter().enumerate() {
pools[i].memory_type = ash::vk::MemoryType {
property_flags: memory_type.property_flags.into(),
heap_index: memory_type.heap_index,
};
}
let block_sizes = {
let mut sizes = ArrayVec::new(memory_heaps.len(), [0; MAX_MEMORY_HEAPS]);
for (i, memory_heap) in memory_heaps.iter().enumerate() {
let idx = match block_sizes.binary_search_by_key(&memory_heap.size, |&(t, _)| t) {
Ok(idx) => idx,
Err(idx) => idx.saturating_sub(1),
};
sizes[i] = block_sizes[idx].1;
// VUID-vkAllocateMemory-pAllocateInfo-01713
assert!(sizes[i] <= memory_heap.size);
}
sizes
};
let export_handle_types = {
let mut types = ArrayVec::new(
export_handle_types.len(),
[ExternalMemoryHandleTypes::empty(); MAX_MEMORY_TYPES],
);
types.copy_from_slice(export_handle_types);
types
};
// VUID-VkMemoryAllocateInfo-flags-03331
device_address &= device.enabled_features().buffer_device_address
&& !device.enabled_extensions().ext_buffer_device_address;
// Providers of `VkMemoryAllocateFlags`
device_address &=
device.api_version() >= Version::V1_1 || device.enabled_extensions().khr_device_group;
let mut memory_type_bits = u32::MAX;
for (index, MemoryType { property_flags, .. }) in memory_types.iter().enumerate() {
if property_flags.lazily_allocated
|| property_flags.protected
|| property_flags.device_coherent
|| property_flags.device_uncached
|| property_flags.rdma_capable
{
// VUID-VkMemoryAllocateInfo-memoryTypeIndex-01872
// VUID-vkAllocateMemory-deviceCoherentMemory-02790
// Lazily allocated memory would just cause problems for suballocation in general.
memory_type_bits &= !(1 << index);
}
}
let max_memory_allocation_count = device
.physical_device()
.properties()
.max_memory_allocation_count;
let max_allocations = max_memory_allocation_count / 4 * 3;
GenericMemoryAllocator {
device,
pools,
block_sizes,
allocation_type,
dedicated_allocation,
export_handle_types,
flags: MemoryAllocateFlags {
device_address,
..Default::default()
},
memory_type_bits,
max_allocations,
}
}
fn validate_allocate_from_type(
&self,
memory_type_index: u32,
create_info: &SuballocationCreateInfo,
) {
let memory_type = &self.pools[memory_type_index as usize].memory_type;
// VUID-VkMemoryAllocateInfo-memoryTypeIndex-01872
assert!(
memory_type
.property_flags
.contains(ash::vk::MemoryPropertyFlags::PROTECTED)
&& !self.device.enabled_features().protected_memory,
"attempted to allocate from a protected memory type without the `protected_memory` \
feature being enabled on the device",
);
// VUID-vkAllocateMemory-deviceCoherentMemory-02790
assert!(
memory_type
.property_flags
.contains(ash::vk::MemoryPropertyFlags::DEVICE_COHERENT_AMD)
&& !self.device.enabled_features().device_coherent_memory,
"attempted to allocate memory from a device-coherent memory type without the \
`device_coherent_memory` feature being enabled on the device",
);
create_info.validate();
}
fn validate_allocate(&self, create_info: &AllocationCreateInfo<'_>) {
let &AllocationCreateInfo {
requirements,
allocation_type: _,
usage: _,
allocate_preference: _,
dedicated_allocation,
_ne: _,
} = create_info;
SuballocationCreateInfo::from(create_info.clone()).validate();
assert!(requirements.memory_type_bits != 0);
assert!(requirements.memory_type_bits < 1 << self.pools.len());
if let Some(dedicated_allocation) = dedicated_allocation {
match dedicated_allocation {
DedicatedAllocation::Buffer(buffer) => {
// VUID-VkMemoryDedicatedAllocateInfo-commonparent
assert_eq!(&self.device, buffer.device());
let required_size = buffer.memory_requirements().size;
// VUID-VkMemoryDedicatedAllocateInfo-buffer-02965
assert!(requirements.size != required_size);
}
DedicatedAllocation::Image(image) => {
// VUID-VkMemoryDedicatedAllocateInfo-commonparent
assert_eq!(&self.device, image.device());
let required_size = image.memory_requirements()[0].size;
// VUID-VkMemoryDedicatedAllocateInfo-image-02964
assert!(requirements.size != required_size);
}
}
}
// VUID-VkMemoryAllocateInfo-pNext-00639
// VUID-VkExportMemoryAllocateInfo-handleTypes-00656
// Can't validate, must be ensured by user
}
}
unsafe impl<S: Suballocator> MemoryAllocator for GenericMemoryAllocator<S> {
fn find_memory_type_index(
&self,
memory_type_bits: u32,
filter: MemoryTypeFilter,
) -> Option<u32> {
let required_flags = filter.required_flags.into();
let preferred_flags = filter.preferred_flags.into();
let not_preferred_flags = filter.not_preferred_flags.into();
self.pools
.iter()
.map(|pool| pool.memory_type.property_flags)
.enumerate()
// Filter out memory types which are supported by the memory type bits and have the
// required flags set.
.filter(|&(index, flags)| {
memory_type_bits & (1 << index) != 0 && flags & required_flags == required_flags
})
// Rank memory types with more of the preferred flags higher, and ones with more of the
// not preferred flags lower.
.min_by_key(|&(_, flags)| {
(!flags & preferred_flags).as_raw().count_ones()
+ (flags & not_preferred_flags).as_raw().count_ones()
})
.map(|(index, _)| index as u32)
}
/// Allocates memory from a specific memory type.
///
/// # Panics
///
/// - Panics if `memory_type_index` is not less than the number of available memory types.
/// - Panics if `memory_type_index` refers to a memory type which has the [`protected`] flag set
/// and the [`protected_memory`] feature is not enabled on the device.
/// - Panics if `memory_type_index` refers to a memory type which has the [`device_coherent`]
/// flag set and the [`device_coherent_memory`] feature is not enabled on the device.
/// - Panics if `create_info.size` is zero.
/// - Panics if `create_info.alignment` is zero.
/// - Panics if `create_info.alignment` is not a power of two.
///
/// # Errors
///
/// - Returns an error if allocating a new block is required and failed. This can be one of the
/// OOM errors or [`TooManyObjects`].
/// - Returns [`BlockSizeExceeded`] if `create_info.size` is greater than the block size
/// corresponding to the heap that the memory type corresponding to `memory_type_index`
/// resides in.
/// - Returns [`SuballocatorBlockSizeExceeded`] if `S` is `PoolAllocator<BLOCK_SIZE>` and
/// `create_info.size` is greater than `BLOCK_SIZE`.
///
/// [`protected`]: MemoryPropertyFlags::protected
/// [`protected_memory`]: crate::device::Features::protected_memory
/// [`device_coherent`]: MemoryPropertyFlags::device_coherent
/// [`device_coherent_memory`]: crate::device::Features::device_coherent_memory
/// [`TooManyObjects`]: VulkanError::TooManyObjects
/// [`BlockSizeExceeded`]: AllocationCreationError::BlockSizeExceeded
/// [`SuballocatorBlockSizeExceeded`]: AllocationCreationError::SuballocatorBlockSizeExceeded
fn allocate_from_type(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
) -> Result<MemoryAlloc, AllocationCreationError> {
self.validate_allocate_from_type(memory_type_index, &create_info);
if self.pools[memory_type_index as usize]
.memory_type
.property_flags
.contains(ash::vk::MemoryPropertyFlags::LAZILY_ALLOCATED)
{
return unsafe {
self.allocate_dedicated_unchecked(
memory_type_index,
create_info.size,
None,
if !self.export_handle_types.is_empty() {
self.export_handle_types[memory_type_index as usize]
} else {
ExternalMemoryHandleTypes::empty()
},
)
};
}
unsafe { self.allocate_from_type_unchecked(memory_type_index, create_info, false) }
}
unsafe fn allocate_from_type_unchecked(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
never_allocate: bool,
) -> Result<MemoryAlloc, AllocationCreationError> {
let SuballocationCreateInfo {
size,
alignment: _,
allocation_type: _,
_ne: _,
} = create_info;
let pool = &self.pools[memory_type_index as usize];
let block_size = self.block_sizes[pool.memory_type.heap_index as usize];
if size > block_size {
return Err(AllocationCreationError::BlockSizeExceeded);
}
let mut blocks = if S::IS_BLOCKING {
// If the allocation algorithm needs to block, then there's no point in trying to avoid
// locks here either. In that case the best strategy is to take full advantage of it by
// always taking an exclusive lock, which lets us sort the blocks by free size. If you
// as a user want to avoid locks, simply don't share the allocator between threads. You
// can create as many allocators as you wish, but keep in mind that that will waste a
// huge amount of memory unless you configure your block sizes properly!
let mut blocks = pool.blocks.write();
blocks.sort_by_key(Suballocator::free_size);
let (Ok(idx) | Err(idx)) = blocks.binary_search_by_key(&size, Suballocator::free_size);
for block in &blocks[idx..] {
match block.allocate_unchecked(create_info.clone()) {
Ok(alloc) => return Ok(alloc),
Err(SuballocationCreationError::BlockSizeExceeded) => {
return Err(AllocationCreationError::SuballocatorBlockSizeExceeded);
}
Err(_) => {}
}
}
blocks
} else {
// If the allocation algorithm is lock-free, then we should avoid taking an exclusive
// lock unless it is absolutely neccessary (meaning, only when allocating a new
// `DeviceMemory` block and inserting it into a pool). This has the disadvantage that
// traversing the pool is O(n), which is not a problem since the number of blocks is
// expected to be small. If there are more than 10 blocks in a pool then that's a
// configuration error. Also, sorting the blocks before each allocation would be less
// efficient because to get the free size of the `PoolAllocator` and `BumpAllocator`
// has the same performance as trying to allocate.
let blocks = pool.blocks.read();
// Search in reverse order because we always append new blocks at the end.
for block in blocks.iter().rev() {
match block.allocate_unchecked(create_info.clone()) {
Ok(alloc) => return Ok(alloc),
// This can happen when using the `PoolAllocator<BLOCK_SIZE>` if the allocation
// size is greater than `BLOCK_SIZE`.
Err(SuballocationCreationError::BlockSizeExceeded) => {
return Err(AllocationCreationError::SuballocatorBlockSizeExceeded);
}
Err(_) => {}
}
}
let len = blocks.len();
drop(blocks);
let blocks = pool.blocks.write();
if blocks.len() > len {
// Another thread beat us to it and inserted a fresh block, try to allocate from it.
match blocks[len].allocate_unchecked(create_info.clone()) {
Ok(alloc) => return Ok(alloc),
// This can happen if this is the first block that was inserted and when using
// the `PoolAllocator<BLOCK_SIZE>` if the allocation size is greater than
// `BLOCK_SIZE`.
Err(SuballocationCreationError::BlockSizeExceeded) => {
return Err(AllocationCreationError::SuballocatorBlockSizeExceeded);
}
Err(_) => {}
}
}
blocks
};
// For bump allocators, first do a garbage sweep and try to allocate again.
if S::NEEDS_CLEANUP {
blocks.iter_mut().for_each(Suballocator::cleanup);
blocks.sort_unstable_by_key(Suballocator::free_size);
if let Some(block) = blocks.last() {
if let Ok(alloc) = block.allocate_unchecked(create_info.clone()) {
return Ok(alloc);
}
}
}
if never_allocate {
return Err(AllocationCreationError::OutOfPoolMemory);
}
// The pool doesn't have enough real estate, so we need a new block.
let block = {
let export_handle_types = if !self.export_handle_types.is_empty() {
self.export_handle_types[memory_type_index as usize]
} else {
ExternalMemoryHandleTypes::empty()
};
let mut i = 0;
loop {
let allocate_info = MemoryAllocateInfo {
allocation_size: block_size >> i,
memory_type_index,
export_handle_types,
dedicated_allocation: None,
flags: self.flags,
..Default::default()
};
match DeviceMemory::allocate_unchecked(self.device.clone(), allocate_info, None) {
Ok(device_memory) => {
break S::new(MemoryAlloc::new(device_memory)?);
}
// Retry up to 3 times, halving the allocation size each time.
Err(VulkanError::OutOfHostMemory | VulkanError::OutOfDeviceMemory) if i < 3 => {
i += 1;
}
Err(err) => return Err(err.into()),
}
}
};
blocks.push(block);
let block = blocks.last().unwrap();
match block.allocate_unchecked(create_info) {
Ok(alloc) => Ok(alloc),
// This can happen if the block ended up smaller than advertised because there wasn't
// enough memory.
Err(SuballocationCreationError::OutOfRegionMemory) => Err(
AllocationCreationError::VulkanError(VulkanError::OutOfDeviceMemory),
),
// This can not happen as the block is fresher than Febreze and we're still holding an
// exclusive lock.
Err(SuballocationCreationError::FragmentedRegion) => unreachable!(),
// This can happen if this is the first block that was inserted and when using the
// `PoolAllocator<BLOCK_SIZE>` if the allocation size is greater than `BLOCK_SIZE`.
Err(SuballocationCreationError::BlockSizeExceeded) => {
Err(AllocationCreationError::SuballocatorBlockSizeExceeded)
}
}
}
/// Allocates memory according to requirements.
///
/// # Panics
///
/// - Panics if `create_info.requirements.size` is zero.
/// - Panics if `create_info.requirements.alignment` is zero.
/// - Panics if `create_info.requirements.alignment` is not a power of two.
/// - Panics if `create_info.requirements.memory_type_bits` is zero.
/// - Panics if `create_info.requirements.memory_type_bits` is not less than 2<sup>*n*</sup>
/// where *n* is the number of available memory types.
/// - Panics if `create_info.dedicated_allocation` is `Some` and
/// `create_info.requirements.size` doesn't match the memory requirements of the resource.
/// - Panics if finding a suitable memory type failed. This only happens if the
/// `create_info.requirements` correspond to those of an optimal image but
/// `create_info.usage` is not [`MemoryUsage::GpuOnly`].
///
/// # Errors
///
/// - Returns an error if allocating a new block is required and failed. This can be one of the
/// OOM errors or [`TooManyObjects`].
/// - Returns [`OutOfPoolMemory`] if `create_info.allocate_preference` is
/// [`MemoryAllocatePreference::NeverAllocate`] and none of the pools of suitable memory
/// types have enough free space.
/// - Returns [`DedicatedAllocationRequired`] if `create_info.allocate_preference` is
/// [`MemoryAllocatePreference::NeverAllocate`] and
/// `create_info.requirements.requires_dedicated_allocation` is `true`.
/// - Returns [`BlockSizeExceeded`] if `create_info.allocate_preference` is
/// [`MemoryAllocatePreference::NeverAllocate`] and `create_info.requirements.size` is greater
/// than the block size for all heaps of suitable memory types.
/// - Returns [`SuballocatorBlockSizeExceeded`] if `S` is `PoolAllocator<BLOCK_SIZE>` and
/// `create_info.size` is greater than `BLOCK_SIZE` and a dedicated allocation was not
/// created.
///
/// [`device_local`]: MemoryPropertyFlags::device_local
/// [`host_visible`]: MemoryPropertyFlags::host_visible
/// [`TooManyObjects`]: VulkanError::TooManyObjects
/// [`SuballocatorBlockSizeExceeded`]: AllocationCreationError::SuballocatorBlockSizeExceeded
/// [`OutOfPoolMemory`]: AllocationCreationError::OutOfPoolMemory
/// [`DedicatedAllocationRequired`]: AllocationCreationError::DedicatedAllocationRequired
/// [`BlockSizeExceeded`]: AllocationCreationError::BlockSizeExceeded
fn allocate(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError> {
self.validate_allocate(&create_info);
unsafe { self.allocate_unchecked(create_info) }
}
unsafe fn allocate_unchecked(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError> {
let AllocationCreateInfo {
requirements:
MemoryRequirements {
size,
alignment: _,
mut memory_type_bits,
mut prefers_dedicated_allocation,
requires_dedicated_allocation,
},
allocation_type: _,
usage,
allocate_preference,
mut dedicated_allocation,
_ne: _,
} = create_info;
let create_info = SuballocationCreateInfo::from(create_info);
memory_type_bits &= self.memory_type_bits;
let filter = usage.into();
let mut memory_type_index = self
.find_memory_type_index(memory_type_bits, filter)
.expect("couldn't find a suitable memory type");
if !self.dedicated_allocation {
dedicated_allocation = None;
}
let export_handle_types = if self.export_handle_types.is_empty() {
ExternalMemoryHandleTypes::empty()
} else {
self.export_handle_types[memory_type_index as usize]
};
loop {
let memory_type = self.pools[memory_type_index as usize].memory_type;
let block_size = self.block_sizes[memory_type.heap_index as usize];
let res = match allocate_preference {
MemoryAllocatePreference::Unknown => {
if requires_dedicated_allocation {
self.allocate_dedicated_unchecked(
memory_type_index,
size,
dedicated_allocation,
export_handle_types,
)
} else {
if size > block_size / 2 {
prefers_dedicated_allocation = true;
}
if self.device.allocation_count() > self.max_allocations
&& size <= block_size
{
prefers_dedicated_allocation = false;
}
if prefers_dedicated_allocation {
self.allocate_dedicated_unchecked(
memory_type_index,
size,
dedicated_allocation,
export_handle_types,
)
// Fall back to suballocation.
.or_else(|err| {
if size <= block_size {
self.allocate_from_type_unchecked(
memory_type_index,
create_info.clone(),
true, // A dedicated allocation already failed.
)
.map_err(|_| err)
} else {
Err(err)
}
})
} else {
self.allocate_from_type_unchecked(
memory_type_index,
create_info.clone(),
false,
)
// Fall back to dedicated allocation. It is possible that the 1/8 block
// size tried was greater than the allocation size, so there's hope.
.or_else(|_| {
self.allocate_dedicated_unchecked(
memory_type_index,
size,
dedicated_allocation,
export_handle_types,
)
})
}
}
}
MemoryAllocatePreference::NeverAllocate => {
if requires_dedicated_allocation {
return Err(AllocationCreationError::DedicatedAllocationRequired);
}
self.allocate_from_type_unchecked(memory_type_index, create_info.clone(), true)
}
MemoryAllocatePreference::AlwaysAllocate => self.allocate_dedicated_unchecked(
memory_type_index,
size,
dedicated_allocation,
export_handle_types,
),
};
match res {
Ok(alloc) => return Ok(alloc),
// This is not recoverable.
Err(AllocationCreationError::SuballocatorBlockSizeExceeded) => {
return Err(AllocationCreationError::SuballocatorBlockSizeExceeded);
}
// Try a different memory type.
Err(err) => {
memory_type_bits &= !(1 << memory_type_index);
memory_type_index = self
.find_memory_type_index(memory_type_bits, filter)
.ok_or(err)?;
}
}
}
}
unsafe fn allocate_dedicated_unchecked(
&self,
memory_type_index: u32,
allocation_size: DeviceSize,
mut dedicated_allocation: Option<DedicatedAllocation<'_>>,
export_handle_types: ExternalMemoryHandleTypes,
) -> Result<MemoryAlloc, AllocationCreationError> {
// Providers of `VkMemoryDedicatedAllocateInfo`
if !(self.device.api_version() >= Version::V1_1
|| self.device.enabled_extensions().khr_dedicated_allocation)
{
dedicated_allocation = None;
}
let allocate_info = MemoryAllocateInfo {
allocation_size,
memory_type_index,
dedicated_allocation,
export_handle_types,
flags: self.flags,
..Default::default()
};
let mut alloc = MemoryAlloc::new(
DeviceMemory::allocate_unchecked(self.device.clone(), allocate_info, None)
.map_err(AllocationCreationError::from)?,
)?;
alloc.set_allocation_type(self.allocation_type);
Ok(alloc)
}
}
unsafe impl<S: Suballocator> MemoryAllocator for Arc<GenericMemoryAllocator<S>> {
fn find_memory_type_index(
&self,
memory_type_bits: u32,
filter: MemoryTypeFilter,
) -> Option<u32> {
(**self).find_memory_type_index(memory_type_bits, filter)
}
fn allocate_from_type(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
) -> Result<MemoryAlloc, AllocationCreationError> {
(**self).allocate_from_type(memory_type_index, create_info)
}
unsafe fn allocate_from_type_unchecked(
&self,
memory_type_index: u32,
create_info: SuballocationCreateInfo,
never_allocate: bool,
) -> Result<MemoryAlloc, AllocationCreationError> {
(**self).allocate_from_type_unchecked(memory_type_index, create_info, never_allocate)
}
fn allocate(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError> {
(**self).allocate(create_info)
}
unsafe fn allocate_unchecked(
&self,
create_info: AllocationCreateInfo<'_>,
) -> Result<MemoryAlloc, AllocationCreationError> {
(**self).allocate_unchecked(create_info)
}
unsafe fn allocate_dedicated_unchecked(
&self,
memory_type_index: u32,
allocation_size: DeviceSize,
dedicated_allocation: Option<DedicatedAllocation<'_>>,
export_handle_types: ExternalMemoryHandleTypes,
) -> Result<MemoryAlloc, AllocationCreationError> {
(**self).allocate_dedicated_unchecked(
memory_type_index,
allocation_size,
dedicated_allocation,
export_handle_types,
)
}
}
unsafe impl<S: Suballocator> DeviceOwned for GenericMemoryAllocator<S> {
fn device(&self) -> &Arc<Device> {
&self.device
}
}
/// Parameters to create a new [`GenericMemoryAllocator`].
#[derive(Clone, Debug)]
pub struct GenericMemoryAllocatorCreateInfo<'b, 'e> {
/// Lets you configure the block sizes for various heap size classes.
///
/// Each entry is a pair of the threshold for the heap size and the block size that should be
/// used for that heap. Must be sorted by threshold and all thresholds must be unique. Must
/// contain a baseline threshold of 0.
///
/// The allocator keeps a pool of [`DeviceMemory`] blocks for each memory type, so each memory
/// type that resides in a heap whose size crosses one of the thresholds will use the
/// corresponding block size. If multiple thresholds apply to a given heap, the block size
/// corresponding to the largest threshold is chosen.
///
/// The block size is going to be the maximum size of a `DeviceMemory` block that is tried. If
/// allocating a block with the size fails, the allocator tries 1/2, 1/4 and 1/8 of the block
/// size in that order until one succeeds, else a dedicated allocation is attempted for the
/// allocation. If an allocation is created with a size greater than half the block size it is
/// always made a dedicated allocation. All of this doesn't apply when using
/// [`MemoryAllocatePreference::NeverAllocate`] however.
///
/// The default value is `&[]`, which must be overridden.
pub block_sizes: &'b [(Threshold, BlockSize)],
/// The allocation type that should be used for root allocations.
///
/// You only need to worry about this if you're using [`PoolAllocator`] as the suballocator, as
/// all suballocations that the pool allocator makes inherit their allocation type from the
/// parent allocation. For the [`FreeListAllocator`] and the [`BuddyAllocator`] this must be
/// [`AllocationType::Unknown`] otherwise you will get panics. It does not matter what this is
/// when using the [`BumpAllocator`].
///
/// The default value is [`AllocationType::Unknown`].
pub allocation_type: AllocationType,
/// Whether the allocator should use the dedicated allocation APIs.
///
/// This means that when the allocator decides that an allocation should not be suballocated,
/// but rather have its own block of [`DeviceMemory`], that that allocation will be made a
/// dedicated allocation. Otherwise they are still made free-standing ([root]) allocations,
/// just not [dedicated] ones.
///
/// Dedicated allocations are an optimization which may result in better performance, so there
/// really is no reason to disable this option, unless the restrictions that they bring with
/// them are a problem. Namely, a dedicated allocation must only be used for the resource it
/// was created for. Meaning that [reusing the memory] for something else is not possible,
/// [suballocating it] is not possible, and [aliasing it] is also not possible.
///
/// This option is silently ignored (treated as `false`) if the device API version is below 1.1
/// and the [`khr_dedicated_allocation`] extension is not enabled on the device.
///
/// The default value is `true`.
///
/// [root]: MemoryAlloc::is_root
/// [dedicated]: MemoryAlloc::is_dedicated
/// [reusing the memory]: MemoryAlloc::try_unwrap
/// [suballocating it]: Suballocator
/// [aliasing it]: MemoryAlloc::alias
/// [`khr_dedicated_allocation`]: crate::device::DeviceExtensions::khr_dedicated_allocation
pub dedicated_allocation: bool,
/// Lets you configure the external memory handle types that the [`DeviceMemory`] blocks will
/// be allocated with.
///
/// Must be either empty or contain one element for each memory type. When `DeviceMemory` is
/// allocated, the external handle types corresponding to the memory type index are looked up
/// here and used for the allocation.
///
/// The default value is `&[]`.
pub export_handle_types: &'e [ExternalMemoryHandleTypes],
/// Whether the allocator should allocate the [`DeviceMemory`] blocks with the
/// [`device_address`] flag set.
///
/// This is required if you want to allocate memory for buffers that have the
/// [`shader_device_address`] usage set. For this option too, there is no reason to disable it.
///
/// This option is silently ignored (treated as `false`) if the [`buffer_device_address`]
/// feature is not enabled on the device or if the [`ext_buffer_device_address`] extension is
/// enabled on the device. It is also ignored if the device API version is below 1.1 and the
/// [`khr_device_group`] extension is not enabled on the device.
///
/// The default value is `true`.
///
/// [`device_address`]: MemoryAllocateFlags::device_address
/// [`shader_device_address`]: crate::buffer::BufferUsage::shader_device_address
/// [`buffer_device_address`]: crate::device::Features::buffer_device_address
/// [`ext_buffer_device_address`]: crate::device::DeviceExtensions::ext_buffer_device_address
/// [`khr_device_group`]: crate::device::DeviceExtensions::khr_device_group
pub device_address: bool,
pub _ne: crate::NonExhaustive,
}
pub type Threshold = DeviceSize;
pub type BlockSize = DeviceSize;
impl Default for GenericMemoryAllocatorCreateInfo<'_, '_> {
#[inline]
fn default() -> Self {
GenericMemoryAllocatorCreateInfo {
block_sizes: &[],
allocation_type: AllocationType::Unknown,
dedicated_allocation: true,
export_handle_types: &[],
device_address: true,
_ne: crate::NonExhaustive(()),
}
}
}
/// Error that can be returned when creating a [`GenericMemoryAllocator`].
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum GenericMemoryAllocatorCreationError {
RequirementNotMet {
required_for: &'static str,
requires_one_of: RequiresOneOf,
},
}
impl Error for GenericMemoryAllocatorCreationError {}
impl Display for GenericMemoryAllocatorCreationError {
fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), FmtError> {
match self {
Self::RequirementNotMet {
required_for,
requires_one_of,
} => write!(
f,
"a requirement was not met for: {}; requires one of: {}",
required_for, requires_one_of,
),
}
}
}
impl From<RequirementNotMet> for GenericMemoryAllocatorCreationError {
fn from(err: RequirementNotMet) -> Self {
Self::RequirementNotMet {
required_for: err.required_for,
requires_one_of: err.requires_one_of,
}
}
}
mod array_vec {
use std::ops::{Deref, DerefMut};
/// Minimal implementation of an `ArrayVec`. Useful when a `Vec` is needed but there is a known
/// limit on the number of elements, so that it can occupy real estate on the stack.
#[derive(Clone, Copy, Debug)]
pub(super) struct ArrayVec<T, const N: usize> {
len: usize,
data: [T; N],
}
impl<T, const N: usize> ArrayVec<T, N> {
pub fn new(len: usize, data: [T; N]) -> Self {
assert!(len <= N);
ArrayVec { len, data }
}
}
impl<T, const N: usize> Deref for ArrayVec<T, N> {
type Target = [T];
fn deref(&self) -> &Self::Target {
// SAFETY: `self.len <= N`.
unsafe { self.data.get_unchecked(0..self.len) }
}
}
impl<T, const N: usize> DerefMut for ArrayVec<T, N> {
fn deref_mut(&mut self) -> &mut Self::Target {
// SAFETY: `self.len <= N`.
unsafe { self.data.get_unchecked_mut(0..self.len) }
}
}
}