Expand description
🗜 presser
Utilities to help make copying data around into raw, possibly-uninitialized buffers easier and safer.
Motivation
presser
can help you when copying data into and reading data out of raw buffers. One primary use-case is copying data into
graphics-api-allocated buffers which will then be accessed by the GPU. Common methods for doing this
right now in Rust can often invoke UB in subtle and hard-to-see ways. For example, viewing an allocated
but uninitialized buffer as an &mut [u8]
is instantly undefined behavior*, and transmute
ing even a
T: Copy
type which has any padding bytes in its layout as a &[u8]
to be the source of a copy is
also instantly undefined behavior, in both cases because it is invalid to create a reference to an invalid
value (and uninitialized memory is an invalid u8
), even if your code never actually accesses that memory.
This immediately makes what seems like the most straightforward way to copy data into buffers unsound 😬
presser
helps with this by allowing you to view raw allocated memory of some size as a “Slab
” of memory and then
provides safe, valid ways to copy data into that memory, and helpers to make reading data back out of that buffer
less fraight. For example, you could implement Slab
for your
GPU-allocated buffer type, or use the built-in RawAllocation
workflow described below, then use
copy_to_offset_with_align
to copy any T: Copy
data into that buffer safely for use on the GPU.
Of course, if your T
doesn’t have the correct layout the GPU expects, accessing it on the GPU side may still be
unsound or at least give an error.
* If you’re currently thinking to yourself “bah! what’s the issue? surely an uninit u8 is just any random bit pattern
and that’s fine we don’t care,” check out this blog post by
@RalfJung, one of the people leading the effort to better define Rust’s memory and execution model. As is explored
in that blog post, an uninit piece of memory is not simply an arbitrary bit pattern, it is a wholly separate
state about a piece of memory, outside of its value, which lets the compiler perform optimizations that reorder,
delete, and otherwise change the actual execution flow of your program in ways that cannot be described simply
by “the value could have some possible bit pattern”. LLVM and Clang are changing themselves to require special
noundef
attribute to perform many important optimizations that are otherwise unsound. For a concrete example
of the sorts of problems this can cause,
see this issue @scottmcm hit.
Introduction
The main idea is to implement Slab
on raw-buffer-esque-types (see the Slab
safety docs),
which then enables the use of the other functions within the crate.
For built-in slab types, see RawAllocation
, HeapSlab
, and make_stack_slab
.
Depending on your use case, you may be able to implement Slab
directly for your buffer type, or it may
be more convenient or necessary to create a wrapping struct that borrows your raw buffer type and in turn
implements Slab
. For an example of this, see RawAllocation
and BorrowedRawAllocation
, which you
may also use directly. The idea is to create a RawAllocation
to your buffer, which you then borrow into
a BorrowedRawAllocation
(which implements Slab
) by calling the unsafe function
RawAllocation::borrow_as_slab
Once you have a slab, you can use the helper functions provided at the crate root, for example,
copy_to_offset
and copy_to_offset_with_align
.
Example
#[derive(Clone, Copy)]
#[repr(C)]
struct MyDataStruct {
a: u8,
b: u32,
}
let my_data = MyDataStruct { a: 0, b: 42 };
// allocate an uninit buffer of some size
let my_buffer: MyBufferType = some_api.alloc_buffer_size(2048);
// use `RawAllocation` helper to allow access to a presser `Slab`.
// alternatively, you could implement the `Slab` on `MyByfferType` directly if that
// type is owned by your code!
let raw_allocation = presser::RawAllocation::from_raw_parts(my_buffer.ptr(), my_buffer.size());
// here we assert that we have exclusive access to the data in the buffer, and get the actual
// `Slab` to use to copy into.
let slab = unsafe { raw_allocation.borrow_as_slab(); }
// now we may safely copy `my_data` into `my_buffer`, starting at a minimum offset of 0 into the buffer
let copy_record = presser::copy_to_offset(&my_data, &mut slab, 0)?;
// note that due to alignment requirements of `my_data`, the *actual* start of the bytes of
// `my_data` may be placed at a different offset than requested. so, we check the returned
// `CopyRecord` to check the actual start offset of the copied data.
let actual_start_offset = copy_record.copy_start_offset;
// we may later (*unsafely*) read back our data. note that the read helpers provided by presser
// are mostly unsafe. They do help protect you from some common footguns, but you still ultimately need
// to guarantee you put the proper data where you're telling it you put the proper data.
let my_copied_data_in_my_buffer: &MyDataStruct = unsafe {
presser::read_at_offset(&slab, actual_start_offset)?
};
#[no_std]
This crate supports no_std
environments by building without the ‘std
’ feature. This will limit some
of the fuctions the crate can perform.
Safety
An important note is that obeying the safety rules specified in the Slab
safety documentation
only guarantees safety for the direct results of the copy operations performed by the
helper functions exported at the crate root (and the safe functions on Slab
). However,
it is not guaranteed that operations which would previously have been safe to perform
using same backing memory that the Slab
you copied into used are still safe.
For example, say you have a fully-initialized
chunk of bytes (like a Vec<u8>
), which you (unsafely*) view as a Slab
, and then (safely) perform a copy
operation into using copy_to_offset
. If the T
you copied into it has any padding bytes in
its memory layout, then the memory locations where those padding bytes now exist in the underlying Vec
’s
memory must now be treated as uninitialized. As such, taking any view into that byte vector which
relies on those newly-uninit bytes being initialized to be valid (for example, taking a &[u8]
slice of the Vec
which includes those bytes, even if your code never actually reads from that slice)
is now instant undefined behavior.
* Note: this is unsafe because, as exemplified, you may copy uninit data into the buffer. Hence, care should
be taken when implementing Slab
and then providing a safe interface on top of a low level buffer type.
Structs
- Represents the unique borrow of a contiguous piece of a single allocation with some layout that is used as a data copying destination. May be wholly or partially uninitialized.
- Record of the results of a copy operation
- HeapSlab
std
- Represents a contiguous piece of a single allocation with some layout. May be wholly or partially uninitialized.
Enums
- An error that may occur during a copy or read operation.
Traits
- Represents a contiguous piece of a single allocation with some layout that is used as a data copying destination or reading source. May be wholly or partially uninitialized.
Functions
- Clones the elements from
src
todst
, returning a mutable reference to the now initialized contents ofdst
. Any already initialized elements will not be dropped. - Copies from
src
iterator into the memory represented bydst
starting at a minimum location ofstart_offset
bytes past the start ofdst
. - Like
copy_from_iter_to_offset_with_align_packed
except that it will return an error and no data will be copied if the suppliedstart_offset
doesn’t meet the computed alignment requirements. - Like
copy_from_iter_to_offset_with_align
except that alignment between elements yielded by the iterator will ignoremin_alignment
and rather only be aligned to the alignment ofT
. - Copies from
slice
into the memory represented bydst
starting at a minimum location ofstart_offset
bytes past the start ofself
. - Copies from
slice
into the memory represented bydst
starting at exactlystart_offset
bytes past the start ofself
. - Copies from
slice
into the memory represented bydst
starting at a minimum location ofstart_offset
bytes past the start ofdst
. - Copies from
slice
into the memory represented bydst
starting at exactlystart_offset
bytes past the start ofdst
and with minimum alignmentmin_alignment
. - Copies the elements from
src
todst
, returning a mutable reference to the now initialized contents ofdst
. - Copies
src
into the memory represented bydst
starting at a minimum location ofstart_offset
bytes past the start ofdst
. - Copies
src
into the memory represented bydst
starting at exactlystart_offset
bytes past the start ofdst
- Copies
src
into the memory represented bydst
starting at a minimum location ofstart_offset
bytes past the start ofdst
and with minimum alignmentmin_alignment
. - Copies
src
into the memory represented bydst
starting at exactlystart_offset
bytes past the start ofdst
and with minimum alignmentmin_alignment
. If the requested parameters would be violated by computed alignment requirements, an error will be returned. - Gets a mutable reference to a
MaybeUninit<T>
withinslab
atoffset
. - Gets a mutable reference to a
MaybeUninit<T>
withinslab
atoffset
, not checking any requirements. - Gets a
&mut [MaybeUninit<T>]
withinslab
atoffset
. - Gets a
&mut [MaybeUninit<T>]
withinslab
atoffset
, not checking any requirements. - Make a
[MaybeUninit<T>; N]
on the stack, which implementsSlab
and can therefore be used with many of the helpers provided by this crate. - Takes a
Vec
and unsafely resizes it to the given length, returning a mutable slice toMaybeUninit<T>
for each item in the newly-resizedVec
. - Gets a shared reference to a
T
withinslab
atoffset
. - Gets a mutable reference to a
T
withinslab
atoffset
. - Gets a mutable reference to a
T
withinslab
atoffset
, not checking any requirements. - Gets a shared reference to a
T
withinslab
atoffset
, not checking any requirements. - Reads a
&[T]
withinslab
atoffset
. - Reads a
&mut [T]
withinslab
atoffset
. - Reads a
&mut [T]
withinslab
atoffset
, not checking any requirements. - Reads a
&[T]
withinslab
atoffset
, not checking any requirements. - Helper to read back data from an ffi function which expects a pointer into which it will write a
T
. - Helper to read back data from an ffi function which expects a pointer into which it will write a slice (in C language, an array) of
T
s.