1//// 2Copyright 2011-2016 Beman Dawes 3 4Distributed under the Boost Software License, Version 1.0. 5(http://www.boost.org/LICENSE_1_0.txt) 6//// 7 8[#buffers] 9# Endian Buffer Types 10:idprefix: buffers_ 11 12## Introduction 13 14The internal byte order of arithmetic types is traditionally called 15*endianness*. See the http://en.wikipedia.org/wiki/Endian[Wikipedia] for a full 16exploration of *endianness*, including definitions of *big endian* and *little 17endian*. 18 19Header `boost/endian/buffers.hpp` provides `endian_buffer`, a portable endian 20integer binary buffer class template with control over byte order, value type, 21size, and alignment independent of the platform's native endianness. Typedefs 22provide easy-to-use names for common configurations. 23 24Use cases primarily involve data portability, either via files or network 25connections, but these byte-holders may also be used to reduce memory use, file 26size, or network activity since they provide binary numeric sizes not otherwise 27available. 28 29Class `endian_buffer` is aimed at users who wish explicit control over when 30endianness conversions occur. It also serves as the base class for the 31<<arithmetic,endian_arithmetic>> class template, which is aimed at users who 32wish fully automatic endianness conversion and direct support for all normal 33arithmetic operations. 34 35## Example 36 37The `example/endian_example.cpp` program writes a binary file containing 38four-byte, big-endian and little-endian integers: 39 40``` 41#include <iostream> 42#include <cstdio> 43#include <boost/endian/buffers.hpp> // see Synopsis below 44#include <boost/static_assert.hpp> 45 46using namespace boost::endian; 47 48namespace 49{ 50 // This is an extract from a very widely used GIS file format. 51 // Why the designer decided to mix big and little endians in 52 // the same file is not known. But this is a real-world format 53 // and users wishing to write low level code manipulating these 54 // files have to deal with the mixed endianness. 55 56 struct header 57 { 58 big_int32_buf_t file_code; 59 big_int32_buf_t file_length; 60 little_int32_buf_t version; 61 little_int32_buf_t shape_type; 62 }; 63 64 const char* filename = "test.dat"; 65} 66 67int main(int, char* []) 68{ 69 header h; 70 71 BOOST_STATIC_ASSERT(sizeof(h) == 16U); // reality check 72 73 h.file_code = 0x01020304; 74 h.file_length = sizeof(header); 75 h.version = 1; 76 h.shape_type = 0x01020304; 77 78 // Low-level I/O such as POSIX read/write or <cstdio> 79 // fread/fwrite is sometimes used for binary file operations 80 // when ultimate efficiency is important. Such I/O is often 81 // performed in some C++ wrapper class, but to drive home the 82 // point that endian integers are often used in fairly 83 // low-level code that does bulk I/O operations, <cstdio> 84 // fopen/fwrite is used for I/O in this example. 85 86 std::FILE* fi = std::fopen(filename, "wb"); // MUST BE BINARY 87 88 if (!fi) 89 { 90 std::cout << "could not open " << filename << '\n'; 91 return 1; 92 } 93 94 if (std::fwrite(&h, sizeof(header), 1, fi) != 1) 95 { 96 std::cout << "write failure for " << filename << '\n'; 97 return 1; 98 } 99 100 std::fclose(fi); 101 102 std::cout << "created file " << filename << '\n'; 103 104 return 0; 105} 106``` 107 108After compiling and executing `example/endian_example.cpp`, a hex dump of 109`test.dat` shows: 110 111``` 11201020304 00000010 01000000 04030201 113``` 114 115Notice that the first two 32-bit integers are big endian while the second two 116are little endian, even though the machine this was compiled and run on was 117little endian. 118 119## Limitations 120 121Requires `<climits>`, `CHAR_BIT == 8`. If `CHAR_BIT` is some other value, 122compilation will result in an `#error`. This restriction is in place because the 123design, implementation, testing, and documentation has only considered issues 124related to 8-bit bytes, and there have been no real-world use cases presented 125for other sizes. 126 127In {cpp}03, `endian_buffer` does not meet the requirements for POD types because 128it has constructors and a private data member. This means that 129common use cases are relying on unspecified behavior in that the {cpp} Standard 130does not guarantee memory layout for non-POD types. This has not been a problem 131in practice since all known {cpp} compilers lay out memory as if `endian` were 132a POD type. In {cpp}11, it is possible to specify the default constructor as 133trivial, and private data members and base classes no longer disqualify a type 134from being a POD type. Thus under {cpp}11, `endian_buffer` will no longer be 135relying on unspecified behavior. 136 137## Feature set 138 139* Big endian| little endian | native endian byte ordering. 140* Signed | unsigned 141* Unaligned | aligned 142* 1-8 byte (unaligned) | 1, 2, 4, 8 byte (aligned) 143* Choice of value type 144 145## Enums and typedefs 146 147Two scoped enums are provided: 148 149``` 150enum class order { big, little, native }; 151 152enum class align { no, yes }; 153``` 154 155One class template is provided: 156 157``` 158template <order Order, typename T, std::size_t Nbits, 159 align Align = align::no> 160class endian_buffer; 161``` 162 163Typedefs, such as `big_int32_buf_t`, provide convenient naming conventions for 164common use cases: 165 166[%header,cols=5*] 167|=== 168|Name |Alignment |Endianness |Sign |Sizes in bits (n) 169|`big_intN_buf_t` |no |big |signed |8,16,24,32,40,48,56,64 170|`big_uintN_buf_t` |no |big |unsigned |8,16,24,32,40,48,56,64 171|`little_intN_buf_t` |no |little |signed |8,16,24,32,40,48,56,64 172|`little_uintN_buf_t` |no |little |unsigned |8,16,24,32,40,48,56,64 173|`native_intN_buf_t` |no |native |signed |8,16,24,32,40,48,56,64 174|`native_uintN_buf_t` |no |native |unsigned |8,16,24,32,40,48,56,64 175|`big_intN_buf_at` |yes |big |signed |8,16,32,64 176|`big_uintN_buf_at` |yes |big |unsigned |8,16,32,64 177|`little_intN_buf_at` |yes |little |signed |8,16,32,64 178|`little_uintN_buf_at` |yes |little |unsigned |8,16,32,64 179|=== 180 181The unaligned types do not cause compilers to insert padding bytes in classes 182and structs. This is an important characteristic that can be exploited to 183minimize wasted space in memory, files, and network transmissions. 184 185CAUTION: Code that uses aligned types is possibly non-portable because alignment 186requirements vary between hardware architectures and because alignment may be 187affected by compiler switches or pragmas. For example, alignment of an 64-bit 188integer may be to a 32-bit boundary on a 32-bit machine and to a 64-bit boundary 189on a 64-bit machine. Furthermore, aligned types are only available on 190architectures with 8, 16, 32, and 64-bit integer types. 191 192TIP: Prefer unaligned buffer types. 193 194TIP: Protect yourself against alignment ills. For example: 195[none] 196{blank}:: 197+ 198``` 199static_assert(sizeof(containing_struct) == 12, "sizeof(containing_struct) is wrong"); 200``` 201 202Note: One-byte big and little buffer types have identical layout on all 203platforms, so they never actually reverse endianness. They are provided to 204enable generic code, and to improve code readability and searchability. 205 206## Class template `endian_buffer` 207 208An `endian_buffer` is a byte-holder for arithmetic types with 209user-specified endianness, value type, size, and alignment. 210 211### Synopsis 212 213``` 214namespace boost 215{ 216 namespace endian 217 { 218 // C++11 features emulated if not available 219 220 enum class align { no, yes }; 221 222 template <order Order, class T, std::size_t Nbits, 223 align Align = align::no> 224 class endian_buffer 225 { 226 public: 227 228 typedef T value_type; 229 230 endian_buffer() noexcept = default; 231 explicit endian_buffer(T v) noexcept; 232 233 endian_buffer& operator=(T v) noexcept; 234 value_type value() const noexcept; 235 unsigned char* data() noexcept; 236 unsigned char const* data() const noexcept; 237 238 private: 239 240 unsigned char value_[Nbits / CHAR_BIT]; // exposition only 241 }; 242 243 // stream inserter 244 template <class charT, class traits, order Order, class T, 245 std::size_t n_bits, align Align> 246 std::basic_ostream<charT, traits>& 247 operator<<(std::basic_ostream<charT, traits>& os, 248 const endian_buffer<Order, T, n_bits, Align>& x); 249 250 // stream extractor 251 template <class charT, class traits, order Order, class T, 252 std::size_t n_bits, align A> 253 std::basic_istream<charT, traits>& 254 operator>>(std::basic_istream<charT, traits>& is, 255 endian_buffer<Order, T, n_bits, Align>& x); 256 257 // typedefs 258 259 // unaligned big endian signed integer buffers 260 typedef endian_buffer<order::big, int_least8_t, 8> big_int8_buf_t; 261 typedef endian_buffer<order::big, int_least16_t, 16> big_int16_buf_t; 262 typedef endian_buffer<order::big, int_least32_t, 24> big_int24_buf_t; 263 typedef endian_buffer<order::big, int_least32_t, 32> big_int32_buf_t; 264 typedef endian_buffer<order::big, int_least64_t, 40> big_int40_buf_t; 265 typedef endian_buffer<order::big, int_least64_t, 48> big_int48_buf_t; 266 typedef endian_buffer<order::big, int_least64_t, 56> big_int56_buf_t; 267 typedef endian_buffer<order::big, int_least64_t, 64> big_int64_buf_t; 268 269 // unaligned big endian unsigned integer buffers 270 typedef endian_buffer<order::big, uint_least8_t, 8> big_uint8_buf_t; 271 typedef endian_buffer<order::big, uint_least16_t, 16> big_uint16_buf_t; 272 typedef endian_buffer<order::big, uint_least32_t, 24> big_uint24_buf_t; 273 typedef endian_buffer<order::big, uint_least32_t, 32> big_uint32_buf_t; 274 typedef endian_buffer<order::big, uint_least64_t, 40> big_uint40_buf_t; 275 typedef endian_buffer<order::big, uint_least64_t, 48> big_uint48_buf_t; 276 typedef endian_buffer<order::big, uint_least64_t, 56> big_uint56_buf_t; 277 typedef endian_buffer<order::big, uint_least64_t, 64> big_uint64_buf_t; 278 279 // unaligned big endian floating point buffers 280 typedef endian_buffer<order::big, float, 32> big_float32_buf_t; 281 typedef endian_buffer<order::big, double, 64> big_float64_buf_t; 282 283 // unaligned little endian signed integer buffers 284 typedef endian_buffer<order::little, int_least8_t, 8> little_int8_buf_t; 285 typedef endian_buffer<order::little, int_least16_t, 16> little_int16_buf_t; 286 typedef endian_buffer<order::little, int_least32_t, 24> little_int24_buf_t; 287 typedef endian_buffer<order::little, int_least32_t, 32> little_int32_buf_t; 288 typedef endian_buffer<order::little, int_least64_t, 40> little_int40_buf_t; 289 typedef endian_buffer<order::little, int_least64_t, 48> little_int48_buf_t; 290 typedef endian_buffer<order::little, int_least64_t, 56> little_int56_buf_t; 291 typedef endian_buffer<order::little, int_least64_t, 64> little_int64_buf_t; 292 293 // unaligned little endian unsigned integer buffers 294 typedef endian_buffer<order::little, uint_least8_t, 8> little_uint8_buf_t; 295 typedef endian_buffer<order::little, uint_least16_t, 16> little_uint16_buf_t; 296 typedef endian_buffer<order::little, uint_least32_t, 24> little_uint24_buf_t; 297 typedef endian_buffer<order::little, uint_least32_t, 32> little_uint32_buf_t; 298 typedef endian_buffer<order::little, uint_least64_t, 40> little_uint40_buf_t; 299 typedef endian_buffer<order::little, uint_least64_t, 48> little_uint48_buf_t; 300 typedef endian_buffer<order::little, uint_least64_t, 56> little_uint56_buf_t; 301 typedef endian_buffer<order::little, uint_least64_t, 64> little_uint64_buf_t; 302 303 // unaligned little endian floating point buffers 304 typedef endian_buffer<order::little, float, 32> little_float32_buf_t; 305 typedef endian_buffer<order::little, double, 64> little_float64_buf_t; 306 307 // unaligned native endian signed integer types 308 typedef endian_buffer<order::native, int_least8_t, 8> native_int8_buf_t; 309 typedef endian_buffer<order::native, int_least16_t, 16> native_int16_buf_t; 310 typedef endian_buffer<order::native, int_least32_t, 24> native_int24_buf_t; 311 typedef endian_buffer<order::native, int_least32_t, 32> native_int32_buf_t; 312 typedef endian_buffer<order::native, int_least64_t, 40> native_int40_buf_t; 313 typedef endian_buffer<order::native, int_least64_t, 48> native_int48_buf_t; 314 typedef endian_buffer<order::native, int_least64_t, 56> native_int56_buf_t; 315 typedef endian_buffer<order::native, int_least64_t, 64> native_int64_buf_t; 316 317 // unaligned native endian unsigned integer types 318 typedef endian_buffer<order::native, uint_least8_t, 8> native_uint8_buf_t; 319 typedef endian_buffer<order::native, uint_least16_t, 16> native_uint16_buf_t; 320 typedef endian_buffer<order::native, uint_least32_t, 24> native_uint24_buf_t; 321 typedef endian_buffer<order::native, uint_least32_t, 32> native_uint32_buf_t; 322 typedef endian_buffer<order::native, uint_least64_t, 40> native_uint40_buf_t; 323 typedef endian_buffer<order::native, uint_least64_t, 48> native_uint48_buf_t; 324 typedef endian_buffer<order::native, uint_least64_t, 56> native_uint56_buf_t; 325 typedef endian_buffer<order::native, uint_least64_t, 64> native_uint64_buf_t; 326 327 // unaligned native endian floating point types 328 typedef endian_buffer<order::native, float, 32> native_float32_buf_t; 329 typedef endian_buffer<order::native, double, 64> native_float64_buf_t; 330 331 // aligned big endian signed integer buffers 332 typedef endian_buffer<order::big, int8_t, 8, align::yes> big_int8_buf_at; 333 typedef endian_buffer<order::big, int16_t, 16, align::yes> big_int16_buf_at; 334 typedef endian_buffer<order::big, int32_t, 32, align::yes> big_int32_buf_at; 335 typedef endian_buffer<order::big, int64_t, 64, align::yes> big_int64_buf_at; 336 337 // aligned big endian unsigned integer buffers 338 typedef endian_buffer<order::big, uint8_t, 8, align::yes> big_uint8_buf_at; 339 typedef endian_buffer<order::big, uint16_t, 16, align::yes> big_uint16_buf_at; 340 typedef endian_buffer<order::big, uint32_t, 32, align::yes> big_uint32_buf_at; 341 typedef endian_buffer<order::big, uint64_t, 64, align::yes> big_uint64_buf_at; 342 343 // aligned big endian floating point buffers 344 typedef endian_buffer<order::big, float, 32, align::yes> big_float32_buf_at; 345 typedef endian_buffer<order::big, double, 64, align::yes> big_float64_buf_at; 346 347 // aligned little endian signed integer buffers 348 typedef endian_buffer<order::little, int8_t, 8, align::yes> little_int8_buf_at; 349 typedef endian_buffer<order::little, int16_t, 16, align::yes> little_int16_buf_at; 350 typedef endian_buffer<order::little, int32_t, 32, align::yes> little_int32_buf_at; 351 typedef endian_buffer<order::little, int64_t, 64, align::yes> little_int64_buf_at; 352 353 // aligned little endian unsigned integer buffers 354 typedef endian_buffer<order::little, uint8_t, 8, align::yes> little_uint8_buf_at; 355 typedef endian_buffer<order::little, uint16_t, 16, align::yes> little_uint16_buf_at; 356 typedef endian_buffer<order::little, uint32_t, 32, align::yes> little_uint32_buf_at; 357 typedef endian_buffer<order::little, uint64_t, 64, align::yes> little_uint64_buf_at; 358 359 // aligned little endian floating point buffers 360 typedef endian_buffer<order::little, float, 32, align::yes> little_float32_buf_at; 361 typedef endian_buffer<order::little, double, 64, align::yes> little_float64_buf_at; 362 363 // aligned native endian typedefs are not provided because 364 // <cstdint> types are superior for this use case 365 366 } // namespace endian 367} // namespace boost 368``` 369 370The expository data member `value_` stores the current value of the 371`endian_buffer` object as a sequence of bytes ordered as specified by the 372`Order` template parameter. The `CHAR_BIT` macro is defined in `<climits>`. 373The only supported value of `CHAR_BIT` is 8. 374 375The valid values of `Nbits` are as follows: 376 377* When `sizeof(T)` is 1, `Nbits` shall be 8; 378* When `sizeof(T)` is 2, `Nbits` shall be 16; 379* When `sizeof(T)` is 4, `Nbits` shall be 24 or 32; 380* When `sizeof(T)` is 8, `Nbits` shall be 40, 48, 56, or 64. 381 382Other values of `sizeof(T)` are not supported. 383 384When `Nbits` is equal to `sizeof(T)*8`, `T` must be a trivially copyable type 385(such as `float`) that is assumed to have the same endianness as `uintNbits_t`. 386 387When `Nbits` is less than `sizeof(T)*8`, `T` must be either a standard integral 388type ({cpp}std, [basic.fundamental]) or an `enum`. 389 390### Members 391 392``` 393endian_buffer() noexcept = default; 394``` 395[none] 396* {blank} 397+ 398Effects:: Constructs an uninitialized object. 399 400``` 401explicit endian_buffer(T v) noexcept; 402``` 403[none] 404* {blank} 405+ 406Effects:: `endian_store<T, Nbits/8, Order>( value_, v )`. 407 408``` 409endian_buffer& operator=(T v) noexcept; 410``` 411[none] 412* {blank} 413+ 414Effects:: `endian_store<T, Nbits/8, Order>( value_, v )`. 415Returns:: `*this`. 416 417``` 418value_type value() const noexcept; 419``` 420[none] 421* {blank} 422+ 423Returns:: `endian_load<T, Nbits/8, Order>( value_ )`. 424 425``` 426unsigned char* data() noexcept; 427``` 428``` 429unsigned char const* data() const noexcept; 430``` 431[none] 432* {blank} 433+ 434Returns:: 435 A pointer to the first byte of `value_`. 436 437### Non-member functions 438 439``` 440template <class charT, class traits, order Order, class T, 441 std::size_t n_bits, align Align> 442std::basic_ostream<charT, traits>& operator<<(std::basic_ostream<charT, traits>& os, 443 const endian_buffer<Order, T, n_bits, Align>& x); 444``` 445[none] 446* {blank} 447+ 448Returns:: `os << x.value()`. 449 450``` 451template <class charT, class traits, order Order, class T, 452 std::size_t n_bits, align A> 453std::basic_istream<charT, traits>& operator>>(std::basic_istream<charT, traits>& is, 454 endian_buffer<Order, T, n_bits, Align>& x); 455``` 456[none] 457* {blank} 458+ 459Effects:: As if: 460+ 461``` 462T i; 463if (is >> i) 464 x = i; 465``` 466Returns:: `is`. 467 468## FAQ 469 470See the <<overview_faq,Overview FAQ>> for a library-wide FAQ. 471 472Why not just use Boost.Serialization?:: 473Serialization involves a conversion for every object involved in I/O. Endian 474integers require no conversion or copying. They are already in the desired 475format for binary I/O. Thus they can be read or written in bulk. 476 477Are endian types PODs?:: 478Yes for {cpp}11. No for {cpp}03, although several 479<<buffers_compilation,macros>> are available to force PODness in all cases. 480 481What are the implications of endian integer types not being PODs with {cpp}03 compilers?:: 482They can't be used in unions. Also, compilers aren't required to align or lay 483out storage in portable ways, although this potential problem hasn't prevented 484use of Boost.Endian with real compilers. 485 486What good is native endianness?:: 487It provides alignment and size guarantees not available from the built-in 488types. It eases generic programming. 489 490Why bother with the aligned endian types?:: 491Aligned integer operations may be faster (as much as 10 to 20 times faster) if 492the endianness and alignment of the type matches the endianness and alignment 493requirements of the machine. The code, however, is likely to be somewhat less 494portable than with the unaligned types. 495 496## Design considerations for Boost.Endian buffers 497 498* Must be suitable for I/O - in other words, must be memcpyable. 499* Must provide exactly the size and internal byte ordering specified. 500* Must work correctly when the internal integer representation has more bits 501that the sum of the bits in the external byte representation. Sign extension 502must work correctly when the internal integer representation type has more 503bits than the sum of the bits in the external bytes. For example, using 504a 64-bit integer internally to represent 40-bit (5 byte) numbers must work for 505both positive and negative values. 506* Must work correctly (including using the same defined external 507representation) regardless of whether a compiler treats char as signed or 508unsigned. 509* Unaligned types must not cause compilers to insert padding bytes. 510* The implementation should supply optimizations with great care. Experience 511has shown that optimizations of endian integers often become pessimizations 512when changing machines or compilers. Pessimizations can also happen when 513changing compiler switches, compiler versions, or CPU models of the same 514architecture. 515 516## {cpp}11 517 518The availability of the {cpp}11 519http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted 520Functions] feature is detected automatically, and will be used if present to 521ensure that objects of `class endian_buffer` are trivial, and thus 522PODs. 523 524## Compilation 525 526Boost.Endian is implemented entirely within headers, with no need to link to 527any Boost object libraries. 528 529Several macros allow user control over features: 530 531* `BOOST_ENDIAN_NO_CTORS` causes `class endian_buffer` to have no 532constructors. The intended use is for compiling user code that must be 533portable between compilers regardless of {cpp}11 534http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted 535Functions] support. Use of constructors will always fail, 536* `BOOST_ENDIAN_FORCE_PODNESS` causes `BOOST_ENDIAN_NO_CTORS` to be defined if 537the compiler does not support {cpp}11 538http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm[Defaulted 539Functions]. This is ensures that objects of `class endian_buffer` are PODs, and 540so can be used in {cpp}03 unions. In {cpp}11, `class endian_buffer` objects are 541PODs, even though they have constructors, so can always be used in unions. 542