Embedded Alchemy

Send feedback »

Alchemy is a collection of independent library components that specifically relate to efficient low-level constructs used with embedded and network programming.

The most recent entries as well as Alchemy topics to be posted soon:
Alchemy: Documentation[^]

I just completed my Masters Degree in Cybersecurity at Johns Hopkins University. I plan to resume Alchemy's development. I plan to use my newly acquired knowledge to add constructs that will help improve the security of devices built for the Internet of (Insecure) Things.

Send feedback »

For this entry, I would like to introduce the type_traits header file. This file contains utility templates that greatly simplify work when writing template-based libraries. This is especially true for libraries that employ template meta-programming. The header file is available with C++ TR1 and above. This library provides tools to identify types, their qualifying properties and even peel-off properties one-by-one programmatically at compile-time. There are also transformation meta-functions that provide basic meta-programming operations such as a compile-time conditional.

The definitions in type_traits will save us a lot of time implementing Alchemy. As I introduce some of the tools that are available in the header, I will also demonstrate how these operations can be implemented. This will help you understand how to construct variations of the same type of solution when applying it in a different context. As an example of this, I will create a construct that behaves similarly to the tertiary operator, but is evaluated at compile-time.

Types and Values

Creating C++ template meta-programs are essentially functional programs. Functional programs compute with mathematical expressions. You will receive the same result each time you call a function with a specific set of parameters. As expressions are declarative, state and mutable data are generally avoided. Meta-programs are structured in a way to require the compiler to calculate the results of the expressions as part of compilation. The two constructs that we have to work with are integer constants, to hold calculation results, and types, which we can use like function calls.

Define a Value

Meta-program constants are declared and initialized statically. Therefore, these values are limited to integer-types. We cannot use string literals or floating-points because these types of static constants cannot be initialized in place with the class or struct definition. Sometimes you will see code declared with enumerations. I believe this is to prevent meta-programming objects from using space. It is possible for users to take the address of static constants, and if this occurs, the compiler must allocate space for the constant, even if the object is never instantiated. It is not possible to take the address of an enumeration declaration. Therefore no memory must be allocated when an enumeration is used.

Since I started using type_traits, I don't worry about it so much. I use the integral_constant template to define values. By convention, values are given the name value. This is important in generic programming to allow the object development to remain loosely coupled and independent of the design of other objects. The example below demonstrates how the integral_constant is typically used. Please note that all of these constructs live in the std:: namespace, which I will be omitting in these examples.

C++

 // Implement the pow function to calculate // exponent multiplication in a meta-function. template < size_t BaseT, size_t ExpT > struct Pow   : integral_constant< size_t,                         BaseT * Pow< BaseT , ExpT-1>::value > { };   // Specialization: Terminator for exp 1. template < size_t BaseT > struct Pow< BaseT, 1 >   : integral_constant< size_t, BaseT > { };   // Specialization: Special case for exp 0. template < size_t BaseT > struct Pow< BaseT, 0 >   : integral_constant< size_t, 1 > { };

Call (Instantiate) our meta-function:

C++

 int x = Result< 3,3 >::value; // 27

integral_constant

All that is required to implement the integral_constant, is a definition of a static constant in the template struct. Structs are generally used because of their default public member access. Here is the most basic implementation:

C++

 template < typename T,            T ValT> struct integral_constant {   static const T value = ValT; };

Yes it's that simple. The implementation in the C++ Standard Library goes a little bit further for convenience. Just as with the STL Containers, typedefs are created for the value_type and type of the object. There is also a value conversion operator to implicitly convert the object to the value_type. Here is the complete implementation:

C++

 template < typename T,            T ValT> struct integral_constant {   typedef T                           value_type;   typedef integral_constant< T, ValT> type;     static const T value = ValT;     operator value_type() const   {     return value;   } };

Two typedefs have been created to reduce the amount of code required to perform Boolean logic with meta-programs:

C++

 typedef integral_constant< bool, true >  true_type; typedef integral_constant< bool, false > false_type;

Compile-time Decisions

With meta-programming there is only one way to define a variable, and there are many ways to create decision making logic. Let's start with one that is very useful for making decisions.

is_same

This template allows you to test if two types are the same type.

C++

 template < typename T,            typename U > struct is_same   : false_type { };   // Specialization: When types are the same template < typename T > struct is_same< T,T >   : true_type { };

The compiler always looks for the best fit. That way, when multiple templates would be suitable, only the best match will be selected, if that is possible. In this case, the best match for when both types are the exact same, is our specialization that indicates true.

conditional

It's time to define a branching construct to enable us to make choices based on type. The conditional template is the moral equivalent of the if-statement for imperative C++.

C++

 // The default implementation represents false template < bool Predicate,            typename T,            typename F > struct conditional {   typedef F type; };   // Specialization: Handles true template < typename T,            typename F > struct conditional < true, T, F > {   typedef T type; };

Applying These Techniques

I have just demonstrated how three of the constructs defined in the type_traits header could be implemented. The techniques used to implement these constructs are used repeatedly to create solutions for evermore complex problems. I would like to demonstrate a construct that I use quite often in my own code, which is both built upon the templates I just discussed, and implemented with the same techniques used to implement those templates.

value_if

While the conditional template will define a type based on the result a Boolean expression, I commonly want to conditionally define a value based on the result of a Boolean expression. Therefore, I implemented the value_if template. This makes use of the integral_constant template and a similar implementation as was used to create the conditional template. This gives me another tool to simplify the complex parametric expressions that I often encounter.

C++

 template < bool Predicate,            typename T,            T TrueValue,            T FalseValue > struct value_if   : integral_constant< T, FalseValue > { };   // Specialization: True Case template < typename T,            T TrueValue,            T FalseValue > struct value_if < true, T, TrueValue, FalseValue >   : integral_constant< T, TrueValue > { };

Summary

I just introduced you to the type_traits header in C++. If you have not yet discovered this header, you should check it out. It can be very useful, even if you are not creating template meta-programs. Here is a reference link to the header from cppreference.com[^].

With the basic constructs that I introduced in this entry, I should now be able to create more sophisticated ways to interact with the Typelist[^] that I previously discussed for Alchemy. With the simple techniques used above, we should be able to implement template expressions that will query a Typelist type by index, get the size of the type at an index, and similarly calculate the offset in bytes from the beginning of the Typelist. The offset will be one of the most important pieces of information to know for the Alchemy implementation.

Type Lists

Send feedback »

Previously I had discussed the tuple data type. The tuple is a general purpose container that can be comprised of any sequence of types. Both the types and the values can be accessed by index or traversing similar to a linked list.

The TypeList is a category of types that are very similar to the tuple, except no data is stored within the type list. Whether the final implementation is constructed in the form of a linked list, an array, or even a tree, they are all typically referred to as Typelists. I believe that the Typelist construct is credited to Andrei Alexandrescu. He first published an article in the C/C++ Users Journal, but a more thorough description can be found in his book, Modern C++ Design.

Note:
For an implementation of a TypeList with Modern C++, check out this entry:
C++: Template Meta-Programming 2.0[^]

There is No Spoon

What happens when you instantiate a Typelist?

Nothing.

Besides, that's not the point of creating a Typelist container. The overall purpose is to collect, manage, and traverse type information programmatically. Originating with generic programming methods in C++, type lists are implemented with templates. Type lists are extremely simple, and yet can be used to solve a number of problems that would require enormous amounts of hand-written code to solve. I will use the Typelist in my solution for Alchemy.

Let's start by defining a Typelist.:

C++

 template < typename T, typename U > struct Typelist {   typedef T        head_t;   typedef U        tail_t; };

There are no data or functions defined in a Typelist, only type definitions. We only need these two elements, because as we saw with the Tuple, more complex sets of types can be created by chaining Typelists together with the tail. Here is an example of more complex Typelist definition:

C++

 // Typelist with an 8, 16, 32, and 64 bit integer defined. typedef    Typelist< char ,    Typelist< short,    Typelist< int, long int > > > integral_t;

When I first saw this, I thought two things:

1. How can this be useful, there is no data
2. The syntax is extremely verbose, I wouldn't want to use that

Remember in The Matrix when Neo goes to meet The Oracle, and he is waiting with the other "potentials"?! While he waits, he watches a young prodigy bending a spoon with his mind. He urges Neo to try, and offers some advice:

Spoon boy: Do not try and bend the spoon. That's impossible. Instead... only try to realize the truth.
Neo: What truth?
Spoon boy: There is no spoon.
Neo: There is no spoon?
Spoon boy: Then you'll see, that it is not the spoon that bends, it is only yourself.

Hmmm, maybe that's a bit too abstract, but, that's pretty much what it's like working with a Typelist. There is no data. Now take your red pill and let's move on.

Simplify the Definition

Let's make the definition simpler to work with. Working with a single Typelist definition of variable length seems much simpler to me than having to enter this repeated nesting of Typelist structures. Something like this:

C++

 typedef Typelist< char, short, int, long int > integral_t; // Or format like a C struct definition: typedef Typelist <    char,   short,   int,   long int > integral_t;

This could be usable, and it is easily achieved with template specialization or variadic templates. The solution based on template specialization is much more verbose, however, it is also more portable. I have also seen comments on compiler limits placed on the number of fields supported by variadic templates, but I do not have any personal experience with hitting limits myself. This is something I will probably explore in the future. For now, I will start the implementation with a template specialization solution.

Specialization

For this type of solution, we must select a maximum number of elements that we want or expect to be used. This is one of the drawbacks of specialization compared to the variadic approach. The forward declaration of the full Typelist would look like this:

C++

 template < typename T0, typename T1, ..., typename Tn > struct Typelist;

We cannot go much further until we resolve the next missing concept.

The Terminator

Similar to a linked-list, we will need some sort of indicator to mark the end of the Typelist. This terminator will also be used in the specialized definitions of Typelist to give us a consistent way to define the large number of definitions that will be created. With meta-programming, we do not have variables, only types and constants. Since a Typelist is constructed entirely from types, we should use a type to define the terminator:

C++

 struct empty {};

With a defined terminator, here is what the outer definition for a 10-node Type list:

C++

 template < typename T0,         typename T1 = empty,             typename T2 = empty, typename T3 = empty,             typename T4 = empty, typename T5 = empty,            typename T6 = empty, typename T7 = empty,             typename T8 = empty, typename T9 = empty > struct Typelist {   // TBD ... };

Here is the outer definition for a specialization with two items:

C++

 template < typename T0, typename T1 > struct Typelist< T0, T1 > {   // TBD ... };

Implementation

Believe it or not, we have already seen the implementation that goes inside of the template definitions shown above. The only exception is we will rename what we previously called a Typelist to a Typenode. Otherwise, the implementation becomes the typedef that we created. By convention, we will name the typedef, type. For reference, constant values are called, value, in template meta-programming. This consistency provides a very generic and compatible way for separate objects that were not designed together, to still inter-operate.

C++

 template < typename T, typename U > struct Typenode {   typedef T        head_t;   typedef U        tail_t; }; // Typelist primary template definition. template < typename T0,         typename T1 = empty,             typename T2 = empty, typename T3 = empty,             typename T4 = empty, typename T5 = empty,            typename T6 = empty, typename T7 = empty,             typename T8 = empty, typename T9 = empty > struct Typelist {   typedef Typenode< T0,           Typenode< T1,           Typenode< T2,           Typenode< T3,           Typenode< T4,           Typenode< T5,           Typenode< T6,           Typenode< T7,           Typenode< T8,           Typenode< T9 > > > > > > > > > >     type; };

Here is the implementation for the two node specialization:

C++

 template < typename T0, typename T1 > struct Typelist < T0, T1 > {   typedef Typenode< T0,           Typenode< T1> >   type; };

Simple? Yes. Verbose? Yes. Is it worth it? I believe it will be. Besides, since this definition will be hidden away from users, I would feel comfortable defining some code-generating MACROs to take care of the verbose and repetitive internal definitions. However, I will demonstrate that another time.

Using a Typelist

We have simplified the interface required of the user to define a Typelist, however, interacting with one at this point is still cumbersome. For example, consider the definition of the integral_t Typelist defined above. If we wanted to create a variable using the type in the 3rd slot (index 2, int), this syntax would be required:

C++

 integral_t::type::tail_t::tail_t::head_t third_type = 0;

These are the stories that grown-up C programmers tell little C++ programmers growing up to prevent the use of some of the most effective aspects of the language. The next few entries will be focused on extracting data out of the Typelist in a clean and simple way. However, let's tackle one solution right now.

How Many Elements?

Length is a common piece of information that is desired from containers. This is no different for type-containers. How can we determine the number of elements in the integral_t we have been using in this entry? We will write a meta-function. Unfortunately, I have not yet demonstrated many of the techniques in meta-function development. This means we will need to develop a Length function that matches the signature for every specialization that was created for the Typelist definition itself. Otherwise, we could create a meta-function that actively traverses the Typenodes searching for a terminator. We will revisit and solve this problem with a more elegant solution.

The solution will be very simple, and very verbose, but still, very simple. Since we must define a Length implementation for each Typelist specialization that we created, we know in each specialization how many parameters types were defined. We can take advantage of that information to create our simple solution:

C++

 template < typename ContainerT > struct Length;   template <> struct Length < empty > {   enum { value = 0 }; }   template < typename T0 > struct Length < Typelist < T0 > > {   enum { value = 1; } }   template &amp< typename T0, typename T1 > struct Length &amp< Typelist < T0, T1 > > {   enum { value = 2; } }

Again, with some preprocessor MACROS, generating all of the variations could be greatly simplified. For now, I would like to get enough infrastructure in place to determine how effective this entire approach will be at solving the goals of Alchemy. Here is a sample that demonstrates querying the number of elements in the integral_t type.

C++

 // Calls the meta-function Length  // to get the number of items in integral_t.  size_t count = Length< integral_t >::value;   // count now equals 4

Summary

I have now introduced the Typelist meta-programming construct. This is a simple type-only type declaration, which contains quite a bit of untapped potential for us. However, without a basic set of meta-programming constructs to build upon, our only option for implementing the Length meta-function is to resort to one function implementation for the total number of elements that can be held in the Typelist. This is not a very maintainable solution, but it won't be this way for long. I will be introducing a few simple constructs to make decisions based on type at compile-time, in order to create a more elegant and maintainable implementation of Length. From there I will continue forward and show how to query the type of parameter at a specified index, get the size of that type and many other meta-functions we can use to inspect our new data structure.

C++: Tuple

3 feedbacks »

During my design analysis for my Network Alchemy implementation I thought that the tuple may be the answer to allow me to iterate over the types defined in a network message definition. Tuples are data structures that are a generalization of the std::pair. Rather than having a limitation of 2 items in a tuple, potentially any number of items can be constructed within a custom tuple definition. The upper-limit will most likely be associated with the limit of your compilers ability to recurse down within a nested template structure. Conceptually, a tuple is similar to a compile-time linked list. If a tuple were implemented in terms of the std::pair, it would be constructed like this:

C++

 #include < utility > using std::pair;    pair< int ,        pair< float,              pair< char, long >           >     > tuple;

The previous block of code defines a structure that contains 4 elements, where the head of the list is a defined type, and the tail is the remainder of the list. This particular tuple defines a type that contains: int, float, char, and a long.

What exactly is a tuple?

Before I can get into detail for how a tuple is useful, I think it would be best to show the difference between a pair and a tuple, especially if you are already familiar with the std::pair. The most common way to be introduced to the pair, is when you use std::map. The key-value pairs are actually stored in the map as an instance of an std::pair. There are only two parameters in the pair; the names of these parameters are first and second. This values represent the first and second items, respectively, which are defined in the pair. As this is a parameterized type, the parameter types of first and second is entirely dependent on each instantiation of std::pair.

Until recently, I have rarely use the pair outside of my usage of the std::map, primarily because of the accessor name, first and second. As I stated earlier, the tuple is a generalization of the pair. Accessing the different fields of a tuple differs from the pair, in that you reference an entry by its position, similar to an array. There is a parameterize function called, get that returns the value of an element of a specified index in the tuple. Similarly, there is a function called tuple_element that can be used to query the type of the element at a specified index. Here is an example that demonstrates the tuple:

C++

 #include < iostream > #include < tuple >   using std::tuple_element; using std::get;   // The types are deduced to construct the tuple. typedef std::tuple< int , char, size_t> tuple_t; tuple_t data = std::make_tuple (10,'a', 100);   tuple_element< 0,tuple_t >::type valueA = get< 0 >(data); tuple_element< 1,tuple_t >::type valueB = get< 1 >(data); tuple_element< 2,tuple_t >::type valueC = get< 2 >(data);   // The types and values for each of the parameters: // int    valueA = 10; // char   valueB = 'a'; // size_t valueC = 100;

There are only a few other functions to be aware of with regards to the tuple:

• tuple_size: Returns the number of elements in the tuple
• tuple_cat: Concatenates two tuples to create a third tuple type
• tie: Unpacks arguments from a tuple, and constructs a new tuple

Tie is the only function that I think needs further explanation. There is a companion type used with tie called, ignore. This is used to skip over elements in the incoming tuple to manipulate the final tuple that is constructed from the call to tie. Ultimately, this is a utility function to more conveniently construct a new type from an existing tuple type. After all, the moral equivalent to variables in template meta-programming is a new type.

How can a tuple be useful?

Let's return back to Alchemy and discuss the tuple in terms of our proposed library. Remember in the previous entry that I was searching for a mechanism that would allow me to iterate over each of the types defined in a message structure?! Well it appears that the tuple is a mechanism that can provide those capabilities for us. For example, consider if a tuple was defined with all of the elements that should appear in a message packet structure. With the handle full of functions provided by the C++ Standard Library we can:

• Determine the number of elements in the tuple with tuple_size
• Iterate the tuple to determine the type of an element at each index with tuple_element
• Obtain the type of an element at a specified index with get
• Most importantly, we can now reliably calculate the offset that a parameter should be placed in a raw buffer with this set of meta-functions

OffsetOf &lt; Idx, TupleT >

Let's analyze what I just defined to calculate the offset in bytes, for the specified position of a parameter in a tuple definition. The very first thing that I chose to do for this implementation was to create a forward declaration of this OffsetOf utility function. Simply to introduce the format of the template before we analyze the more complex aspects of this construct. This template requires a positive integer value to specify the index of the element, and a fully defined tuple. Because it's a template declaration, any type can actually be placed in the second parameter TupleT. However, the first parameter's type is already defined as size_t, so a positive integral value must be specified for the first parameter. I will demonstrate shortly that the actual types that can be specified are slightly more restrictive than I first indicated.

C++

 //  Forward Declaration ********** template < size_t Index, typename TupleT> struct OffsetOf;

The second struct defined in the first block requires us to take off our imperative programming hat, and switch over to use our functional programming hat. This hat has a very awkward fit at first; it will probably feel a little tight and lead to headaches, unless you have worn one while programming a different language such as F#, Haskel or Lisp. My advice, don't fight it, when you break this hat in, you will be a better programmer for having done so. It opens your mind to a new way of thinking about problems. You will be able to see potential solutions from multiple perspectives.

As many meta-template solutions tend to be composed of recursive implementation, the second item in the first block contains the terminating case. That is, a template specialization is created for an element whose specified index is 0. The offset for all elements at index 0, is at 0-bytes. One of the characteristics of functional programming is the inability to create mutable state within a statement. We are able to define new functions, and placeholder variables of sorts. The former is accomplished by declaring a new type, the latter by defining a constant. This is the reason an enumeration named value is assigned the value 0; to hold the calculated offset for the element at index 0. To access the value of the index, the OffsetOf type must be specified with the desired index, and the enumeration value must be referenced to get access to the value.

C++

 //  Forward Declaration ********** template < size_t Index, typename TupleT> struct OffsetOf;   //  ******************************* /// The item at index 0 will always have an offset at 0.  ///  template < typename TupleT > struct OffsetOf< 0, TupleT > {   enum { value = 0 }; };

C++

 //  Get the offset of element 0 ** size_t offset = OffsetOf<0, tuple_t>::value;   // offset now equals: 0

By convention, constant values are given the name value and type definitions for sub-types are given a name that indicates it is a type of some sorts. For example, look at the documentation for the C++ Standard containers such as vector and map. These containers have many typedefs to declare the value_type, size_type, allocator_type; then there are other general purpose names used for reference, pointer, iterator and const versions for these types.

Specialization for a specific tuple type

C++

 template < typename TupleT> struct OffsetOf< IdxT, TupleT > {   typedef TupleT  container;   typedef std::tuple_element< IdxT-1,                                container>::type    prev;     enum { value = OffsetOf<(IdxT)-1, container>::value                 + sizeof(prev)}; };

There is truly only one statement in the declaration of OffsetOf and that is the definition for the enumeration, value. The two typedefs, container, and prev are solely added for improved readability. There is, however, the possibility this extra type definitions could also prove useful later on. None-the-less, readability is my primary reasoning for adding them here. The calculation of the offset is a straight-forward recursive implementation, and the zero-defined specialization will terminate the loop. Essentially the offset of the previous element is calculated plus the size of that element to calculate the offset of this element. However, to calculate the offset of the previous element, we must know the offset and size of the element prior to that one as well. This behavior recurses until the zero terminator is reached. Then the values bubble back up, and the calculated offset is defined.

All of this behavior occurs at compile-time, with this calculations actually being performed by the compiler. No run-time code is generated from these definitions, only constant values that will be substituted in place anytime an instance of one of these templates is defined. While it is possible that heavy reliance on code written with templates can slow down the compile-times, it has been my experience that the increase in time is negligible. Also, for recursive solution such as this, many compilers have improved to the point where they store the values of previous template instantiation calculations. Therefore, the second time iterating through the loop for calculated offsets, the answer to the previous item will already be known, and a simple lookup is performed to get the value. I will continue go into deeper detail regarding the work that occurs behind the scenes another time.

Will tuples work for Alchemy>

I believe the answer to this question is "Yes." However, I do not think they will be the best fit. As I have been exploring how to use tuples, including accessing values, I think it would be difficult to achieve the natural struct-type syntax I have original set out to achieve. Primarily because of the syntax that is required to access the values from a tuple, the std::get function Therefore, I am going to put this idea to the side, and explore some other options. If the other options do not turn out to be as good as the tuple, I can always return to this idea. I want to point out that I believe the std::tuple would provide a much more robust and portable solution that the offsetof MACRO we explored in the last entry.

Next Step

We will explore what I believe to be the most promising approach in the next entry, TypeLists. I do not know the full etymology for the term, however, many of the sources that I have read credit Andrei Alexandrescu with creating the term in his book Modern C++ Design, which I believe was previously mentioned in an article he wrote that was published in Dr. Dobbs Journal. At TypeList is a similar template-type construct, that is strictly a type. No data is contained in the TypeList itself. This will give us the freedom to choose how to represent and manage all of the data, and at the same time be able to iterate over each of the field types as required.

Desired Alchemy Syntax

Send feedback »

When I create a new library, I like to approach the design from two different directions. The first is the traditional route, analyzing the functionality that I need and designing a suitable interface to access those features. I also may write a set of pseudo-code that demonstrates what it would look like to use the library. Generally with writing unit tests I get to cover this second option. It is all the better if I write the tests while developing the interface. Then I discover if the interface is a clunky collection of garbage or an joy to work with.

Alchemy is a unique API, in that I don't actually want there to appear to be a library at work. One goal that I set for alchemy is to facilitate the proper handling of network data without requiring much work on the users part. In this sense, I need to work backwards. I would like to take the proposed syntax, and attempt to design my library API to meet the target syntax. The expressive ability of C++ is one of the things that I really like at the language.

What Does It Currently Look Like?

Before I attempt to create a new API to break old habits, I would like to inspect what the old habits look like. If I can make my API similar to the old way, hopefully it will be adopted more easily. The technique that I am going to mimic, is one that is not guaranteed to be portable, yet it works often enough that its use is quite common. This practice is declaring a structure, using the #pragma pack(1) option, and casting raw char* to the structure type. Then the fields are accessed by their name specified in the struct. Here is an example of this technique:

C++

 #pragma pack(1)  struct MsgFormat  {   uint32_t firstValue;    int16_t secondValue;    char thirdValue;  };  #pragma pack()

C++

 MsgFormat msg;  ::memset(&msg, 0, sizeof(MsgFormat));  int retVal = ::recv(s, &msg, sizeof(msg), 0);  // Error checking ...    // Convert to the proper byte-order  ConvertMsgToHost(msg);  uint32_t valueA = msg.firstValue;  int16_t valueB = msg.secondValue;  char valueC = msg.thirdValue;

This code is simple and straight-forward linear defined logic. There is nothing is entirely wrong with this type of code. However, things change over time, and what is a few lines of code now could morph into thousands of lines of similar code. I did not show the implementation for the function ConvertMsgToHost. As the format of MsgFormat is updated, the implementation of ConvertMsgToHost must be updated as well. Most likely, there is also a corresponding function called ConvertMsgToNetwork that will need to be updated as well.

When the size of the code grows to include dozens to possibly hundreds of messages, linearly defined logic creates lots of places for little bugs to hide. Moreover, these are not bugs that are introduced to the code, they suddenly appear due to forgetting to update location B because the definition in location A changed. What you don't know CAN definitely hurt you, especially in this situation.

What Should It Look Like?

After looking at common implementations for serialized data, I have a basic idea what I want to try to create. My goals again are to create a portable and robust library to facilitate reliable and maintainable network communication implementations. I want integration of this library to be easy, and the maintenance effort to be minimal. With those factors in mind, here is a pseudo-code representation of how I would like to be able to use Network Alchemy:

Message Definition:

C++

 class MsgFormat    : AlchemyMsg  {    uint32_t firstValue;    int16_t secondValue;    char thirdValue; };

Message Usage:

C++

 MsgFormat msg;  input >> msg;  // Something like this should also be possible:  // recv(s, msg.buffer(), msg.size(), 0);    // Error checking ...    // Convert to the proper byte-order  msg.to_host();  uint32_t valueA = msg.firstValue;  int16_t valueB = msg.secondValue;  char valueC = msg.thirdValue;

Notice how the parameters are still accessible by their name, and its not a function call. This is highly desirable for the code that I anticipate replacing with Alchemy. Data access like this is common all throughout the code base I have in mind. However, if this cannot be achieved cleanly, then I would be okay with a function call like one of these:

C++

 uint32_t valueA = msg.firstValue();  int16_t valueB = msg.get_secondValue();

However, I am not going to settle until I at least investigate what the possibilities are, and how much work it would take to implement code structured around my first choice.

Analysis

It appears to me that there are three challenges to overcome in order to create a library that can be used like my target syntax.

1. Safely and portably access data from a raw buffer with an objects data members
2. Provide automatic byte-order conversion (How to know what fields to convert, and when?).
3. Make the usage syntax similar to that of accessing data members in a struct

Item number one is where much of the trouble originates and a major motivation for this library. My initial thought is focused on the position of data fields within a structure, compared to where they maybe found in the raw buffer. Ideally they will match up 1:1, however, in practice we already know that does not happen due to the difference in hardware implementations. The sizeof operator is available and it gives us a part of the information that we need if we know what type or the name of the field that we are dealing with.

What if we take the address of a member data field in the structure, and subtract the address of the structure itself away from the members address; that would give us the offset of that field from the start of the structure. This is, in fact, what the offsetof MACRO does that can be found in the header file stddef.h. Here is an example of how offsetof could be implemented:

C++

 #define osffsetof(s,m) \    (size_t)&reinterpret_cast( (((s *)0)->m)) )

Unfortunately, when I consult the C++ standard, it indicates that I should make no assumption for how the fields are represented in memory. The compiler is free to organize the data for classes and structures anyway it sees fit. This freedom is given to the compiler writers to allow them to determine the most optimal layout for a data structure on the target platform. As users of structs we should use the named value interfaces and not attempt to poke down under the covers abstracted by the compiler. Also, the behavior of the offsetof MACRO is undefined for bit-fields, and that is something that I would like to possibly support in the future.

Although I am fairly confident that most of what I need to do could be solved with a method like this, I will move on to a different approach, at least for now. offsetof will not create the most portable and robust solution. I am going to skip item one, and move on to two and three. They seem like they will require the same technique to overcome the challenges presented by both of these items. Knowing that C++ is capable of some very expressive syntax for user-defined objects due to operator overloading, I believe the approach to take to create a solution for all three items will be to create a data type to abstract individual fields, a proxy member

Proxy Member

A proxy member object will need to be defined as a type that matches the corresponding type that the proxy points to. We can probably overload the conversion operator for the proxy member's type. This will create the natural syntax of accessing a data member, even though it is actually a function call, the conversion operator. If we can get the user to call any type of function, we can handle the byte-order conversion and data accesses safe and portable. We will still need a way to automatically convert the data for each field type. If we could guarantee that the user would access every single data member, every single time a message went on or came off the wire we would have a solution. However, this seems unrealistic. So again, we will return to the automatic conversion of the data's byte-order.

Here is an example of an overloaded conversion operator for a C++ object:

C++

 struct ProxyMemberLong {   long m_data;     ProxyMemberLong(long value)     : m_data(value);   { }       operator long()   {     return m_data;   } };    // Usage: ProxyMemberLong pm(100);   // Implicitly use the long conversion  // operator to assign 100 to value; Long value = pm;

Generally it is wise to stay away from overloading the conversion operator, as well as value constructors. This is because the compiler is allowed to look for constructors and conversion operations on types that could potentially create a fit for a data type passed into a function. Insidious problems can appear due to the compiler implicitly converting an object from one type to another that was not intended. This type of issue is very difficult to track down. However, this looks like a promising solution, so I would like to at least investigate this path a little further.

I would also like to mention that you can use the explicit keyword with the constructors to prevent the compiler from implicitly using the constructor as a way to convert from one type to your objects type. In this case, the constructor needs to be explicitly called in order to be used. As of C++ 11 the explicit keyword is also available for use with the conversion operator. I will not be using explicit with the conversion operator because I do want the implicit conversion to occur for most of the situations. This is an issue we will need to keep in mind and revisit when we are further along to test for any potential problems this could cause.

Automatic Processing of All Members

For languages that support reflection, this would be a simple problem to solve. Reflection is a mechanism that allows a construct to query about itself to learn things like the names of function calls and data members. Since we are going to move forward and seek a solution that uses a proxy member object, we know that we will be able to perform the required processing before fields are accessed, but we cannot guarantee they will be accesses. We need a way to be able to iterate over all of the child members of a message object.

With an iterator we will not need to know the name of the parameter. Most likely we will have a pointer or a reference to the member data that we need to process. How can we go about doing that? The tuple from the C++ Standard Library seems like a promising candidate. The std::tuple is much like the std::pair, except that the number of sub-field types is not limited to two. In fact, the std::tuple behaves a lot like a compile-time linked list. Each entry has a head and tail node, allowing you to traverse to the next node in the tuple set. There are also quite a few utility functions provided that allow you to access an entry in the tuple with an index. The tuple seems very promising.

Next Step

In the next entry for Alchemy, I will explore how the tuple might be used to solve the challenges I am working with for this library. This will give us a chance to inspect some of the valuable and useful components in the C++ Standard Library that I have not used before, and possibly apply it to solve a problem. Once we verify this is a viable direction to continue with, we can develop the code for Network Alchemy a bit further.

An Eye on Refactoring

Send feedback »

Recently I ran across the article, "Is Design Dead?", from Martin Fowler's blog. Martin addresses the common misconception that design is discouraged in Extreme Programming. He describes the overall purpose of design in software and how the design emerges as part of the development phase. As more is learned about the problem domain, better decisions can be made and the design evolves with the implementation. Refactoring code is an activity that helps facilitate success with this approach.

At a different time I was learning about the structure of the human eye and some of the potential paths in evolution that could have led to the structure of our eyes today. I learned some surprising things with regards to how our eyes are structured. The evolved structure is counter-intuitive to what I would have imagined, especially when compared to the structure of a digital camera. I took a step back and starting thinking about the evolutionary path of the eye and compared it to typical events I have experienced during a software development cycle. I wanted to share the conclusions that I reached when I thought about these two topics.

Software Design Provides Flexibility

Design is an important aspect for any engineering discipline. Good designs allow for the development of a project to be divided into independent tasks. One of the most important benefits that can be realized from software design, is the ability to remain flexible to change throughout the project. Software is so valuable for that simple reason, the flexibility and relatively ease of change. Now I say relatively because if the project is not developed and managed properly, the structure of a program can become rigid and more resistant to change.

The tendency to resist change as a project progresses is what drives the concept of the Software Change Curve. This curve says that each progressive phase of development becomes exponentially more expensive to change the software from its original design. A concept that may only cost one dollar to analyze may be orders of magnitude more expensive to introduce after the product as been released.

The Waterfall process spends a large amount of time up-front attempting to analyze and plan to the current known set of requirements. Time can also be spent to try to identify the most volatile areas of the requirements to create flexibility at the points that may change later in the project. However, this is still a difficult task, which requires the ability to predict the requirements that are likely to change. No amount of planning can foresee all potential issues that will occur.

The XP approach of evolutionary design asserts that it is possible to flatten the software change curve by making design corrections during the development process. Some of the practices in XP are intended to facilitate the corresponding changes in software to match the updates to the design; I am referring to testing, continuous integration (CI), and refactoring. Testing and CI help synchronize a teams efforts when integrating changes with each other. Software refactoring is the practice that provides the agility to efficiently alter the software by simplifying the volatile regions of code.

Structure of the Eye

So how does the eye relate to all of this? Let me first describe the structure of the eye, and I will draw the connection for the structure of the eye and its similarity to software development. In many ways the eye is structured like any type of camera that has a lens. At the front there is the lens, and the back of the eye is the retina. The retina is the light-sensitive tissue that detects light and sends signals to the brain for processing. The component in a camera that corresponds to the retina would be the CCD or CMOS image sensor.

The CCD sensor faces the lens and captures incoming photons and generates an electrical charge proportional to the amount of light that it receives. The retina functions in a very similar manner with two exceptions.

1. Chemical and electrical nerves are triggered to send signals to the brain.
2. The photoreceptors in the retina are at the back of the retina.

Let's focus on item two. The optical nerves that transmit signals to the brain come out of the retina facing the lens. Therefore, Any light captured must travel through the layers of nerve fibers and other cells before they are detected by the photoreceptors. This also means there must be a location in the retina where the optical nerve passes back through the retina to the back of the eye and on to the brain, and there is; this point is called the Macula. It creates a blind spot in that area of vision for our eyes, that our brain fills in the missing piece, but that's another story.

Here is an illustration from Gray's Anatomy, which shows the forward facing part of the retina at the top, and the photoreceptors at the bottom:

The Code-and-Fix Process

The "Code-and-Fix" development process is what many software development projects devolve into at some-point in their lifetime; even if it's only a temporary situation. Basically, it evaluates what functionality is needed next, code the solution, and fix an errors that are detected. Once a feature is completed, the next feature is implemented in a similar fashion. After a non-descript period of time passes, and numerous cycles of this process the code base appears to have taken on a life of its own. The current project no longer matches any initial design work that may have been done before the project started. The source code has essentially evolved to meet the current needs.

Many people mistaken the "Code-and-Fix" process, with the practice of Evolutionary Design. That's because they are very similar to one another with the exception of one thing, Evolutionary Design prescribes that you must refactor regularly to simplify the current solution. By taking this step, your design decisions can be simplified by removing complexity from your system, and reintroducing flexibility. Remember, flexibility to change is what will help flatten the software-development curve. Refactoring is key to keeping your code nimble and adaptable to solve the next set of problems.

Back to Eyes

I now want to use the evolution of the eye as analogy of a "Code-and-Fix" development cycle without refactoring. I consider myself lucky to not require corrective lenses to see clearly, especially with so much time staring at a computer screen. Through my eyes, I think that evolution arrived at a pretty good implementation for an orbital ocular organ for me to sense light, track and focus on objects, and even adapt between high-contrast and low-light conditions.

Now consider how many millions of years vision took to evolve to this level. Also, the path that has led up to my eyes today, also has billions of other very similar versions evolving with each generation. The number of less-optimal mutations of the eye, which never came to be, must be orders of magnitude larger than the number of eyes currently on this planet. That is a lot of trial-and-error.

Refactoring

I think of "Code-and-Fix" as "Trial-and-error". This is a generally an expensive process to use with engineering. "Trial-and-error" becomes more expensive as the software development cycle progresses into the next stage. If you can remain cognizant of the issues that you have run into and apply this wisdom and experience towards altering the current implementation to be most beneficial to adapt to the next set of features, you have refactoring.

There is a little more to it than that, however, not much more. Refactoring source code is just like, well, it's just like factoring a fraction into a mixed number. You have a simpler form of the same thing. Refactoring is not intended to mean rework, re-implement, or create version 2.0. You are merely supposed to simplify the structure of your program while keeping the same functionality set. This is where unit tests are very valuable to help insure you do not break things that were previously working. A good rule of thumb, if you have to change your unit tests when you refactor code, you are probably doing more than refactoring.

Regardless, you must continue to analyze and simplify the structure of your software if you want to keep the software development curve relatively flat. Otherwise, the code will become rigid, and resistant to the addition of new features free of defects. This is essentially the cause for the exponential growth of cost in the curve. Taking that little extra time to reorganize your code as you develop can payoff by allowing new features to be more easily developed. That makes more time for you do other things such as take on new challenges, or surf the Internet.

Summary

Evolution is a complex process that takes a lot of time, more time than we are given to complete a project. Evolution takes a lot of resources because it is a process that is based on Trial-and-Error. Code-and-Fix often is a Trial-and-Error process as well. There are more efficient ways to develop code. I would like to add that I do not think there is a single process that is suitable for every situation. Whether you design completely at the beginning, or let your design evolve throughout the development of your project, one of your goals is to keep your code base flexible. A code base that can easily adapt to new requirements and unforeseen circumstances, is a valuable code base indeed. The cost of development during progressive phases of development can remain relatively flat, as opposed to increasing exponentially. Because It will be easier to modify, easier to understand, and easier to maintain. Continual refactoring is a key to remaining nimble.

Byte Sex

Send feedback »

Good! Now that I have your attention let's solve a relatively simple problem, byte sex. A less sensational name for this concept is byte endianess. This is one of those concepts that you should at least be aware of, even if you don't have to pay much attention to it in your day-to-day work. Each CPU architecture has it's own definition for memory. One of these properties is the endianess format of data registers. This is the first issue that I address for Network Alchemy.

Endianess

Endianess defines the order bytes are arranged within a word in machine hardware. There are two commonly used types, big-endian and little-endian. Other formats do exist, however, they are rare. Endianess is often mentioned with the context of network communication. Be aware that it must actually be addressed for any medium used to transfer data from one machine to another. This includes binary file formats.

• Big Endian: Big Endian format places the most significant byte to the left of the byte sequence. Essentially, the byte sequence starts with the most significant byte, and ends with the least significant byte. This maps to the same format that we represent numbers.
• Little Endian: The least significant byte is at the left of the byte sequence.
• Network Byte-order: The byte-order format used to transfer data between two different machines. Big-endian byte-order format is used for network byte-order by convention.
• Host Byte-order: The local machine's byte order format.

The description given at Wikipedia contains a more thorough treatment of this subject.

A small set of functions are usually provided with the socket libraries for a platform. These functions will perform a byte-order conversion for integers represented with multi-byte formats. The set of supported functions varies between platforms. You can generally count on this set of four to exist:

• htons: Converts a 16-bit integer (short) from host to network byte-order.
• ntohs: Converts a 16-bit integer (short) from network to host byte-order.
• htonl: Converts a 32-bit integer (long) from host to network byte-order.
• ntohl: Converts a 32-bit integer (long) from network to host byte-order.

C++

 unsigned short s_before = 0xC0DE;  unsigned long  l_before  = 0x600D1DEA;  unsigned short s_after = htons(s_before);  unsigned long  l_after = htonl(l_before);

 Endianess Type Before Before (hex) After After (hex) Big short 0xC0DE 49374 0xC0DE 49374 Big long 0x600D1DEA 1611472362 0x600D1DEA 1611472362 Little short 0xC0DE 49374 0xDEC0 57024 Little long 0x600D1DEA 1611472362 0xEA1D0D60 246222176

Little-endian systems are the only systems that will require a modification of data. The call to htonXX() functions will not generate any code on big-endian type systems. The results in the table will be the same if the same data is input to the ntohXX(). In fact, the implementation for ntohs() could be implemented like this:

C++

 inline unsigned short ntohs(   unsigned short netshort ) {   return htons(netshort); }

Potential Problems

I have experienced these two issues, related to the byte-order conversion functions, creeping into a project. This message structure and byte-order conversion function will be used for reference to demonstrate these problems.

C++

 struct DataMsg {   int   count;   short index;   unsigned char code;   unsigned long value; };   void ConvertMsgToHost(DataMsg &msg) {   msg.count = (long) ntohl( (unsigned long)msg.count);   msg.index = (short)ntohs((unsigned short)msg.index);    msg.value = ntohl(msg.value); }

Inconsistent Conversion Calls

If a strong pattern of implementation is not created and followed with the message transport process, more care will be required to keep track of the state of a message buffer. Has the buffer already been converted to network byte-order? If a buffer that has already been converted to network byte-order is converted a second time, the result will be returned to the host byte-order.

Mistake #1

C++

 // Calling the convert function an even number of times // will return the data to its original byte-order: ConvertMsgToHost(msg);   // ... later called on the same message. ConvertMsgToHost(msg);

Mistake #2

C++

 // Developer is unaware that // ConvertMsgToHost exists or is called:  ConvertMsgToHost(msg);   // ... later // ERROR: The result returned is in network order. long value = ntohl(msg.value);

Field Type Changes

Many times the field types in a message my change. It is possible for the type of field to be changed, but the byte-order conversion function to remain unchanged. This is especially true because of the tendency to use explicit casts with these functions. If this change goes undetected, data values will be truncated and incorrect values will be transported across the network.

C++

 // Noted types are changed in the message format below: struct DataMsg {   int   count;   size_t index;        // Changed from short   unsigned short code; // Changed from unsigned char   unsigned long value; };   // The mistake occurs if the conversion // function is not properly updated: void ConvertMsgToHost(DataMsg &msg) {   msg.count = (long) ntohl( (unsigned long)msg.count);     // The explicit cast will force this call to succeed.   // Error: data is truncated and in wrong order   msg.index = (short)ntohs((unsigned short)msg.index);     msg.value = ntohl(msg.value);     // The value msg.code will not be properly converted  }

Generic Solution

One way to improve the maintainability of your code is to use solutions that are highly consistent. When compared to the traditional approach, our solution will have a more consistent implementation by the method that we handle the context sensitive information of the solution. I am referring to: 1) the data-type to be converted, 2) the host byte-order, 3) the target byte-order of the data. If any of the field types in our DataMsg struct changes, our converter function will continue to be valid without any required changes. The EndianSwap function simply knows what to do. This does not yet resolve the issue of inconsistently converted messages. We will address that after we have a robust and consistent method to swap the byte-order of data fields.

C++

 void ConvertMsgToHost(DataMsg &msg) {   msg.count = EndianSwap(msg.count);   msg.index = EndianSwap(msg.index);    msg.Code  = EndianSwap(msg.code);    msg.value = EndianSwap(msg.value); }

We can create a function like EndianSwap that will take the appropriate action regardless of the type of field passed into it. The socket byte conversion functions are written to be compatible with C. Therefore, a different name must be given to each function, for each type supported. Function overloading in C++ will allow us to create a set of functions that can be used to generically convert the byte-order of many different types of fields. This still leaves the problem of calling the convert function an even number of times returns the data format to it's original type. We will revisit this once I create the first part of the solution.

Byte-Order Swap

Because a large variety of types may be encoded in data, we will start with a template-based approach to create a generic EndianSwap function. This will allow us to create sets of solutions and reuse them, rather than duplicating code and giving a new function prototype to each function. The base implementation will provide an empty solution that simply returns the input value. The compiler will optimize away the function call and result assignment. This effectively turns this call into a no-op:

C++

 template < typename T > inline  T EndianSwap(T input) {   return input; }

A specialization of this template will be implemented for each type that requires byte-order conversion logic. Here is the implementation for unsigned 16-bit and unsigned 32-bit integers.

C++

 template < > inline  uint16_t EndianSwap(uint16_t input) {   return  (input << convert ::k_8bits)         | (input >> convert ::k_8bits); }   template < >  inline  uint32_t EndianSwap(uint32_t input) {   return  (input  <<  convert ::k_24bits)         | ((input >> convert::k_8bits) & 0x0000FF00)         | ((input <<  convert ::k_8bits) & 0x00FF0000)         | (input >>  convert::k_24bits); }

I chose to implement my own set of byte-order swap functions for a couple of reasons.

1. To remain independent of system socket libraries. The functions are not portable implemented or available
2. There is now only one byte-order conversion function rather than two with different names
3. Added flexibility; The socket functions become no-ops on big-endian solutions, where as EndianSwap will always swap.

Another way we will improve upon the existing byte-order conversion functions is by providing a specialization for the signed types. This is necessary to eliminate the need to provide a cast with calls to these functions. The implementations for the signed versions can be implemented in terms of the unsigned definitions. However, care must be taken to avoid conditions that would create an integer overflow condition. Overflows with signed integers results in truncated data.

C++

 template < > inline  int16_t EndianSwap(int16_t input) {   return static_cast < int32_t >(      EndianSwap(static_cast < uint16_t >(input))   ); }   template < >  inline  int32_t EndianSwap(int32_t input) {   return static_cast < int32_t >(      EndianSwap(static_cast < uint32_t >(input))   ); }

Manage Context Information

We now have a consistent method to swap the byte-order for any data-type that we choose. However, we still need to account for the other types of context sensitive information for this solution to be useful. We have two additional pieces of context information:

• Big Endian / Little Endian
• Host Byte-order / Network Byte-order

Two types of context information with binary values means we will have 4 possible solutions. Here is a first attempt to create a solution that incorporates all of the pieces:

Constants deduced from compiler settings and platform header files:

C++

 /// Platform Endian types  enum Endianess {   k_big_endian    = 0,   k_little_endian = 1     };   /// Constant indicates machine endianess. const Endianess k_endianess = Endianess(NA_ENDIANESS);

Host-order conversion function

C++

 template < typename T> inline  T ToHostOrder(T input) {   if (k_little_endian == k_endianess)   {     return EndianSwap(input);   }     else   {     return input;   } }   // A function called ToNetworkOrder is also // implemented with the identical implementation.

At this point we have two separate functions, that both handle two cases. This manages our four distinct possibilities for combinations of context-sensitive information. However, this is not the final solution. In fact, this solution has all of the same issues as the socket library calls, except we can handle any type of data. We have even added a runtime cost to the implementation with an if statement.

Improve the Solution

We are going to put the compiler to work for us to improve this solution. Templates are an excellent option when you have all of the information you need at compile-time. The solution in the previous step used two runtime mechanisms to manage the context-sensitive information, a function call and a conditional. Let's create a template that will let the compiler decide if it is necessary to swap the order of bytes for a data-type rather than a runtime conditional statement.

Generic EndianSwap template to swap based upon the endian-type of the machine.

C++

 // Default template implementation // This version always performs a swap. template < typename T, bool isSwap > struct EndianType {   static   T SwapOrder(const T& value)   {     return EndianSwap(value);   } };

EndianSwap specialization that does not swap the order of bytes.

C++

 // The false value is set for the isSwap field // This is the selector to prevent the byte-order swap template < typename T > struct EndianType < T,false> {   static   T SwapOrder(const T& value)   {     return value;   } };

Template meta-programming solutions require all of the information to be known up front, and any decisions need to be calculated by the compiler. Therefore, constants and data-types become the primary mechanisms used to store information rather than variables. The compiler must have all of the answers fixed in-place to determine which pieces to use for construction of the logical statements. We now have a new struct with a static function to conditionally swap the order of bytes in a data-type. Let's connect it to the final piece of the solution.

We will shift another piece of the context-sensitive information from a variable to a type. The message endian byte-order will be encoded in this type. This object handles for conversion of data that is in host byte-order. The ToHost function call is a no-op, and the ToNetwork will provide the byte-order conversion.

C++

 template < Endianess E > struct HostByteOrderT {   static const     Endianess order = E;     static const     bool       isHost = true;     template < typename T>   static   T ToNetwork(const T& input)   {     return EndianType < T,                         (k_big_endian != order)                       >::SwapOrder(input);   }     template < typename T>   static   T ToHost(const T& input)   {     return input;   } };

Here is the corresponding implementation for the network byte-order type.

C++

 template < Endianess E > struct NetworkByteOrderT {   static const     Endianess order = E;     static const     bool       isHost = false;     template < typename T>   static   T ToHost(const T& input)   {     return EndianType < T,                         (k_big_endian != order)                       >::SwapOrder(input);   }     template < typename T>   static   T ToNetwork(const T& input)   {     return input;   } };

These two typdefs will simplify usage:

C++

 typedef HostByteOrderT < k_endianess > HostByteOrder; typedef NetByteOrderT  < k_endianess > NetByteOrder;

Usage

All of the pieces are in place to have the compiler generate context-sensitive code, based upon the machine architecture, and the desired target endian byte-order. It may not be obvious, but all of the components exist for us create a conversion function to safely convert data between byte-order types. We can even prevent a mishap from occurring by inadvertently converting a data-type from host-order twice.

The key is within the types that allow us to indicate of a value is in host or network byte-order. To create a message whose byte-order is properly managed, encode its endian byte-order with a template.

C++

 template < typename T> struct DataMsg   : T {   int   count;   short index;   unsigned char code;   unsigned long value; };   typedef DataMsg< HostByteOrder > DataMsg_Host; typedef DataMsg< NetByteOrder >  DataMsg_Net;

The corresponding conversion function for network-to-host conversion:

C++

 // This function will only convert  // network types to host-order. // The output is into a host-order type. template < typename T> void ConvertMsgToHost(   const DataMsg_Net  &in,         DataMsg_Host &out ) {   out.count = DataMsg < T >::ToHost(in.count);   out.code  = DataMsg < T >::ToHost(in.code);    out.index = DataMsg < T >::ToHost(in.index);    out.value = DataMsg < T >::ToHost(in.value); }

Summary

The solution created in this entry is a fairly independent piece of code. This goal of this solution is to provide a generic structure for how byte-order conversions are performed. This will ensure they are performed consistently over time, and hopefully reduce the chances of misuse. The topic of IPC is already dancing near the edges of type-safe data. Anything that we can do to keep type information around as long as possible will help improve the quality and correctness of our software.

Before we can progress much further with the Network Alchemy library, I will need to provide some foundation in template meta-programming concepts. Therefore, the next few entries related to this library will be focused on concepts, and a design for the next phase of the library.

My Unit Test Environment

Send feedback »

I discussed the merits of selecting a suitable unit test framework for your development project in my previous post. I described the qualities that I found most valuable in the test framework that I use, CxxTest. The qualities are xUnit framework, portability, simplicity, and flexibility. There are other frameworks for C++ that contain many more features, however, rarely have I had to expand on what is provided in order to test something. Moreover, CxxTest is lightweight enough, that other test tools can be integrated with it, such as Google's GMock library.

Resources

I want to describe my test environment in detail since this will be the same environment that I will use to build and verify Network Alchemy.

Visual Studio Project Wizard: CxxTest

Integrated Tools

I saw benefits immediately when I first started experimenting with unit tests. I wrote a set of tests for a specialized string class I had recently written. That exercise alone helped me uncover 3 or 4 bugs that had existed in the class even though it had been in use for over two years. The thing that took a little longer to uncover was how cumbersome it became to create new unit test projects. Ideally you create a new unit test suite for each object.

I thought I that I needed to make this aspect simpler in order to have a chance of this being adopted by the other developers on my team. Therefore, I created a Visual Studio Wizard to quickly generate a new test project for me, as well as a class wizard to add a new test suite to the project. A project can contain multiple suites, however, I would only put multiple suites in the same project if they were closely related. It's best to minimize dependencies in your test suites, because of the tendency for this to help minimize the dependencies in your objects.

I also setup a Make system to be able to build my projects on Linux and other platforms. I have added these Visual Studio project wizards to my site for you to download and use if you are interested. More details are located below to describe tool requirements, installation and usage instructions. These files are located in the base directory of the zip file resources. You can use these Make files as a reference to integrate CxxTest into your own build environment.

• \Makefile: The top-level make file
• \Makefile.unittest: A make file specific to each unit test.

Requirements

This list describes what tools are required to use this integrated test environment that I have developed.

CxxTest

The CxxTest framework needs to be installed on your system:

Python

The Python scripting engine must be installed to generate the test runners:

Visual Studio

I have successfully used the CxxTest Project Wizard integrates with these versions of Visual Studio. I have verified the wizard works properly in Visual Studio 2013 Express. I believe it should also work in previous Express versions, possibly with minor modification:

• Visual Studio 2005
• Visual Studio 2008
• Visual Studio 2010
• Visual Studio 2012
• Visual Studio 2013

Installation

I have customized a JavaScript file that was originally included with the WTL installation wizard. This script is in main folder of the unit test zip file. Running this script will search for installations of Visual Studio on your system by looking up a key in the registry. If it detects an installation the appropriate wizard files will be copied into the C++ wizard directories of that install. A message box will appear when the script completes. This will list all of the versions of Visual Studio that were detected and for which the wizard was installed.

In the past I was able to create a very flexible project configuration that allowed the tool locations to be configured with a Visual Studio Property Sheet. This is either a .vsprops or a .props file depending on which version of Visual Studio you are using. Unfortunately, the API that I used in the wizard to associate this configuration property sheet with the project is not supported in JavaScript for Visual Studio 2010 and newer. Therefore, this became a manual step in my own environment. I have still distributed the property sheet files that I created with the wizard. However, I have chosen a solution that is less flexible but provides more convenience in test creation.

There are a variables that I have placed at the top of a few files that are responsible for configuring the location to the Python and CxxTest installations. These are the variables and files to modify before you run the installation script:

• /files/AppWiz/Scripts/1033/default.js: Project Wizard Install script. Set both of these paths to indicate where python and CxxTest have been installed so the build tool can access them during compilation.

var pythonPath = "C:\Python34\python.exe";
var cxxTestPath = "C:\Development\CxxTest";
• /SetupCxxTest_VS.js (Optional): Install script for registered versions of Visual Studio. Change this to set the name of the category the test wizard appears in Visual Studio under C++ Wizards.

var categoryDir = "CodeOfTheDamned";
• /SetupCxxTest_VS - Express.js (Optional): Install script for Express versions of Visual Studio. Change this to set the name of the category the test wizard appears in Visual Studio under C++ Wizards.

var categoryDir = "CodeOfTheDamned";

Usage

There are two types of wizards that I have developed to simplify the creation of unit test suites for Visual Studio and CxxTest.

Unit Test Project Wizard

The project wizard is essential for the start of any project. Since CxxTest requires a two step process to generate, then run the unit tests, I have configured this type of project to handle this steps as one. Therefore, all the developer is left to do is develop their code in the unit test harness as they would if it were any simple driver program. Except the boilerplate code for a test harness is already setup.

1. Create a new project. Use the CxxTest Wizard from the category you have configured.

2. Give the test suite class a name and your basic comments.

Once the wizard has completed, you will have a shell project, similar to a default console project that Visual Studio creates. This project will be a unit test suite named by you, with file comments at the top. There are stub entries for the constructor, setup and teardown routines, and a single test case. Here is the structure of the project:

• /TestSuite
• (Place all utility and shared test suite files in this directory)
• /Src
• (All test suite header definition files will be placed in here)
• TestSuite.h
• /Generated
• (Files in this directory are automatically generated)
• TestSuiteRunner.cpp

If all of the tools are configured properly, you should be able to successfully build the project. Output similar to this should be expected:

1> ------ Build started: Project: TestSuite1, Configuration: Debug Win32 ------
1>  A subdirectory or file ./Src/Generated/ already exists
1>  NewTestSuiteRunner.cpp
1>  TestSuite1.vcxproj -> c:\output\TestSuite1.exe
1>  Running 1 test.OK!
========== Build: 1 succeeded, 0 failed, 0 skipped ==========

If you would like to add a second test suite to an existing test project you can use the class wizard to generate another test suite object file. You will see the same dialog as the project wizard, which asks you for an object name, comments and author. The difference is this object will be added to your selected project. You can access the class wizard from the following menu:

This time the type of project is called a Test Fixture.

Summary

The relatively little amount of time I spent to better integrate my test environment with my development environment has been time well spent for me and my team. I have been using basically this same unit test wizard since 2009, and I have used it with all of the major releases of Visual Studio since 2005. Whatever tool you select for your development, make sure that it is easy to use. The point is to encourage the development and maintenance of unit tests for your software; not to introduce yet another tool or step in the process for the developers to fight.

Unit Test Frameworks

Send feedback »

If you ask a group of 10 software engineers to develop unit tests for the same object, you will end up with 10 unique approaches to testing that object. Now imagine each engineer was given a different object. This was my experience with unit testing before I discovered how useful and how much more valuable your unit tests become when they are developed with a test framework. A unit test framework provides a consistent foundation to build all of your tests upon. The method to setup and run each and every test will be the same. Most importantly, the output provided by each test will be the same, no matter who developed the test.

In a previously posted an entry called, The Purpose of a Unit Test, I clarified the benefits of writing and maintaining the tests, who does the work, and why you should seriously consider developing unit tests as well. I mentioned unit test frameworks and test harnesses, but I did not provide any details. In this entry I give a high-level overview of unit test frameworks and a quick start sample with the test framework that I use.

Unit Test Frameworks

Similar to most any tool in our industry, unit test frameworks can incite heated debates about why "My tool is better than yours." For the first half of my career I used simple driver programs to exercise functionality of a new object I was developing. As development progressed, I depended on my test driver less and less, and eventually the driver failed to compile successfully. I had always considered writing a driver framework to make things easier, however, testing was not something I was extremely interested in up to that point in my career.

Once I discovered a few great books written about unit testing, I quickly learned of better approaches to testing software. Some of the new knowledge I had gained taught me that writing and maintaining unit tests does not have to be that much extra work. When I knew what to look for, I found that a plethora of high quality test frameworks already existed. xUnit is the style of test framework that I use. To my knowledge SUnit is the first framework of this type, which was written for Smalltalk by Kent Beck. JUnit seems to be the most well known framework that tests JAVA.

A very detailed list of test frameworks can be found at Wikipedia. The number of frameworks listed is enormous, and it appears there are for frameworks to choose from for C++ than any other language. I assume this is due to the variety of different C++ compilers, platforms and environments C/C++ is used in. Some of the frameworks are commercial products, most are released as open source tools. Here are a few of the frameworks that I am familiar with or I have at least inspected to see if I like them or not:

• Boost Test Library
• CppUnit / CppUnitLite
• CxxTest
• Visual Studio

Qualities

There are many factors could play into what will make a unit test policy successful with your development team; some of these factors are the development platform, the type of project and even the culture of the company. These factors affect which qualities will be valuable when selecting a software unit test frameworks. I am sure there are many more than I list below. However, this is the short list of qualities that I considered when I selected a test framework, and it has served me well.

xUnit Format

There is more literature written on the xUnit test frameworks than any other style that exists. In fact, it appears that frameworks are either classified as xUnit or not. This is important because this test paradigm is propagated across many languages. Therefore, if you program in more than one language, or would like to choose a tool that new developers to your company will be able to come up to speed the quickest you should select an xUnit-style framework.

Portability

Even though Visual Studio is my editor of choice and some really nice test tools have been integrated into it with the latest versions, I have chosen a framework that is not dependent on any one platform. I prefer to write portable code for the majority of my work. Moreover, C++ itself is a portable language. There is no reason I should bound all of my tests to a Windows platform that requires Visual Studio.

Simplicity

Any tool that you try to introduce into a development process should be relatively easy to use. Otherwise, developers will get fed up, and create a new process of their own. Since unit tests are thought of by many as extra work, setting up a new test harness and writing tests should not feel like work. Ease of use will lower the resistance for your team to at least try the tool, and increase the chances of successful adoption.

Flexibility

Any type of framework is supposed to give you a basis to build upon. Many frameworks can meet your needs as long as you develop the way the authors of the framework want you to. Microsoft Foundation Classes (MFC) is a good example of this. If you stick with their document/view model you should have no problems. The further your application moves away from that model, the more painful it becomes to use MFC. When selecting a unit test framework you will want flexibility to be able to write tests however you want. Figuring out how to test some pieces of software is very challenging. A framework that requires tests to be written a certain way could make certain types of tests damn near impossible.

I suggest that you do a little bit of research of your own, and weigh the pros and cons for each framework before deciding upon one. They all have strengths and weaknesses, therefore I don't believe that any one of these is better than the others for all situations.

CxxTest

I have built my test environment and process around the CxxTest framework. When I chose this one, I worked with a wide variety of compilers, which many had poor support for some of C++'s features. CxxTest had the least compiler requirements and the framework is only a handful of header files so there are no external libraries that need to be linked or distributed with the tests. Many of the test harnesses developed for C++ early on required RTTI; this feature definitely wasn't supported on the compiler that I started using CxxTest. The syntax for creating unit tests is natural looking C++, which I think makes the tests much easier to read and maintain. Finally, it is one of the frameworks that does not require any special registration lists to be created to run tests that are written; this eliminates some of the work that would make testing more cumbersome.

The one thing that I consider a drawback is that Python is required to use CxxTest. Adding tools to a build environment makes it more complex, and one of my goals has always been to keep things as simple as possible. Fortunately, Python is easily accessible for a variety of platforms, so this has never been a problem. Python is used to generate the test runner with the registered tests, which eliminates the extra work to register the tests manually. When you work on large projects and have thousands of tests, this feature is priceless.

CxxTest is actively maintained at SourceForge.net/CxxTest

Quick start guide

I want to give you a quick summary for how to get started using CxxTest. You can access the full documentation online at http://cxxtest.com/guide.html. The documentation is also available in PDF format. There are four basic things to know when starting with CxxTest.

All of the features from CxxTest can be accessed with the header file:

C++

 #include < cxxtest/TestSuite.h>

2. Create a test fixture class

A test fixture, or harness, is a class that contains all of the logic to pragmatically run your unit tests. All of the test framework functionality is in the class CxxTest::TestSuite. You must derive your test fixture publicly from the TestSuite class. You are free to name your class however you prefer.

C++

 class TestFixture : public CxxTest::TestSuite {   // ... };

3. Test functions

All functions in your fixture that are to be a unit test must start with the word test. This is a case-insensitive name, therefore you can use CamelCase, snake_case, or even _1337_Ca53 as long as the function name starts with "Test" and it is a valid symbol name in C++. Also, test functions must take no parameters and return void. Finally, all test functions must be publicly accessible:

C++

 public: void Test(void);     // Examples   void TestMethod(void);   void test_method(void);   void TeSt_1337_mEtH0D(void);

4. (Optional) Setup / Teardown

As with the xUnit test philosophy, there exists a pair of virtual functions that you can override in your test suite, setUp and tearDown. Setup is called before each and every test to perform environment initialization that must occur for every test in your test suite. Its counterpart, Teardown, is called after every test to cleanup before the next test. I highly encourage you to factor code out from your tests into utility functions when you see duplication. When all of your tests require the same initialization/shutdown sequences, move the code into these two member functions. It may only be two or three lines per test, however, when you reach 20, 50, 100 and more a change in the initialization of each test becomes significant.

C++

 void setUp(void); void tearDown(void);

Short Example

I have written a short example to demonstrate what a complete test would look like. Reference the implementation for this function that indicates if a positive integer is a leap year or not:

C++

 bool IsLeapYear(size_t year) {   // 1. Leap years are divisible by 4   if (year % 4)    return true;     // 3. Except century mark years.   if (year % 100)   {     // 4. However, every 4th century      //    is also a leap year.     return !(year % 400);   }     // 2.    return true; }

Each individual unit test should aim to verify a single code path or piece of logic. I have marked each of the code paths with a numbered comment in the IsLeapYear function. I have identified four unique paths through this function, and implemented four functions to test each of the paths.

C++

 // IsLeapYearTestSuite.h   #include < cxxtest/TestSuite.h> class IsLeapYearTestSuite : public CxxTest::TestSuite {   public:   // Test 1   void TestIsLeapYear_False_NotMod()   {     // Verify 3 instances of this case.     TS_ASSERT_EQUALS(false, IsLeapYear(2013));     TS_ASSERT_EQUALS(false, IsLeapYear(2014));     TS_ASSERT_EQUALS(false, IsLeapYear(2015));   }     // Test 2   void TestIsLeapYear_True()   {     TS_ASSERT_EQUALS(true, IsLeapYear(2016));   }     // Test 3   void TestIsLeapYear_False_Mod100()   {     // These century marks do not qualify.     TS_ASSERT_EQUALS(false, IsLeapYear(700));     TS_ASSERT_EQUALS(false, IsLeapYear(1800));     TS_ASSERT_EQUALS(false, IsLeapYear(2900));   }     // Test 4   void TestIsLeapYear_True_Mod400()   {     // Verify 3 instances of this case.     TS_ASSERT_EQUALS(true, IsLeapYear(2000));     TS_ASSERT_EQUALS(true, IsLeapYear(2400));   } };

Before this test can be compiled, the python script must be run against the test suite header file to generate the test runner:

python.exe --runner=ParenPrinter
-o IsLeapYearTestRunner.cpp
IsLeapYearTestSuite.h

Now you can compile the output cpp file that you specify, along with any other dependency files that your test may require. For this example simply put the IsLeapYear function in the test suite header file. This is the output you should expect to see when all of the tests pass:

Running 4 tests....OK!

If the second test were to fail, the output would appear like this:

Running 4 tests.
In IsLeapYearTestSuite::TestIsLeapYear_True:
IsLeapYearTestSuite.h(19):
Error: Expected (true == IsLeapYear(2016)),
found (true != false)
..
Failed 1 of 4 tests
Success rate: 75%

When I execute the tests within Visual Studio, if any errors occur I can double click on the error and I am taken to the file and line the error occurred, much like is done for compiler errors. The output printers can be customized to display the output however you prefer. One of the output printers that comes with CxxTest is an XmlPrinter that is compatible with the Continuous Integration tools that are common, such as Jenkins.

Summary

I believe that unit tests are almost as misunderstood as how to effectively apply object-oriented design principles, which I will tackle another time. It seems that most everyone you ask will agree that testing is important, a smaller group will agree that unit tests are important, and even fewer put this concept to practice. Select a unit test framework that meets the needs of your organization's development practices to maximize the value of any tests that are developed for your code.

Introduction to Network Alchemy

Send feedback »

While many of the principles of developing robust software are easy to explain, it is much more difficult to know how and when to apply these principles. Practice and learning from mistakes is generally the most productive way to understand these principles. However, it is much more desirable to understand the principles before a flawed system is built; an example from the physical world is the Tacoma Narrows bridge. Therefore I am starting a journey to demonstrate how to create robust software. This is not a simple task that can be summarized in a magazine article, a chapter in a book or a reference application with full source code.

This journey starts now, and I expect to continue over the next few months in a series of entries. I will demonstrate and document the design and development of a small library intended to improve the quality of network communication software. Developing code for network data transmission is very cumbersome and error prone. The bugs that appear are generally subtle defects that may only appear once you use a new compiler or move to a new operating system or hardware platform. As with any type of software, there are also the bugs that can be introduced by changes. So let's begin by first identifying what we need and problems we are trying to avoid.

Inter-process Communication

The core topic of discussion is Inter-process Communication (IPC). IPC is required when to processes do not share the same memory pool by default. IPC mechanisms are used to communicate between the two separate processes, which takes the form of message transmissions and data transfers. There are many mediums that are used to communicate between two programs such as:

• Shared Memory Pools
• Pipes
• Network Sockets
• Files

Regardless of the medium used to transfer data, the process remains the same for preparing the data. This process is repetitive, mundane, and very error prone. This is the portion of the IPC process that I will focus on first. Let's breakdown the typical tasks that are required to communicate with IPC.

Serialize Structured Data

Data has many forms. A set of numbers, complex classes, raw buffers and bitmap images are examples of information that may be transferred between processes. One thing that is important to remember, is that two different programs will most likely structure the data differently. Even though the information is the same, the difference in compilers, languages, operating systems and hardware platforms may represent data in a different way than the program that originally created the data. Therefore, it is very important to convert the data into a well-defined format shared between the two endpoints of the IPC session. A packet format definition is a standard method that is commonly used to define these serialized message formats.

Interpret Serialized Data

Once the data has been transferred and received by the destination endpoint, it must be interpreted. This process is the opposite process of serialization. However, the consumer of the data does not necessarily need to be the same author of the source endpoint that created the data. Therefore, the packet format definition should be used to interpret the data that has been received. A common implementation is to take the data from the binary buffer and assign the values into fields of a structure for ease of access. This sort of approach makes the code much clearer to understand.

Translate Data Messages

Another common operation is to take an incoming message in one format and convert it to a different message format before further processing. This is usually a simple task, however if you have a large number of messages to convert, the task can quickly become mundane. In my experience, mundane tasks that require a large amount of repetitive code are very error prone done to copy/paste errors.

This is a great example of a problem that occurs frequently and a brute force approach is used to solve the problem. This should be a major concern from a maintenance perspective, because this solution generates an enormous amount of code to maintain. There is high probability that errors will be introduced if fundamental changes are ever made to this body of code. to this body of code. I plan on creating a simple and maintainable solution to this problem as part of network alchemy.

Transmit Data

Transmitting the data is the reason we started this journey. Transmission of data is simple in concept, but in practice there are many potential pitfalls. The problems that must be handled depends on the medium used to transfer the data. Fortunately, many libraries have already been written to tackle these issues. Therefore, my primary focus will be to prepare the data for use with an API designed to transfer serialized data. Because of this, it will most likely be necessary to research a few of these APIs to be sure that our library can be integrated with relative ease.

Common Mistakes

We have identified the primary goals of our library. However, I also would like to take into account experience gained from similar projects. If I don't have the personal experience related to a project I at least try to heed the wisdom offered to me by others. I always have the option to choose later whether the advice is useful or relevant to my situation. In this particular situation I have both made these mistakes myself as well as found and fixed similar bugs causes by others. I would like to what I can to eliminate these issues from appearing in the future.

Byte Order Conversions

Byte order conversion is necessary when data is transferred between two different machines with different endianess. As long as your application or library will only run on a single platform type this should not be a concern. However, because of the nature that software tends to outlive its original purpose, I would not ignore this issue. Your application will work properly for the original platform, but if it is ever to be run on two different types of platforms, the issues will begin to appear.

Standard convention is to convert numbers to Network byte order (Big Endian format) when transferring over the wire. Data extracted from the wire should then be converted to Host byte order. For Big Endian systems the general operation is a no-op, and the work really only needs to be performed for Little Endian platforms. A few functions have been created to minimize the amount of duplicate code that must be written:

C++

 // Host to Network Conversion unsigned short htons(unsigned short hostShort); // 16-bit unsigned long  htonl(unsigned long  hostLong); // 32-bit   // Network to Host Conversion unsigned short ntohs(unsigned short netShort); // 16-bit unsigned long  ntohl(unsigned long  netLong); // 32-bit   // No consistent and portable implementations exist for  // 64-bit integers, float and double values. // However, it is important that these numbers  // be converted as well.

The issue will appear if the data you transfer uses number fields with two or more bytes for representation when the correct endianess is not used for data transfer. The symptoms that appear can be as subtle as incorrect numbers are received, to more immediate and apparent issues such as the program or thread crashes. To further obfuscate the problem, it is quite common to use the correct conversion functions in most places, however, a single instance is incorrectly converted in the one of potentially hundreds of uses of these functions. During maintenance this often occurs when changing the size of a field but failing to change the conversion function used to swap the byte order before transport or interpretation.

Abuse of the Type System

Serializing data between different processes requires a predefined agreement, protocol, to dictate what should be sent and received. Data that is read from any source should be questioned and verified before it is accepted and further processed. This is true whether the source is a network communication stream, a file, and especially user input. The type information for the data structures we work with can not be statically enforced by the compiler until we are able to verify it, and put it back into a well-defined and reliable type. More often than not, I have observed implementations that are content to pass around pointers to buffers and simply trust the contents encoded in the buffer.

Here is a very simple example to illustrate the concept. An incoming message is received and assumed to be one of the types of messages that can be decoded by the switch statement. The first byte of the message indicates the type of message, and from there the void* is blindly cast to the 1 of 256 possible message types indicated by the first byte.

C++

 void ProcessMsg(void* pVoid) {   if (!pVoid)   {     return;   }     char* pType = (char*)pVoid;   switch (*pType)   {   case k_typeA:     ProcessMessageA((MessageA*)pVoid);     break;   case k_typeB:     ProcessMessageB((MessageB*)pVoid);     break;   // ...   } }

Any address could be passed into this function, which would then be interpreted as if it were a valid message. This particular problem cannot be completely eliminated simply because eventually the information is reduced to a block of encoded bytes. However, much can be done to both improve the robustness of the code as well as continue to provide convenience to the developer serializing data for transport. It is important that we keep as much type information around through as much of the code as possible. When the type system is subverted, even more elusive bugs can creep into your system. It is possible to create code that appears to work correctly on one system but becomes extremely difficult to port to other platforms, and although the program works on the original system, it may be running at a reduced performance level.

Properly Aligned Memory Access

One of the key features of the C++ is its ability to allow us to develop at a high layer of abstraction or drop down and code right on top of the hardware. This allows it to run efficiently in all types of environments. Unfortunately when you peek below the abstractions provided by the language it is important that you understand what you are now responsible for that the language normally provides for you. A memory access restriction is one of these concepts that are often hidden by the language. When these rules are broken, the best you can hope for is decreased performance; more likely an unaligned memory access exception will be triggered by the processor.

Generally a processor will require a address to be aligned along the same granularity based on the size of data to be read. By definition, reading a single byte is always aligned with the appropriate address. However, a 16-bit (2 byte) value must be located at an address that is divisible by a power of two. Similarly a 32-bit (4 byte) value must be aligned at an address divisible by 4. This rule is not absolute due to the different ways processors perform memory access. Many processors will automatically make the adjustment to create a 4 byte read as a series of smaller single or double byte operations and combine the results in the expected 4 byte address. This convenience comes at a cost of efficiency. If a processor does not handle misaligned access, an exception will be raised.

Running into this issue often confounds developers when they run into this issue. Thinking, "It works on system A, why does it crash on system B?" There is nothing syntactically wrong with the code, and even a thorough inspection of the code would not reveal any glaring issues with the code. I demonstrate two common implementations that I have run across when working with serialized data communications that can potentially lead to a misaligned memory access violation.

Both examples use this structure definition to provide a simplified access to the fields of the message. This structure is purposely designed to create a layout where field2 is not aligned on a 4-byte boundary:

C++

 struct MsgFormat {   unsigned short field1;     // offset 0   unsigned long  field2;     // offset 2        unsigned char  field3;     // offset 6 };                           // Total Size: 7 bytes

Blind Copy

The code below assumes that the MsgFormat structure is layer out in memory exactly as the fields would be if you added up the size of each field and offset it from the beginning of the structure. Note also that this example ignores byte-order conversion.

C++

 // Receive data... const size_t k_size = sizeof(MsgFormat); char buffer[k_size];   // Use memcpy to transfer the data  // into structure for convenience. MsgFormat packet; ::memcpy(&MsgFormat, buffer. k_size);   short value_a = packet.field1;    // May cause an unaligned memory access violation //   Or  // The expected may not be held in field2.   long  value_b = packet.field2;

The problem is the compiler is allowed to take certain liberties with the layout of data structures in memory in order to produce optimal code. This usually done by adding extra padding bytes throughout the structure to ensure the individual data fields are optimally aligned for the target architecture. This is what the structure will most likely look like in memory:

C++

 struct MsgFormat {   unsigned short field1;   // offset 0   unsigned short padding;  // offset 2   unsigned long  field2;   // offset 4   unsigned char  field3;   // offset 8 };                         // Total Size: 9 bytes

Dereferencing Unaligned Memory

This example demonstrates a different way to trigger the unaligned memory access violation. We recognize that byte-order conversion has not been properly handled in the previous example, so lets rectify that:

C++

 // Receive data... const size_t k_size = sizeof(MsgFormat); char buffer[k_size];   // Appropriately convert the data  // to host order before assigning it // into the message structure. MsgFormat packet;   Packet.field1 = ntohs(*(unsigned short*) buffer);   // The pointer assigned to p_field2 // is aligned on a 2-byte boundary rather than 4-bytes. // Dereferencing this address will cause an  // access violation on architectures that  // require 4-byte alignment. char* p_field2 =  buffer + sizeof(unsigned short); Packet.field2  = ntohl (*(unsigned long *)p_field2);

Even more insidious memory alignment issues can appear when message definitions become large sets of nested structures. This is a good design decision in many ways, it will help organize and abstract groups of data. However, care must be taken to ensure adding fields in a deeply nested child does not cause memory alignment issues. One must always be aware of the possibility for pointers passed into a function may not be properly aligned. In practice I think this is a responsibility that should fall upon the function caller as this is a rare but very real possibility to be aware of.

One final practice that could be a cause of future pain is pointers embedded in the message structure. It is common to see offset fields indicate the location of a block of data in a message structure. This can be done safely, as long as validity checks are made along the way.

Message Buffer Management

Much of the code that I have worked with did a fine job of managing buffers appropriately. Allocations are freed properly as expected. Buffer lengths are verified. However, it appears that much of the misused techniques that I described above, are actually driven by the developers attempt to reduce memory allocations and eliminate copying. Therefore, another goal that I will try to integrate into Alchemy, is efficient and flexible buffer management. Specifically aiming towards minimal allocations, minimal copying of buffers, and robust memory management.

Summary

We have identified a handful of common problems that we would like to address for the Network Alchemy library. The next step is to explore some strategies to determine what could feasibly solve these problems in a realistic and economical way. I will soon add an entry that describes the unit-test framework that I prefer, as well as the tools I have developed over the years to help improve my productivity of developing in an independent test harness.

C++: using and namespace

Send feedback »

using and namespace are two of the most useful C++ keywords when it comes to simplifying syntax, and clarifying your intentions with the code. You should understand the value and flexibility these constructs will add to your software and it maintenance. The benefits are realized in the form of organization, readability, and adaptability of your code. Integration with 3rd party libraries, code from different teams, and even the ability to simplify names of constructs in your programs are all situations where these two keywords will help. Beware, these keywords can also cause unnecessary pain when used incorrectly. There are some very simple rules to keep in mind, and you can avoid these headaches.

At its core, The Compiler, is an automaton that works to translate our code that is mostly human readable,  a form understood by your target platform. These programs are works of art in and of themselves. They have become very complex to address our complex needs in both our languages and the advances in computing in the last few decades. For C/C++, the compiled module is not capable of running on the computer yet, the linker needs to get involved.

The compiler create a separate compiled module for each source file (.c, .cpp, .cc) that is in your program. Each compiled module contains a set of symbols that are used to reference the code and data in your program. The symbols created in these modules will be one of three different types:

1. Internal Symbol: An element that is completely defined and used internally in the module.
2. Exported. Symbol: An element this is defined internally to this module, and advertised as accessible for other modules.
3. Imported Symbol: An element that is used within a module, however the definition is contained with another module. This is indicated with the extern qualifier.

Now it's time for The Linker to take each individual module and link them together; similar to stitching together the individual patches in a quilt. The Linker combines all of the individual modules, resolving any dependencies that were indicated by The Compiler. If a module is expecting to import a symbol, the linker will attempt to find that symbol in the other set of modules.

If all works out well, every module that is expecting to import a symbol will now have location to reference that symbol. If a symbol cannot be found, you will receive a linker error indicating "Missing Symbol". Alternatively, if a symbol is defined in multiple modules The Linker will not be able to determine which symbol is the correct symbol to associate with the import module. The Linker will issue a "Duplicate Symbol" error.

Namespaces

The duplicate symbol linker error can occur for many reasons, such as:

• A function is implemented in a header file without the inline keyword.
• A global variable or function with the same name is found in two separate source code modules.
• Adding a 3rd party library that defines a symbol in one of its modules that match a symbol in your code.

The first two items on the list are relatively easy to fix. Simply change the name of your variable or function. Generally a convention is adopted, and all of the names of functions and variables end up with a prefix that specifies the module. Something similar to this:

C++

 // HelpDialog.cpp   int g_helpDialogId; int g_helpTopic; int g_helpSubTopic;   int HelpCreateDialog() {   // ... }

This solution works. However, it's cumbersome, won't solve the issue of a 3rd party library that creates the same symbol and finally, it's simply unnecessary in C++. Place these declarations in a namespace. This will give the code a context that will help make your symbols unique:

C++

 // HelpDialog.cpp   namespace help {   int g_dialogId; int g_topic; int g_subTopic;   int CreateDialog() {   // ... }   } // namespace help

The symbols in the code above no longer exist in the globally scoped namespace. To access the symbols, the name must be qualified with help::, similarly to referencing a static symbol in a class definition. Yes, it is still entirely possible for a 3rd party library to use the same namespace. Namespaces can be nested. Therefore to avoid a symbol collision such as this, place the help namespace into a namespace specified for your library or application:

C++

 namespace netlib { namespace help {   // ... Symbols, Code,     } // namespace help } // namespace netlib

Namespaces Are Open

Unlike a class definition, a namespace's declaration is open. This means that multiple blocks can be defined for a namespace and the combined set of declarations will live in a single namespace. Multiple blocks can appear in the same file and separate blocks can be in multiple files. It is possible for a namespace block to spread across two library modules, however, the separate libraries would need to be compiled by compiler that uses the same name-mangling algorithm. For those that are unaware, name-mangling is the term used to describe the adornments the C++ compiler gives to a symbol to support a feature such as function overloading.

C++

 namespace code { namespace detail { // Forward declare support functions symbols int VerifySyntax(const string &path); }    // Main implementation   namespace detail { // New symbols can be defined and added bool has_error = false; // Implement functions   int VerifySyntax(const string &path) {   // ... }   } // namespace detail } // namespace code

The Unnamed Namespace

The static keyword is used In C to declare a global variable or a function, and limit its scope to the current source file. This method is also supported in C++ for backward compatibility. However, there is a better way hide access to globally scoped symbols; use the unnamed namespace. This is simply a defined namespace that is given a unique name, only accessible to the compiler. To reference symbols in this namespace, access it as if it lived in the global namespace. Each module is given their own unnamed namespace. Therefore it is not possible to access unnamed namespace symbols defined in a different module.

C++

 namespace // unnamed { int g_count; } // namespace (unnamed)   // Access a variable in the unnamed namespace // as if it were defined in the globally scoped namespace int GetCount() {   return g_count; }   } // namespace (unnamed)

The code above is an example for how to protect access to global variables. If you desire a different source module to be able to access the variable, create a function for other modules to call to gain access to the global variable. This helps keep control of how the variable is used, and control how the value of the variable is changed.

Alias a Namespace

Namespaces share the same rules defined for naming functions and variables. Potentially long namespace names could be created to properly/uniquely describe a set of code. For example:

C++

 namespace CodeOfTheDamned { namespace network { enum Interface {   k_type1 = 1,   k_type2,   k_type3  }   class Buffer {   // ... }   } // namespace network } // namespace CodeOfTheDamned

The fully scoped names that these definitions create could become quite cumbersome to deal with.

C++

 CodeOfTheDamned::network::Interface intf = CodeOfTheDamned::network::k_type1;   If (CodeOfTheDamned::network::k_type1 == intf) {   CodeOfTheDamned::network::Buffer buffer;   // ... }

Compare and contrast this with the code that did not use namespaces:

C++

 Interface intf = k_type1;   If (k_type1 == intf) {   Buffer buffer;   // ... }

Possibly the top item on my list for creating maintainable software is to make using existing declarations easy to understand and use. Typing a long cumbersome prefix is not easy. I like to keep my namespace names between 2 to 4 characters long. Even still the effort required to specify a fully qualified path becomes painful again once you hit the second nested namespace; 3 namespaces or more is just sadistic. Enter, the namespace alias. This syntax allows you to redeclare an existing namespace with an alias that may be simpler to use. For Example:

C++

 // Namespace Alias Syntax namespace cod  = CodeOfTheDamned; namespace dnet = cod::network;   // Example of new usage If (dnet::k_type1 == intf) {   dnet::Buffer buffer;   // ... }

This is much nicer, simple, convenient. We can do better though. There is one other keyword in C++ that helps simplify the usage of namespaces when organizing your code, using.

Using

using allows a name that is defined in a different declarative region to be defined in the same declarative region, which using appears. More simply stated, using adds a definition from some other namespace to the same namespace using is declared.

C++

 // Syntax for using // Bring a single item into this namespace using std::cout; using CodeOfTheDamned::network::Buffer;   // Now these symbols are in this namespace as well as their original namespace: cout << "Hello World"; Buffer buffer;

Using with Namespaces

The ability to bring a symbol far far away from another namespace is greatly simplified with using. using can also bring the contents of an entire namespace into the current declarative region. However, this particular usage should be used sparingly because to avoid defeating the purpose of namespaces. The contents of two namespaces are combined together. One absolute rule that I would recommend for your code guidelines, is to prohibit the use of using in header files to bring namespaces into the global namespace.

C++

 // Syntax for using // Bring a single item into this namespace using namespace std; using namespace CodeOfTheDamned;   // The entire std namespace has been brought into this scope cout << "Hello World" << endl; // The CodeOfTheDamned namespace was brought to us. // However, qualifying with the network sub-namespace // will still be required. network::Buffer buffer;

My preferred use of using simple declarations at the top of the function allows me to quickly see which symbols I am pulling in to use within the function, and I simplify the code at the same time. Only the specific symbols I intend to use are brought into the scope of the function. I limit what is imported, to the set of symbols that are actually used.:

C++

 // using within a function definition // Forward declaration void Process(int number);   void ProcessList(NumberList &numbers) {   using std::for_each;     // Preparations ...     for_each(numbers.begin(),            numbers.end(),            Process);   // ... }

using Within a Class

using can be used within a class declaration. Unfortunately it cannot be used to bring namespace definitions into the class scope. using is used within a class scope to bring definitions from a base class into the scope of a derived class without requiring explicit qualification. Another feature to note, is the accessibility of a declaration can be modified in a base class with using.

C++

 // Syntax for using // Bring a symbol from a base class into this class scope. class Base { public:   int value;     // ... };   class Derived   : private Base { public:   // Base::value will continue to be accessible   // in the public interface, even though all    // of the Base classes constructs are hidden.   using Base::value;   };

This feature becomes necessary when working heavily with templates. If you have a template class that derives from a base class template, the compiler will not look in the base class for a symbol. This is to error on the side of caution and generate errors sooner in the compile process rather than later. The using keyword is one way to provide a hint to the compiler that it can find a symbol in the base template type.

Summary

using and namespace are two very useful declarations to be aware of in C++ to help create a balance between portability, adaptability and ease of coding. The ability to define namespaces allows code symbols from separate libraries to be segregated to prevent name collisions when using libraries developed by multiple development teams. The keyword using allows the developer to bring specific elements from a namespace into the current declarative scope for convenience.

A little care must be taken to ensure that over-zealous use of the using keyword does not undermine any organizational structure created with namespace. However, with the introduction of a few conventions to your coding standards, the effort required to properly organize your code into logical units that avoid name collisions can be kept to a minimum. The importance that you invest in a namespace structure increases with likelihood that your code is to be ported across multiple platforms, to use 3rd party libraries, or to be sold as a library. I believe the results are well worth little effort required.

Contact / Help. ©2017 by Paul Watt; Charon adapted from work by daroz. CMS / cheap web hosting / adsense.
Design & icons by N.Design Studio. Skin by Tender Feelings / Skin Faktory.