Embedded Alchemy

Alchemy Send feedback »

Alchemy is a collection of independent library components that specifically relate to efficient low-level constructs used with embedded and network programming.

The latest version of Embedded Alchemy[^] can be found on GitHub.
The most recent entries as well as Alchemy topics to be posted soon:
Alchemy: Documentation[^]

I just completed my Masters Degree in Cybersecurity at Johns Hopkins University. I plan to resume Alchemy's development. I plan to use my newly acquired knowledge to add constructs that will help improve the security of devices built for the Internet of (Insecure) Things.

Alchemy: BitLists Mk2

CodeProject, C++, maintainability, Alchemy, design, engineering Send feedback »

A continuation of a series of blog entries that documents the design and implementation process of a library. The library is called, Network Alchemy[^]. Alchemy performs low-level data serialization with compile-time reflection. It is written in C++ using template meta-programming.

With my first attempt at creating Alchemy, I created an object that emulated the behavior of bit-fields, yet still resulted in a packed-bit format that was ABI compatible for portable wire transfer protocols. You can read about my design and development experiences regarding the first attempt here Alchemy: BitLists Mk1[^].

My first attempt truly was the epitome of Make it work. Because I didn't even know if what I was attempting was possible. After I released it, I quickly received feedback regarding defects, additional feature requests, and even reported problems with it's poor performance. This pass represents the Make it right phase.


Feedback is an important aspect for any software development project that has any value, and would like to continue to provide value. Feedback appears in many types of formats including:

  • Bug Reports
  • Feature Requests
  • Usability
  • Performance Reports
  • Resource Usage

I realized there were limitations with the first implementation of the BitLists in Alchemy. However, I had created a simple proof-of-concept implementation for version 1, and I needed to finish the serialization and reflection features for the library to provide any value at all.

Imagine my surprise when I received the first requests to expand the capabilities of the BitLists from supporting 8 sub-fields to 32. This was already the most expensive structure as far as size in the library, and expanding its bit-field support to 32 quadrupled the size. Therefore, I knew I would have to revisit the design and implementation of this feature rather quickly.

Re-engineering the BitList

My primary concern with the first implementation of BitList was all of the predefined data fields that it contained. For each possible bit-field managed by the BitList, there was an instance of that variable in the object; whether that field was required or not. The primary reason, all of this work is occurring at compile-time. My initial choices appeared to be:

  1. Predeclare all of the fields up-front
  2. Create a different implementation for each BitList length

I often mull over projects in my head while I'm driving, walking and during other idle situations. It didn't take long for me to realize that I had already solved this problem of variable field length in other contexts of Alchemy. I solved it using list objects, just like the TypeList.

With the TypeList, I still only had a single implementation that reserves a slot for up to 32 fields. However, the use of code generatng MACROs and default template parameters allowed me to make this solution transparent to the user. My situation with the BitList is slightly different than the TypeList; I did have an idea to account for this difference.

BTW, if you are wondering why a list-based approach did not occur to me earlier, I had originally called this object a BitSet. I changed the name because of this new approach.

The new approach

My new approach would be to create a host, or head, list object that managed all of the external interfaces and details. Internally, I would build up the list one node at a time until the desired number of entries were defined within the BitList itself. The difference between this BitList and the TypeList is the TypeList does not contain any data or variables that can be referenced by the user. The BitList on the other hand, is an object that is fully defined at compile-time and performs real work at run-time.

The final structure is an embedded linked-list. Similar to the TypeList, each node contains a way to reference the next node in the list. This could possibly become a problem that would impede performance for a run-time based solution. However, once the compiler processes this structure, the code will perform as if it is a flattened array.

I estimated the modifications to be relatively benign to all of the objects outside of the new BitList because it was not dependent on any external objects. Moreover, the meta-functions that I created to calculate the type, size, and offsets of the message fields were generic templates that would recognize the new BitList as the previous BitSet.

The implementation

Modifying the code to use this new format was not terribly difficult. I did struggle with the recursive nature of the algorithm to be able to acheive the same results as the original BitSet. The primary issue that I encountered was how to correctly terminate the list. I would end up with one too-many sub-fields, or the bit-count would not be correct.

This is a situation where one can truly experience the value of having a strong collection of unit-tests written for an object. Very few of my tests required any type of adjustment, and I could quickly tell if my refactoring efforts made the system at least as good as it was before, or worse.

Let's take a brief look at the internals of this implementation.

The BitList host

Similar to my first attempt, I created a base class to root the variable to hold the data and basic contruction logic for these fields. This type also helps determine when the end of the list has been reach. However, this class cannot be the sole terminator as I will demonstrate momentarily.


template< typename T,
          typename FieldsT >;
struct BasicBitList
  : public bitfield_trait
  typedef BitField< 0, 0, value_type >;    nil_bits_t;
  value_type            *m_pData;
  value_type            temp_data;
  // Can only be instantiated by a derived class.
    : m_pData(&temp_data)
    , temp_data(0)
  { }
  BasicBitList(value_type &data_field)
    : temp_data(0)
    m_pData = &data_field;

One more thing I would like to point out that never sat well with me, were the two data fields that I defined. The first is a pointer called, m_pData, which actually points to the real storage location for the bit-field data. This pointer is then passed into each bit-field node to allow the nodes to all reference the same piece of data.

The second defined value is called, temp_data, and acts as a place-holder for an uninitialized BitList. This allows the constructor to always assign the field, m_pData, to a valid memory location. It also allows the use of references in the bit-field node objects, which then avoids the need to check for NULL pointers. Still, this part of my implementation always felt like slapping a band-aid on a symptom, rather than solving the actual problem. I am able to eventually remove this band-aid, but not until my third iteration of the BitList.

Let's move from the base class to the top-level host for the BitList, BitFieldList. This first section of logic shows the declaration of the class, common typedefs and constants. This is the class that acts as the head; of the linked-list structure. I am able to make all of the activity resolve itself at compile-time by deriving from a BitFieldNode, rather than declaring the BitFieldNode as an actual data member of this class.

The parameterized type, RootT, accepts the BasicBitList shown previously. This is passed through each BitFieldNode in the linked list until the terminator. This allows each BitFieldNode to have access to the same member data that all of the BitFieldNodes share.


template< typename RootT,
            typename SeqT>
struct BitFieldList
  : public BitFieldNode< RootT, 0, 0, SeqT >
  typedef typename
    RootT::value_type                      value_type;
  typedef typename
    BitFieldNode< RootT, 0, 0, SeqT >      base_type;
  //  Constants *********************************
    const size_t k_size  = sizeof(value_type);

The only other definitions for this class are the set of constructors that initiate the cascading initialization process that occurs as each derived BitFieldNode is initialized.


// Default constructor
    : base_type()
    RootT::m_data = 0;
  // Const Value constructor,
  // Typically used for temporary instances.
  // This call will use internal memory to store values.
  BitFieldList(const value_type &data_field)
    : base_type()
  // Value constructor.
  BitFieldList(value_type &data_field)
    : base_type(data_field)
  { }

The BitFieldNode

This class is the innovation of the second iteration of the BitList object. Rather than storing a fixed collection of these types in the container object, a compile-time linked-list is created by deriving the current node from the next node in the sequence. This would most likely cause a very inefficient implementation if it were a solution based on dynamic polymorphism. However, I found through analysis of the compiled code that the overall structure is flattened out by the compiler, and becomes as efficient as if they were all declared as peers in the same container.


template< typename RootT,
            size_t    IndexT,
            size_t    OffsetT,
            typename  SeqT>
struct BitFieldNode
  : BitFieldNode< RootT,
                  OffsetT+Front< SeqT >::value,
                  typename GetTail< SeqT >::type
  //  Typedef **************************************
  typedef typename
    GetTail< SeqT >::type                  tail_type;
  typedef typename
    BitFieldNode < RootT,
                   OffsetT+Front< SeqT >::value,
                 >                         base_type;
  typedef typename
    RootT::value_type                      value_type;

While the BitFieldNode derives from the next node in the sequence, this instance defines a BitField object to manage the mask and shift operations to operate on the appropriate bits in the common storage data. The BitField object remains unchanged from the version I used in the first BitList implementation. It is also the same version that I have previously written about.

The salient detail to recognize in the constructors, is the base class is first initialized, then the local BitField is initialized. Remember the base type will eventually reach the BasicBitList that houses and initializes the actual storage data for this entire structure. Therefore on the return pass, each BitFieldNode is initialized with valid data starting from the tail node and finishing at the head.


// Default Constructor.
    : base_type()
    , m_field( RootT::GetFieldAddress(m_field) )
  // Value constructor
  BitFieldNode(value_type &data_field)
    : base_type(data_field)
    , m_field(GetFieldAddress< IndexT >(m_field))
    // assigns the reference of data_field
    // to the data member, m_data.
  BitField< OffsetT,
            Front< SeqT >;::value,
            value_type >            &m_field;

The BitFieldNode Terminator

At the end of the set of nodes in the linked-list, is a specialization of the BitFieldNode that derives from the passed in BasicBitList rather than the next node in the chain. I cannot recall the exact troubles that I ran into trying to find the correct set of definitions to properly terminate this list. I do remember that I continued to switch back-and-forth between having an incorrect number of sub-nodes in the list and an incorrect offset for each field.

The code below is the final correct solution at which I arrived. The first block contains the declaration and the constructors.


template< typename  RootT,
            size_t    IndexT,
            size_t    OffsetT
struct BitFieldNode< RootT, IndexT, OffsetT, MT>
  : public RootT
  //  Typedefs *******************************
  typedef typename
    RootT::value_type              value_type;
  BitFieldNode(const BitFieldNode &rhs)
    : RootT(rhs)
  { }
  BitFieldNode(value_type &data_field)
    : RootT(data_field)
  { }

The function below is something that I carried over from the first implementation as well. It allows internal references to the actual BitField objects to be queried by index. The basic mechanism that allows the compile-time reflection to occur over this type.


// A parameterized method that allows the
// BitList heirarchy to pragmatically access
// the different bit-fields defined in the list
// without knowledge of the name of the field.
template< size_t    Idx,
          typename  field_t >
field_t& GetFieldAddress(const field_t&)
  typedef FieldIndex< Idx, RootT, field_t::k_size> field_index_t;
  // Calls a template member function "GetField"
  //  found in the base class "RootT".
  // The static_cast from this to a type "RootT"
  // is required to give a hint to the compiler where
  // to look for the "GetField" function.
    static_cast< RootT* >(this)->template
                     GetField< field_index_t >(field_t());

The results

Overall I was pleased with this solution, especially compared to my first-pass attempt. This version felt more like a Make it Right type of solution, rather than the starting phase of Make it Work. A few portions of the implementation still didn't sit right with me. For example, an extra dead-weight temporary to ensure that I never dereferenced a NULL pointer.

There is one other source of pain that I experienced with this implementation. This is a result of the deep-derivation that I used to create the compile-time linked list. When I was working through the defects and creating new tests, I spent a fair amount of time in the debugger inspecting the contents of BitList variables and the data found in each node. Unfortunately, this data structure was not very convenient to troubleshoot.

Here are two snapshots of what the following BitList definition looks like when expanded in the Visual Studio debugger:


HG_BEGIN_BITLIST (uint32_t, mixed_set)
  HG_BIT_FIELD   (0,   first,   5)
  HG_BIT_FIELD   (1,   second,  4)
  HG_BIT_FIELD   (2,   third,   3)
  HG_BIT_FIELD   (3,   fourth,  2)
  HG_BIT_FIELD   (4,   fifth,   1)
  HG_BIT_FIELD   (5,   sixth,   16)
  HG_BIT_FIELD   (6,   seventh, 1)

It is easy to see that the value stored in the entire BitList is 0xfeedcodf. However, that is also the value displayed when the bit-fields are expanded. Therefore the debug data viewer provides much less value than I had hoped it would.

This view is an expanded view of just the type names for each field in the derived BitFieldNode chain.

Needless to say this irritated me enough that I eventually created some custom Visual Studio visualizers that I added to Alchemy to correct this inconvenience.


The usage of the BitList changed from the point-of-view of the final user. However, I was much more satisfied with this solution than the original solution. This solution expands gracefully to any length the user defines (within reason). The amount of space required for each field is reduced to only require space if a BitFieldNode is defined. Therefore, only as much space is allocated as is required for the data structure.

Unfortunately, this data type still remains the most expensive data type that is supported by Alchemy. One pointer is added for each BitField specified in the BitList. This amounts to 4*32 = 128 bytes on a 32-bit system for a BitList with 32 1-bit flags. This is in addition to the other overhead incurred for each type.

I was pushing off my worries about performance as I was not ready yet to write the benchmark tests. I will admit I was concerned, and as you will soon find out, I discovered some very compelling reasons to make my third attempt at implementing the BitList.

Coupling and Cohesion

adaptability, CodeProject, maintainability, design 9 feedbacks »

Coupling and Cohesion are two properties that deserve your attention when you design software. These are important because they have a direct effect on how flexible and maintainable your software will be as your software continues to e developed. Will you be able to reuse it? Will you be able to adapt it? Will you need a shoe-horn to force that new feature in the future? These questions are much simpler to answer when you can properly gauge the level of Coupling and Cohesion of your components.


Coupling and Cohesion

The source of the terms Coupling and Cohesion originated from the 1968 National Symposium on Modular Programming by Larry Constantine. A more commonly referenced source is from a later published paper called, Structured Software Design (1974), Larry Constantine, Glenford Myers, and Wayne Stevens. This paper discusses characteristics of “good” programming practices that can reduce maintenance and modification costs for software.

Loose (Low) coupling and High (Tight) cohesion are the desirable properties for components in software. Low coupling allows components to be used independently from other components. High cohesion increases the probability that a component can be reused in more places, by limiting its capabilities to small well-defined tasks. For those of you keeping score at home, high cohesion is the realization of the S, Single Responsibility, in SOLID object-oriented design.



A measure of the degree to which a component depends upon other components of the system.

Given two components X and Y:
If X changes interface interface how much code in Y must be changed?

Low coupling is desired because it allows components to be used and modified independently from one and other. 

The next two rows demonstrate coupling dependencies.


Very generic coupling, the most compatible bricks.




A measure of how strongly related the various functions of a single software component are.

Given a component W:
How focused is W at fulfilling a single responsibility?

High cohesion is desired because:

  • Leads to a simpler minimal class that embodies one concept
  • A minimal class is easier to comprehend and use
  • Therefore it is more likely that this class will be reused 

These pieces demonstrate cohesion over a range of interfaces.

Highest cohesion:


Lowest Cohesion:



Reuse of Code

For decades many great minds have envisioned software development to require much less effort than it requires today. Each new process, concept, language or framework has high hopes to be that solution. For example, consider the number of projects built specifically to make JavaScript more manageable. We want to encapsulate the details and abstract away the complexity to provide a clean interface.

It’s important to set your expectations for what to consider reusable code. Without a clear set of expectations, how will you know when you have achieved your goals. More importantly, how can you formulate a design strategy or a reproducible process to consistently create reusable code.

Not all code is reusable

To further clarify this statement, not all code should be reused.

Why Not?

Because the majority of software that is written is designed to fulfill a very specific purpose. The challenge is to find the balance point between generic reusable building blocks, and a tool or application that meets a need. Generic software components are very valuable, however, they are almost useless by themselves.

What good is a linked-list without a mechanism to populate the data in the list, process the data to perform useful outcomes, and later inspect the data. Another example could be a button control for a GUI that handles the name, display and animations of the button. However, it is the actions that are triggered by the button that are important to the end user.

Good design and practices should be used for all software that you write. However, recognize and differentiate which portions of your application are generic and likely to be reused compared to the code that is very specialized. Both of these are of equal importance.

The value of reusable code

For every statement of code that you write or use, you should ask the question:

Does this add value or risk?

Reusable code only provides value If it:

  • Is easy to learn and use
  • Requires less work -> time -> money to develop your final target
  • Has already been tested and proven to be robust
  • Results with a project that is at least as robust as the code that it is built upon

If any one of these qualities is undetermined, you may be adding risk along with any perceived value for the reused code.

Increase the potential for code reuse

Here are some design guidelines that can help increase the potential for your code and components to be reusable:

Easy to use correctly, hard to use incorrectly

If you are fan of Scott Meyers’, Effective C++, you will probably recognize this from Item 18:

“Ideally, if an attempted use of an interface won’t do what the client expects, the code won’t compile; and of the code that does compile, it will do what the client wants”

Good interfaces are easy to use correctly and hard to use incorrectly. The example that is used to demonstrate this concept in Effective C++ uses a calendar date class to illustrate the point. Start with a constructor such as this:


class Date
  Date(int month, int day, int year);

This is a reasonable declaration if you live in the United States, which uses the format month/day/year. However, many other countries prefer the format day/month/year. Other possibilities exist such as the ISO 8601 date order with four-digit years: YYYY-MM-DD (introduced in ISO 2014), which is specifically chosen to be unambiguous.

Another problem with this interface is there are three integer input parameters. Without refering to some form of documentation, it may be impossible to deduce the order of the parameters. These are some of the possibilities for how the interface could be misused:


Date d1(2015, 30, 3); // Non-US date format
Date d2(30, 3, 2015); // Listing the day first
Date d3(1, 30, 2015); // There's only 28 days in Feb.
                      // On yeah, our months are 0-based.

One possible approach to solving all of the issues above is to create new types to improve clarity and enforce compile-time correctness.


class Month
  static Month Jan() { return Month(1);}
  static Month Feb() { return Month(2);}
  explicit Month(int m) { ... }

Declaring the only constructor privately forces the user to use the static member functions to create new instances of a Month object. Similar definitions could be created for the day and year parameters. The improved usage would look like this:


Date d(Month::Feb(), Day(20), Year(2015));

To encourage correct use, design interfaces that are consistent and behaviorally compatible.

Use these techniques to help prevent errors:

  • Create new types
  • Restrict operations on those types
  • Constrain object values
  • Eliminate client resource management

Create interfaces that are minimal and complete

Strive to create the minimal number of operations in your interface to access the complete behavior of the object. Minimal objects embody one concept at the right level of granularity. Also, minimal interfaces are easier to comprehend, and more likely to be used and reused.

Monolithic classes will generally dilute encapsulation. Instead of doing one thing well, they end up providing many things that are mediocre. It is more difficult to make these objects correct and error-safe because they tackle multiple responsibilities. These types of objects usually result from an attempt to deliver a complete solution for a problem.

Encapsulate - Hide/Protect Information

Encapsulation is one of the most valuable aspects of object-oriented programming. Encapsulation is necessary to create a proper abstract design. The purpose of abstraction is to simplify concepts by hiding the details of the abstraction.

A proper abstraction will handle the dirty details required to achieve a correct and error-free implementation. The hidden details verify the invariants of your object each command and when internal state or data is changed. Abstractions become even more effective when a number of internal variables can be managed, and a reduced set of values are presented through the public interface.

3 made up facts… Eh, but probably true

  1. Programmers like to get the job done
  2. Programmers are inventive
  3. If a programmer finds a way to complete a task, even if it requires them to blatantly ignore the comment below, they will still write to the data; Because it will get the job done (refer to factioid 1)


class HopelesslyCoupledFromTheStart
  // This is a very sensitive value, you can read but
  // do NOT write to this value even to get the job done!
  double m_calibrationOffset;  

Data Members

Declare all data members private. Private data is teh best means a class can use to protect its invariants now as well as in the future as it adapts to possible changes.

Public data members move the responsibility to maintain and guarantee the invariants from the object to all of the code that accesses the abstraction. Freely providing access to an objects data members effectively negates most of the benefits gained from creating the abstraction.

Metaphorical Thought Experiment

WARNING: Visualize this thought experiment at your own risk.

Next time you are considering creating a public data member, think of the human body as an abstraction of a much more complicated biological organism. One where the designer declared all of the data members public rather than private.

  • Your internal organs would now all be external organs ripe for the picking of any black-market human organ peddeler.
  • All figures of speech related to the body must be interpretted literally: “What’s on your mind?”
  • There will be many more Do-it-yourself doctors.
  • Negates the benefits of being declared a friend.
  • Couples do not need to worry about tight coupling, however, they will need to worry about becoming tangled.
  • Humans in space?! Forget it!

The inferiority of non-private data members

Non-private data members are almost always inferior to even simple pass-through getter/setter methods. These methods provide the flexibility in the future to adapt to change as well as these benefits:

  • Verify Invariants
  • Lazy Evaluation / Data Caching
  • Instrumentation
  • Decouple for Future Modifications

Protected data members suffer from the same disadvantages as public data members. The responsibility to maintain the invariants is shared with the unbounded set of objects which includes:

  • Every class currently derived from this class
  • Every class that will be derived from this class in the future

The one exception is the C-style struct or plain-old data (POD), which is used as a behaviorless aggregate and contains no member functions.

Use the weakest coupling relationship possible

Let’s examine the types of relationships that are possible in C++. The list below is ordered by a measure of coupling, tightest to loosest:

  • friend-ship
  • public inheritance
  • Nonpublic inheritance
  • Composition (member variable)

Choose the coupling relationship that provides the weakest level of coupling possible. This will provide more flexibility when something needs to be changed in the future.

Prefer writing non-member non-friend functions

This will improve encapsulation by minimizing dependencies, therefore, reduces coupling. The function implementation cannot be modified to depend on the private member data of the class. It can only be implemented based on what is publically accessible.

This will also break apart monolithic classes. The separable functionality can then be further encapsulated by composition.

Example: std::string contains 103 public member functions, 71 could be non-member non-friend implementations.

Use inheritance to achieve substitutability

Inheritance is a wonderful tool available with object-oriented programming. However, it should not be selected as a way to reuse code. Instead, use inheritance as a way to facilitate polymorphism.(For those of you keeping score at home, this concept is the L, Liskov substitutability principle, in SOLID object-oriented design.)

When you consider inheritance, seriously contemplate the is-a relationship between your objects. Think of is-a as “Can be substituted with…".

Public inheritance facilitates polymorphism, which implies substitutability:

Original Object … Can be substituted with: … Not with:

Working with your objects will feel cumbersome if you inherit without substitutability.

Don’t inherit to reuse; inherit to be reused.

    From Item 37 in C++ Coding Standards, Herb Sutter and Andrei Alexandrescu 

Both composition and private inheritance means is-implemented-in-terms-of.

So which one do you use?

Use composition whenever possible. Choose private inheritance when necessary. It becomes necessary when private or protected access to information is required.

Remember, use the solution that creates the weakest coupling relationship:

public Inheritance


public BrickX { };


color_t m_color;
More Composition


block_t m_blocks[2];
pin_t m_pins[2];

Don’t underestimate the power of composition

Robust composition is easier to achieve compared to inheritance, especially with cohesive interfaces.


Minding the concepts of coupling and cohesion can lead to more robust software that avoids fragility and is more maintainable.

  • Simple definitions and implementations are easier to comprehend and use
  • Simple objects can be reused to create more useful objects with composition
  • Only use public inheritance when derived objects can truly be substituted for the base object
  • Every solution in a computer program is built upon a collection of much simpler solutions

Legos images, trademarks and copyrights belong to the LEGO Group.


communication, CodeProject 1 feedback »

I learned a new term today, bikeshedding. After reading about this term, I reached clarity regarding a number of phenomena that I have experienced as a software developer. This term is an example of Parkinson's law of triviality, which was defined by, Cyril Northcote Parkinson, in one of his books, Parkinson's Law: The Pursuit of Progress (1958).

Parkinson's law of triviality:
The time spent on any item of the agenda will be in inverse proportion to the sum involved.

Parkinson's Law

The law of triviality is actually only a corollary to the broader adage, Parkinson's Law, which I am almost certain that you have heard before:

Parkinson's Law:
Work expands so as to fill the time available for its completion.

This book is a series of cynical essays that satirizes government bureaucracies and their tendency to increase in size over time, until they eventually become irrelevant. Parkinson's law of triviality is an example of a committee's deliberations for the expenses of a nuclear power plant, a bicycle shed and refreshments.


Why is it called bikeshedding?

Parkinson dramatizes the committee's discussion, and selected a nuclear reactor because of it's vast difference in complexity and expense compared to the bicycle shed. A nuclear reactor is so complex that the average person cannot completely understand it. Contrast this with the relative simplicity of a bike shed. Anyone can visualize a bike shed, and therefore anyone that want's to contribute will add suggestions to show their participation.

Simpler topics will elicit more discussion and debate. Agreement is hardest to achieve for simpler technical problems or organizational topics that can be argued indefinitely.

The Agenda

Let's return to the dramatization of the committee's discussion. Here are the results for each of the three topics:

  1. The contract for the reactor with a cost of £10 million passes in two and a half minutes because it is to complex and technical.
  2. The bicycle shed costs £350 and discussion lasts forty-five minutes as everyone in the committee can grasp the number £350. Also, the shed is simple enough that each member has an opinion regarding the building materials and the possibility of saving £50. They are all pleased with themselves with this item concluded.
  3. Finally, a discussion regarding £21 for the yearly supply of refreshments takes seventy-five minutes. The conclusion is to as the Secretary to gather more information and they will resolve this issue next meeting.

There is a funny email on The Internet called, Why Should I Care What Color the Bikeshed Is?[^] written by, Poul-Henning Kamp. He prefaces the email with this summary:

The really, really short answer is that you should not. The somewhat longer answer is that just because you are capable of building a bikeshed does not mean you should stop others from building one just because you do not like the color they plan to paint it. This is a metaphor indicating that you need not argue about every little feature just because you know enough to do so. Some people have commented that the amount of noise generated by a change is inversely proportional to the complexity of the change.


What sort of clarity did I find? A few months ago I wrote a few entries regarding software design patterns. I briefly mentioned the commonly vilified Singleton and promised to follow up with a more detailed exploration of the Singleton. While collecting my thoughts and determine what I wanted to say about the Singleton, I experienced The Singleton Induced Epiphany[^].

I have always had a hard time understanding why so many developers are quick to criticize this design pattern, and I have heard very few criticisms of any other software design pattern. I reached two conclusions by the end of that post:

  1. The majority of the arguments against the Singleton are criticisms about the languages used to implement the Singleton. For instance up until recently C++ did not have a construct built into the language to be able to create a portable and thread-safe implementation of the Singleton, and therefore the Singleton is bad.
  2. The Singleton is the simplest design pattern, and most likely the only design pattern that everyone understands completely. Therefore, it is the only design pattern everyone feels comfortable arguing about.

At the time I was unaware that a very succinct term existed to describe the phenomenon that I observed, and now I know that it is called bikeshedding.


Now that you know of bikeshedding you should be able to recognize this phenomenon. You can then start to curb the pointless discussions about the minutiae and redirect everyone's time and attention to the more important and technical discussions at hand.

Can One-size Fit All?

general, communication, CodeProject Send feedback »

What does one-size fits all mean? I suppose there are two way to answer that question, 1) How well do you want it to fit? 2) What do you mean by 'all'? For example, a clothing manufacturer is content with the percentage of the population that their product will fit, such as a T-shirt, hat or Snuggie. All in this case is just a sub-set of the actual population. It should also be stated that the Snuggie will only fit perfectly and look stylish on the most charismatic of cult leaders.

Snuggie - One Size Fits All

Next time you read an article about programming best practices, consider if the one-size fits all mentality is used by the author. What is their experience level, programming background, project history, or even industry? These are some of the factors that determine how well their advice can be applied to your situation. Especially, if their article only seems to consider situations similar to theirs.


Ultimately, I want to illustrate that it is necessary to understand the context and background used by the author to describe a programming methodology or a set of best practices. I am referring to the process du jour such as SCRUM, XP, Waterfall, 6 Sigma, Crouching Tiger, d20, GURPS or what have you. Assuming that the author has found value in the development process of which they are writing, they have most likely applied it with success in a narrow range of projects compared to what exists.

I am not denigrating anyone's experience. I only assert that it is not possible to create a process that is the best possible practice for every type of development project.

Sometimes when I read an article extolling the virtues of a process such as SCRUM, the concepts that are presented seem very foreign to me. This is in spite of having been a part of SCRUM teams that were both successful and unsuccessful in their goals. In a situation like this I find that I become a bit defensive as I read thinking "There is no way that would!"

That is until I stop and consider the context of the article. Is the author an application developer, web developer, or a rocket surgeon for NASA? When you start to consider the author's perspective and the context for which they are writing, it becomes much easier to identify the value in the concepts they are describing. Only then is it possible to take their lessons learned and try to apply them to your context and situation.

Methodologies such as SCRUM, XP and even Waterfall have provided value and been successfully applied for a variety of teams and development projects. Also realize that these same methodologies have failed miserably for all types of development situations. Why?


I believe there are valuable aspects to all development methodologies. However, many of them are specifically designed for use in a certain context. What works well for a web development team might not directly transfer well to a team that develops products for government contracts. Some tailoring and adaptation will most likely be required to see any of the benefits.

Each step prescribed by the process is presumably added to solve a particular problem that has occurred repeatedly during development. While all software development looks the same from the point-of-view of the programmer in the chair, it is naïve to think that this is true for all stages of development in all contexts where software is developed. Many of the articles that I have read either make this naïve assumption, or do not clearly state the context for which the article is written. I have seen this for both average programmers that want to share their experience and the most respected developers in the industry that speak in terms of absolute best practices.

Everyone is trying to find a way to make their daily chores simpler with more reliable development processes. This is especially true for teams where one must coordinate activities with others. This can lead to a lot of floundering for the majority of the population that is not sure what they need to improve their development processes. They discover a new process, but fail to tailor it to their specific situation and wonder why it feels awkward, or they are not seeing the improvements that they expected.

I suspect this is one reason why we often read of the complaints where teams blindly follow the steps prescribed by a development process and it simply feels like a waste of time. These teams have failed to adapt the process to their particular situation.

Experience Level

Our level of experience affects how we are able absorb apply concepts with respect to a subject. The Dreyfus Model of Skill Acquisition is one model of this concept. Essentially, it describes the progression from a Novice to an Expert. As a person becomes more proficient with a skill, they are capable of moving away from rigid sets of rules and guidelines towards an intuitive grasp of the situation based upon on deep tacit understanding.


As a Novice, we generally rely on a rigid set of rules or plans to lead us to a successful outcome. A practitioner at this level is not expected to exercise much, if any, discretionary judgment to succeed. This is very similar to most baking cookbooks. The recipes are described in steps, and the quality of baked good often depends on how closely the recipe was followed. Attempting to alter the recipe without understanding the purpose of each ingredient and each step could lead to something that is not edible.

Intermediate Levels

More possibilities become available to a student/practitioner as they gain experience. With a greater collection of skills, they can begin to recognize patterns and differentiate between similar situations that require different solutions. As an example let's consider the differences between making the mix for a batch of muffins vs. cookies. The list of ingredients for chocolate chip cookies looks very similar to muffins. In fact, without careful inspection, all of the recipes in a cook book look similar due to their common layout and format. However, there is an enormous difference between the two types of recipes.

With the understanding of a few concepts in baking beyond beginner level, it is possible to slightly alter existing recipes to create a different consistency. Muffins use the "Muffin Method", which combines all of the dry ingredients in one bowl and the wet ingredients in a second bowl. When the oven has reached it's target temperature, the ingredients from both bowls are combined and briefly stirred, placed in the pan, then finally baked in the oven. The ingredients are not combined until the oven is ready, in part because the muffins use baking powder as one of its leaveners. When the acid in the egg whites is mixed with the dry ingredients the baking powder is activated for a brief period of time and will create air bubbles. Once the batter reaches a high enough temperature the batter will set and lock the bubbles in place creating a fluffy texture.

Basic chocolate chip cookies use the "Creaming Method". This is a step where the butter and sugar are beat together to create billions of tiny air bubbles within the mixture. This combination of ingredients adds structure to the cookies because the sugar holds the air until the cookie mix is set in the oven.

What happens if the "Creaming Method" is not strictly followed? The cookies will still probably turn out, however, they will not have the chewy consistency one would expect of this traditional cookie recipe. If the butter is completely melted you will end up with the thin crispy cookie (hey, maybe that's what you wanted). The chewy consistency is also achieved by the gluten that is created when mixing the flour with the water-based ingredients in the mix. To create a less dense cookie you could change the type of flour to eliminate the gluten.


Expert bakers will simply "know" a few dozen baking recipes and ingredient ratios off the top of their head and know how altering the ratio of the ingredients is likely to affect the final baked good. From this collection of knowledge they would be able to create a plethora of new recipes to suit the needs of each situation they encounter.

Unfortunately, experts will often know a subject so well that some times they are not able to explain how they know the answer, "they just do." This can be intuition leading them to the answer. They have encountered the problem so many times they recognize the pattern and know the right solution. This can lead to problems for less experienced practitioners learning from an expert. Even with a decent amount of experience we are not always able to see all of the connections to arrive at the expert's conclusion. This is one reason why I dislike the phrase "I will leave this as an exercise for the reader."

A Perfect Fit

Software design patterns are similar to recipes for a software development, though not quite. I would compare them more to different techniques in baking such as the "Muffin Method" and the "Creaming Method". You choose a design pattern based on the problem you are trying to solve. Each design pattern is known for providing a good solution to certain classes of problems.

A particular implementation of the design pattern would be comparable to a recipe. The Singleton design pattern is capable of creating solutions more sophisticated than an object that controls it's own creation and limits itself to a single instance. Just as you adapt software design patterns to fit your needs when you develop a software solution, so should you adapt the development processes that you choose to use in your organization.

One final example to illustrate this point. For decades large corporations have been adopting and incorporating SAP database systems to help optimize the logistics of running their company. When these solutions are bought, they require large multi-million dollar investments to customize these systems for the company. The SAP software is already written. However, custom integration is required to make the system usable and useful to the company.

The software development process that you select should be no different. Yes, all SCRUM processes will look similar when compared between different companies that practice this process. However, I would expect there are subtle details that differ between two companies that successfully manage their projects with SCRUM. I believe these subtle differences could be important enough that if one company were to change to the other companies process they would no longer be as successful, or even worse, they no longer succeed.


It is important to analyze "best practices" and development processes before adopting them to determine how well they fit your organization. I have worked for a half-dozen companies in my career, and none of them setup their build process exactly the same, even though many of them used the same tools. In order to get a best-fit for development processes and practices I would expect some level of customization to be required to be as effective as possible.

C++: enable_if

CodeProject, C++, knowledge Send feedback »

A few weeks ago I wrote an entry on SFINAE[^], and I mentioned enable_if. At that point I had never been able to successfully apply enable_if. I knew this construct was possible because of SFINAE, however, I was letting the differences between SFINAE and template specialization confuse me. I now have a better understanding and wanted to share my findings.

What is it?

As I mentioned before enable_if is a construct that is made possible because of SFINAE. enable_if is a meta-function that can help you conditionally remove functions from the overload resolution performed on the type-traits supplied to the function.

You can find examples of it's usage in the Standard C++ Library within the containers themselves, such as std::vector:


template< class _Iter>
  typename enable_if< _Is_iterator&lt;_Iter>::value,
assign(_Iter _First, _Iter _Last)
{ ... }
assign(size_type _Count, const value_type& _Val)
{ ... }

previous block of code accomplish? The assign call will always exist if you want to specify a count and supply a value that is a value_type. However, the conditional version will only be declared if the parameterized type has a type_trait that evaluates to an iterator-type. This adds flexibility to definition of this function, and allows many different instantiations of this function; as long as the type is an iterator.

Also note that the template definition has a completely different signature than the other non-parameterized version of assign. This is an example of overload resolution. I think it is important to point this out, because I was getting caught up into how it should be used. As I perused the Internet forums, it seems that many others are confused on this point as well.

How it works

Let's continue to look at the std::vector. If the parameterized type is not an iterator type, the enable_if construct will not allow the parameterized type to exist. Because this occurs in the template parameter selection phase, it uses SFINAE to eliminate this function as a candidate to participate in the overloaded selection of assign.

Let's simplify this example even further. Imagine the second version of assign does not exist; the only definition is the parameterized version that uses enable_if. Because it is a template, potentially many instantiations of this function will be produced. However, the use of enable_if ensures that the function will only be generated for types that pass the iterator type-trait test.

This protects the code from attempting to generate the assign function when integers, floats or even user defined object types. Essentially, this usage is a form of concept checking that forces a compiler error if the type is not an iterator. This prevents invalid code with types that appears to compile cleanly from getting a chance to run if it will produce undefined behavior.

Another Example

Previously, when I attempted to use enable_if, I tried to apply it in a way that reduced to this:


class Sample
  template< typename =
    typename std::enable_if< true >::type >
  int process()
    return 1;
  template< typename =
    typename std::enable_if< false >::type >
  int process()
    return 0;

Hopefully something appears wrong to you in the example above. If we eliminate the template declarations, and the function implementations, this is what remains:


//template< typename =
  //  typename std::enable_if< true >::type >
  int process()
  // ...
  //template< typename =
  //  typename std::enable_if< false >::type >
  int process()
  // ...
  // This declaration will always trigger an error for a non-existent type:
  //     std::enable_if< false >::type

When the compiler processes the object, it may not instantiate every function, however, it does catalog every definition. Even though there is a definition for a false case that should not result in the instantiation of that version, it does not affect the overload resolution of the functions. The enable_if definition needs to contain a type that is dependent on the expression. This line will always be invalid:

Types of Usage

There are four places enable_if can be used:

Function return type

This is a common form. However, it cannot be used with constructors and destructors. Typically one of the input parameters must be dependent on a parameterized type to enable multiple function signatures. Overload resolution cannot occur on the return-type alone.


// Only calculate GCD for
// type T that is an integer.
template< typename T >
typename std::enable_if< std::is_integral< T >::value, T >::type
gcd(T lhs, T rhs)
  // ...

Function argument

This format looks a little awkward to me. However, it works for all cases except operator overloading.


// Only calculate GCD for
// type T that is an integer.
template&lt; typename T >
T gcd(T lhs,
      T rhs,
      typename std::enable_if< std::is_integral< T >::value, T >::type* = 0
  // ...

Class template parameter

This example is from cppreference.com, and it indicates that static_assert may be a better solution that the class template parameter. Nonetheless, here it is:


// A is enabled via a template parameter
template< class T, class Enable = void>
class A; // undefined
template< class T>
class A&lt; T , typename std::enable_if< std ::is_floating_point< T>::value >::type> {
}; // note: for this use case, static_assert may be more appropriate
// usage:
//  A< int > a1; // compile-time error
    A< double > a1; // OK

Function template parameter

This is probably the cleanest and most universal form the enable_if construct can be used. There are no explicit types for the user to define because the template parameter will be deduced. While the same amount of code is required in this form, the declaration is moved from the function parameter list to the template parameter list.

Personally I like this form the most.


// Only calculate GCD for
// type T that is an integer.
template< typename T,
          typename std::enable_if< std::is_integral< T >::value,
                                  T >::type*=nullptr >
T gcd(T lhs, T rhs)
  // ...


When learning new tools, it is often difficult to see how they are applied. We spent years practicing in math, as we attempted to solve the numerous story problems with each new mathematic abstraction taught to use. Writing code and using libraries is very similar. enable_if is a construct that is best used to help properly define functions that meet concept criteria for a valid implementation. Think of it as a mechanism to help eliminate invalid overload possibilities rather than a tool to allow the selection between different types and your application of this meta-function should become simpler.

Alchemy: BitLists Mk1

CodeProject, C++, Alchemy, design Send feedback »

A continuation of a series of blog entries that documents the design and implementation process of a library. The library is called, Network Alchemy[^]. Alchemy performs low-level data serialization. It is written in C++ using template meta-programming.

At this point, I was still in the make it work phase of implementation. However, I learned quite a bit from experiencing the progression of design and multiple implementations. So much that I thought it was valuable enough to share.

After the initial creation of the Datum type, it was relatively simple to abstract the design and implement the Bit-Field. object. The next step was to combine the collection of BitField objects into a single group that could be statically-verified as type-safe. This was also relatively simple. Again I just had to model the implementation of the message structure. I called this collection a BitList. Primarily to differentiate it from the std::bitset.

The real trouble appeared when I began the integration of the BitList with the Message in a way that was similar to the Datum. At this point, I started to see the potential this library could provide. Through experience, I have learned that it is important to get the programming interface correct in the initial design. Any mistakes that were made, or code that is not as clean as you would like can be refactored or even re-implemented behind the original interface.

The BitList

My initial approach was straight-forward. As meta-programming was new to me, I looked for opportunities to take advantage of the compiler to verify invariants where ever I could apply them. Therefore, there is a type_check that verifies the number of bits specified by the user does not exceed the capacity of the data type used for storage.


template < typename BaseT,
    size_t B0,   size_t B1=0, size_t B2=0, size_t B3=0,
    size_t B4=0, size_t B5=0, size_t B6=0, size_t B7=0
struct BitSet
  // Are you receiving a compiler Error:
  //    'Hg::type_check < false >' : base class undefined?
  // The sum of the provided bits must be <= sizeof(T) in bits
  : type_check< (B0+B1+B2+B3+B4+B5+B6+B7)
              <=(sizeof(typename BaseT::value_type)*8)>
  , public BaseT

I thought the best approach at the time was to store a reference to the primary value, to which all of the BitFields referred. This was an attempt to avoid a mechanism where the child calls the parent for the value then the operation is performed. I was focused on the general usage of the object, and didn’t pay too much attention to the construction of this type. As you can see, the construction of this BitList is quite expensive compared to the amount of information that is encoded in the type.


//  Typedef **************************************
  typedef typename
    BaseT::value_type                      value_type;
  BitList(value_type &amp;data_field)
    : m_data(data_field)
    , m_field_0(GetFieldAddress< 0 >(m_field_0))
    , m_field_1(GetFieldAddress< 1 >(m_field_1))
    , m_field_2(GetFieldAddress< 2 >(m_field_2))
    , m_field_3(GetFieldAddress< 3 >(m_field_3))
    , m_field_4(GetFieldAddress< 4 >(m_field_4))
    , m_field_5(GetFieldAddress< 5 >(m_field_5))
    , m_field_6(GetFieldAddress< 6 >(m_field_6))
    , m_field_7(GetFieldAddress< 7 >(m_field_7))
    m_field_0.attach(m_data);  // The assignment in the
    m_field_1.attach(m_data);  // constructor assigns the
    m_field_2.attach(m_data);  // reference of data_field to
    m_field_3.attach(m_data);  // the data member, m_data.

There are two parts to the previous section of code, the initializer list, and the constructor body. The initializer list is used to associate address of the BitField object with the local reference held by the BitList container. This is how the container is able to enumerate over the values in the class. The second part attaches a reference of the internal data field with the child fields.

Because of this twisted setup, the majority of what remains is simple and looks much like the other value classes I have previously shown. The one oddity that remains, is a set of static constants defined to help navigate the set of sub-fields. The code below depends on the knowledge of the size and relative offset for each value.


static const size_t k_size_0   = B0;
  static const size_t k_offset_0 = 0;
  static const size_t k_size_1   = B1;
  static const size_t k_offset_1 = value_if< (B1>0), size_t, k_size_0, 0 >::value;
  static const size_t k_size_2   = B2;
  static const size_t k_offset_2 = value_if< (B2>0 ), size_t,
                                            k_offset_1 + k_size_1,
  static const size_t k_size_3   = B3;
  static const size_t k_offset_3 = value_if< (B3>0 ), size_t,
                                            k_offset_2 + k_size_2,
  // ... continue through 7

Those definitions allow the actual value types fields to be defined with the code below:


value_type                                     &m_data;
  Bit_field< k_offset_0, B0, value_type>          &m_field_0;
  Bit_field< k_offset_1, B1, value_type>          &m_field_1;
  Bit_field< k_offset_2, B2, value_type>          &m_field_2;
  Bit_field< k_offset_3, B3, value_type>          &m_field_3;
  Bit_field< k_offset_4, B4, value_type>          &m_field_4;
  Bit_field< k_offset_5, B5, value_type>          &m_field_5;
  Bit_field< k_offset_6, B6, value_type>          &m_field_6;
  Bit_field< k_offset_7, B7, value_type>          &m_field_7;

Unfortunately with this design, there will always be eight sub-fields per BitList object, regardless of the actual number of bit-fields that are used. Imagine my surprise when shortly after I released this into the wild, I had requests to support 32 bit-fields per Bitlist. At that point I started to think of ways that I could eliminate all of these unused references. Unfortunately, I was not able to implement my ideas until the second version of this type. So those ideas will have to wait.


To organize the Hg::BitList similarly to the message required an outer container to be defined for the user type. I am not going to show all of the details because they are not that interesting. Also, the current implementation for this feature in Alchemy barely resembles this concept at all. Here is a high-level overview of it’s original structure.

Essentially, the MACRO below is a euphemism for the morass of partially defined parameterized errors waiting to be compiled.


// MACRO declaration of a BitList
HG_BEGIN_BIT_LIST(uint32_t, info)
  HG_BIT_FIELD(0, 4, first)
  HG_BIT_FIELD(1, 4, second)
  HG_BIT_FIELD(2, 4, third)
  HG_BIT_FIELD(3, 4, fourth)
  HG_BIT_FIELD(4, 4, fifth)
  HG_BIT_FIELD(5, 4, sixth)
  HG_BIT_FIELD(6, 4, seventh)
  HG_BIT_FIELD(7, 4, eighth)

The name info defines a new BitList type the user will place in the type-list definition that defines the overall message format.

In the end, I spent much more time on this section of code that I am comfortable admitting. So let me just show you a few snippets of what the MACRO definition hides and move away from this.


// ...
// The header starts the declaration of a new
// container type that becomes the BitList.
struct C                                                 \
    : public Base_bit_list                                 \
  {                                                        \
    typedef C                             this_type; \
    typedef T                             value_type;  \
    typedef Bit_field<0,0,value_type>     nil_bits_t;\
    enum { k_offset_0 = 0 };                               \
    template < typename IndexT,                           \
              typename BitT>                                \
    BitT&amp; GetField(const BitT &amp;)                         \
    { return GetFieldData(BitT()); }                       \
    nil_bits_t&amp; GetFieldData(const nil_bits_t&amp;)            \
    {                                                      \
      static nil_bits_t nil_bits;                          \
      return nil_bits;                                     \

For each field that is defined, a collection of definitions is created by the MACRO that is similar to this:


typedef FieldIndex< 2, this_type,4> idx_2;
  typedef Bit_field  < k_offset_2, 4, value_type > third_t;
  enum { k_offset_3 = k_offset_2 + N };
  third_t   third;
  third_t&amp; GetFieldData(const third_t&amp;)
  { return P; }

-The definition above declares a new Hg::BitField type that meets that criteria for the proper index, offset and size of that field. A new instance of that time is defined with the name the user has specified, and a function is provided that returns a reference to that field. The function requires a const reference of an object of the same type as the desired field (I have since realized that pointers are much less expensive than const references and just as effective). Since all of the fields have different indices and offsets there will be no problems with duplicate types.

The type definitions use a well-defined format that can be programmatically indexed. Therefore developing functions to operate over the fields will be possible. However, there is still more work to integrate this type into the Alchemy framework.

Type-deduced definitions

Throughout the development of this project, the template definitions continued to grow longer and more complex. In many cases, they weren’t that difficult to deal with at all. Rather they were tedious and required the parameterized types to be propagated through the depths of the interfaces to which they applied. It didn’t take long for me to recognize that I could wrap all of the complex definitions into parent-type, and reference my types much like the meta-functions I have been creating. The only difference is in this case I truly do want a simpler definition of the type.

Here is what the definition of a Hg::BitSet with 8 fields that each used 4 bits would look like:


Hg::BitSet< uint32_t, 4,4,4,4,4,4,4,4 >   info;

Because the type information is packaged in the convenient message type-list, all of the details required to reconstruct the Hg::BitList definition is still available, even when packaged like this:


template< size_t    Idx,
          typename  format_type>
struct Format_bitset
    typename Hg::TypeAt < Idx,
                        >::type                   base_t;
  typedef Hg::BitSet< base_t,
                    >                             type;

The type-list and index make the Hg::BitList much more accessible, considering all of the other utility functions and definitions that have been created to support the fundamental types.

Integrating as a Datum

At first glance this does not appear to be a difficult issue to resolve. The primary issue is we need to make the Hg::BitList compatible with the existing Hg::Datum type. For a basic template, it would already meet that requirement. Through the use of static-polymorphism, a type only needs to contain the same member data and functions for all of the types to be compatible in an algorithm. They are not required to be tied together through inheritance, which is the basis of dynamic-polymorphism.

The difficulty I ran into, was the logic that I had created to programmatically navigate each field of a message were specialized for the Hg::Datum type. To specialize on a second distinct type would cause ripples through the entire API, and cause almost complete duplicate implementations for alternative specialization. This seems to defeat the purpose of templates. So I searched for a solution to bring the two types together.


The solution I used for this first version was to create a class, Hg::BitListDatum, which derived from both of these classes. Here is the declaration of the type:


template< size_t IdxT,
          typename format_type
struct BitListDatum
  : public Hg::Format_bitlist< IdxT, format_type >::type
  , public Hg::Datum< IdxT, format_type >
{ };

The purpose of this class is to become a shallow adapter that is compatible with the definitions and types already defined for the Hg::Datum. The only challenge that I ran into was creating a definition that would allow the Hg::BitList type to be returned for the user to interact with, but still be a Hg::Datum. This is what I created:


typedef typename
    Hg::Datum < Idx,
              >               base_type;
  operator base_type&amp;()
    return *static_cast< base_type *>(this);
  operator value_type() const
    return static_cast< const base_type*>(this)->operator value_type();

The base_type operator properly handles the interactions where a reference to the Hg::Datum type is required. In all other cases, the user can interact with the value_type operator that returns access to the child Hg::BitField objects by name.


After dynamically-sized arrays, the support of bit-fields has been the most tedious feature to implement in Alchemy. In fact, I do not think implementing support for arrays would have gone nearly as smoothly for me if I hadn’t learned the lessons from implementing bit-field support.

Bit-fields are a feature that are not used much. However, they tend to appear quite frequently when dealing with network protocols or data-storage formats. Once you consider that the bit-field feature is not well defined for C and C++, it becomes clear that an alternate solution for providing bit-level access to the data is important. Mask and shift code is tedious and error prone, not to mention difficult to read if you do not see it often. Considering that the simplification of software maintenance is one of the primary goals of Alchemy, providing a feature such as the Hg::BitList is well worth the effort.

Alchemy: BitField

portability, CodeProject, C++, Alchemy, design Send feedback »

This is an entry for the continuing series of blog entries that documents the design and implementation process of a library. This library is called, Network Alchemy[^]. Alchemy performs data serialization and it is written in C++.

After I had proven to myself that serializing data with a template meta-program was feasible, I started to research what features would be required to make this a useful library. After the fundamental data types, I noticed that sub-byte access of bits appeared quite often in network packet formats, bit-fields. I have also worked in my share of code that implemented these packet formats using the bit-field feature in C/C++.

I decided support for accessing values at the bit-level needed to be a feature of Alchemy. I will explain why, as well as show my first attempt to make this work.

Bit-fields with C++

Bit-fields are like that questionable plate of left-overs in your fridge, yet you choose to eat it anyway. Personally, I avoid both.

In case you are not familiar with bit-fields, here is a brief example:


struct Bits
  char isSet:1;
  char type:3;
  char flags:4;  
// This allows these three fields to be packed into the
// single char and accessed with the field names.

The compiler will automatically apply the appropriate mask to the data field to return the value of the named bit-field. This is very convenient. However, there is a catch; the original C specification left the implementation underneath undefined for the compiler implementer to decide. C++ then inherited the same rules. The reason this may be an issue is the bit-fields are not guaranteed to be packed in the order and location that we define them. Therefore, when they are serialized, they will not be correct according to your format specification.

Basically, bit-fields are not portable. Therefore, they should be avoided in code that is intended to be portable, or produces data that should be portable.


There is not much logic that is required to extract the correct bits defined in a structure similar to bit-fields. I had just finished an implementation of message fields that act like regular types. My plan was to attempt use the same approach with the Hg::BitField, to allow the user to access each field with a specific variable name. If that did not work, my fallback plan was to simply convert the variable-like syntax to use a function instead.

Masking Bits

As I mentioned earlier, the logic to mask bits out of an integer is just a few lines of code. Let's assume we have a pre-defined mask and we want to change the value of the bit the mask applies to. Here is the code:


void BitField::set_masked_value(const uint32_t &rhs)
    // Member data defined for this bit-field instance.
    // const uint32_t k_mask = 0x0000FF00;
    // const uint32_t k_offset = 8;
    // Mask off the bits we are about to set,
    // by inverting the mask, and leaving all
    // of the other bits alone.
    m_value &= ~k_mask;
    // Shift the new value by the appropriate offset,
    // and combine that value with the mask.
    // This leaves two values with all of the bits
    // mutually exclusive. Or them together to set the value.
    m_value |= k_mask & (rhs << k_offset );

To go in the opposite direction is a bit simpler with only one line of code:


size_t BitField::mask_value()
    return (m_value & k_mask) >> k_offset;

Calculate Mask and Offset

Hopefully you have seen logic like this before. It may even have been tucked away in a MACRO or something. How do we calculate the mask?

  1. Take the number of bits in the value, and calculate: 2^n-1
  2. Determine the starting position of the mask in the value, this becomes the offset.
  3. Shift the value calculated in step one to the left by the offset.

At run-time we could calculate the mask on-the-fly with this code snippet:


// Where:
  //   n is the number of bits in the mask
  //   start_pos is the index of the first mask bit.
  uint32_t bits   = std::pow(2, n) - 1;
  uint32_t offset = start_pos;
  uint32_t mask   = bits << offset ;

Convert to compile-time

The only part of the expression above that currently requires run-time processing is the std::pow() function. More than likely you have already seen a template meta-program implementation of this call, if not at least a recursive implementation. Here is the implementation that I use:

Start with a top-level function.


// T is used to define the type of variable.
// An instance of T is immediately created to hold var.
// Finally the exponent.
template&lt; typename T, T var, unsigned char exp >
struct pow
  // This expression will take the current var and
  // multiply it by the remainder of the calculation.
  // Create a recursive instance with exp - 1.
  static const T value = var * pow< T, var, exp-1 >::value;
This is the terminating specialization. When the exponent reaches 1, simply return the value.


template< typename T, T var >
struct pow< T , var, 1 >
  static const T value = var;
However, we also need to handle a special case, when 0 is passed in, the resulting value becomes 1.


template< typename T, T var >
struct pow< T , var, 0 >
  static const T value = 1;

The mask expression

Now it is possible to define a constant at compile-time that represents the value of the mask. Two values will need to be passed into the definition of the Hg::BitField template declaration, 1) the size of the field, 2) the offset.


template< size_t OffsetT,
          size_t CountT,
          class  T = uint8_t>
struct BitField
  static const T k_offset = OffsetT;
  static const T k_size   = CountT;
  //  The BitField Mask can be described with this formula:
  //  Mask = (2 ^ (number of bits))-1 << shifted to the left
  //  by offset bits.
  static const T k_mask   =
    T( ((Hg::pow< size_t, 2, k_size>::value)-1) << k_offset );
  /// ...

Hg::BitField declaration

This is the point where I need to front-load my explanation with an excuse because it appears so obvious to me now. However, in the experimentation phase, it is most important to even find a working solution. I subscribe to the mantra:

Make It Work, Make It Right, Make it Fast

This sage advice has been part of the Unix Way for many years, and more recently attributed to Kent Beck with respect to Test-Driven Development. This is the end of the excuse. Spoiler alert, I reworked this class when I reached the benchmark phase and witnessed it's abysmal performance. Hey, It Worked Right, there was no reason to revisit it until I discovered it was an issue.

The initial idea was to create a collector object called a Hg::BitSet, and this would contain the collection of Hg::BitField instances, and also store the actual data buffer. The bit-field objects would only contain a pointer to the value in the bit-set. Originally it was a reference, but I reached a limitation where the Hg::BitField had to be default constructable, and thus the reference initialized to a default value.

The remainder of the class definition:


// ...
  BitField() : mp_value(0) { }
  explicit BitField(T& value) : mp_value(&value) { }
  void Attach(T& storage)  { mp_value = &storage; }
  operator T()
  { return  mp_value
          ? mask_value()
          : 0;
  T operator=(const T& rhs)
  { if (!mp_value) return 0;
    *mp_value &= ~k_mask;
    *mp_value |= k_mask
              & (rhs <<  k_offset);
    return mask_value();
  T* mp_value;
  T mask_value()
  { return (*mp_value & k_mask) >> k_offset;}

That's it!


I am trying to create slightly shorter entries so they can be digested more quickly, and I will be able to update more frequently. The unintended side-effect is that it has also allowed me to focus a bit better.

This entry described the intended purpose and design goals of the Hg::BitField class. In my next Alchemy entry I will show how I integrated the bit-field instance into its container. Then I will describe the hurdles that I started to encounter as my data types became more complex.

What Is a Software Architect?

general, leadership, communication, CodeProject, engineering Send feedback »

There is so much confusion surrounding the purpose of a Software Architect and the value they provide and what they are supposed to do. So much so, that it seems the title is being used less and less by companies and replaced with a different title such as principal or staff. I assume this is due to the perception there must be a way to distinguish a level above senior, which is handed out after only about three years of experience.


This has been a topic that I have wanted to discuss for quite some time. However, up until I read this question "Are there any Software Architects here?[^]" at CodeProject.com, I had been more compelled to focus on the more technical aspects of software development.

This is an excellent post that has multiple questions. If you were to take a quick peek at the question (it's ok, just come back), you would see about 80 responses in the discussion; with a wide variety of facts, opinions, and jokes related to software architecture and software development in general. Furthermore, many of the posts differ in opinion and even contradict each other.

I define the role of Software Architect in this entry. The definition is based upon my nearly two decades of experience developing software. Up to this point in my career, I have worked for small companies where I was the only software developer; all of the way up to companies where I was one of hundreds of developers and about one-thousand engineers of many engineering disciplines. Through the years I have performed just about every role found in the Software Development Lifecycle (SDLC) including a software architect.

The Basics of the Role

The Software Architect is not another stepping stone above Senior Software Engineer or Software Engineer V. They do work closely with people in these other roles, and in smaller companies they may also function as a Software Engineer V. However, the role of Software Architect is an entirely different position, and it has different responsibilities that are so much different than the task of writing software.

The software architect is a technical leader for the development team, and they are also the figurehead of the product for which they are responsible. One reason they become a figurehead for the product, is because the organization should make them responsible for the technical aspects of the product.

This makes them a good point of contact for information regarding the product, especially for personnel in the organization that may not be closely tied with the product. They may not always be the best person to answer a question, but they most certainly know who is.

As a leader, the architect is responsible for foster unity and trust among their team for success. The technical aspect requires the architect to mentor and guide the team along the way toward maintainable product with high quality. Moreover, they provide technical guidance and recommendations to the customer, which quite often is their own management team.

The Value

The architect adds value to the product and the organization by providing structure to both the design and implementation of the system, as well as the flow of information. If you are a fan of Fred Brooks and his book, The Mythical Man Month, you will recall that he states the implementation of a system tends to mimic the communication structure of your organization. In my experience, this has always proven to be true.

Improved Quality and Stability

Design by committee can work. However, you often need a group of like-minded individuals that do not differ in opinion too drastically for this succeed. In the extreme circumstances, if there are wildly divergent opinions, the group must be willing to forego their egos and accept the decisions that are made even when the decision is the polar opposite of what they believe to be the best decision. When all else fails, there is a final decision-maker at the top.

Thus enters the architect. They are after all, responsible for the product that is ultimately designed and built. They will help the team make decisions that are aligned with the macro-scale goals for the organization. While the development team themselves focus on the design decisions that are most beneficial at the micro-scale of the feature or system for which they are responsible.


A good flow of information is often required for harmony and success among a group. This can lead to other desirable benefits such as a high-level of morale and greater productivity. Effective communication is essential. Technical endeavors especially suffer when teams do not communicate effectively.

Development may move along steadily during the initial stages. However, if the integration of the components was not planned well in advance, progress will screech to a halt. Adjustments will need to be made for each component to be joined correctly. The architect can monitor the progress and direction for each developer or team, and provide guidance early on to correct course for the intended target.

The management team and key stake-holders communicate with the architect to remain in touch with the technical aspects of the system that require their attention. Such as clarifications on requirements, or missing resources that are necessary to continue to make progress on the project.

One problem that I see talented engineers struggle with, is to be able to adjust the language and content of their message to match their audience. Generally, they tend to be too technical and provide too many extraneous details for the important message to be properly conveyed. A good architect helps bridge this communication gap. They are able to effectively use the best style of communication to convey the details and properly persuade their audience.


At this point I would like to diverge and briefly and discuss the different roles that are required or exist for a software development project. Primarily because I often run across the question "What does an architect do that a developer doesn't?" This will also provide some context to refer to for the final section in which I will list desirable qualities you should look for in a software architect.

I mentioned there are many roles in software development. Each role is responsible for performing certain tasks. The problem is, the responsibilities for each role differ based on the organization. The size, culture and industry can affect how responsibilities are organized.

A company with a small staff may assign multiple tasks to its employees. To be able to complete all of the work in a timely fashion, the different roles may be responsible for overlapping tasks. While a large organization and development team may have many people that focus on performing a single task. All of the while, both companies may use the same titles to describe the roles of their employees.

It is important to know what your role and responsibilities are for any organization to be able to succeed at your own job; not to forget the success of the project and company as well. Therefore, I think it makes more sense to discuss the basic tasks that must be accomplished or commonly are included with a companies SDLC.


Here is a non-comprehensive list of the technical tasks that are required to develop software.

  1. Analysis: Determining what you want, need, or should build
  2. Requirements Gathering: Create a definition of what is to be built.
  3. Planning: Create the budget and the schedule. This task is not entirely technical, but the schedule and budget will be more accurate with technical guidance.
  4. Coordination: This is a bit abstract, however, it basically covers all communication and resource scheduling.
  5. Design: Create plans for the structure and operation of your product.
  6. Development: Build the product.
  7. Verification: Verify that it meets requirements, specifications and quality levels.
  8. Source Control Management: Organization of the resources required to build the product.
  9. Documentation: Useful or necessary information to the creators and users.

The only task listed above that is actually necessary is development. However, the process of development and the quality of the product suffers when the other tasks are omitted. Also, this same effect can occur when the level of effort is reduced for any of the other tasks.

Example: Analysis

How difficult is it to build a jigsaw puzzle when you do not even know what the final image is? It is possible. However, it is much more difficult without knowing what you are aiming for.

Let's complicate the example further. How much more difficult does it become to assemble the puzzle when:

  • Extraneous pieces from another puzzle are added to your pile of loose pieces
  • The number of pieces in the puzzle is not known
  • No corner pieces exist
  • No edge pieces exist

Example: Coordination

When you run in a three-legged race, the team that is most coordinated will have a great advantage. As the amount coordination increases, the less amount of time the team-members will spend fighting each other trying to find a rhythm.

The same concept holds true when you have more than a single person performing tasks on a project. These are the types of things that should be targeted through communication and coordination:

  • Agreement between the groups for how their systems will interoperate.
  • Scheduling project tasks to minimize conflicts and dependencies. If each task is dependent linearly upon another, it will not make sense to have more than one person on the project at a time.
  • Good communication throughout the project helps spot potential issues and address them before they are too difficult to tackle.
  • If the schedule requires your software team to develop at the same time the hardware is being developed, some alternate and temporary solution will be required for the developers. Virtual machines or reference hardware maybe acquired for software to use until the hardware is available. This is also an effective solution when there is a shortage of real hardware compared to the number of developers on your team.

Return to Roles

I will describe the set of tasks the software architect may perform for a small to moderately sized team. We will define this as about 20 people, with 8 or so are developers. The remainder of the team includes product managers, personnel management, marketing, sales, and quality control.

Analysis, Requirements. Planning and Documentation

The architect should be involved from the beginning. Once you know there may be a product to create, or a new engineering cycle on an existing project, bring the architect in. They can assist with the analysis, requirements gathering, and planning tasks. They can spend a fraction of their time acting as a consultant for these tasks. They can help advise and make sure the right questions get asked, the proper information is gathered, and only realistic promises are made. When it comes to creating documentation, the architect and development team should be accessible to answer questions for the technical writers.


As a leadership role, one of the primary responsibilities of the developer will be to communicate. More so listening than speaking. One of the most valuable things that an architect, or a software lead and manager for that matter, can do is to make sure their team understands what they are building. Then make sure each individual understands how their work will contribute to the final project.

I have seen projects go from extremely behind schedule, to finishing on time, after a new software lead was put into place. On one particular project, the developers had a vague idea what they were building, but on the surface it only seemed like another version of the five things they had built before. The new lead spent the first few days going over the project, its goals, what made it so cool, and verified each member of the team knew what they were responsible for. At that point, they moved forward with excitement and a new understanding. When some engineers finished their work, they jumped in and helped in other areas where possible. Do not underestimate the value of proper team coordination.


Design is definitely something that the software architect will be involved with. However, I think that most people start to misunderstand the responsibility of the architect when the task of design is mentioned. The perception that I see the most often is that the architect designs the entire system; and this can cause some angst among developers that believe this is the case.

The cause of any angst may be because the developers believe their control and creative freedom will be limited; hopefully that is not the case. I have a bit more to say about this later in the essay.

The architect should take responsibility for the overall structure of the program. High-level behavior, control-flow and technologies to be utilized will all be determined by the architect. Other software developers are given the responsibility to design smaller sections of the program at a higher level of detail. Their design should be guided by the structure the architect puts in place.


Any work performed up to this point is mostly focused on making this stage run smoothly. Hopefully all of the assumptions are correct, and the requirements are fixed. If not, the team must adapt. This is where the architect's responsibilities really become important. A foundation designed to be flexible with logically separated modules will have a better chance of adapting to any surprises that appear during development.

The architect is most likely to play more of a support role during this phase than a primary producer. That is because their time will be spent on guiding, mentoring and guiding the developers, as well as inspecting the results, and providing support where it will be most valuable. Moreover, they will keep the technical teams aligned with one and other, as well as the business and management teams informed of progress.

Furthermore, any surprises or new discoveries that appear along the way can also be adequately managed by the architect. On the other hand, any events occur that require the architects attention could cause further delays if their time is scheduled nearly 100% towards a development work items.


This is one phase where I believe the architect should have as little to do with this as possible. The reason being, you want an independently verified product. QA teams usually get the raw end of the deal when it comes to verifying and shipping software. If the programmers overrun on their schedule, QA is often tasked with finding creative ways of making the quality of the software better, faster.

If the architect is involved, the integrity of the verification process could be compromised. QA is a control gate to verify that things work correctly before they are released to the customer. The last thing you want is the architect and developers influencing QA's findings; making excuses for why something is not a bug, or at least why it should be classified as a level 4 superficial bug, rather than a level 2 major.

The architect should only coordinate with QA in order to get QA the resources they need to properly perform this task.

Source Control Management

There is so much that can be said about software SCM, but I am not going to. that is because it is a complicated task that deserves an entire essay of its own. The bottom line is the architect must be involved in the SCM for the product. It is crucial during development. It is damn near critical that SCM is handled according to a policy the architect defines (based upon the guidelines and strategies of the organization of course.)

Some products live a single existence, and slowly evolve with each release. Others spring into existence, and core components are reused to build other projects, which the process is then repeated ad infinitum. If there is no one managing the source code appropriately, or the way it is managed does not work for everyone, you may just get forked[^].

An Effective Software Architect

I was going to write a section titled Qualities. However, qualities are subjective, and you could probably guess the items on the list. It would look like every job ad for a software architect, and nearly the same for every software engineer. Just fill in the blank for the desired number of years of experience.

Because that list of qualities is so predictable and repeated everywhere, it would not add any more value to conveying the purpose and value of a software architect. Therefore, I thought it would be better to describe a few actions an Effective Software Architect would perform and the potential benefits that can be realized.

Focus on the Future

Jargon litters our lives. When job ads say "work in a fast-paced environment", that's a nice way of saying, "We want it right now!" No matter how quickly a piece of software can be developed, it still isn't quick enough. That is where many projects go awry; they are too focused on just for today. Tomorrow always becomes another today. Next thing you know you're programming in a swamp of code.

The software architect should focus on the long-term direction of a product, and execute towards the short-term goals of the business. It is good to have goals, they help provide direction. Every step forward may not lead directly to the goal. However, if the goal is always kept in mind when design decisions are made, the goal will become more attainable.

Shortcuts will be taken, and technical debt will accrue on the product. There never seems to be enough time to go back and correct all of those blemishes. However, the design can be made to mitigate short-term decisions and still provide a stable path towards the goals of the product.

Management and Business Development need to support the Software Architect by providing them with goals and information regarding the product. This will help the architect to develop a vision for the project and help guide it towards long-term goals; despite all of the rash and short-term decisions that are usually made in software development.

Create Unity

This is a more focused description of the coordination task above. This action is focused on building and maintaining a level of trust between all of the development team, as well as the management and other stake-holders that are involved with the project. As I mentioned earlier, the software product tends to mimic the communication style of the organization. Therefore a more unified team is more likely to develop a program that is unified in structure and quality. There is more to unity than the final structure of the program.


You may think that it is odd that I started an essay discussing computer programming, and I now I am on the topic of trust. For a team to work well together, they must trust each other. They must also trust their leaders that are directing them. The software architect is in a perfect position to help foster trust.

They are a technical leader, which means that they focus their attention on the technology and the way it is used. The role itself does not manage people. However, some organizations make the architect the manager of the developers as well. I believe this is a mistake because it throws away this opportunity to have a role to mediate situations of mistrust between development and management. This is especially true with respect to the technical aspect of the project.

The list below briefly describes how trust can be fostered with the other roles in the organization:

  • Developers:

    Listen to the development team. Incorporate their ideas into the design where they may fit, and be sure that they get credit for their ideas and work. When someone shares ideas, then implements them or sees someone else implement them and get the credit, they will tend to share less.

    That's unfortunate, because the variety of experience and ideas that come from a collective group potentially provides more ideas to choose from when searching for a solution.

  • Management:

    A software architect needs the trust of management to succeed for two simple reasons:

    1. Funding:
    2. The managers control the purse strings of the business, and they decide where money is invested. If the trust of management is lost, they may be less likely to invest in your product. Developers may be peeled off and moved to another project that is deemed a higher priority. In the worst case, they decide they no longer want to fund you or your position.

    3. Support:
    4. The software architect is the steward of the product; they take all of the responsibility yet do not own it. In many cases, you will simply do the research, and present the facts, possibly providing a few options for management to choose from. However, for the topics that matter the most, persuasion of management to support your initiatives may be crucial. It is much easier to persuade someone when you are in good favor with them. Moreover, you may find yourself in a conflict with the development team, and without the support of management, you may lose that battle.

  • Business Development and Marketing:

    This group if interested parties becomes important to the architect if their software is a product that is owned and sold by the company. Having a good line of communication with the groups that drive the business's growth is extremely important. Information is the most valuable thing in our industry; a lack of information leaves us to speculate. It's better to in line with the initiatives set by the company, if for no other reason than you may spot problems before too many resources have been invested. Change becomes much more difficult at that point.

    One other potential benefit of creating a strong relationship with these groups is they can gain a better grasp of what your product does. This is important because it allows them to consider the existing capabilities of your product when they are looking to convert a business opportunity into a sale. You flow information upstream, they flow information and potentially more sales back downstream.

  • Customers:

    The customer is the reason we write the software in the first place. The best way to earn trust with the customer is to listen to them. Sometimes it may seem like they don't know what they're talking about. However, it could simply be because you two are using different terms to mean the same things.

    Therefore, it is important to clarify what you understand the customer is telling you. A quick way to lose their trust, as well as their confidence in you, is to ignore the advice or requests of the customer. If you can't do something or decide that you are not going to do it, have that discussion with them rather than avoid them.

Let's compare trust to technology, or a software codebase. You need to maintain relationships, otherwise that bond of trust starts to weaken. I am sure there already is a fancy term to describe this like trustfulness debt. If not, there you go. Open and transparent communication will help develop trust. Returning and visiting with your contacts periodically is another way to help maintain that bond. When trust is lost, it is very difficult to earn back.

Utilize the Teams Abilities

In a way, this is the Software Architect showing trust in their development team. People expect the Software Architect to be the smartest person on the team, or the most knowledgeable. That does not have to be the case, and usually it depends on the question or topic that you are referring to. Architects are good at seeing the Big Picture. In most cases I would expect there to be a number of developers with finer skills when referring to a deep domain topic, such as device drivers.

Say the architect is responsible for the development of an operating system kernel. There are many generalizations and design decisions that can be made to create structure for the kernel. Then a bit more thought would go towards a driver model and its implementation to simplify that task. When you reach the implementation for each particular type of driver, there are different nuances between file-system drivers, network drivers, and port I/O drivers. At some point, you will reach the limit of the architect's knowledge and expertise, and you will reach the realm of the domain expert.

Some engineers like to learn every minute detail about a topic, no matter how abstruse; that, is the domain expert. Typically they do not do well in the role of architect because they tend to get caught up in the low-level details when it is not necessary; think, Depth-first Search. Nonetheless this is the perfect candidate for the architect to trust and depend on when knowledge and advice is required on the expert's domain of specialty.


When the word architect is used, it is usually associated with design, and every programmer already does design (whether it is formal or on-the-fly is another discussion.) I think it would be best to help clarify the role of the position if effort was made to emphasize structure when discussing the software architect, with regards to both team and the software product. This may help disambiguate the purpose and value a software architect provides and distinguish it from the next step up after senior engineer.

There is much more to this role that dictating how the program should be built, which if you read the entire article you now know that isn't even one of the architect's responsibilities.


general, adaptability, CodeProject, C++, design 5 feedbacks »

This post will focus on the concept of SFINAE, Substitution Failure Is Not An Error. This is a core concept that is one of the reasons templates are even possible. This concept is related exclusively to the processing of templates. It is referred to as SFINAE by the community, and this entry focuses on the two important aspects of SFINAE:

  1. Why it is crucial to the flexibility of C++ templates and the programming of generics
  2. How you can use it to your advantage

What the hell is it?

This is a term exclusively used in C++, which specifies that an invalid template parameter substitution itself is not an error. This is a situation specifically related to overload resolution of the considered candidates. Let me restate this without the official language jargon.

If there exists a collection of potential candidates for template substitution, even if a candidate is not a valid substitution, this will not trigger an error. It is simply eliminated from the list of potential candidates. However, if there are no candidates that can successfully meet the substitution criteria, then an error will be triggered.

Template type selection

SFINAE < Example >

I gave a vague description that somewhat resembled a statement in set theory. I also added a the Venn-diagram to hopefully add more clarity. However, there is nothing like seeing a demonstration in action to illustrate a vague concept. This concept is valid for class templates as well.

Below I have created a few overloaded function definitions. I have also created two template types that use the same name (overloaded), but have completely different structures. This example demonstrates the reason for the original rule:


struct Field
  typedef double type;
template <typename T>
typename T::type Scalar(typename T::type value)  
  return value * 4;
template <typename T>
T Scalar(T value)
  return value * 3;

The first case below, is the simpler example. It only requires, and accepts type where the value can be extracted implicitly from the type passed in; such as the intrinsic types, or types that provide a value conversion operator. More details have been annotated above each function.


int main()
  // The version that requires a sub-field called "type"
  // will be excluded as a possibility for this version.
  cout << "Field type: " << Scalar<int> (5) <<"\n";
  // In this case, the version that contains that
  // sub-field is the only valid type.
  cout << "Field type: " << Scalar<Field>(5) << "\n";


Field type: 15
Field type: 20


SFINAE was added to the language to make templates usable for fundamental purposes. It was envisioned that a string class may want to overload the operator+ function or something similar for an unordered collection object. However, it did not take long for the programmers to discover some hidden powers.

The power that was unlocked by SFINAE was the ability to programmatically determine the type of an object, and force a particular overload based on the type in use. This means that a template implementation is capable of querying for information about it's type and qualifiers at compile-time. This is similar to the feature that many languages have called reflection. Although, reflection occurs at run-time (and also incurs a run-time penalty).


I am not aware of a name for this static-form of reflection. If there is could someone comment and le me know what it is called. If it hasn't been name I think it should be something similar to reflection, but it is still a separate concept.

When I think of static, I think of "fixed-in-place" or not moving. Meditation would fit quite well, it's just not that cool. Very similar to that is ponder. I thought about using introspection, but that is just a more pretentious form of reflection.

Then it hit me. Rumination! That would be perfect. It's a verb that means to meditate or muse; ponder. There is also a secondary meaning for ruminate: To chew the cud; much like the compiler does. Regardless, it's always fun to create new buzzwords. Remember, Rumination.

Innovative Uses

I make heavy use of SFINAE in my implementation of Network Alchemy. Mostly the features provided by the < type_traits > header. The construct std::enable_if is built upon the concept of SFINAE. I am ashamed to admit, that I have not been able to understand and successfully apply std::enable_if yet. I have crossed many situations that it seemed like it would be an elegant fit. When I figure it out, I will be sure to distill what I learn, and explain it so you can understand it too. (I understand enable_if[^] now.)

Useful applications of SFINAE

To read a book, an article or blog entry and find something genuinely new and useful that I have an immediate need for is fantastic. I find it extremely irritating when there is not enough effort put into the examples that usually accompany the brief explanation. This makes the information in the article nearly useless. This is even more irritating if the examples are less complicated than what I could create with my limited understanding of the topic to begin with.

A situation is extremely frustrating when I believe that I have found a good solution, yet I cannot articulate the idea to apply it. So unless you get extremely frustrated by useful examples applied to real-world problems, I hope these next few sections excite you.

Ruminate Idiom

We will create a meta-function that can make a decision based on the type of T. To start we will need to introduce the basis on which the idiom is built. A scenario is required where there are a set of choices, and only one of the choices is valid. Let's start with the sample, and continue to build until we reach a solution.

We will need two different types that are of a different size


template < typename T >
struct yes_t
{ char buffer[2]; };
typedef char no_t;

We will also need two component that are common in meta-programming:

  1. the sizeof operator
  2. static-member constanst

We define a meta-function template, that will setup a test between the two types using the size of operator to determine which type was selected. This will give us the ability to make a binary decision in the meta-function.


template < typename T >
struct conditional
  template < typename U >
  static yes_t  < /* conditional on U */ > selector(U);
  static no_t selector(...);
  static T* this_t();
  static const bool value =
    sizeof(selector(*this_t())) != sizeof(no_t);

We started with static declarations of the two types that I defined earlier. However, there is no defined conditional test for the yes_t template, yet. It is also important to understand that the template parameter name must be something different than the name used in the templates outer parameter. Otherwise the template parameter for the object would be used and SFINAE would not apply.

The lowest type in the order of precedence for C++ is .... At first glance this looks odd. However, think of it as the catch all type. If the conditional statement for yes_t produces an invalid type definition, the no_t type will be used for the declaration of the selector function.

It is important to note that it is not necessary to define the function implementations for selector because they will never actually be executed. Therefore, it is not required by the linker. We also use an arbitrary function, selector, that returns T, rather than a function that invokes T(), because T may not have a default constructor.

It is also possible to declare the selector function to take a pointer to T. However, a pointer type will allow void to become valid as a void*. Also, any type of reference will trigger an error because pointers to references are illegal. This is one area where there is no single best way to declare the types. You may need to add other specializations to cover any corner cases. Keep these alternatives in mind if you receive compiler warnings with the form I presented above.

More Detail

You were just presented a few facts, a bit of code, and another random mix of facts. Let's tie all of this information together to help you understand how it works.

  • SFINAE will not allow a template substitution error halt the compiling process
  • Inside of the meta-function we have created to specializations that accept T
  • We have selected type definitions that will help us determine if a condition is true based upon the type. (An example condition will be shown next).
  • We also added a catch-all declaration for the types that do not meet the conditional criteria (...)
  • The stub function this_t() has been created to be used in a sizeof expression. The sizeof expression compares the two worker types to the no_t type to determine the result of our conditional.

The next section contains a concrete example conditional that is based on the type U.


Months ago I wrote about the standard header file, Type Traits[^]. This file contains some of the most useful templates for correctly creating templates that correctly support a wide range of types.

The classification of a type can be determined, such as differentiating between a Plain-old data (POD) struct and struct with classes. Determine if a type is const or volatile, if it's an lvalue, pointer or reference. Let me demonstrate how to tell if a type is a class type or not. Class types are the compound data structures, class, struct, and union.

What we need in the conditional template parameter is something that can differentiate these types from any other type. These types are the only types that it is legal to make a pointer to a scope operator ::*. The :: operator resolves to the global namespace.

Here is the definition of this template meta-function:


template < typename T >
struct is_class
  template < typename U >
  static yes_t  < int U::* > selector(U);
  static no_t selector(...);
  static T* this_t();
  static const bool value =
    sizeof(selector(*this_t())) != sizeof(no_t);

Separate classes based on member declarations

Sometimes it becomes beneficial to determine if an object supports a certain function before you attempt to use that feature. An example would be the find() member function that is part of the associative containers in the C++ Standard Library. This is because it is recommended that you should prefer to call the member function of a container over the generic algorithm in the library.

Let's first present an example, then I'll demonstrate how you can take advantage and apply this call:


template < typename T >
struct has_find
  // Identify by using a pointer to a member
  template < typename U >
  static yes_t  <U::find*> selector(U);
  static no_t selector(...);
  static T* this_t();
  static const bool value =
    sizeof(selector(*this_t())) != sizeof(no_t);

Applying the meta-function

The call to std::find() is very generic, however, it can be inconvenient. Also, imagine we want to build a generic function ourselves that will allow any type of container to be used. We could encapsulate the std::find() call itself in a more convenient usage. Then build a single version of the generic function, as opposed to creating specializations of the implementation.

This type of approach will allow us to encapsulate the pain-point in our function that would cause the implementation of a specialization for each type our generic function is intended to support.

We will need to create one instance of our meta-function for each branch that exist in the final chain of calls. However, once this is done, the same meta-function can be combined in any number of generic ways to build bigger and more complex expressions.


namespace CotD
template < typename C, bool hasfind>
struct call_find
   bool operator()(C& container, C::value_type& value, C::iterator &result)
    result = container.find(container.begin(),
                            value) != container.end();
    return result != container.end();
} // namespace CotD


namespace CotD
template <typename C>
struct call_find <C, false>
  bool operator()(C& container, C::value_type& value, C::iterator &result)
    result = std::find( container.begin(),
    return result != container.end();
} // namespace CotD

This is a very simple function. In my experience, the small, generic, and cohesive functions and objects are the ones that are most likely to be reused. With this function, we can now use it in a more specific context, which should still remain generic for any type of std container:


namespace CotD
template < typename T >
void generic_call(T& container)
  T::value_type target;
  // ... Code that determines the value ...
  T::iterator item;
  if (!CotD::call_find<T, has_find<T>::value>(container, target, item))
  // ... item is valid, continue logic ...
} // namespace CotD


int main(int argc, _TCHAR* argv[])
  bool res = has_find<SetInt>::value;
  call_find<SetInt, has_find<SetInt>::value> set_call;
  SetInt::iterator set_iter;
  SetInt s;
  set_call(s, 0, set_iter);
  call_find<VecInt, has_find<VecInt>::value> vec_call;
  VecInt::iterator vec_iter;
  VecInt v;
  vec_call(v, 0, vec_iter);
Run this code


T::find() called
std::find() called

We made it possible to create a more complex generic function with the creation of the small helper function , CotD::find(). The resulting CotD::generic_call is agnostic to the type of container that is passed to it.

This allowed us to avoid the duplication of code for the larger function, CotD::generic_call, due to template specializations.

There is also a great chance that the helper template will be optimized by the compiler to eliminate the unreachable branch due to the types being pre-determined and fixed when the template is instantiated.


Substitution Failure Is Not An Error (SFINAE), is a subtle addition that was added to C++ make the overload resolution possible with templates. Just like the other basic functions and classes. However, this subtle compiler feature opens the doors of possibility for many applications in generic C++ programming.

Are You Mocking Me?!

general, adaptability, portability, CodeProject, maintainability Send feedback »

It seems that every developer has their own way of doing things. I know I have my own methodologies, and some probably are not the simplest or the best (that I am aware of). I have continued to refine me design, development, test and software support skills through my career.

I recognize that everyone has their own experiences, so I usually do not question or try to change someone else's process. I will attempt to suggest if I think it might help. However, sometimes I just have to ask, "are you sure you know what you are doing?" For this entry I want to focus on unit testing, specifically with Mock Objects.


Are you sure you know what you are doing?

What do you mean by "Mock"?

I want to seriously focus on a type of unit-test technique, one that is so misused, that I would even go so far as to call it an anti-technique. This is the use of Mock Objects.

Mock objects and functions can fill in an important gap when a unit-test is attempting to eliminate dependencies or to avoid the use of an expensive resource. Many mock libraries make it pretty damn simple to mock the behavior of your codes dependencies. This is especially true when the library is integrated into your development environment and will generate much of the code for you.

There are other approaches that exist, which I believe are a better first choice. Whatever method you ultimately chose should depend on the goal of your testing. I would like to ruminate on some of my experiences with Mock Objects as well as provide some alternate possibilities for clipping dependencies for unit tests.

Unit Testing

Mock Objects are only a small element of the larger topic of unit-testing. Therefore, I think it's prudent to provide a brief overview of unit testing to try to set the context of this discussion, as well as align our understanding. Unit testing is a small isolated test written by the programmer of a code unit, which I will universally refer to as a system.

You can find a detailed explanation of what a unit test is and how it provides value in this entry: The Purpose of a Unit Test[^].

It is very important to try to isolate your System Under Test (SUT) from as many dependencies as possible. This will help you differentiate between problems that are caused by your code and those caused by its dependencies. In the book xUnit Patterns, Gerrard Meszaros, introduces the concept of a Test Double used to stand-in for these dependencies. I have seen many different names used to describe test doubles, such as dummy, fake, mock, and stub. I think that it is important to clarify some vocabulary before we continue.

The best definitions that I have found and use today come from this excellent blog entry by, Martin Fowler, Mocks Aren't Stubs. Martin defines a set of terms that I will use to differentiate the individual types of test doubles.

  • Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
  • Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
  • Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
  • Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive

Martin's blog entry above is also an excellent source for learning a deeper understanding between the two general types of testing that I will talk about next.

So a Mock is just a different type of test double?!

No, not really.

Besides replacing a dependency, mock objects add assertions to their implementation. This allows a test to report if a function was called, if a set of functions were called in order, or even if a function was call that should not have been called. Compare this to simple fake objects, and now fake objects look like cheap knock-offs (as opposed to high-end knock-offs). The result becomes a form of behavioral verification with the addition of these assertions.

Mock objects can be a very valuable tool for use with software unit tests. Many unit test frameworks now also include or support some form of mocking framework as well. These frameworks simplify the creation of mock objects for your unit tests. A few frameworks that I am aware of are easyMock and jMock for JAVA, nMock with .Net and GoogleMock if you use GoogleTest to verify C++ code.

Behavior verification

Mock objects verify the behavior of software. For a particular test you may expect your object to be called twice and you can specify the values that are returned for each call. Expectations can be set within the description of your mock declaration, and if those expectations are violated, the mock will trigger an error with the framework. The expected behavior is specified directly in the definition of the mock object. This in turn will most likely dictate how the actual system must be implemented in order to pass the test. Here is a simple example in which a member function of an object registers the object for a series of callbacks:


// No, I'm not a Singleton.
// I'm an Emporium, so quit asking.
class CallbackEmporium
  // Provides functions to register callbacks
TheCallbackEmporium& TheEmporium()
{ ... }
// Target SUT
void Object::Register()
  TheEmporium().SetCallbackTypeA( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );

Clearly the only way to validate in this function is through verifying its behavior. There is no return value, so that cannot be verified. Therefore, a mock object will be required to verify the register function. With the syntax of a Mock framework, this will be a snap, because we add the assertion right in the declaration of the Mock for the test, and that's it!

Data verification

Ultimately the function object::Register() is interested to know if the two proper callbacks were registered with TheEmporium. So if you nodded your head in agreement in the previous section when I said "Clearly the only way...", I suggest you stop after you read sentences that read like that and challenge the author's statement. Most certainly there are other ways to verify, and here is one of them.

1 point if you paused after that trick sentence, 2 points if you are reserving judgment for evidence to back up my statement.

It would still be best if we have a stand-in object to replace TheEmporium. However, If there is some way for use to verify after the SUT call, that the expected callback functions were registered in the correct parameters of TheEmporium, then we do not need a mock object. We have verified the final data of the system was as expected, not that the program executed to a prescribed behavior.

Why does it matter?

Tight Coupling between the test and the implementation.

Suppose you wrote your mock object to verify the code in this way:


// This is a mocked yet serious syntax
// for a mock-object statement to verify Register().
Mocker.call( SetCallbackTypeA()
      .call( SetCallbackTypeB()

That will test the function as-is currently implemented. However, if the implementation of Object::Register were implemented like anyone of these, the test may report a failure, even though the intended and correct results were achieved by the SUT.


void Object::Register()
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );
  TheEmporium().SetCallbackTypeA( Object::StaticCallA, this );


// Too many calls to one of the functions
void Object::Register()
  TheEmporium().SetCallbackTypeA( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );


// Call each function twice:
// Assign incorrect values first.
// Then call a second time with the correct values.
void Object::Register()
  TheEmporium().SetCallbackTypeA( Object::StaticCallB, this );
  TheEmporium().SetCallbackTypeA( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );


// Assign incorrect values first.
// Then call a second time with the correct values.void Object::Register()
  TheEmporium().SetCallbackTypeA( Object::StaticCallB, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeA( Object::StaticCallA, this );
  TheEmporium().SetCallbackTypeB( Object::StaticCallB, this );

All four of these implementations would have continued to remain valid for the data validation form of the test. Because the correct results were assigned to the proper values at the return of the SUT.

When the Mock Object Patronizes You

Irony. Don't you love it?!

Mock objects can get you very far successfully. In fact, you may get towards the very end of your development phase and you have unit tests around every object and function. You are wrapping up your component integration phase. Things are not working as expected. These are some that I have personally observed:

  • The compiler complains about a missing definition
  • The linker (for the C and C++ folks) complains about undefined symbols have been referenced
  • This is a network application, everything compiles and links, the program loads and doesn't crash. It doesn't do anything else either. You connect a debugger, it is not sending traffic.

I have seen developers become so enthusiastic with how simple mock objects made developing tests for them, that they virtually created an entire mock implementation. When it compiled and was executed, critical core components had a minimal or empty implementation. All of the critical logic was complete and verified. However, the glue that binds the application together, the utility classes, had not been implemented. They remained stubs.


There are many ways to solve problems. Each type of solution provides value in its own way. Some are simple, others elegant, while others sit and spin in a tight loop to use up extra processing cycles because the program doesn't work properly on faster CPUs. Just be aware of what value you are gaining from the approaches that you take towards your solution. If it's the only way you know, that is a good first step. Remember to keep looking, because their may be better solutions in some way that you value more.

Contact / Help. ©2017 by Paul Watt; Charon adapted from work by daroz. CMS / cheap web hosting / adsense.
Design & icons by N.Design Studio. Skin by Tender Feelings / Skin Faktory.