Alchemy is a collection of independent library components that specifically relate to efficient low-level constructs used with embedded and network programming.The latest version of Embedded Alchemy[^] can be found on GitHub.
The most recent entries as well as Alchemy topics to be posted soon:
I am currently working on a Masters Degree in Cybersecurity at Johns Hopkins University. Therefore, I have not been able to devote much time to the further development of Alchemy. I will be finished with my degree in the Spring of 2017 and I plan to resume Alchemy's development. I plan to use my newly acquired knowledge to add constructs that will help improve the security of devices built for the Internet of (Insecure) Things.
Coupling and Cohesion are two properties that deserve your attention when you design software. These are important because they have a direct effect on how flexible and maintainable your software will be as your software continues to e developed. Will you be able to reuse it? Will you be able to adapt it? Will you need a shoe-horn to force that new feature in the future? These questions are much simpler to answer when you can properly gauge the level of Coupling and Cohesion of your components.
Coupling and Cohesion
The source of the terms Coupling and Cohesion originated from the 1968 National Symposium on Modular Programming by Larry Constantine. A more commonly referenced source is from a later published paper called, Structured Software Design (1974), Larry Constantine, Glenford Myers, and Wayne Stevens. This paper discusses characteristics of “good” programming practices that can reduce maintenance and modification costs for software.
Loose (Low) coupling and High (Tight) cohesion are the desirable properties for components in software. Low coupling allows components to be used independently from other components. High cohesion increases the probability that a component can be reused in more places, by limiting its capabilities to small well-defined tasks. For those of you keeping score at home, high cohesion is the realization of the S, Single Responsibility, in SOLID object-oriented design.
A measure of the degree to which a component depends upon other components of the system.
|Given two components X and Y:|
|If X changes||→||how much code in Y must be changed?|
Low coupling is desired because it allows components to be used and modified independently from one and other.
|The next two rows demonstrate coupling dependencies.|
Very generic coupling, the most compatible bricks.
A measure of how strongly related the various functions of a single software component are.
|Given a component W:|
|How focused is W at fulfilling a single responsibility?|
High cohesion is desired because:
- Leads to a simpler minimal class that embodies one concept
- A minimal class is easier to comprehend and use
- Therefore it is more likely that this class will be reused
These pieces demonstrate cohesion over a range of interfaces.
Reuse of Code
It’s important to set your expectations for what to consider reusable code. Without a clear set of expectations, how will you know when you have achieved your goals. More importantly, how can you formulate a design strategy or a reproducible process to consistently create reusable code.
Not all code is reusable
To further clarify this statement, not all code should be reused.
Because the majority of software that is written is designed to fulfill a very specific purpose. The challenge is to find the balance point between generic reusable building blocks, and a tool or application that meets a need. Generic software components are very valuable, however, they are almost useless by themselves.
What good is a linked-list without a mechanism to populate the data in the list, process the data to perform useful outcomes, and later inspect the data. Another example could be a button control for a GUI that handles the name, display and animations of the button. However, it is the actions that are triggered by the button that are important to the end user.
Good design and practices should be used for all software that you write. However, recognize and differentiate which portions of your application are generic and likely to be reused compared to the code that is very specialized. Both of these are of equal importance.
The value of reusable code
For every statement of code that you write or use, you should ask the question:
Does this add value or risk?
Reusable code only provides value If it:
- Is easy to learn and use
- Requires less work -> time -> money to develop your final target
- Has already been tested and proven to be robust
- Results with a project that is at least as robust as the code that it is built upon
If any one of these qualities is undetermined, you may be adding risk along with any perceived value for the reused code.
Increase the potential for code reuse
Here are some design guidelines that can help increase the potential for your code and components to be reusable:
Easy to use correctly, hard to use incorrectly
If you are fan of Scott Meyers’, Effective C++, you will probably recognize this from Item 18:
“Ideally, if an attempted use of an interface won’t do what the client expects, the code won’t compile; and of the code that does compile, it will do what the client wants”
Good interfaces are easy to use correctly and hard to use incorrectly. The example that is used to demonstrate this concept in Effective C++ uses a calendar date class to illustrate the point. Start with a constructor such as this:
This is a reasonable declaration if you live in the United States, which uses the format month/day/year. However, many other countries prefer the format day/month/year. Other possibilities exist such as the ISO 8601 date order with four-digit years: YYYY-MM-DD (introduced in ISO 2014), which is specifically chosen to be unambiguous.
Another problem with this interface is there are three integer input parameters. Without refering to some form of documentation, it may be impossible to deduce the order of the parameters. These are some of the possibilities for how the interface could be misused:
One possible approach to solving all of the issues above is to create new types to improve clarity and enforce compile-time correctness.
Declaring the only constructor privately forces the user to use the static member functions to create new instances of a Month object. Similar definitions could be created for the day and year parameters. The improved usage would look like this:
To encourage correct use, design interfaces that are consistent and behaviorally compatible.
Use these techniques to help prevent errors:
- Create new types
- Restrict operations on those types
- Constrain object values
- Eliminate client resource management
Create interfaces that are minimal and complete
Strive to create the minimal number of operations in your interface to access the complete behavior of the object. Minimal objects embody one concept at the right level of granularity. Also, minimal interfaces are easier to comprehend, and more likely to be used and reused.
Monolithic classes will generally dilute encapsulation. Instead of doing one thing well, they end up providing many things that are mediocre. It is more difficult to make these objects correct and error-safe because they tackle multiple responsibilities. These types of objects usually result from an attempt to deliver a complete solution for a problem.
Encapsulate - Hide/Protect Information
Encapsulation is one of the most valuable aspects of object-oriented programming. Encapsulation is necessary to create a proper abstract design. The purpose of abstraction is to simplify concepts by hiding the details of the abstraction.
A proper abstraction will handle the dirty details required to achieve a correct and error-free implementation. The hidden details verify the invariants of your object each command and when internal state or data is changed. Abstractions become even more effective when a number of internal variables can be managed, and a reduced set of values are presented through the public interface.
3 made up facts… Eh, but probably true
- Programmers like to get the job done
- Programmers are inventive
- If a programmer finds a way to complete a task, even if it requires them to blatantly ignore the comment below, they will still write to the data; Because it will get the job done (refer to factioid 1)
Declare all data members private. Private data is teh best means a class can use to protect its invariants now as well as in the future as it adapts to possible changes.
Public data members move the responsibility to maintain and guarantee the invariants from the object to all of the code that accesses the abstraction. Freely providing access to an objects data members effectively negates most of the benefits gained from creating the abstraction.
Metaphorical Thought Experiment
WARNING: Visualize this thought experiment at your own risk.
Next time you are considering creating a public data member, think of the human body as an abstraction of a much more complicated biological organism. One where the designer declared all of the data members public rather than private.
- Your internal organs would now all be external organs ripe for the picking of any black-market human organ peddeler.
- All figures of speech related to the body must be interpretted literally: “What’s on your mind?”
- There will be many more Do-it-yourself doctors.
- Negates the benefits of being declared a
- Couples do not need to worry about tight coupling, however, they will need to worry about becoming tangled.
- Humans in space?! Forget it!
The inferiority of non-
private data members
private data members are almost always inferior to even simple pass-through getter/setter methods. These methods provide the flexibility in the future to adapt to change as well as these benefits:
- Verify Invariants
- Lazy Evaluation / Data Caching
- Decouple for Future Modifications
Protected data members suffer from the same disadvantages as
public data members. The responsibility to maintain the invariants is shared with the unbounded set of objects which includes:
- Every class currently derived from this class
- Every class that will be derived from this class in the future
The one exception is the C-style struct or plain-old data (POD), which is used as a behaviorless aggregate and contains no member functions.
Use the weakest coupling relationship possible
Let’s examine the types of relationships that are possible in C++. The list below is ordered by a measure of coupling, tightest to loosest:
- Nonpublic inheritance
- Composition (member variable)
Choose the coupling relationship that provides the weakest level of coupling possible. This will provide more flexibility when something needs to be changed in the future.
Prefer writing non-member non-friend functions
This will improve encapsulation by minimizing dependencies, therefore, reduces coupling. The function implementation cannot be modified to depend on the private member data of the class. It can only be implemented based on what is publically accessible.
This will also break apart monolithic classes. The separable functionality can then be further encapsulated by composition.
Example: std::string contains 103
public member functions, 71 could be non-member non-
Use inheritance to achieve substitutability
Inheritance is a wonderful tool available with object-oriented programming. However, it should not be selected as a way to reuse code. Instead, use inheritance as a way to facilitate polymorphism.(For those of you keeping score at home, this concept is the L, Liskov substitutability principle, in SOLID object-oriented design.)
When you consider inheritance, seriously contemplate the is-a relationship between your objects. Think of is-a as “Can be substituted with…".
Public inheritance facilitates polymorphism, which implies substitutability:
|Original Object||… Can be substituted with:||… Not with:|
Working with your objects will feel cumbersome if you inherit without substitutability.
Don’t inherit to reuse; inherit to be reused.
From Item 37 in C++ Coding Standards, Herb Sutter and Andrei Alexandrescu
Both composition and
private inheritance means is-implemented-in-terms-of.
So which one do you use?
Use composition whenever possible. Choose
private inheritance when necessary. It becomes necessary when
protected access to information is required.
Remember, use the solution that creates the weakest coupling relationship:
Don’t underestimate the power of composition
Robust composition is easier to achieve compared to inheritance, especially with cohesive interfaces.
Minding the concepts of coupling and cohesion can lead to more robust software that avoids fragility and is more maintainable.
- Simple definitions and implementations are easier to comprehend and use
- Simple objects can be reused to create more useful objects with composition
- Only use public inheritance when derived objects can truly be substituted for the base object
- Every solution in a computer program is built upon a collection of much simpler solutions
Legos images, trademarks and copyrights belong to the LEGO Group.
I learned a new term today, bikeshedding. After reading about this term, I reached clarity regarding a number of phenomena that I have experienced as a software developer. This term is an example of Parkinson's law of triviality, which was defined by, Cyril Northcote Parkinson, in one of his books, Parkinson's Law: The Pursuit of Progress (1958).
Parkinson's law of triviality:
The time spent on any item of the agenda will be in inverse proportion to the sum involved.
The law of triviality is actually only a corollary to the broader adage, Parkinson's Law, which I am almost certain that you have heard before:
Work expands so as to fill the time available for its completion.
This book is a series of cynical essays that satirizes government bureaucracies and their tendency to increase in size over time, until they eventually become irrelevant. Parkinson's law of triviality is an example of a committee's deliberations for the expenses of a nuclear power plant, a bicycle shed and refreshments.
Why is it called bikeshedding?
Parkinson dramatizes the committee's discussion, and selected a nuclear reactor because of it's vast difference in complexity and expense compared to the bicycle shed. A nuclear reactor is so complex that the average person cannot completely understand it. Contrast this with the relative simplicity of a bike shed. Anyone can visualize a bike shed, and therefore anyone that want's to contribute will add suggestions to show their participation.
Simpler topics will elicit more discussion and debate. Agreement is hardest to achieve for simpler technical problems or organizational topics that can be argued indefinitely.
Let's return to the dramatization of the committee's discussion. Here are the results for each of the three topics:
- The contract for the reactor with a cost of £10 million passes in two and a half minutes because it is to complex and technical.
- The bicycle shed costs £350 and discussion lasts forty-five minutes as everyone in the committee can grasp the number £350. Also, the shed is simple enough that each member has an opinion regarding the building materials and the possibility of saving £50. They are all pleased with themselves with this item concluded.
- Finally, a discussion regarding £21 for the yearly supply of refreshments takes seventy-five minutes. The conclusion is to as the Secretary to gather more information and they will resolve this issue next meeting.
The really, really short answer is that you should not. The somewhat longer answer is that just because you are capable of building a bikeshed does not mean you should stop others from building one just because you do not like the color they plan to paint it. This is a metaphor indicating that you need not argue about every little feature just because you know enough to do so. Some people have commented that the amount of noise generated by a change is inversely proportional to the complexity of the change.
What sort of clarity did I find? A few months ago I wrote a few entries regarding software design patterns. I briefly mentioned the commonly vilified Singleton and promised to follow up with a more detailed exploration of the Singleton. While collecting my thoughts and determine what I wanted to say about the Singleton, I experienced The Singleton Induced Epiphany[^].
I have always had a hard time understanding why so many developers are quick to criticize this design pattern, and I have heard very few criticisms of any other software design pattern. I reached two conclusions by the end of that post:
- The majority of the arguments against the Singleton are criticisms about the languages used to implement the Singleton. For instance up until recently C++ did not have a construct built into the language to be able to create a portable and thread-safe implementation of the Singleton, and therefore the Singleton is bad.
- The Singleton is the simplest design pattern, and most likely the only design pattern that everyone understands completely. Therefore, it is the only design pattern everyone feels comfortable arguing about.
At the time I was unaware that a very succinct term existed to describe the phenomenon that I observed, and now I know that it is called bikeshedding.
Now that you know of bikeshedding you should be able to recognize this phenomenon. You can then start to curb the pointless discussions about the minutiae and redirect everyone's time and attention to the more important and technical discussions at hand.
What does one-size fits all mean? I suppose there are two way to answer that question, 1) How well do you want it to fit? 2) What do you mean by 'all'? For example, a clothing manufacturer is content with the percentage of the population that their product will fit, such as a T-shirt, hat or Snuggie. All in this case is just a sub-set of the actual population. It should also be stated that the Snuggie will only fit perfectly and look stylish on the most charismatic of cult leaders.
Next time you read an article about programming best practices, consider if the one-size fits all mentality is used by the author. What is their experience level, programming background, project history, or even industry? These are some of the factors that determine how well their advice can be applied to your situation. Especially, if their article only seems to consider situations similar to theirs.
Ultimately, I want to illustrate that it is necessary to understand the context and background used by the author to describe a programming methodology or a set of best practices. I am referring to the process du jour such as SCRUM, XP, Waterfall, 6 Sigma, Crouching Tiger, d20, GURPS or what have you. Assuming that the author has found value in the development process of which they are writing, they have most likely applied it with success in a narrow range of projects compared to what exists.
I am not denigrating anyone's experience. I only assert that it is not possible to create a process that is the best possible practice for every type of development project.
Sometimes when I read an article extolling the virtues of a process such as SCRUM, the concepts that are presented seem very foreign to me. This is in spite of having been a part of SCRUM teams that were both successful and unsuccessful in their goals. In a situation like this I find that I become a bit defensive as I read thinking "There is no way that would!"
That is until I stop and consider the context of the article. Is the author an application developer, web developer, or a rocket surgeon for NASA? When you start to consider the author's perspective and the context for which they are writing, it becomes much easier to identify the value in the concepts they are describing. Only then is it possible to take their lessons learned and try to apply them to your context and situation.
Methodologies such as SCRUM, XP and even Waterfall have provided value and been successfully applied for a variety of teams and development projects. Also realize that these same methodologies have failed miserably for all types of development situations. Why?
I believe there are valuable aspects to all development methodologies. However, many of them are specifically designed for use in a certain context. What works well for a web development team might not directly transfer well to a team that develops products for government contracts. Some tailoring and adaptation will most likely be required to see any of the benefits.
Each step prescribed by the process is presumably added to solve a particular problem that has occurred repeatedly during development. While all software development looks the same from the point-of-view of the programmer in the chair, it is naïve to think that this is true for all stages of development in all contexts where software is developed. Many of the articles that I have read either make this naïve assumption, or do not clearly state the context for which the article is written. I have seen this for both average programmers that want to share their experience and the most respected developers in the industry that speak in terms of absolute best practices.
Everyone is trying to find a way to make their daily chores simpler with more reliable development processes. This is especially true for teams where one must coordinate activities with others. This can lead to a lot of floundering for the majority of the population that is not sure what they need to improve their development processes. They discover a new process, but fail to tailor it to their specific situation and wonder why it feels awkward, or they are not seeing the improvements that they expected.
I suspect this is one reason why we often read of the complaints where teams blindly follow the steps prescribed by a development process and it simply feels like a waste of time. These teams have failed to adapt the process to their particular situation.
Our level of experience affects how we are able absorb apply concepts with respect to a subject. The Dreyfus Model of Skill Acquisition is one model of this concept. Essentially, it describes the progression from a Novice to an Expert. As a person becomes more proficient with a skill, they are capable of moving away from rigid sets of rules and guidelines towards an intuitive grasp of the situation based upon on deep tacit understanding.
As a Novice, we generally rely on a rigid set of rules or plans to lead us to a successful outcome. A practitioner at this level is not expected to exercise much, if any, discretionary judgment to succeed. This is very similar to most baking cookbooks. The recipes are described in steps, and the quality of baked good often depends on how closely the recipe was followed. Attempting to alter the recipe without understanding the purpose of each ingredient and each step could lead to something that is not edible.
More possibilities become available to a student/practitioner as they gain experience. With a greater collection of skills, they can begin to recognize patterns and differentiate between similar situations that require different solutions. As an example let's consider the differences between making the mix for a batch of muffins vs. cookies. The list of ingredients for chocolate chip cookies looks very similar to muffins. In fact, without careful inspection, all of the recipes in a cook book look similar due to their common layout and format. However, there is an enormous difference between the two types of recipes.
With the understanding of a few concepts in baking beyond beginner level, it is possible to slightly alter existing recipes to create a different consistency. Muffins use the "Muffin Method", which combines all of the dry ingredients in one bowl and the wet ingredients in a second bowl. When the oven has reached it's target temperature, the ingredients from both bowls are combined and briefly stirred, placed in the pan, then finally baked in the oven. The ingredients are not combined until the oven is ready, in part because the muffins use baking powder as one of its leaveners. When the acid in the egg whites is mixed with the dry ingredients the baking powder is activated for a brief period of time and will create air bubbles. Once the batter reaches a high enough temperature the batter will set and lock the bubbles in place creating a fluffy texture.
Basic chocolate chip cookies use the "Creaming Method". This is a step where the butter and sugar are beat together to create billions of tiny air bubbles within the mixture. This combination of ingredients adds structure to the cookies because the sugar holds the air until the cookie mix is set in the oven.
What happens if the "Creaming Method" is not strictly followed? The cookies will still probably turn out, however, they will not have the chewy consistency one would expect of this traditional cookie recipe. If the butter is completely melted you will end up with the thin crispy cookie (hey, maybe that's what you wanted). The chewy consistency is also achieved by the gluten that is created when mixing the flour with the water-based ingredients in the mix. To create a less dense cookie you could change the type of flour to eliminate the gluten.
Expert bakers will simply "know" a few dozen baking recipes and ingredient ratios off the top of their head and know how altering the ratio of the ingredients is likely to affect the final baked good. From this collection of knowledge they would be able to create a plethora of new recipes to suit the needs of each situation they encounter.
Unfortunately, experts will often know a subject so well that some times they are not able to explain how they know the answer, "they just do." This can be intuition leading them to the answer. They have encountered the problem so many times they recognize the pattern and know the right solution. This can lead to problems for less experienced practitioners learning from an expert. Even with a decent amount of experience we are not always able to see all of the connections to arrive at the expert's conclusion. This is one reason why I dislike the phrase "I will leave this as an exercise for the reader."
A Perfect Fit
Software design patterns are similar to recipes for a software development, though not quite. I would compare them more to different techniques in baking such as the "Muffin Method" and the "Creaming Method". You choose a design pattern based on the problem you are trying to solve. Each design pattern is known for providing a good solution to certain classes of problems.
A particular implementation of the design pattern would be comparable to a recipe. The Singleton design pattern is capable of creating solutions more sophisticated than an object that controls it's own creation and limits itself to a single instance. Just as you adapt software design patterns to fit your needs when you develop a software solution, so should you adapt the development processes that you choose to use in your organization.
One final example to illustrate this point. For decades large corporations have been adopting and incorporating SAP database systems to help optimize the logistics of running their company. When these solutions are bought, they require large multi-million dollar investments to customize these systems for the company. The SAP software is already written. However, custom integration is required to make the system usable and useful to the company.
The software development process that you select should be no different. Yes, all SCRUM processes will look similar when compared between different companies that practice this process. However, I would expect there are subtle details that differ between two companies that successfully manage their projects with SCRUM. I believe these subtle differences could be important enough that if one company were to change to the other companies process they would no longer be as successful, or even worse, they no longer succeed.
It is important to analyze "best practices" and development processes before adopting them to determine how well they fit your organization. I have worked for a half-dozen companies in my career, and none of them setup their build process exactly the same, even though many of them used the same tools. In order to get a best-fit for development processes and practices I would expect some level of customization to be required to be as effective as possible.
A few weeks ago I wrote an entry on SFINAE[^], and I mentioned
enable_if. At that point I had never been able to successfully apply
enable_if. I knew this construct was possible because of SFINAE, however, I was letting the differences between SFINAE and template specialization confuse me. I now have a better understanding and wanted to share my findings.
What is it?
As I mentioned before
enable_if is a construct that is made possible because of SFINAE.
enable_if is a meta-function that can help you conditionally remove functions from the overload resolution performed on the type-traits supplied to the function.
You can find examples of it's usage in the Standard C++ Library within the containers themselves, such as
previous block of code accomplish? The assign call will always exist if you want to specify a count and supply a value that is a
value_type. However, the conditional version will only be declared if the parameterized type has a type_trait that evaluates to an iterator-type. This adds flexibility to definition of this function, and allows many different instantiations of this function; as long as the type is an iterator.
Also note that the template definition has a completely different signature than the other non-parameterized version of assign. This is an example of overload resolution. I think it is important to point this out, because I was getting caught up into how it should be used. As I perused the Internet forums, it seems that many others are confused on this point as well.
How it works
Let's continue to look at the
std::vector. If the parameterized type is not an iterator type, the
enable_if construct will not allow the parameterized type to exist. Because this occurs in the template parameter selection phase, it uses SFINAE to eliminate this function as a candidate to participate in the overloaded selection of assign.
Let's simplify this example even further. Imagine the second version of
assign does not exist; the only definition is the parameterized version that uses
enable_if. Because it is a template, potentially many instantiations of this function will be produced. However, the use of
enable_if ensures that the function will only be generated for types that pass the iterator type-trait test.
This protects the code from attempting to generate the assign function when integers, floats or even user defined object types. Essentially, this usage is a form of concept checking that forces a compiler error if the type is not an iterator. This prevents invalid code with types that appears to compile cleanly from getting a chance to run if it will produce undefined behavior.
Previously, when I attempted to use
enable_if, I tried to apply it in a way that reduced to this:
Hopefully something appears wrong to you in the example above. If we eliminate the template declarations, and the function implementations, this is what remains:
When the compiler processes the object, it may not instantiate every function, however, it does catalog every definition. Even though there is a definition for a false case that should not result in the instantiation of that version, it does not affect the overload resolution of the functions. The
enable_if definition needs to contain a type that is dependent on the expression. This line will always be invalid:
Types of Usage
There are four places
enable_if can be used:
Function return type
This is a common form. However, it cannot be used with constructors and destructors. Typically one of the input parameters must be dependent on a parameterized type to enable multiple function signatures. Overload resolution cannot occur on the return-type alone.
This format looks a little awkward to me. However, it works for all cases except operator overloading.
Class template parameter
This example is from cppreference.com, and it indicates that
static_assert may be a better solution that the class template parameter. Nonetheless, here it is:
Function template parameter
This is probably the cleanest and most universal form the
enable_if construct can be used. There are no explicit types for the user to define because the template parameter will be deduced. While the same amount of code is required in this form, the declaration is moved from the function parameter list to the template parameter list.
Personally I like this form the most.
When learning new tools, it is often difficult to see how they are applied. We spent years practicing in math, as we attempted to solve the numerous story problems with each new mathematic abstraction taught to use. Writing code and using libraries is very similar.
enable_if is a construct that is best used to help properly define functions that meet concept criteria for a valid implementation. Think of it as a mechanism to help eliminate invalid overload possibilities rather than a tool to allow the selection between different types and your application of this meta-function should become simpler.
A continuation of a series of blog entries that documents the design and implementation process of a library. The library is called, Network Alchemy[^]. Alchemy performs low-level data serialization. It is written in C++ using template meta-programming.
At this point, I was still in the make it work phase of implementation. However, I learned quite a bit from experiencing the progression of design and multiple implementations. So much that I thought it was valuable enough to share.
After the initial creation of the
Datum type, it was relatively simple to abstract the design and implement the Bit-Field. object. The next step was to combine the collection of
BitField objects into a single group that could be statically-verified as type-safe. This was also relatively simple. Again I just had to model the implementation of the message structure. I called this collection a
BitList. Primarily to differentiate it from the
The real trouble appeared when I began the integration of the
BitList with the
Message in a way that was similar to the
Datum. At this point, I started to see the potential this library could provide. Through experience, I have learned that it is important to get the programming interface correct in the initial design. Any mistakes that were made, or code that is not as clean as you would like can be refactored or even re-implemented behind the original interface.
My initial approach was straight-forward. As meta-programming was new to me, I looked for opportunities to take advantage of the compiler to verify invariants where ever I could apply them. Therefore, there is a type_check that verifies the number of bits specified by the user does not exceed the capacity of the data type used for storage.
I thought the best approach at the time was to store a reference to the primary value, to which all of the
BitFields referred. This was an attempt to avoid a mechanism where the child calls the parent for the value then the operation is performed. I was focused on the general usage of the object, and didn’t pay too much attention to the construction of this type. As you can see, the construction of this
BitList is quite expensive compared to the amount of information that is encoded in the type.
There are two parts to the previous section of code, the initializer list, and the constructor body. The initializer list is used to associate address of the BitField object with the local reference held by the BitList container. This is how the container is able to enumerate over the values in the class. The second part attaches a reference of the internal data field with the child fields.
Because of this twisted setup, the majority of what remains is simple and looks much like the other value classes I have previously shown. The one oddity that remains, is a set of static constants defined to help navigate the set of sub-fields. The code below depends on the knowledge of the size and relative offset for each value.
Those definitions allow the actual value types fields to be defined with the code below:
Unfortunately with this design, there will always be eight sub-fields per BitList object, regardless of the actual number of bit-fields that are used. Imagine my surprise when shortly after I released this into the wild, I had requests to support 32 bit-fields per Bitlist. At that point I started to think of ways that I could eliminate all of these unused references. Unfortunately, I was not able to implement my ideas until the second version of this type. So those ideas will have to wait.
To organize the
Hg::BitList similarly to the message required an outer container to be defined for the user type. I am not going to show all of the details because they are not that interesting. Also, the current implementation for this feature in Alchemy barely resembles this concept at all. Here is a high-level overview of it’s original structure.
Essentially, the MACRO below is a euphemism for the morass of partially defined parameterized errors waiting to be compiled.
The name info defines a new
BitList type the user will place in the type-list definition that defines the overall message format.
In the end, I spent much more time on this section of code that I am comfortable admitting. So let me just show you a few snippets of what the MACRO definition hides and move away from this.
For each field that is defined, a collection of definitions is created by the MACRO that is similar to this:
-The definition above declares a new
Hg::BitField type that meets that criteria for the proper index, offset and size of that field. A new instance of that time is defined with the name the user has specified, and a function is provided that returns a reference to that field. The function requires a
const reference of an object of the same type as the desired field (I have since realized that pointers are much less expensive than const references and just as effective). Since all of the fields have different indices and offsets there will be no problems with duplicate types.
The type definitions use a well-defined format that can be programmatically indexed. Therefore developing functions to operate over the fields will be possible. However, there is still more work to integrate this type into the Alchemy framework.
Throughout the development of this project, the template definitions continued to grow longer and more complex. In many cases, they weren’t that difficult to deal with at all. Rather they were tedious and required the parameterized types to be propagated through the depths of the interfaces to which they applied. It didn’t take long for me to recognize that I could wrap all of the complex definitions into parent-type, and reference my types much like the meta-functions I have been creating. The only difference is in this case I truly do want a simpler definition of the type.
Here is what the definition of a
Hg::BitSet with 8 fields that each used 4 bits would look like:
Because the type information is packaged in the convenient message type-list, all of the details required to reconstruct the
Hg::BitList definition is still available, even when packaged like this:
The type-list and index make the
Hg::BitList much more accessible, considering all of the other utility functions and definitions that have been created to support the fundamental types.
Integrating as a Datum
At first glance this does not appear to be a difficult issue to resolve. The primary issue is we need to make the
Hg::BitList compatible with the existing
Hg::Datum type. For a basic template, it would already meet that requirement. Through the use of static-polymorphism, a type only needs to contain the same member data and functions for all of the types to be compatible in an algorithm. They are not required to be tied together through inheritance, which is the basis of dynamic-polymorphism.
The difficulty I ran into, was the logic that I had created to programmatically navigate each field of a message were specialized for the
Hg::Datum type. To specialize on a second distinct type would cause ripples through the entire API, and cause almost complete duplicate implementations for alternative specialization. This seems to defeat the purpose of templates. So I searched for a solution to bring the two types together.
The solution I used for this first version was to create a class,
Hg::BitListDatum, which derived from both of these classes. Here is the declaration of the type:
The purpose of this class is to become a shallow adapter that is compatible with the definitions and types already defined for the
Hg::Datum. The only challenge that I ran into was creating a definition that would allow the
Hg::BitList type to be returned for the user to interact with, but still be a
Hg::Datum. This is what I created:
base_type operator properly handles the interactions where a reference to the
Hg::Datum type is required. In all other cases, the user can interact with the
value_type operator that returns access to the child
Hg::BitField objects by name.
After dynamically-sized arrays, the support of bit-fields has been the most tedious feature to implement in Alchemy. In fact, I do not think implementing support for arrays would have gone nearly as smoothly for me if I hadn’t learned the lessons from implementing bit-field support.
Bit-fields are a feature that are not used much. However, they tend to appear quite frequently when dealing with network protocols or data-storage formats. Once you consider that the bit-field feature is not well defined for C and C++, it becomes clear that an alternate solution for providing bit-level access to the data is important. Mask and shift code is tedious and error prone, not to mention difficult to read if you do not see it often. Considering that the simplification of software maintenance is one of the primary goals of Alchemy, providing a feature such as the
Hg::BitList is well worth the effort.
This is an entry for the continuing series of blog entries that documents the design and implementation process of a library. This library is called, Network Alchemy[^]. Alchemy performs data serialization and it is written in C++.
After I had proven to myself that serializing data with a template meta-program was feasible, I started to research what features would be required to make this a useful library. After the fundamental data types, I noticed that sub-byte access of bits appeared quite often in network packet formats, bit-fields. I have also worked in my share of code that implemented these packet formats using the bit-field feature in C/C++.
I decided support for accessing values at the bit-level needed to be a feature of Alchemy. I will explain why, as well as show my first attempt to make this work.
Bit-fields with C++
Bit-fields are like that questionable plate of left-overs in your fridge, yet you choose to eat it anyway. Personally, I avoid both.
In case you are not familiar with bit-fields, here is a brief example:
The compiler will automatically apply the appropriate mask to the data field to return the value of the named bit-field. This is very convenient. However, there is a catch; the original C specification left the implementation underneath undefined for the compiler implementer to decide. C++ then inherited the same rules. The reason this may be an issue is the bit-fields are not guaranteed to be packed in the order and location that we define them. Therefore, when they are serialized, they will not be correct according to your format specification.
Basically, bit-fields are not portable. Therefore, they should be avoided in code that is intended to be portable, or produces data that should be portable.
There is not much logic that is required to extract the correct bits defined in a structure similar to bit-fields. I had just finished an implementation of message fields that act like regular types. My plan was to attempt use the same approach with the
Hg::BitField, to allow the user to access each field with a specific variable name. If that did not work, my fallback plan was to simply convert the variable-like syntax to use a function instead.
As I mentioned earlier, the logic to mask bits out of an integer is just a few lines of code. Let's assume we have a pre-defined mask and we want to change the value of the bit the mask applies to. Here is the code:
To go in the opposite direction is a bit simpler with only one line of code:
Calculate Mask and Offset
Hopefully you have seen logic like this before. It may even have been tucked away in a MACRO or something. How do we calculate the mask?
- Take the number of bits in the value, and calculate: 2^n-1
- Determine the starting position of the mask in the value, this becomes the offset.
- Shift the value calculated in step one to the left by the offset.
At run-time we could calculate the mask on-the-fly with this code snippet:
Convert to compile-time
The only part of the expression above that currently requires run-time processing is the
std::pow() function. More than likely you have already seen a template meta-program implementation of this call, if not at least a recursive implementation. Here is the implementation that I use:
The mask expression
Now it is possible to define a constant at compile-time that represents the value of the mask. Two values will need to be passed into the definition of the
Hg::BitField template declaration, 1) the size of the field, 2) the offset.
This is the point where I need to front-load my explanation with an excuse because it appears so obvious to me now. However, in the experimentation phase, it is most important to even find a working solution. I subscribe to the mantra:
Make It Work, Make It Right, Make it Fast
This sage advice has been part of the Unix Way for many years, and more recently attributed to Kent Beck with respect to Test-Driven Development. This is the end of the excuse. Spoiler alert, I reworked this class when I reached the benchmark phase and witnessed it's abysmal performance. Hey, It Worked Right, there was no reason to revisit it until I discovered it was an issue.
The initial idea was to create a collector object called a
Hg::BitSet, and this would contain the collection of
Hg::BitField instances, and also store the actual data buffer. The bit-field objects would only contain a pointer to the value in the bit-set. Originally it was a reference, but I reached a limitation where the
Hg::BitField had to be default constructable, and thus the reference initialized to a default value.
I am trying to create slightly shorter entries so they can be digested more quickly, and I will be able to update more frequently. The unintended side-effect is that it has also allowed me to focus a bit better.
This entry described the intended purpose and design goals of the Hg::BitField class. In my next Alchemy entry I will show how I integrated the bit-field instance into its container. Then I will describe the hurdles that I started to encounter as my data types became more complex.
There is so much confusion surrounding the purpose of a Software Architect and the value they provide and what they are supposed to do. So much so, that it seems the title is being used less and less by companies and replaced with a different title such as principal or staff. I assume this is due to the perception there must be a way to distinguish a level above senior, which is handed out after only about three years of experience.
This has been a topic that I have wanted to discuss for quite some time. However, up until I read this question "Are there any Software Architects here?[^]" at CodeProject.com, I had been more compelled to focus on the more technical aspects of software development.
This is an excellent post that has multiple questions. If you were to take a quick peek at the question (it's ok, just come back), you would see about 80 responses in the discussion; with a wide variety of facts, opinions, and jokes related to software architecture and software development in general. Furthermore, many of the posts differ in opinion and even contradict each other.
I define the role of Software Architect in this entry. The definition is based upon my nearly two decades of experience developing software. Up to this point in my career, I have worked for small companies where I was the only software developer; all of the way up to companies where I was one of hundreds of developers and about one-thousand engineers of many engineering disciplines. Through the years I have performed just about every role found in the Software Development Lifecycle (SDLC) including a software architect.
The Basics of the Role
The Software Architect is not another stepping stone above Senior Software Engineer or Software Engineer V. They do work closely with people in these other roles, and in smaller companies they may also function as a Software Engineer V. However, the role of Software Architect is an entirely different position, and it has different responsibilities that are so much different than the task of writing software.
The software architect is a technical leader for the development team, and they are also the figurehead of the product for which they are responsible. One reason they become a figurehead for the product, is because the organization should make them responsible for the technical aspects of the product.
This makes them a good point of contact for information regarding the product, especially for personnel in the organization that may not be closely tied with the product. They may not always be the best person to answer a question, but they most certainly know who is.
As a leader, the architect is responsible for foster unity and trust among their team for success. The technical aspect requires the architect to mentor and guide the team along the way toward maintainable product with high quality. Moreover, they provide technical guidance and recommendations to the customer, which quite often is their own management team.
The architect adds value to the product and the organization by providing structure to both the design and implementation of the system, as well as the flow of information. If you are a fan of Fred Brooks and his book, The Mythical Man Month, you will recall that he states the implementation of a system tends to mimic the communication structure of your organization. In my experience, this has always proven to be true.
Improved Quality and Stability
Design by committee can work. However, you often need a group of like-minded individuals that do not differ in opinion too drastically for this succeed. In the extreme circumstances, if there are wildly divergent opinions, the group must be willing to forego their egos and accept the decisions that are made even when the decision is the polar opposite of what they believe to be the best decision. When all else fails, there is a final decision-maker at the top.
Thus enters the architect. They are after all, responsible for the product that is ultimately designed and built. They will help the team make decisions that are aligned with the macro-scale goals for the organization. While the development team themselves focus on the design decisions that are most beneficial at the micro-scale of the feature or system for which they are responsible.
A good flow of information is often required for harmony and success among a group. This can lead to other desirable benefits such as a high-level of morale and greater productivity. Effective communication is essential. Technical endeavors especially suffer when teams do not communicate effectively.
Development may move along steadily during the initial stages. However, if the integration of the components was not planned well in advance, progress will screech to a halt. Adjustments will need to be made for each component to be joined correctly. The architect can monitor the progress and direction for each developer or team, and provide guidance early on to correct course for the intended target.
The management team and key stake-holders communicate with the architect to remain in touch with the technical aspects of the system that require their attention. Such as clarifications on requirements, or missing resources that are necessary to continue to make progress on the project.
One problem that I see talented engineers struggle with, is to be able to adjust the language and content of their message to match their audience. Generally, they tend to be too technical and provide too many extraneous details for the important message to be properly conveyed. A good architect helps bridge this communication gap. They are able to effectively use the best style of communication to convey the details and properly persuade their audience.
At this point I would like to diverge and briefly and discuss the different roles that are required or exist for a software development project. Primarily because I often run across the question "What does an architect do that a developer doesn't?" This will also provide some context to refer to for the final section in which I will list desirable qualities you should look for in a software architect.
I mentioned there are many roles in software development. Each role is responsible for performing certain tasks. The problem is, the responsibilities for each role differ based on the organization. The size, culture and industry can affect how responsibilities are organized.
A company with a small staff may assign multiple tasks to its employees. To be able to complete all of the work in a timely fashion, the different roles may be responsible for overlapping tasks. While a large organization and development team may have many people that focus on performing a single task. All of the while, both companies may use the same titles to describe the roles of their employees.
It is important to know what your role and responsibilities are for any organization to be able to succeed at your own job; not to forget the success of the project and company as well. Therefore, I think it makes more sense to discuss the basic tasks that must be accomplished or commonly are included with a companies SDLC.
Here is a non-comprehensive list of the technical tasks that are required to develop software.
- Analysis: Determining what you want, need, or should build
- Requirements Gathering: Create a definition of what is to be built.
- Planning: Create the budget and the schedule. This task is not entirely technical, but the schedule and budget will be more accurate with technical guidance.
- Coordination: This is a bit abstract, however, it basically covers all communication and resource scheduling.
- Design: Create plans for the structure and operation of your product.
- Development: Build the product.
- Verification: Verify that it meets requirements, specifications and quality levels.
- Source Control Management: Organization of the resources required to build the product.
- Documentation: Useful or necessary information to the creators and users.
The only task listed above that is actually necessary is development. However, the process of development and the quality of the product suffers when the other tasks are omitted. Also, this same effect can occur when the level of effort is reduced for any of the other tasks.
How difficult is it to build a jigsaw puzzle when you do not even know what the final image is? It is possible. However, it is much more difficult without knowing what you are aiming for.
Let's complicate the example further. How much more difficult does it become to assemble the puzzle when:
- Extraneous pieces from another puzzle are added to your pile of loose pieces
- The number of pieces in the puzzle is not known
- No corner pieces exist
- No edge pieces exist
When you run in a three-legged race, the team that is most coordinated will have a great advantage. As the amount coordination increases, the less amount of time the team-members will spend fighting each other trying to find a rhythm.
The same concept holds true when you have more than a single person performing tasks on a project. These are the types of things that should be targeted through communication and coordination:
- Agreement between the groups for how their systems will interoperate.
- Scheduling project tasks to minimize conflicts and dependencies. If each task is dependent linearly upon another, it will not make sense to have more than one person on the project at a time.
- Good communication throughout the project helps spot potential issues and address them before they are too difficult to tackle.
- If the schedule requires your software team to develop at the same time the hardware is being developed, some alternate and temporary solution will be required for the developers. Virtual machines or reference hardware maybe acquired for software to use until the hardware is available. This is also an effective solution when there is a shortage of real hardware compared to the number of developers on your team.
Return to Roles
I will describe the set of tasks the software architect may perform for a small to moderately sized team. We will define this as about 20 people, with 8 or so are developers. The remainder of the team includes product managers, personnel management, marketing, sales, and quality control.
Analysis, Requirements. Planning and Documentation
The architect should be involved from the beginning. Once you know there may be a product to create, or a new engineering cycle on an existing project, bring the architect in. They can assist with the analysis, requirements gathering, and planning tasks. They can spend a fraction of their time acting as a consultant for these tasks. They can help advise and make sure the right questions get asked, the proper information is gathered, and only realistic promises are made. When it comes to creating documentation, the architect and development team should be accessible to answer questions for the technical writers.
As a leadership role, one of the primary responsibilities of the developer will be to communicate. More so listening than speaking. One of the most valuable things that an architect, or a software lead and manager for that matter, can do is to make sure their team understands what they are building. Then make sure each individual understands how their work will contribute to the final project.
I have seen projects go from extremely behind schedule, to finishing on time, after a new software lead was put into place. On one particular project, the developers had a vague idea what they were building, but on the surface it only seemed like another version of the five things they had built before. The new lead spent the first few days going over the project, its goals, what made it so cool, and verified each member of the team knew what they were responsible for. At that point, they moved forward with excitement and a new understanding. When some engineers finished their work, they jumped in and helped in other areas where possible. Do not underestimate the value of proper team coordination.
Design is definitely something that the software architect will be involved with. However, I think that most people start to misunderstand the responsibility of the architect when the task of design is mentioned. The perception that I see the most often is that the architect designs the entire system; and this can cause some angst among developers that believe this is the case.
The cause of any angst may be because the developers believe their control and creative freedom will be limited; hopefully that is not the case. I have a bit more to say about this later in the essay.
The architect should take responsibility for the overall structure of the program. High-level behavior, control-flow and technologies to be utilized will all be determined by the architect. Other software developers are given the responsibility to design smaller sections of the program at a higher level of detail. Their design should be guided by the structure the architect puts in place.
Any work performed up to this point is mostly focused on making this stage run smoothly. Hopefully all of the assumptions are correct, and the requirements are fixed. If not, the team must adapt. This is where the architect's responsibilities really become important. A foundation designed to be flexible with logically separated modules will have a better chance of adapting to any surprises that appear during development.
The architect is most likely to play more of a support role during this phase than a primary producer. That is because their time will be spent on guiding, mentoring and guiding the developers, as well as inspecting the results, and providing support where it will be most valuable. Moreover, they will keep the technical teams aligned with one and other, as well as the business and management teams informed of progress.
Furthermore, any surprises or new discoveries that appear along the way can also be adequately managed by the architect. On the other hand, any events occur that require the architects attention could cause further delays if their time is scheduled nearly 100% towards a development work items.
This is one phase where I believe the architect should have as little to do with this as possible. The reason being, you want an independently verified product. QA teams usually get the raw end of the deal when it comes to verifying and shipping software. If the programmers overrun on their schedule, QA is often tasked with finding creative ways of making the quality of the software better, faster.
If the architect is involved, the integrity of the verification process could be compromised. QA is a control gate to verify that things work correctly before they are released to the customer. The last thing you want is the architect and developers influencing QA's findings; making excuses for why something is not a bug, or at least why it should be classified as a level 4 superficial bug, rather than a level 2 major.
The architect should only coordinate with QA in order to get QA the resources they need to properly perform this task.
Source Control Management
There is so much that can be said about software SCM, but I am not going to. that is because it is a complicated task that deserves an entire essay of its own. The bottom line is the architect must be involved in the SCM for the product. It is crucial during development. It is damn near critical that SCM is handled according to a policy the architect defines (based upon the guidelines and strategies of the organization of course.)
Some products live a single existence, and slowly evolve with each release. Others spring into existence, and core components are reused to build other projects, which the process is then repeated ad infinitum. If there is no one managing the source code appropriately, or the way it is managed does not work for everyone, you may just get forked[^].
An Effective Software Architect
I was going to write a section titled Qualities. However, qualities are subjective, and you could probably guess the items on the list. It would look like every job ad for a software architect, and nearly the same for every software engineer. Just fill in the blank for the desired number of years of experience.
Because that list of qualities is so predictable and repeated everywhere, it would not add any more value to conveying the purpose and value of a software architect. Therefore, I thought it would be better to describe a few actions an Effective Software Architect would perform and the potential benefits that can be realized.
Focus on the Future
Jargon litters our lives. When job ads say "work in a fast-paced environment", that's a nice way of saying, "We want it right now!" No matter how quickly a piece of software can be developed, it still isn't quick enough. That is where many projects go awry; they are too focused on just for today. Tomorrow always becomes another today. Next thing you know you're programming in a swamp of code.
The software architect should focus on the long-term direction of a product, and execute towards the short-term goals of the business. It is good to have goals, they help provide direction. Every step forward may not lead directly to the goal. However, if the goal is always kept in mind when design decisions are made, the goal will become more attainable.
Shortcuts will be taken, and technical debt will accrue on the product. There never seems to be enough time to go back and correct all of those blemishes. However, the design can be made to mitigate short-term decisions and still provide a stable path towards the goals of the product.
Management and Business Development need to support the Software Architect by providing them with goals and information regarding the product. This will help the architect to develop a vision for the project and help guide it towards long-term goals; despite all of the rash and short-term decisions that are usually made in software development.
This is a more focused description of the coordination task above. This action is focused on building and maintaining a level of trust between all of the development team, as well as the management and other stake-holders that are involved with the project. As I mentioned earlier, the software product tends to mimic the communication style of the organization. Therefore a more unified team is more likely to develop a program that is unified in structure and quality. There is more to unity than the final structure of the program.
You may think that it is odd that I started an essay discussing computer programming, and I now I am on the topic of trust. For a team to work well together, they must trust each other. They must also trust their leaders that are directing them. The software architect is in a perfect position to help foster trust.
They are a technical leader, which means that they focus their attention on the technology and the way it is used. The role itself does not manage people. However, some organizations make the architect the manager of the developers as well. I believe this is a mistake because it throws away this opportunity to have a role to mediate situations of mistrust between development and management. This is especially true with respect to the technical aspect of the project.
The list below briefly describes how trust can be fostered with the other roles in the organization:
Listen to the development team. Incorporate their ideas into the design where they may fit, and be sure that they get credit for their ideas and work. When someone shares ideas, then implements them or sees someone else implement them and get the credit, they will tend to share less.
That's unfortunate, because the variety of experience and ideas that come from a collective group potentially provides more ideas to choose from when searching for a solution.
A software architect needs the trust of management to succeed for two simple reasons:
The managers control the purse strings of the business, and they decide where money is invested. If the trust of management is lost, they may be less likely to invest in your product. Developers may be peeled off and moved to another project that is deemed a higher priority. In the worst case, they decide they no longer want to fund you or your position.
The software architect is the steward of the product; they take all of the responsibility yet do not own it. In many cases, you will simply do the research, and present the facts, possibly providing a few options for management to choose from. However, for the topics that matter the most, persuasion of management to support your initiatives may be crucial. It is much easier to persuade someone when you are in good favor with them. Moreover, you may find yourself in a conflict with the development team, and without the support of management, you may lose that battle.
- Business Development and Marketing:
This group if interested parties becomes important to the architect if their software is a product that is owned and sold by the company. Having a good line of communication with the groups that drive the business's growth is extremely important. Information is the most valuable thing in our industry; a lack of information leaves us to speculate. It's better to in line with the initiatives set by the company, if for no other reason than you may spot problems before too many resources have been invested. Change becomes much more difficult at that point.
One other potential benefit of creating a strong relationship with these groups is they can gain a better grasp of what your product does. This is important because it allows them to consider the existing capabilities of your product when they are looking to convert a business opportunity into a sale. You flow information upstream, they flow information and potentially more sales back downstream.
The customer is the reason we write the software in the first place. The best way to earn trust with the customer is to listen to them. Sometimes it may seem like they don't know what they're talking about. However, it could simply be because you two are using different terms to mean the same things.
Therefore, it is important to clarify what you understand the customer is telling you. A quick way to lose their trust, as well as their confidence in you, is to ignore the advice or requests of the customer. If you can't do something or decide that you are not going to do it, have that discussion with them rather than avoid them.
Let's compare trust to technology, or a software codebase. You need to maintain relationships, otherwise that bond of trust starts to weaken. I am sure there already is a fancy term to describe this like trustfulness debt. If not, there you go. Open and transparent communication will help develop trust. Returning and visiting with your contacts periodically is another way to help maintain that bond. When trust is lost, it is very difficult to earn back.
Utilize the Teams Abilities
In a way, this is the Software Architect showing trust in their development team. People expect the Software Architect to be the smartest person on the team, or the most knowledgeable. That does not have to be the case, and usually it depends on the question or topic that you are referring to. Architects are good at seeing the Big Picture. In most cases I would expect there to be a number of developers with finer skills when referring to a deep domain topic, such as device drivers.
Say the architect is responsible for the development of an operating system kernel. There are many generalizations and design decisions that can be made to create structure for the kernel. Then a bit more thought would go towards a driver model and its implementation to simplify that task. When you reach the implementation for each particular type of driver, there are different nuances between file-system drivers, network drivers, and port I/O drivers. At some point, you will reach the limit of the architect's knowledge and expertise, and you will reach the realm of the domain expert.
Some engineers like to learn every minute detail about a topic, no matter how abstruse; that, is the domain expert. Typically they do not do well in the role of architect because they tend to get caught up in the low-level details when it is not necessary; think, Depth-first Search. Nonetheless this is the perfect candidate for the architect to trust and depend on when knowledge and advice is required on the expert's domain of specialty.
When the word architect is used, it is usually associated with design, and every programmer already does design (whether it is formal or on-the-fly is another discussion.) I think it would be best to help clarify the role of the position if effort was made to emphasize structure when discussing the software architect, with regards to both team and the software product. This may help disambiguate the purpose and value a software architect provides and distinguish it from the next step up after senior engineer.
There is much more to this role that dictating how the program should be built, which if you read the entire article you now know that isn't even one of the architect's responsibilities.
This post will focus on the concept of SFINAE, Substitution Failure Is Not An Error. This is a core concept that is one of the reasons templates are even possible. This concept is related exclusively to the processing of templates. It is referred to as SFINAE by the community, and this entry focuses on the two important aspects of SFINAE:
- Why it is crucial to the flexibility of C++ templates and the programming of generics
- How you can use it to your advantage
What the hell is it?
This is a term exclusively used in C++, which specifies that an invalid template parameter substitution itself is not an error. This is a situation specifically related to overload resolution of the considered candidates. Let me restate this without the official language jargon.
If there exists a collection of potential candidates for template substitution, even if a candidate is not a valid substitution, this will not trigger an error. It is simply eliminated from the list of potential candidates. However, if there are no candidates that can successfully meet the substitution criteria, then an error will be triggered.
SFINAE < Example >
I gave a vague description that somewhat resembled a statement in set theory. I also added a the Venn-diagram to hopefully add more clarity. However, there is nothing like seeing a demonstration in action to illustrate a vague concept. This concept is valid for class templates as well.
Below I have created a few overloaded function definitions. I have also created two template types that use the same name (overloaded), but have completely different structures. This example demonstrates the reason for the original rule:
The first case below, is the simpler example. It only requires, and accepts type where the value can be extracted implicitly from the type passed in; such as the intrinsic types, or types that provide a value conversion operator. More details have been annotated above each function.
Field type: 15 Field type: 20
SFINAE was added to the language to make
templates usable for fundamental purposes. It was envisioned that a string class may want to overload the
operator+ function or something similar for an unordered collection object. However, it did not take long for the programmers to discover some hidden powers.
The power that was unlocked by SFINAE was the ability to programmatically determine the type of an object, and force a particular overload based on the type in use. This means that a template implementation is capable of querying for information about it's type and qualifiers at compile-time. This is similar to the feature that many languages have called reflection. Although, reflection occurs at run-time (and also incurs a run-time penalty).
I am not aware of a name for this static-form of reflection. If there is could someone comment and le me know what it is called. If it hasn't been name I think it should be something similar to reflection, but it is still a separate concept.
When I think of static, I think of "fixed-in-place" or not moving. Meditation would fit quite well, it's just not that cool. Very similar to that is ponder. I thought about using introspection, but that is just a more pretentious form of reflection.
Then it hit me. Rumination! That would be perfect. It's a verb that means to meditate or muse; ponder. There is also a secondary meaning for ruminate: To chew the cud; much like the compiler does. Regardless, it's always fun to create new buzzwords. Remember, Rumination.
I make heavy use of SFINAE in my implementation of Network Alchemy. Mostly the features provided by the < type_traits > header. The construct
std::enable_if is built upon the concept of SFINAE. I am ashamed to admit, that I have not been able to understand and successfully apply
std::enable_if yet. I have crossed many situations that it seemed like it would be an elegant fit. When I figure it out, I will be sure to distill what I learn, and explain it so you can understand it too. (I understand enable_if[^] now.)
Useful applications of SFINAE
To read a book, an article or blog entry and find something genuinely new and useful that I have an immediate need for is fantastic. I find it extremely irritating when there is not enough effort put into the examples that usually accompany the brief explanation. This makes the information in the article nearly useless. This is even more irritating if the examples are less complicated than what I could create with my limited understanding of the topic to begin with.
A situation is extremely frustrating when I believe that I have found a good solution, yet I cannot articulate the idea to apply it. So unless you get extremely frustrated by useful examples applied to real-world problems, I hope these next few sections excite you.
We will create a meta-function that can make a decision based on the type of T. To start we will need to introduce the basis on which the idiom is built. A scenario is required where there are a set of choices, and only one of the choices is valid. Let's start with the sample, and continue to build until we reach a solution.
We will need two different types that are of a different size
We will also need two component that are common in meta-programming:
- static-member constanst
We define a meta-function template, that will setup a test between the two types using the
size of operator to determine which type was selected. This will give us the ability to make a binary decision in the meta-function.
We started with static declarations of the two types that I defined earlier. However, there is no defined conditional test for the
yes_t template, yet. It is also important to understand that the template parameter name must be something different than the name used in the templates outer parameter. Otherwise the template parameter for the object would be used and SFINAE would not apply.
The lowest type in the order of precedence for C++ is
.... At first glance this looks odd. However, think of it as the catch all type. If the conditional statement for
yes_t produces an invalid type definition, the
no_t type will be used for the declaration of the
It is important to note that it is not necessary to define the function implementations for
selector because they will never actually be executed. Therefore, it is not required by the linker. We also use an arbitrary function,
selector, that returns
T, rather than a function that invokes
T may not have a default constructor.
It is also possible to declare the
selector function to take a pointer to
T. However, a pointer type will allow
void to become valid as a
void*. Also, any type of reference will trigger an error because pointers to references are illegal. This is one area where there is no single best way to declare the types. You may need to add other specializations to cover any corner cases. Keep these alternatives in mind if you receive compiler warnings with the form I presented above.
You were just presented a few facts, a bit of code, and another random mix of facts. Let's tie all of this information together to help you understand how it works.
- SFINAE will not allow a template substitution error halt the compiling process
- Inside of the meta-function we have created to specializations that accept
- We have selected type definitions that will help us determine if a condition is true based upon the type. (An example condition will be shown next).
- We also added a catch-all declaration for the types that do not meet the conditional criteria (...)
- The stub function
this_t()has been created to be used in a
sizeofexpression compares the two worker types to the
no_ttype to determine the result of our conditional.
The next section contains a concrete example conditional that is based on the type
The classification of a type can be determined, such as differentiating between a Plain-old data (POD) struct and struct with classes. Determine if a type is const or volatile, if it's an lvalue, pointer or reference. Let me demonstrate how to tell if a type is a class type or not. Class types are the compound data structures, class, struct, and union.
What we need in the conditional template parameter is something that can differentiate these types from any other type. These types are the only types that it is legal to make a pointer to a scope operator
:: operator resolves to the global namespace.
Here is the definition of this template meta-function:
Separate classes based on member declarations
Sometimes it becomes beneficial to determine if an object supports a certain function before you attempt to use that feature. An example would be the
find() member function that is part of the associative containers in the C++ Standard Library. This is because it is recommended that you should prefer to call the member function of a container over the generic algorithm in the library.
Let's first present an example, then I'll demonstrate how you can take advantage and apply this call:
Applying the meta-function
The call to
std::find() is very generic, however, it can be inconvenient. Also, imagine we want to build a generic function ourselves that will allow any type of container to be used. We could encapsulate the
std::find() call itself in a more convenient usage. Then build a single version of the generic function, as opposed to creating specializations of the implementation.
This type of approach will allow us to encapsulate the pain-point in our function that would cause the implementation of a specialization for each type our generic function is intended to support.
We will need to create one instance of our meta-function for each branch that exist in the final chain of calls. However, once this is done, the same meta-function can be combined in any number of generic ways to build bigger and more complex expressions.
This is a very simple function. In my experience, the small, generic, and cohesive functions and objects are the ones that are most likely to be reused. With this function, we can now use it in a more specific context, which should still remain generic for any type of
T::find() called std::find() called
We made it possible to create a more complex generic function with the creation of the small helper function ,
CotD::find(). The resulting
CotD::generic_call is agnostic to the type of container that is passed to it.
This allowed us to avoid the duplication of code for the larger function,
CotD::generic_call, due to template specializations.
There is also a great chance that the helper template will be optimized by the compiler to eliminate the unreachable branch due to the types being pre-determined and fixed when the template is instantiated.
Substitution Failure Is Not An Error (SFINAE), is a subtle addition that was added to C++ make the overload resolution possible with templates. Just like the other basic functions and classes. However, this subtle compiler feature opens the doors of possibility for many applications in generic C++ programming.
It seems that every developer has their own way of doing things. I know I have my own methodologies, and some probably are not the simplest or the best (that I am aware of). I have continued to refine me design, development, test and software support skills through my career.
I recognize that everyone has their own experiences, so I usually do not question or try to change someone else's process. I will attempt to suggest if I think it might help. However, sometimes I just have to ask, "are you sure you know what you are doing?" For this entry I want to focus on unit testing, specifically with Mock Objects.
What do you mean by "Mock"?
I want to seriously focus on a type of unit-test technique, one that is so misused, that I would even go so far as to call it an anti-technique. This is the use of Mock Objects.
Mock objects and functions can fill in an important gap when a unit-test is attempting to eliminate dependencies or to avoid the use of an expensive resource. Many mock libraries make it pretty damn simple to mock the behavior of your codes dependencies. This is especially true when the library is integrated into your development environment and will generate much of the code for you.
There are other approaches that exist, which I believe are a better first choice. Whatever method you ultimately chose should depend on the goal of your testing. I would like to ruminate on some of my experiences with Mock Objects as well as provide some alternate possibilities for clipping dependencies for unit tests.
Mock Objects are only a small element of the larger topic of unit-testing. Therefore, I think it's prudent to provide a brief overview of unit testing to try to set the context of this discussion, as well as align our understanding. Unit testing is a small isolated test written by the programmer of a code unit, which I will universally refer to as a system.
It is very important to try to isolate your System Under Test (SUT) from as many dependencies as possible. This will help you differentiate between problems that are caused by your code and those caused by its dependencies. In the book xUnit Patterns, Gerrard Meszaros, introduces the concept of a Test Double used to stand-in for these dependencies. I have seen many different names used to describe test doubles, such as dummy, fake, mock, and stub. I think that it is important to clarify some vocabulary before we continue.
The best definitions that I have found and use today come from this excellent blog entry by, Martin Fowler, Mocks Aren't Stubs. Martin defines a set of terms that I will use to differentiate the individual types of test doubles.
- Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
- Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
- Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
- Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive
Martin's blog entry above is also an excellent source for learning a deeper understanding between the two general types of testing that I will talk about next.
So a Mock is just a different type of test double?!
No, not really.
Besides replacing a dependency, mock objects add assertions to their implementation. This allows a test to report if a function was called, if a set of functions were called in order, or even if a function was call that should not have been called. Compare this to simple fake objects, and now fake objects look like cheap knock-offs (as opposed to high-end knock-offs). The result becomes a form of behavioral verification with the addition of these assertions.
Mock objects can be a very valuable tool for use with software unit tests. Many unit test frameworks now also include or support some form of mocking framework as well. These frameworks simplify the creation of mock objects for your unit tests. A few frameworks that I am aware of are easyMock and jMock for JAVA, nMock with .Net and GoogleMock if you use GoogleTest to verify C++ code.
Mock objects verify the behavior of software. For a particular test you may expect your object to be called twice and you can specify the values that are returned for each call. Expectations can be set within the description of your mock declaration, and if those expectations are violated, the mock will trigger an error with the framework. The expected behavior is specified directly in the definition of the mock object. This in turn will most likely dictate how the actual system must be implemented in order to pass the test. Here is a simple example in which a member function of an object registers the object for a series of callbacks:
Clearly the only way to validate in this function is through verifying its behavior. There is no return value, so that cannot be verified. Therefore, a mock object will be required to verify the register function. With the syntax of a Mock framework, this will be a snap, because we add the assertion right in the declaration of the Mock for the test, and that's it!
Ultimately the function
object::Register() is interested to know if the two proper callbacks were registered with TheEmporium. So if you nodded your head in agreement in the previous section when I said "Clearly the only way...", I suggest you stop after you read sentences that read like that and challenge the author's statement. Most certainly there are other ways to verify, and here is one of them.
1 point if you paused after that trick sentence, 2 points if you are reserving judgment for evidence to back up my statement.
It would still be best if we have a stand-in object to replace TheEmporium. However, If there is some way for use to verify after the SUT call, that the expected callback functions were registered in the correct parameters of TheEmporium, then we do not need a mock object. We have verified the final data of the system was as expected, not that the program executed to a prescribed behavior.
Why does it matter?
Tight Coupling between the test and the implementation.
Suppose you wrote your mock object to verify the code in this way:
That will test the function as-is currently implemented. However, if the implementation of
Object::Register were implemented like anyone of these, the test may report a failure, even though the intended and correct results were achieved by the SUT.
All four of these implementations would have continued to remain valid for the data validation form of the test. Because the correct results were assigned to the proper values at the return of the SUT.
When the Mock Object Patronizes You
Irony. Don't you love it?!
Mock objects can get you very far successfully. In fact, you may get towards the very end of your development phase and you have unit tests around every object and function. You are wrapping up your component integration phase. Things are not working as expected. These are some that I have personally observed:
- The compiler complains about a missing definition
- The linker (for the C and C++ folks) complains about undefined symbols have been referenced
- This is a network application, everything compiles and links, the program loads and doesn't crash. It doesn't do anything else either. You connect a debugger, it is not sending traffic.
I have seen developers become so enthusiastic with how simple mock objects made developing tests for them, that they virtually created an entire mock implementation. When it compiled and was executed, critical core components had a minimal or empty implementation. All of the critical logic was complete and verified. However, the glue that binds the application together, the utility classes, had not been implemented. They remained stubs.
There are many ways to solve problems. Each type of solution provides value in its own way. Some are simple, others elegant, while others sit and spin in a tight loop to use up extra processing cycles because the program doesn't work properly on faster CPUs. Just be aware of what value you are gaining from the approaches that you take towards your solution. If it's the only way you know, that is a good first step. Remember to keep looking, because their may be better solutions in some way that you value more.
I found the metaphorical Silver Bullet that everyone has been searching for in software development and it worked beautifully on my last project. Unfortunately, I only had one of them. I am pretty sure that I could create another one if I ever have to work with a beast that is similar to my last project. However, I don't think that my bullet would be as effective if the circumstances surrounding the project varies too much from my original one.
No Silver Bullet
To my knowledge, the source of this term is from. Fred Brooks, the author of The Mythical Man Month. He wrote a paper in 1986 called No Silver Bullet -- Essence and Accident in Software Engineering. Brooks posits:
There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.
The concepts and thoughts that I present in this entry are deeply inspired by his thoughts and assertions that he states in his paper.
There have been a number of studies and papers that challenge the assertions that Brooks presents in this paper. To this day, the only way that programmers have increased their productivity by an order of magnitude has been by moving to a higher-level programming language. This comparison is done by comparing the productivity between programmers developing in assembly and C. Then comparing C to the next level, such as C# or JAVA, and then on to scripting languages.
The Silver Bullet
I feel obligated to let you know where the term Silver Bullet originates. The mythical creature, the werewolf, is fabled to only be able to be killed by silver weapons. The modern day equivalent is a bullet made from silver. Depending on which mythology, author, video game or table-top gaming platform you are most fond of, other types of weapons may harm the creature, but only temporarily. Unless it is made from pure silver, it will not completely kill the creature.
For the uninitiated
I feel obligated to interpret this metaphor, which I think is a great metaphor. Your project is the werewolf. For decades, millions of programmers and project managers have been searching for a way to kill, reliably execute the development of a software project. Software development has been notoriously plagued by cost and schedule overruns due to the poor ability of estimating the costs required to develop a project. The silver bullet is that special tool, process, programming language, programmer food or sensory deprivation chamber that tames this beast and allows a project to be developed with some increased form of reliable predictability.
I want to take a scenic detour from the primary topic of this post, to perform an exercise that may help demonstrate my point.
Every since I started programming I have heard debates and also wondered myself, "Is computer programming a science or an art?" I have never heard a convincing argument or definition that I could accept. Each year I find myself further away from what I believe to be a valid definition because I have stumbled upon many other titles that could classify where a computer programmer fits in the caste of our workforce. This is a non-exhaustive list of other labels that seem to have fit at one time or another:
- Form of Engineering
- Linguistic Translator
- Competitive Sport (both formally and informally)
- Modern day form of a Charlatan / Snake-oil salesman / Witch-doctor
- A way to get paid for working with my hobby
To create a precise definition for the practice of computer programming, is much like trying to nail JELLO to the wall. No matter where you go, it will be practiced differently. The culture will be different. The management will be different. The process will be different. The favored framework will be different. Experience can teach us, mold us, and jade us just to name a few ways in which we can change. I have collected a list of my experiences that span my career in an attempt to classify a computer programmer.
I can only speak from my personal research and experiences, and both of those have mostly been focused on the realm of software development. Here is a broad sampling of my experiences related to software development.
I have read scores of books that span many aspects of software development, such as...
- the programming languages du jour
- ...the programming processes du jour
- ...the programming styles du jour
- ...C++, lots of books related to C++
- ...books on the sociology of software development
- ...books only tangentially related to software development because they were right next to the software development section in the bookstore
I have worked for a variety of companies, which...
- ...had many intelligent people
- ...had people of much less intelligence
- ...had enthusiastic learners
- ...had people with passion for technology
- ...had people I always wondered how they passed the stringent hiring process
- ...had bosses with a variety of management styles
- ...aimed for completely different goals
- ...valued results that ranged from the next quarter to ten years from now
Over my career I have also...
- ...learned many things from colleagues
- ...spent time unlearning some things
- ...helped others understand and become better developers,
- ...written a lot of code
- ...learned that that amount of code I have written is too much, there are better ways
- ...deleted a lot of code (after my thumbs, my right pinky is the digit on my hand that I value the most)
- ...lost a lot of code due to power outages, workstation upgrades by IT, and I suspect from BigFoot
- ...repeated myself far too much
- ...gathered many requirements
- ...communicated and miscommunicated with many people
- ...rewritten someone else's code because my way is cleaner
- ...had my cleaner code rewritten back to its original form
- ...had many fantastic ideas
- ...created a fewer number of those fantastic designs
- ...implemented even fewer programs based on those designs
- ...implemented programs based on other peoples designs
- ...criticized the poor quality of untold amounts of code
- ...was humbled when I discovered I wrote some of that code
- ...was shocked when I looked at old code of mine and literally said out loud "This is my code?! When did I learn how to do that, and why don't I remember, because that's something I've always wanted to learn how to do?!"
- ...became too emotionally attached to projects (yes, multiple times)
- ...reached a point where I lost all motivation on a project and it was excruciating to watch how slowly the LEDs would take to blink 3600 (or if you prefer 0xE10) times each hour
- ...learned how to articulate most of my ideas
- ...become a better listener
- ...learned that no matter how similar a situation is, my next experience will always be different
To summarize my experiences
I have worked for a half-dozen companies in a variety of roles. Although there have been similarities, each of these companies still had completely unique talent level, cultures, goals, company values, management styles, reward systems and more. These factors played into how well the teams worked together, the quality of products the company produced, how pleased the customers were and in the end this affected how they defined the role of programmer or software engineer.
There is a very strong observation that I have silently mulled over for a few years now. The digital world has turned the world we know upon its head. In true digital form, there are
10 types of people in this world, those that can grasp the abstract concepts encoded in digital technology, and those that can't.
It seems that most people develop some sort of intuition when it comes to the physical world. You can sense with possibly all five of your senses qualities about any physical object. Will it be soft and cuddly, heavy and smooth, squishy and sticky? Those of you old enough to remember TVs when they were large, heavy and thick; you know Cathode Ray Tube (CRT) technology. For some reason, we have this instinct to whack the TV on the side. One or two of those usually did the trick. Why? The mechanical components like the vacuum tubes were becoming unseated. A firm jolt let them settle correctly back into place.
Now consider a digital circuit. To all but experts even, it is not possible to tell if that circuit card is running properly. I had a flat-screen TV give out, and before I decided to throw it out, thought I would look online to see if there was a solution, and there was. It turned out to be cheap capacitors going bad on the power board. I opened the TV up to replace the bad capacitors, and luckily for me that is all that it was. I would have had no idea if something had gone wrong on the IC with more complex components. No amount of whacking on the side of a digital TV or computer monitor will resolve the issue.
The ability to grasp the abstract digital concepts is a valuable talent. Other engineering disciplines such as electrical, chemical, astro-physics are like magic to most people not in those fields. However, there is one sound basis on which they all are based upon, and that is physics. These fields are based on the laws and models that human-kind developed to approximate the best we can understand about our physical world.
What is computing based upon? The physics that allow the electrical circuits to switch at lightening speed in a defined pattern to create some effect, generally a calculation. It's time to consider how we define these patterns to compute.
Computer programming is an activity that articulates an amorphous abstract thought, in as precise of a manner as to be interpreted and executed by a computer. We are converting this thought pulsing in our minds into a language of sorts, to communicate with a digital creature. That is fascinating.
What is troubling, is that 8 developers (we'll go back to base 10) can be sitting in a room, listening to a set of requirements. Then independently recreate that list of requirements. Quite often the result is 9 distinct lists of requirements (remember the original requirements). If these 8 developers went off to converse with their digital pet, each of them will create very similar programs and results, yet they will all differ because of the programmers interpretation. That is not even considering programmers that do not quite understand all of the nuances of the language with which they are commanding their computer.
What was the question again?
What is Software Engineering?
It's mostly an engineering discipline and also has a strong foundation in science.
It can be a form of art, but mostly only appreciated by other programmers that understand the elegance and beauty of what the program does.
I think it is definitely a trade or a craft. The more you practice, the better you become and natural talent can sure help as well.
It was only recently that I considered the communication/linguistic/translation concept. Especially when abstract thoughts and concepts are factored into how those ideas are translated. Math is very similar in its abstract concepts and models that we have created. However, math is also much more well defined than computer programming.
To me, programming is very much like writing an essay on a book only using mathematical notations.
The digital nature and complexity of computer programs allow us to become charlatans. It's possible to tell your managers and customers that your program does the right thing, even though it's only good enough to appear to do the right thing. If they find the bug, we'll create a patch for that; maybe.
Software Engineering is many things, most of them great. However, it is still a relatively new discipline humans are attempting to master. It does not work very well to compare this profession to other engineering professions, because we are not bound by the laws of physics. Human creativity and stupidity are the forces that limit what can be done with computers.
Back to the Silver Bullet
I think we should continue to develop new tools, processes and energy drinks that will help developers write better code. I also think that communication is an aspect that really needs to be explored to further solidify the definition of this profession.
In order to improve our processes, everyone that has a stake in the project must consider the differences between the next project, and the previous project. A team that works well and communicates effectively at a company that is doing well and has a great culture, will out perform that exact same team at a company with layoffs looming in the near future. It would be interesting to take the great team working at the great company and scale the project and personnel size up by 5 times.
What will still be good?
What could go wrong?
What will need to change for the new team to succeed on this project?
Success through feedback
I think it's possible for the new team to succeed. However, they have to consider the differences of their new project compared to their past ones. They can't expect processes that worked well for a team of four, to work equally as well with a team of 20 without making adjustments. When the work is underway, the team will need to be observant and use the new information they receive to adjust their processes to ensure success.
There is no Silver Bullet. Well, maybe one or two. But not every project is a werewolf. If you run into BigFoot, don't waste your silver bullet, just protect your data.