Cloud Engineering

Apr 03 2020

Creating Robust Test Automation for Microservices

Any and all projects that a software engineer joins will come in one of two forms: greenfield or legacy codebases. In the majority of cases, projects will fall into the realm of the legacy repositories. As a software engineer, it is our responsibility to be able to strategically navigate our way through either type of project by looking objectively at the opportunities to improve the code base, lower the cognitive load for software engineering and make a determination to advise on better design strategies. But, chances are, there is a problem. Before architecture or design refactors can be taken its best to take a pulse on the health of your platform End to End (E2E). The reason being, lurking in a new or existing platform is likely a common ailment of a modern microservices approach – the inability to test the platform E2E across microservices that are, by design, commonly engineered by different teams over time.

I recently worked on a greenfield platform and nearly 6 months into the project, one of the most critical parts of the project had fallen several months behind from a quality assurance perspective. It was no longer possible for QA to catch up, nor was it possible for QA to engineer and execute E2E testing to complete common user journeys throughout the enterprise system. To solve this conundrum, E2E data generation tools needed to be created so that the QA team could keep upbuilding and testing every scenario and edge case. As I see it, there are three main requirements for an E2E account and data generation tool. The tool should:

1) Create test accounts with mock data for each microservice

2) Link those accounts between up and downs stream microservices

3) Provide easy to access APIs that are self-documenting

Using a tool like Swagger, QA can use the API description for REST API, i.e. OpenAPI Specification (formerly Swagger Specification) to view the available endpoints and operations to create accounts, generate test data, authenticate, authorize and “connect the microservices”.

By creating tools for E2E testing, our QA team was able to eliminate the hassle of trying to figure out which upstream and downstream microservices needed to be called to ensure that the required accounts and data were available and set up properly to ensure a successful test of all scenarios i.e. based upon the variety of different data types, user permissions, user information and also covering the negative test cases. The QA team was able to catch up and write their entire suite of test scenarios generating the matching accounts and data to satisfy those requirements. The net result of having built an E2E test generation tool was automated tests could be produced exponentially quicker and the tests themselves are more resilient to failure.

Even though the microservices pattern continues to gain traction, developing E2E testing tools that generate accounts and test data across an enterprise platform will likely still remain a pain point. That having been said, the purpose of this article is to raise awareness. Software Engineering Leadership has an opportunity to step up and ensure that the enterprise system that they look after is healthy E2E. I can’t think of a better way to maintain a healthy system than to ensure accounts and data in the lower environments actually work and unblock testing end-to-end.

James Woods, Web Engineering Practice Lead

Feb 04 2020

Letting Go of the Formality

It wasn’t that long ago that suits, ties, knee-length skirts, heels and pantyhose were required wardrobe staples. In fact, many enterprise Fortune 100s upheld a formal dress code as part of their HR Policy Handbooks. Goldman Sachs, the last bastion of formality on Wall Street, finally relaxed its dress code and offered a “flexible style” option. They only did so last year. Their updated policy permitted men to choose garments other than suits. Admittedly, this made things more complicated for women but that’s a topic for another blog.

Since we’re agile practitioners here at InRhythm, it made sense to look at our own approaches to talent and acquisition. Not to mention retention. Practices and policies drive corporate culture. Here, Agile Platinum Principle One: resist formality, must serve as one of the pillars of our talent strategy.

If your company doesn’t offer a culture that’s aligned with the needs and expectations of today’s job seekers, then, as Hiring Manager or Chief People Officer, you’re going to have a really hard time. At best, you’re going to struggle and slog through your days and recruiting campaigns. At worst, you’re not going to have any people to serve as Chief over!

The link between informality and tech

Ask anyone within the technology field, or external to it, to describe or draw a person in tech, be it a software developer or a startup founder. The universal response is a male, typically Caucasian, dressed in sloppy jeans and a hoodie. Where did this level of informality originate?

Many would cite Lou Gerstner’s (former CEO of IBM) decision to roll out “casual dress.” Even the New York Times balked with a headline that read, “Black Jeans Invade Big Blue.” Shortly thereafter, Hewlett Packard (HP) introduced “Blue Sky Days” which permitted employees to wear jeans to work on Fridays. From there, dress code devolved or evolved – it depends on your perspective – into what we see today.

However, the Platinum Principal of resisting formality is not restricted to dress code. It extends into everything that we do. Formality also has implications with unconscious bias, how we greet and interact with people, the forms that we require candidates to complete and so on. Tone and style go hand in hand.

How are we resisting formality?

We are constantly taking steps to increase our awareness and connection with our prospective employees, candidates, contractors, staff and employees. Efforts are made to understand what they need so that we can craft our talent, acquisition and retention strategies around their needs and expectations versus requiring that they conform to ours. Here, we have embraced an agile culture across the company, not just in our product development efforts.

As agile craftsmen, with respect to HR recruiting and retention strategies, we have adopted an approach that is more responsive than it is prescriptive. Our applications are deliberately designed with brevity in mind to expedite the hiring process. We measure how long it takes from the time that an application is submitted to an open job posting until someone is seated in the role. A conscientious effort, with monitoring, is made to ensure that we are constantly moving our candidates through the hiring process as swiftly as they seek to do so.

Another way that may come as a bit of a surprise is the effort we make to create a relaxed culture that allows for flexibility and downtime. We understand the enormous pressure and stress that our employees feel as they do all that they can to deliver on our clients’ needs. It’s okay to push through a couple of scrums or back-to-back sprints but it’s not sustainable. Over the holidays, we relaxed with an in-house office party and gave people some bonus time. This week is Valentine’s Day: celebrate it or not, you don’t need to make it about sending flowers and chocolates to your partner. Use it as a reminder to resist formality, take some time and reach out to whomever it is that you care about, be it a parent or sibling or a friend.

Resist formality! Make strides to emulate the people and culture around you. Forcing people into a process defined by its rigor and inflexibility is the antithesis of what a company should do. Particularly a company who’s purpose is to bring agile methodologies and success to our clients.

Stu Weiser,

Director of Talent Acquisition
Image created by katemangostar – www.freepik.com

Jan 18 2020

Configuration Automation Tools: Orchestrating Your Deployment

“We must accept human error as inevitable – and design around that fact.” – Donald Berwick

In our Technology field, buzz words come and go. One day we’re discussing how databases are the new best thing to the world of Agile Development and the next we see programming languages, frameworks, and methodologies bit the dust like an old western movie.

But one unchanging aspect of technology and its journey is the People who are an irreplaceable part of the creation, the demise, and the popularity of any given technology. This modern-day-world calls for a close to perfection execution, which humans cannot always account for. How does that affect us, the developers and creators, when we are called to building perfect products?

Removing Human Error

Automation automation automation – and of course throw in some orchestration deployment and configuration management. Leaving the buzz words behind the jest of all of this, “new technology frontier”, is removing human error. This is removing the dependencies of tribal knowledge when it pertains to application and system administration job duties. Those job duties are performed in a repetitive fashion. The job duties are usually consolidated into various custom scripts, leaving a lot of those scripted actions the ability to be boxed up and reused over and over again.

For example, what steps does it take to deploy an application to a server instance?

Download and Install the various languages and/or framework libraries the application usages.
Download, Install, Configure the Web server that the application will use.
Download, Install, and Configure the Database that the application will use.
And finally Test to see if all the steps are installed and configured correctly.

Last but not least run application tests to make sure it is running as expected. Those are the highlight steps for deploying an application to a bare metal, cloud, or virtual server. With a further review, there are numerous upon numerous additional steps. Something simplistic nevertheless highly catastrophic is a possibility of a typo. Consider this case, maybe in this scenario, I meant to type

cd /var/ect/ansible; rm -rf *

but instead I forgot the cd execute command and only ran

rm -rf /

Now my whole drive is erased. Surprisingly enough this can happen quite often even though I am pretty sure the source of truth of my drive is backed up. However, as Murphy Law states, what can happen will happen. And let’s be honest, redoing previous actions that could have been avoided is a down right pain.

No Hidden/Tribal Knowledge

Let’s look back at those steps to deploy an application to an environment. There are a lot of gotchas and/or small intermediary steps involved. For instance, I only listed the high-level steps. I hand these steps to a new engineer or even a junior engineer fresh out of college. The chances are these engineers will have to ask several questions on “How to”.

These series of questions will eventually lead to someone like myself saying “just let me do it”. This is how one scenario can lead to years upon years of hidden or tribal knowledge, as one may call it. To harbor information like a fugitive on the run is getting more and more exposed in the IT/Software industry. The information should be a source of truth maintained in a repository database, that is easy and intuitive to leverage

Implementation Technology Overview

What does that source of truth entail? Can we not skip the documentation of information and go straight into the execution of the steps onto a given system? Or create scripts to reconfigure the application if there was ever a need to? Those questions have been proposed several times and solutions have been formulated several times into the form of extensive and comprehensive build tools/frameworks. These tools are used throughout the industry to solve the problem of orchestrated development, configuration automation and management.

Furthermore, DevOps tools such as: Chef, Puppet, and Ansible are well mature automation/orchestration tools. Each tool will provide you with enough architecture flexibility to virtually handle any use case you present. Let’s take a slight dive into each one of these technologies.

Puppet

Puppet is the first widely used Configuration Automation and Orchestration software dating back to its initial release in 2005. Puppet uses the Master and Slave paradigm to control X amount of machines. The Ruby language is the script language say, for executing commands in a destination environment.

The “Puppet Agents”(Slave) are modularized distinct components to be deployed to a server. This can be used for the creation of the server(ie. web server, database, application) in its destination environment. The Puppet enterprise (Master) is comprised of all the inner workings to manage, secure, and organize agents.

Puppet Architecture

Documentation

Chef

Chef is somewhat similar to Puppet. The core language used within Chef’s abstract module components is Ruby. Chef has several layers of management for your infrastructure automation needs. The Chef workstation is the primary area for managing the various Chef components. The Chef components consist of cookbooks, recipes, and nodes. The naming convention actually follows what you would consider these nouns in cooking to be with technical context of course.

Recipes are collections of configurations for a given system, virtual, bare metal, or cloud environment. Chef calls those different environments nodes. Cookbook contains recipes and other configurations for application deployment and control mechanisms for the different Chef clients. Lastly, the whole collections of cookbooks are maintained in a Supermarket.

Chef Architecture

Documentation

Ansible

Ansible is the newest mainstream automation/configuration management tool on the market. Therefore, Ansible uses more modern programming languages and configurations concepts and tools. Python is the programming language used in this framework. One of the modern and fastest up-and-coming template languages is YAML. YAML is programming language agnostic and is a subset of the ever so popular JSON. YAML is used within Ansible to describe an Ansible playbook.

Ansible playbook contains the steps that need to be executed on a given system. Once the Ansible playbook is intact configuration or further manipulation of the host can be executed through Ansible API – which is implemented in Python. There are several other components within Ansible technology such as modules, plugins, and inventory.

Ansible Architecture

Documentation

Reusability

After covering a couple of the Configuration Automation and Development tools on the market you can see the vast amount of flexibility available in eliminating those repeatable steps from human error. This software’s framework promotes reusable software within an organization – which is the most viable. The ability to scale your application development environment and environmental infrastructure is critical.

The learning curve may be deeper than using plain old bash scripts but the structure and integrity of using a proven tool and ease of maintenance outweigh the learning curve.

Check it for yourself, reorganize and propel your CI/CD process with Configuration Automation and Orchestrated Deployments and let me know what you think about this article in the comments below.

Raphael Weaver

Sr. Software Engineer

Nov 11 2019

Practice What You Preach

As agile practitioners and thought leaders in software engineering, when we work onsite at our clients’ offices, it is important that we deliver our best. After all, as consultants, we are viewed as experts brought in with the expectation that we instrument and equip them with high velocity teams. In this regard, as we heard from our CEO, Gunjan Doshi, in last weeks Learning & Growth Newsletter, we are agile craftsmen. Our challenge is how we can balance learning on-the-fly while integrated onsite with our clients and managing their project expectations.

It is one thing to operate as a living lab when you are shielded inside the walls of your company’s office. With the right culture, mistakes will be expected and valued. When they result in a learning opportunity they ultimately drive a new insight into efficiency and judgement, however, when you are working onsite with a client, it can be an entirely different experience. Mistakes can result in injured productivity or be viewed as an inhibitor to progress and improving velocity.

Clients understand that software development is partly intuitive as much as it is scientific, and even masters of the craft will occasionally misjudge. Agile methodologies instill the mindset that we constantly iterate upon and improve our solution as well as our process. This not only holds true in what we craft, but also from within ourselves. As agile craftsmen it is our responsibility to analyze our own efforts, understand and mitigate risks, decipher inefficiencies in complexity, and test and implement modern approaches, all on behalf of our clients.

As each of us continues on our own path of learning, an opportunity to support each other within our living lab presents itself. We are a community traveling in the same direction. Such says the African proverb, “If you want to go fast, go alone. If you want to go far, go together”. We are bar-raisers and we travel together. In this sense, all of our clients can benefit from any of our thought leaders.

There will be times where we struggle. Perhaps, through a buggy release or during the release of a new product with unanticipated difficulty. We must take feedback with fortitude and cannot be afraid to pivot our strategies, evaluate new implementations, test and ultimately learn.

Learning by doing is often the best form of instruction. Agile craftsmen must recognize that failure is a part of the development process. As Robin Arzon would say, “Failure is the platform to which we build upon”. Often times success only provides validation and feedback is an essential aspect of agile development supporting the continuous loop through which we design, implement, test, and learn.

Earlier this summer, the much anticipated release of Apache Kafka 2.3, a vastly popular component in microservice architecture, enabling streaming and real-time processing of records, was met with its own challenges. One of the persistent bugs was related to the user unsubscribe functionality. This was obviously a critical part of their offering today given the growing requirements related to data privacy and simply an essential aspect of any offering to ensure customer satisfaction. I’m sure everyone will agree, in today’s modern service oriented technology landscape, that the customer is in control and products must incorporate their needs if they wish to remain in business.

It’s simply not easy to project our own living lab culture when onsite with a client. The pressure is already high to meet deadlines and each of our own efforts is codependent on others’ to execute and deliver. Another factor can be pride. Nobody wants to err and be labeled as the bottleneck.

So how will you manage to accomplish this? Observation and communication. Follow what the signs say in the subway stations and airports. “If you see something, say something.” This is actually universal advice and is highly applicable to agile development efforts. By remaining sharply aware of what you’re doing and how that work is impacting those around you, your sprint, project, and team, you will be in a better position to anticipate potential challenges and outcomes. You can then communicate these as you predict them or as they are happening so that you can work collaboratively with your client and your colleagues onsite to address them. Sometimes simply managing expectations can afford you the opportunity to make mistakes, and as we know, making them earlier saves future development efforts.

Failing is a part of the process and not having the visibility and foresight to identify and cross-communicate about potential issues creates risk and impedes high-velocity teams. Agile craftsmen are expected to be nimble minded, to think and learn fast. We are expected to err on occasion and it’s all about how we handle that situational awareness and how we are communicating to others so that we all learn and grow together.

Cloud Engineering

Removing Human Error

No Hidden/Tribal Knowledge

Implementation Technology Overview

Puppet

Documentation

Chef

Documentation

Ansible

Documentation

Reusability

Footer

Interested in learning more?