InRhythm

Your partners in accelerated digital transformation

Learning and Development

Lightning Talk: Creating a Proxy with Node + Express

tldr;

I created a Node and Express proxy that rams together two websites / apps. If you are ever asked to develop code for a site you don’t have access to, this could save the day.

  1. Download the code:
    https://github.com/mattbillard/code-collider-v2
  2. Watch the video:

My most recent hobby project involved building a Node and Express proxy to ram two websites or apps together. I dubbed it “The Code Collider.” 

The idea of the Code Collider was born about two years ago when I was given a rather difficult coding challenge at a client. Another department had a website and we were being asked to iframe it. As you might know, iframes have a lot of “interesting nuances” as they might politely be called, one of which is that if a user right-clicks on a link and chooses “Open in new tab,” they will break out of the parent’s iframe and go directly to the iframed child site. Of all things, I was asked to write some code to prevent this and provide my solution to the other team. There was just one catch: I had no way of running the other team’s site in my local environment and thus (or so I initially thought) no easy way to test my code as I wrote it. 

This situation comes up more often than you’d think. There are several reasons a developer might not be able to run a site or app in their local environment. Perhaps it’s complex to set up, involves a lot of permissions, or the other team just can’t be bothered to spare the time to help you out. Regardless of the reason, I did eventually strike upon a novel way to solve the problem.

After what felt like a meandering odyssey of Stack Overflow posts, I managed to cobble together a Node and Express project that took the other team’s site, piped it through a proxy, injected my script into it, and served it to my browser. Even though my innovation worked and I was able to deliver the final code requested, there was just one thing that bothered me: I always had the nagging feeling that I had only gotten the proxy to work through a mixture of stubborn persistence and blind luck. I resolved to rebuild it from scratch in my free time one day, piece by piece, actually understanding how the solution worked – and that as it turns out is exactly what I did.

The Architecture

Here’s a diagram of how it works:

The browser makes a request to our Node Express proxy server where we then have three scenarios:

  1. If the user requested an HTML page, we need to combine the page from website 1 and 2. The proxy first asks website 1 and then website 2 for its HTML. It then modifies the two HTML files, combines them, and returns the result to the browser. (Details on how this works below.)
  2. The HTML page will then request the CSS, JavaScript, and other assets it requires. These requests will again go through the proxy which will pass on the requests. If website 1 has the asset, great, the proxy will return it to the browser. 
  3. If website 1 does not have the asset, the proxy will then ask website 2 and return it to the browser.

Here are screenshots of the result:

In this example, we’ll use the real InRhythm.com website as the target into which we’ll inject some local code (in this case a basic Create React App project). The final result on the right is an actual screenshot of the 2 websites living together in the same browser window. Now this is pretty exciting; the target website and the code that you are developing could be anything!

How it Works

  1. As mentioned above, website 1 and 2’s HTML are combined. This involves a few steps. Webpages can’t have 2 doctype, html, head, or body tags, so we’ll use some regex to strip those. Now that website 2’s HTML is ready, we’ll inject it before website 1’s closing </body> tag. 
  1. And here is the code that shows our modifications to website 1’s HTML. It shows a few things.

    Firstly, many websites have full ‘absolute URLs’ for their links. They look like this: https://www.inrhythm.com/who-we-are/
    The problem is if the user clicks on this, they’ll be taken away from our proxy and go to the target website. We’ll solve this by removing all www.something.com bit and keep just the bit after the slash.

    Secondly, we show injecting the CSS discussed above to remove backgrounds and allow clicks to pass through website 2 to website 1. (Keep in mind this will probably be slightly different depending on the two sites you are combining.)

    Thirdly, we show injecting the HTML from website 2 we modified from above before the closing </body> tag.

Gotchas to Avoid

As mentioned above, it took a lot of research as well as trial and error to solve all the technical hurdles this project presented. Here are some of the main ones below.

  1. Websites usually compress or “Gzip” their content. Normally this is a great thing. It means less data is transferred and websites load more quickly. This is not going to work for our case however. You can’t parse, manipulate, and modify HTML if it looks like gibberish. The solution was actually quite simple: as it turns out, there’s a header you can send with your request to ask the server not to Gzip anything.
  1. There’s another header we need to send however. Because we’re using a proxy, all requests are going to have the header ‘host’ set to ‘localhost’. Now this is probably not a problem for most sites, but to the target server, this doesn’t look like a very normal request, and indeed, I did find some websites responded oddly and returned pages that looked nothing like I expected. The solution again was quite simple, just modify one of the headers of our request. 
  1. Now as you’ve probably gathered from above, we’ve modified our requests quite a lot, and as a result this is going to cause the browser to do some odd things. The solution to this problem is to delete the ‘content-length’ header before the proxy sends your browser any response. This will stop the browser from truncating the response and removing all the hard work we did. 
  1. Finally, I’ll mention one other problem I solved. If you are combining sites that use https, the proxy might complain that the SSL certificates don’t match what it’s expecting. Turns out it’s rather easy to relax this with the following code: 

Conclusion

That’s it. Feel free to download the code, follow the instructions in the readme, and open your browser to localhost:8000 . You should be able to see two web sites rammed together. 

Creating Robust Test Automation for Microservices

Any and all projects that a software engineer joins will come in one of two forms: greenfield or legacy codebases. In the majority of cases, projects will fall into the realm of the legacy repositories. As a software engineer, it is our responsibility to be able to strategically navigate our way through either type of project by looking objectively at the opportunities to improve the code base, lower the cognitive load for software engineering and make a determination to advise on better design strategies. But, chances are, there is a problem. Before architecture or design refactors can be taken its best to take a pulse on the health of your platform End to End (E2E). The reason being, lurking in a new or existing platform is likely a common ailment of a modern microservices approach – the inability to test the platform E2E across microservices that are, by design, commonly engineered by different teams over time.

I recently worked on a greenfield platform and nearly 6 months into the project, one of the most critical parts of the project had fallen several months behind from a quality assurance perspective. It was no longer possible for QA to catch up, nor was it possible for QA to engineer and execute E2E testing to complete common user journeys throughout the enterprise system. To solve this conundrum, E2E data generation tools needed to be created so that the QA team could keep upbuilding and testing every scenario and edge case. As I see it, there are three main requirements for an E2E account and data generation tool. The tool should:

1) Create test accounts with mock data for each microservice

2) Link those accounts between up and downs stream microservices

3) Provide easy to access APIs that are self-documenting 

Using a tool like Swagger, QA can use the API description for REST API, i.e. OpenAPI Specification (formerly Swagger Specification) to view the available endpoints and operations to create accounts, generate test data, authenticate, authorize and “connect the microservices”.

By creating tools for E2E testing, our QA team was able to eliminate the hassle of trying to figure out which upstream and downstream microservices needed to be called to ensure that the required accounts and data were available and set up properly to ensure a successful test of all scenarios i.e. based upon the variety of different data types, user permissions, user information and also covering the negative test cases. The QA team was able to catch up and write their entire suite of test scenarios generating the matching accounts and data to satisfy those requirements. The net result of having built an E2E test generation tool was automated tests could be produced exponentially quicker and the tests themselves are more resilient to failure. 

Even though the microservices pattern continues to gain traction, developing E2E testing tools that generate accounts and test data across an enterprise platform will likely still remain a pain point. That having been said, the purpose of this article is to raise awareness. Software Engineering Leadership has an opportunity to step up and ensure that the enterprise system that they look after is healthy E2E. I can’t think of a better way to maintain a healthy system than to ensure accounts and data in the lower environments actually work and unblock testing end-to-end. 

James Woods, Web Engineering Practice Lead

Letting Go of the Formality

It wasn’t that long ago that suits, ties, knee-length skirts, heels and pantyhose were required wardrobe staples. In fact, many enterprise Fortune 100s upheld a formal dress code as part of their HR Policy Handbooks. Goldman Sachs, the last bastion of formality on Wall Street, finally relaxed its dress code and offered a “flexible style” option. They only did so last year. Their updated policy permitted men to choose garments other than suits. Admittedly, this made things more complicated for women but that’s a topic for another blog.

Since we’re agile practitioners here at InRhythm, it made sense to look at our own approaches to talent and acquisition. Not to mention retention. Practices and policies drive corporate culture. Here, Agile Platinum Principle One: resist formality, must serve as one of the pillars of our talent strategy.

If your company doesn’t offer a culture that’s aligned with the needs and expectations of today’s job seekers, then, as Hiring Manager or Chief People Officer, you’re going to have a really hard time. At best, you’re going to struggle and slog through your days and recruiting campaigns. At worst, you’re not going to have any people to serve as Chief over!

The link between informality and tech

Ask anyone within the technology field, or external to it, to describe or draw a person in tech, be it a software developer or a startup founder. The universal response is a male, typically Caucasian, dressed in sloppy jeans and a hoodie. Where did this level of informality originate?

Many would cite Lou Gerstner’s (former CEO of IBM) decision to roll out “casual dress.” Even the New York Times balked with a headline that read, “Black Jeans Invade Big Blue.” Shortly thereafter, Hewlett Packard (HP) introduced “Blue Sky Days” which permitted employees to wear jeans to work on Fridays. From there, dress code devolved or evolved – it depends on your perspective – into what we see today.

However, the Platinum Principal of resisting formality is not restricted to dress code. It extends into everything that we do. Formality also has implications with unconscious bias, how we greet and interact with people, the forms that we require candidates to complete and so on. Tone and style go hand in hand.

How are we resisting formality?

We are constantly taking steps to increase our awareness and connection with our prospective employees, candidates, contractors, staff and employees. Efforts are made to understand what they need so that we can craft our talent, acquisition and retention strategies around their needs and expectations versus requiring that they conform to ours. Here, we have embraced an agile culture across the company, not just in our product development efforts.

As agile craftsmen, with respect to HR recruiting and retention strategies, we have adopted an approach that is more responsive than it is prescriptive. Our applications are deliberately designed with brevity in mind to expedite the hiring process. We measure how long it takes from the time that an application is submitted to an open job posting until someone is seated in the role. A conscientious effort, with monitoring, is made to ensure that we are constantly moving our candidates through the hiring process as swiftly as they seek to do so.

Another way that may come as a bit of a surprise is the effort we make to create a relaxed culture that allows for flexibility and downtime. We understand the enormous pressure and stress that our employees feel as they do all that they can to deliver on our clients’ needs. It’s okay to push through a couple of scrums or back-to-back sprints but it’s not sustainable. Over the holidays, we relaxed with an in-house office party and gave people some bonus time. This week is Valentine’s Day: celebrate it or not, you don’t need to make it about sending flowers and chocolates to your partner. Use it as a reminder to resist formality, take some time and reach out to whomever it is that you care about, be it a parent or sibling or a friend.

Resist formality! Make strides to emulate the people and culture around you. Forcing people into a process defined by its rigor and inflexibility is the antithesis of what a company should do. Particularly a company who’s purpose is to bring agile methodologies and success to our clients.

Stu Weiser,

Director of Talent Acquisition
Image created by katemangostar – www.freepik.com

Configuration Automation Tools: Orchestrating Your Deployment

“We must accept human error as inevitable – and design around that fact.” – Donald Berwick

Raphael Weaver

In our Technology field, buzz words come and go. One day we’re discussing how databases are the new best thing to the world of Agile Development and the next we see programming languages, frameworks, and methodologies bit the dust like an old western movie.

But one unchanging aspect of technology and its journey is the People who are an irreplaceable part of the creation, the demise, and the popularity of any given technology. This modern-day-world calls for a close to perfection execution, which humans cannot always account for. How does that affect us, the developers and creators, when we are called to building perfect products? 

Removing Human Error

Automation automation automation – and of course throw in some orchestration deployment and configuration management. Leaving the buzz words behind the jest of all of this, “new technology frontier”, is removing human error. This is removing the dependencies of tribal knowledge when it pertains to application and system administration job duties. Those job duties are performed in a repetitive fashion. The job duties are usually consolidated into various custom scripts, leaving a lot of those scripted actions the ability to be boxed up and reused over and over again.

For example, what steps does it take to deploy an application to a server instance? 

  1. Download and Install the various languages and/or framework libraries the application usages.
  2. Download, Install, Configure the Web server that the application will use.
  3. Download, Install, and Configure the Database that the application will use. 
  4. And finally Test to see if all the steps are installed and configured correctly. 

Last but not least run application tests to make sure it is running as expected. Those are the highlight steps for deploying an application to a bare metal, cloud, or virtual server. With a further review, there are numerous upon numerous additional steps. Something simplistic nevertheless highly catastrophic is a possibility of a typo. Consider this case, maybe in this scenario, I meant to type

cd /var/ect/ansible; rm -rf *

but instead I forgot the cd execute command and only ran

rm -rf /

Now my whole drive is erased. Surprisingly enough this can happen quite often even though I am pretty sure the source of truth of my drive is backed up. However, as Murphy Law states, what can happen will happen. And let’s be honest, redoing previous actions that could have been avoided is a down right pain.

No Hidden/Tribal Knowledge

Let’s look back at those steps to deploy an application to an environment. There are a lot of gotchas and/or small intermediary steps involved. For instance, I only listed the high-level steps. I hand these steps to a new engineer or even a junior engineer fresh out of college. The chances are these engineers will have to ask several questions on  “How to”. 

These series of questions will eventually lead to someone like myself saying “just let me do it”. This is how one scenario can lead to years upon years of hidden or tribal knowledge, as one may call it.  To harbor information like a fugitive on the run is getting more and more exposed in the IT/Software industry. The information should be a source of truth maintained in a repository database, that is easy and intuitive to leverage

Implementation Technology Overview

What does that source of truth entail? Can we not skip the documentation of information and go straight into the execution of the steps onto a given system? Or create scripts to reconfigure the application if there was ever a need to? Those questions have been proposed several times and solutions have been formulated several times into the form of extensive and comprehensive build tools/frameworks. These tools are used throughout the industry to solve the problem of orchestrated development, configuration automation and management. 

Furthermore, DevOps tools such as: Chef, Puppet, and Ansible are well mature automation/orchestration tools. Each tool will provide you with enough architecture flexibility to virtually handle any use case you present. Let’s take a slight dive into each one of these technologies.

Puppet

Puppet is the first widely used Configuration Automation and Orchestration software dating back to its initial release in 2005. Puppet uses the Master and Slave paradigm to control X amount of machines. The Ruby language is the script language say, for executing commands in a destination environment. 

The “Puppet Agents”(Slave) are modularized distinct components to be deployed to a server. This can be used for the creation of the server(ie. web server, database, application) in its destination environment.  The Puppet enterprise (Master) is comprised of all the inner workings to manage, secure, and organize agents.

Puppet Architecture

Documentation 

Chef

Chef is somewhat similar to Puppet. The core language used within Chef’s abstract module components is Ruby. Chef has several layers of management for your infrastructure automation needs. The Chef workstation is the primary area for managing the various Chef components. The Chef components consist of cookbooks, recipes, and nodes. The naming convention actually follows what you would consider these nouns in cooking to be with technical context of course. 

Recipes are collections of configurations for a given system, virtual, bare metal, or cloud environment. Chef calls those different environments nodes. Cookbook contains recipes and other configurations for application deployment and control mechanisms for the different Chef clients. Lastly, the whole collections of cookbooks are maintained in a Supermarket.

Chef Architecture

Documentation

Ansible

Ansible is the newest mainstream automation/configuration management tool on the market. Therefore, Ansible uses more modern programming languages and configurations concepts and tools. Python is the programming language used in this framework. One of the modern and fastest up-and-coming template languages is YAML. YAML is programming language agnostic and is a subset of the ever so popular JSON. YAML is used within Ansible to describe an Ansible playbook. 

Ansible playbook contains the steps that need to be executed on a given system. Once the Ansible playbook is intact configuration or further manipulation of the host can be executed through Ansible API – which is implemented in Python. There are several other components within Ansible technology such as modules, plugins, and inventory. 

Ansible Architecture

Documentation

Reusability

After covering a couple of the Configuration Automation and Development tools on the market you can see the vast amount of flexibility available in eliminating those repeatable steps from human error. This software’s framework promotes reusable software within an organization – which is the most viable. The ability to scale your application development environment and environmental infrastructure is critical. 

The learning curve may be deeper than using plain old bash scripts but the structure and integrity of using a proven tool and ease of maintenance outweigh the learning curve.

Check it for yourself, reorganize and propel your CI/CD process with Configuration Automation and Orchestrated Deployments and let me know what you think about this article in the comments below.

Raphael Weaver

Sr. Software Engineer

Unleashing The Potential Of Your Offsite Team

Thriving vs Surviving

Are You Unleashing Your Team’s Potential? Or, are you an engineering Practice Lead or Manager who’s unconsciously capping your team’s potential? While there may be the odd diabolical leader who intents on deliberately holding his or her team back, I like to think that doing so is not the norm. Leaders of agile craftsmen want their teams to flourish and fly through each sprint. Doing so will require creating an environment and work culture that fosters improvement and unleashes a team’s potential.

With a new year and decade here upon us, the idea of making resolutions to catalyze changes isn’t new. It’s a centuries-old tradition that has begun to fall out of favor. Given that < 8%  of us make it through the year, performance and motivation coaches suggest that we set goals for the year instead of resolutions. By setting a goal, we know which direction we’re headed for and it allows us to “slip back” or go off-course occasionally yet still track towards the goal unlike resolutions, which do not offer the same latitude. 

Here at InRhythm, we’re looking forward to achieving continued success in 2020. With increased efforts to build and empower our 10x teams and to better enable our clients, we have set several key goals for the year ahead.

2020 Goals

  1. Continue to put our customers first and make every effort towards 100% customer satisfaction
  2. Foster a culture that truly embodies the Agile Manifesto
  3. Grow our footprint in new markets and industries 

As the Practice Lead , I’m conscious of how well my team is performing and how the toll of being onsite at clients’ workspaces affects them. I’m also aware that I need to give the team space to work together on a sprint and to figure things out on their own. When they hit an obstacle or begin struggling with forward progress, that’s the time to jump in. Until then, leaders must encourage their teams to experiment and err, support them, and back them when the choices they’ve made may not be the ones anticipated by their employer but they’re exactly the right ones needed for the client.

In addition to giving your team some latitude and space so that they can creatively solve problems without you helicoptering over them, you also need to think about giving them recognition. Of course, nobody wants to celebrate repeated failure, but, if a software developer made an informed decision to try to do something in a new way and everyone could extract lessons learned from that effort, then s/he/they should be celebrated for making a bold move and trying to do something different. It should go without saying that the successes also need to be celebrated. Recognition motivates teams to try harder.

Efforts in the opposite direction will have – no surprise – the opposite effect. For example, a lack of recognition for doing well or finger-pointing when there is an error will demotivate teams and start to put restraints on their ability (and desire) to flourish. Powering up 10x teams requires motivation and support so that members can dig deep and unlock their potential. Motivation should be a weekly or monthly focus and not reserved for the one day each year when companies formally recognize their employees, clients or management team. 

Here at InRhythm, we’re encouraging more dialogue between members and practice leaders and added the role of a liaison designed to facilitate open and honest discussion. We’re creating a culture where it’s safe to respectfully voice an opinion and to try new things that could make a difference for our clients. It’s also about accountability. If a team member come forward and voices a concern or a need that does not get addressed with an effort towards closed-loop communication, that member’s potential becomes thwarted. 

Have you made 2020 all about your team’s potential? Are you subconsciously thwarting it or actively boosting it? Be the practice leader that helps teams thrive – not just survive.