Bringing Valuable Information to DevOps Professionals

DevOps Journal

Subscribe to DevOps Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get DevOps Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


DevOpsJournal Authors: Jyoti Bansal, Yeshim Deniz, AppNeta Blog, Liz McMillan, Pat Romanski

Related Topics: IT Strategy, DevOps Journal

DevOpsJournal: Article

Best Practices for Advanced Deployment Patterns

Industry Experts Discuss Advanced Deployment Patterns

In this episode of our #c9d9 video podcast, our expert panel included: John Harris, Head of DevOps & Solution Architect for LeftShiftIT; Viktor Farcic, Software Architect at Everis; Flynn, solving problems at Datawire.io; and, our very own Anders Wallgren and Sam Fell.

During the episode, we discussed the differences between deployments and releases, as well as advanced deployment patterns and tactics such as rolling, Blue/Green, Canary, Big Bang, feature flags and dark launches.

»Deployment vs. Release

 

Not having to  all at once is necessary if you want to release at , per @_flynn live on 

 

"If you are doing it right, you are probably doing way more deployments than releases" - @anders_wallgren

 

Pipeline is king for seeing how your deployments perform in production, per @johnharris85 on .

 

Flynnflynn-c9d9-devops-podcast provides his definition of releases and deployments, and describes how releases are different in Continuous Delivery: “I tend to think of a release as being a chunk of software that you think is ready to go, and the deployment as the way you actually make it go…I feel that back in the days when everybody was doing big bangs, we tended to use releases as a way of collating all the dependencies together that worked. A given release was the line in the sand where everybody had to adhere to a common set of protocols and a common set of interaction patterns. In the continuous world, it feels much more like you have to arrange it such that the interaction patterns are always there, which implies that if everybody is always adhering to the same interaction patterns then you can deploy stuff whenever you need to, but it also implies that you can’t do things like switch everybody from protocol version 1 to protocol version 2.”

John-Harris-c9d9-devops-podcastFor Harris, the difference between releases and deployments is philosophical: “For me it’s kind of a philosophical split rather than a physical one, I think the release part of it is more in the business, it’s a kind of disconnect – the business should be able to choose when to actually release new features and pieces to their customers, and IT should just enable that. The deployments are just rolling the whole time. If IT decides that this is a release candidate worthy of going out and this is different from a deployment.”

victor-farcicOnce you get to the release stage, all you can do is prepare for the worst and expect the best, says Farcic: “As you soon as you get to the point of releasing or deploying you are already in bad shape, because you don’t know what you are going to put into production. The only thing you can do at that point is put a helmet on your head and hope that nothing is going to fall on top of you.”

Wallgrenanders-wallgren-c9d9 advises to always do deployments before deploying to production: “Hopefully you are doing deployments before you deploy to production. Snowflake environments and snowflake deployment scripts gets us in trouble all the time. Focus more on using the same processes, even if you are going into different environments, then we are a lot more confident at the deploy to production or release time that it’s going to work.”

»Rolling Deployments

 

 

Per @samueldfell - Rehearse the deployment pattern as early on as possible so that you're not trying to create a script only for prod. 

 

anders-wallgren-c9d9Wallgren on the benefits and considerations of rolling deployments: “The one thing about rolling deployments is you get zero-downtime which is brilliant. What you have to put into is you have to architect the product in such a way that you can do this. You have to be able to have multiple versions of most things up and running. The database ends up being the problem when running multiple versions at the same time.”

sam-fellIf you are doing rolling deployments, make sure you are rehearsing your deployment up front, says Fell: “Rehearse the deployment pattern as early as you can in the cycle so that you are not spending hours of time creating some fancy script just for production and having one-sy/two-sy kind of deployment scripts for pre-production because then you are not testing your process.”

victor-farcicFarcic provides his simple definition of rolling deployments: “Rolling deployments are about running multiple instances of something and then updating parts of those instances. At any given point, at least some of the instances are always running. More likely in some period of time you are going to have a mixed release in production.”

 

»Blue/Green Deployments

 

 

 

 

John-Harris-c9d9-devops-podcastHarris provides a definition of Blue/Green deployments: “Blue/Green is where you have two environments which are basically identical and you route all of your customers to one, and then you deploy everything to the second environment, let’s call it Green. Then you flip over everyone using load balancers, DNS pointing to Green let’s say, and then Blue becomes a standby and if anything goes wrong you can quite easily flip back, you can flip the DNS back to Blue and fix whatever terrible thing happened with Green.”

flynn-c9d9-devops-podcast

Thoughts on blue/green deployments in the Cloud, per Flynn: “Even in the Cloud, with blue/green you need to completely deploy the new thing. Even though it doesn’t have to be very expensive in terms of money, it can definitely arrange it so that the deployment itself is fairly expensive in terms of time. If you’re going to deploy the entire second environment, then ideally you would have some way of making yourself comfortable, but your second deployment is working correctly before you switch all the production traffic to it.”

victor-farcicFarcic: “I really like Blue/Green deployments because of testing. It’s the only deployment that truly gives me the ability to test something. If I do a second release, it works, then I change the Proxy – it’s that easy.”

»Canary Deployments

 

Per @johnharris85: Canary deployments are when you deploy a release to a small set of users to gain metrics and refactor as needed 

 

 

flynn-c9d9-devops-podcastCanary deployments are difficult to do in monoliths, says Flynn: “I know that Netflix has done a bunch of stuff with having the infrastructure to switch the way routing happens, it’s all about being able to take a group of people to do this too and they will induce failures and make sure that the system is resilient in that case. It feels like this sort of a thing would be extremely difficult to do in monolith.”

anders-wallgren-c9d9

There are a lot of moving parts in Canary deployments, so having the infrastructure to support them is critical, says Wallgren: “The more in control of your architecture you are at the application level, the easier it is. You are going to have to have an IT infrastructure that supports it because you are going to have to have a way to steer the traffic – whether it’s having the load balancer do it, or your DNS do it – you have to have some sort of way of saying I only want 1% of traffic or only people in Connecticut. Then you need to have the ability to run that stand alone application.”

»Big Bang Deployments

 

 

If you want to crank quality things out quickly, you have to think strategically about  patterns - @anders_wallgren

John-Harris-c9d9-devops-podcastHarris explains Big Bang deployments: “In a Big Bang deployment, once it’s gone, you’ve deployed, you’ve done it, it’s out there, you can’t fix anything or worry about things after that. If anyone is interested in a specific topic, there is a fantastic talk by Gary Gruver, he used to be the head of IT at HP, and he talked about how they implemented continuous delivery for firmware, he said if we can do continuous delivery for firmware then you guys can do it right.”

victor-farcicYou can really use any kind of deployment pattern you want if you are using Continuous delivery, but Big Bang is the most likely choice, says Farcic: “What runs in CD can be Big Bang, it can be Canary, it can be a Rolling deployment. Those using CD are probably very traditional and are probably more likely to do Big Bang, but not because it is CD.”

 

anders-wallgren-c9d9Wallgren relates how installers are treated to how deployment is treated, and why that needs to shift: “What we often do is assign the most junior, hated person the job of running installers. Which is dumb because running installers is really difficult and really important. The problem is anyone who wants to be an installer shouldn’t be allowed to, and I feel to some extent it’s the same way with some people’s deployment patterns – “It’s IT’s problem let them do it” – but you need to think about it better than that. If you really want to crank this stuff out quickly and with quality, then you need to worry about those things. We worry about it less than we should.”

sam-fellMetrics play a major role in any deployment pattern, says Fell: “We can also look at telemetry from inside the product that’s giving us information on what’s the performance like, and all this other information that the web guys use all the time when they are doing a Canary deployment or whatever type of deployment, to make sure that things are square before they keep plowing forward.”

»Feature Flags and Dark Launches

 

.@_flynn advises to ask for a dark launch- How important is it to you that no one is going to see it until they have permission? |

 

 

.@vfarcic says feature-flags are largely related to coding techniques as opposed to deployment techniques 

 

flynn-c9d9-devops-podcastFeature flags can be beneficial, but understand there is a time and place for them, advises Flynn: “Feature flags gives you an opportunity to roll the stuff out, to know that the developer can enable the thing for testing and also know that you have an additional layer preventing that from being something that costumers can see before they are ready for it or before you’re ready for them to see it. On the other hand, from the developers’ point of view, it also tends to involve a little bit more work, particularly for a feature that’s relatively far reaching. There is a time and a place for a lot of these things, and the balance about feature flags has to do with how much hustle is it going to be for the developers to put in a runtime switch that controls this thing, and then balancing that against how important is it to you that you know that nobody is going to see it until you exclusively give them permission.”

victor-farcicFarcic explains: “In most cases, feature flags are more related to coding techniques than deployment techniques. You are going to deploy with one of the techniques we have been talking about, and then once you deploy it (no matter how you deploy it) then your feature flag in your code kicks in.”

sam-fellFell on software and deployments: “Software is eating the world. Well, software is also eating the deployment process. The release process is basically a feature flag, and you have to be careful with that. Your pipeline has an additional vulnerability potential.”

John-Harris-c9d9-devops-podcastFacebook is an example of a company that is good at doing feature flags and dark launches, says Harris: “My favorite story about feature flags and dark launches is Facebook, one of the best companies that does this. It was two or three years ago when they said that 90% of the stuff they are going to release over the next 3 months is already in production and they are going to test it while watching under the hood. I think Facebook Chat got out six months before it was even on for users, and they were actually sending client-side calls faking chat massages to all the users through their service.”

Watch the full episode here:

 

 


Want more Continuous Discussions (#c9d9)?

We hold our #c9d9 podcast every other Tuesday at 10 a.m. PST. Each episode features expert panelists talking about DevOps, Continuous Delivery, Agile and more.

Check out all past episodes and panelists here.

More Stories By Anders Wallgren

Anders Wallgren is Chief Technology Officer of Electric Cloud. Anders brings with him over 25 years of in-depth experience designing and building commercial software. Prior to joining Electric Cloud, Anders held executive positions at Aceva, Archistra, and Impresse. Anders also held management positions at Macromedia (MACR), Common Ground Software and Verity (VRTY), where he played critical technical leadership roles in delivering award winning technologies such as Macromedia’s Director 7 and various Shockwave products.