In this episode of our #c9d9 video podcast, our expert panel included: John Harris, Head of DevOps & Solution Architect for LeftShiftIT; Viktor Farcic, Software Architect at Everis; Flynn, solving problems at Datawire.io; and, our very own Anders Wallgren and Sam Fell.
During the episode, we discussed the differences between deployments and releases, as well as advanced deployment patterns and tactics such as rolling, Blue/Green, Canary, Big Bang, feature flags and dark launches.
Flynn provides his definition of releases and deployments, and describes how releases are different in Continuous Delivery: “I tend to think of a release as being a chunk of software that you think is ready to go, and the deployment as the way you actually make it go…I feel that back in the days when everybody was doing big bangs, we tended to use releases as a way of collating all the dependencies together that worked. A given release was the line in the sand where everybody had to adhere to a common set of protocols and a common set of interaction patterns. In the continuous world, it feels much more like you have to arrange it such that the interaction patterns are always there, which implies that if everybody is always adhering to the same interaction patterns then you can deploy stuff whenever you need to, but it also implies that you can’t do things like switch everybody from protocol version 1 to protocol version 2.”
For Harris, the difference between releases and deployments is philosophical: “For me it’s kind of a philosophical split rather than a physical one, I think the release part of it is more in the business, it’s a kind of disconnect – the business should be able to choose when to actually release new features and pieces to their customers, and IT should just enable that. The deployments are just rolling the whole time. If IT decides that this is a release candidate worthy of going out and this is different from a deployment.”
Once you get to the release stage, all you can do is prepare for the worst and expect the best, says Farcic: “As you soon as you get to the point of releasing or deploying you are already in bad shape, because you don’t know what you are going to put into production. The only thing you can do at that point is put a helmet on your head and hope that nothing is going to fall on top of you.”
Wallgren advises to always do deployments before deploying to production: “Hopefully you are doing deployments before you deploy to production. Snowflake environments and snowflake deployment scripts gets us in trouble all the time. Focus more on using the same processes, even if you are going into different environments, then we are a lot more confident at the deploy to production or release time that it’s going to work.”
Wallgren on the benefits and considerations of rolling deployments: “The one thing about rolling deployments is you get zero-downtime which is brilliant. What you have to put into is you have to architect the product in such a way that you can do this. You have to be able to have multiple versions of most things up and running. The database ends up being the problem when running multiple versions at the same time.”
If you are doing rolling deployments, make sure you are rehearsing your deployment up front, says Fell: “Rehearse the deployment pattern as early as you can in the cycle so that you are not spending hours of time creating some fancy script just for production and having one-sy/two-sy kind of deployment scripts for pre-production because then you are not testing your process.”
Farcic provides his simple definition of rolling deployments: “Rolling deployments are about running multiple instances of something and then updating parts of those instances. At any given point, at least some of the instances are always running. More likely in some period of time you are going to have a mixed release in production.”
Harris provides a definition of Blue/Green deployments: “Blue/Green is where you have two environments which are basically identical and you route all of your customers to one, and then you deploy everything to the second environment, let’s call it Green. Then you flip over everyone using load balancers, DNS pointing to Green let’s say, and then Blue becomes a standby and if anything goes wrong you can quite easily flip back, you can flip the DNS back to Blue and fix whatever terrible thing happened with Green.”
Thoughts on blue/green deployments in the Cloud, per Flynn: “Even in the Cloud, with blue/green you need to completely deploy the new thing. Even though it doesn’t have to be very expensive in terms of money, it can definitely arrange it so that the deployment itself is fairly expensive in terms of time. If you’re going to deploy the entire second environment, then ideally you would have some way of making yourself comfortable, but your second deployment is working correctly before you switch all the production traffic to it.”
Farcic: “I really like Blue/Green deployments because of testing. It’s the only deployment that truly gives me the ability to test something. If I do a second release, it works, then I change the Proxy – it’s that easy.”
Canary deployments are difficult to do in monoliths, says Flynn: “I know that Netflix has done a bunch of stuff with having the infrastructure to switch the way routing happens, it’s all about being able to take a group of people to do this too and they will induce failures and make sure that the system is resilient in that case. It feels like this sort of a thing would be extremely difficult to do in monolith.”
There are a lot of moving parts in Canary deployments, so having the infrastructure to support them is critical, says Wallgren: “The more in control of your architecture you are at the application level, the easier it is. You are going to have to have an IT infrastructure that supports it because you are going to have to have a way to steer the traffic – whether it’s having the load balancer do it, or your DNS do it – you have to have some sort of way of saying I only want 1% of traffic or only people in Connecticut. Then you need to have the ability to run that stand alone application.”
Harris explains Big Bang deployments: “In a Big Bang deployment, once it’s gone, you’ve deployed, you’ve done it, it’s out there, you can’t fix anything or worry about things after that. If anyone is interested in a specific topic, there is a fantastic talk by Gary Gruver, he used to be the head of IT at HP, and he talked about how they implemented continuous delivery for firmware, he said if we can do continuous delivery for firmware then you guys can do it right.”
You can really use any kind of deployment pattern you want if you are using Continuous delivery, but Big Bang is the most likely choice, says Farcic: “What runs in CD can be Big Bang, it can be Canary, it can be a Rolling deployment. Those using CD are probably very traditional and are probably more likely to do Big Bang, but not because it is CD.”
Wallgren relates how installers are treated to how deployment is treated, and why that needs to shift: “What we often do is assign the most junior, hated person the job of running installers. Which is dumb because running installers is really difficult and really important. The problem is anyone who wants to be an installer shouldn’t be allowed to, and I feel to some extent it’s the same way with some people’s deployment patterns – “It’s IT’s problem let them do it” – but you need to think about it better than that. If you really want to crank this stuff out quickly and with quality, then you need to worry about those things. We worry about it less than we should.”
Metrics play a major role in any deployment pattern, says Fell: “We can also look at telemetry from inside the product that’s giving us information on what’s the performance like, and all this other information that the web guys use all the time when they are doing a Canary deployment or whatever type of deployment, to make sure that things are square before they keep plowing forward.”
Feature flags can be beneficial, but understand there is a time and place for them, advises Flynn: “Feature flags gives you an opportunity to roll the stuff out, to know that the developer can enable the thing for testing and also know that you have an additional layer preventing that from being something that costumers can see before they are ready for it or before you’re ready for them to see it. On the other hand, from the developers’ point of view, it also tends to involve a little bit more work, particularly for a feature that’s relatively far reaching. There is a time and a place for a lot of these things, and the balance about feature flags has to do with how much hustle is it going to be for the developers to put in a runtime switch that controls this thing, and then balancing that against how important is it to you that you know that nobody is going to see it until you exclusively give them permission.”
Farcic explains: “In most cases, feature flags are more related to coding techniques than deployment techniques. You are going to deploy with one of the techniques we have been talking about, and then once you deploy it (no matter how you deploy it) then your feature flag in your code kicks in.”
Fell on software and deployments: “Software is eating the world. Well, software is also eating the deployment process. The release process is basically a feature flag, and you have to be careful with that. Your pipeline has an additional vulnerability potential.”
Facebook is an example of a company that is good at doing feature flags and dark launches, says Harris: “My favorite story about feature flags and dark launches is Facebook, one of the best companies that does this. It was two or three years ago when they said that 90% of the stuff they are going to release over the next 3 months is already in production and they are going to test it while watching under the hood. I think Facebook Chat got out six months before it was even on for users, and they were actually sending client-side calls faking chat massages to all the users through their service.”
Watch the full episode here:
Want more Continuous Discussions (#c9d9)?
We hold our #c9d9 podcast every other Tuesday at 10 a.m. PST. Each episode features expert panelists talking about DevOps, Continuous Delivery, Agile and more.
Anders Wallgren is Chief Technology Officer of Electric Cloud. Anders brings with him over 25 years of in-depth experience designing and building commercial software. Prior to joining Electric Cloud, Anders held executive positions at Aceva, Archistra, and Impresse. Anders also held management positions at Macromedia (MACR), Common Ground Software and Verity (VRTY), where he played critical technical leadership roles in delivering award winning technologies such as Macromedia’s Director 7 and various Shockwave products.
yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
DevOps Journal is focused on this critical enterprise IT topic in the world of cloud computing. DevOps Journal brings valuable information to DevOps professionals who are transforming the way enterprise IT is done.
Cloud Computing & All That
It Touches In One Location Cloud Computing - Big Data - Internet of Things
SDDC - WebRTC - DevOps
Cloud computing is become a norm within enterprise IT.
The competition among public cloud providers is red hot, private cloud continues to grab increasing shares of IT budgets, and hybrid cloud strategies are beginning to conquer the enterprise IT world.
Big Data is driving dramatic leaps in resource requirements and capabilities, and now the Internet of Things promises an exponential leap in the size of the Internet and Worldwide Web.
The world of SDX now encompasses Software-Defined Data Centers (SDDCs) as the technology world prepares for the Zettabyte Age.
Add the key topics of WebRTC and DevOps into the mix, and you have three days of pure cloud computing that you simply cannot miss.
Delegates will leave Cloud Expo with dramatically increased understanding the entire scope of the entire cloud computing spectrum from storage to security.
Cloud Expo - the world's most established event - offers a vast selection of 130+ technical and strategic Industry Keynotes, General Sessions, Breakout Sessions, and signature Power Panels. The exhibition floor features 100+ exhibitors offering specific solutions and comprehensive strategies. The floor also features two Demo Theaters that give delegates the opportunity to get even closer to the technology they want to see and the people who offer it.
Attend Cloud Expo. Craft your own custom experience. Learn the latest from the world's best technologists. Find the vendors you want and put them to the test.
See you in New York!
Register Now! Save $1595 on your “Golden Pass”! Call 201.802.3020
or click here to Register
Speaker Opportunities Submit your speaking proposal for the upcoming Cloud Expo in New York
[June 6-8, 2017]
Exhibitor Info Please Call 201.802.3021
events (at) sys-con.com.
carmen (at) sys-con.com.