Bringing Valuable Information to DevOps Professionals

DevOps Journal

Subscribe to DevOps Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get DevOps Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


DevOpsJournal Authors: Liz McMillan, Elizabeth White, Yeshim Deniz, Stackify Blog, Jyoti Bansal

Related Topics: Application Performance Management (APM), Continuous Testing, DevOps Journal

Blog Post

Integrating APM into Testing | @DevOpsSummit @Stackify #DevOps #APM #AI

Does your team have a testing strategy?

Integrating APM into Your Testing Strategy
By Erik Dietrich

Does your team have a testing strategy?  In 2017, I have a hard time imagining that wouldn't at least have some strategy, however rudimentary.  Unlike a couple of decades ago, you hear less and less about people just changing code on the production server and hoping for the best.

At the very least, you probably have a QA group or at least someone who serves in that role before shipping your software.  You write the code, do something to test it, and then ship it once the testers bless it (or at least notate "known issues").

From there, things probably run the gamut among those of you reading.  Some of you probably do what I've described and little more.  Some of you probably have multiple pre-production environments to which a continuous integration setup automatically deploys builds.  Of course, it only deploys those builds assuming all automated unit, integration, and smoke tests pass and assuming that your static analysis doesn't flag any show stopper issues.  Once deployed, a team of highly skilled testers performs exploratory testing.  Or, maybe, you do something somewhere in between.

But, whatever you do, you can always do more.  In fact, I encourage you always to look for new ways to test.  And today I'd like to talk about an idea for just such a thing.  Specifically, I think you can leverage application performance management (APM) software to help your testing efforts.  I say this in spite of the fact that most shops have traditionally taken advantage of these tools only in production.

What You Need to Get Started
First things first.  You can't just download or purchase a tool and head off to the races.  You need a least some degree of sophistication in your existing setup.  And, if you lack this sophistication, I suggest that you acquire it.

You're going to need a non-production environment in which to test your application.  Generally speaking, this means a server (or multiple servers) in which you simulate your production environment.  You don't need to simulate all production conditions, necessarily.  But you can't reasonably do this on Steve's development box because he has the only desktop with 16 gigs of memory.  You're going to need to up your game a little beyond that.

You'll also need for this environment to include reasonable facsimiles of externalities to your application.  Most commonly, this means a database, but it also might involve things like (non-production) endpoints web service calls.  You need to recreate production, both regarding the deployed environment and the things it depends on.

What You Really Ought to Have
From there, you probably want to do a few more things to make this work.  I strongly suggest automating the process taking committed code to deployment in your testing environment(s).  You can get by without it, but a manual process dissuades people from executing it frequently.  And, without frequent execution, you lose early feedback and a good bit of value.

The same reasoning applies to exercising the application.  You could deploy it to the test environment and then do something like having a whole bunch of people log in and start banging around at things.  But you'd do a lot better to automate this to simulate traffic.  You can think of this as an automated smoke test.  And doing it will put stress on the system in a way that manual testers probably can't and certainly won't.  It will also ensure a consistent and methodical approach that you can repeat to troubleshoot.

With all of that squared away, you can start to take advantage of APM in your testing strategy.

First Steps and Getting a Feel for Things
In my consulting, I advocate that people do what they can do tighten feedback loops.  Plan a small slice of value in what you're doing, execute on it, notch a quick win, and go from there.  I advocate this same approach here.

Get an APM tool and set it up in your testing environment.  Play with it and get a feel for its features and how it works.  See what sorts of data it furnishes and how that changes under different circumstances.

You're just seeing what it will tell you about your application at this point.  If you've never really profiled your code before, I can almost promise you that it will find things that astonish you.   Sure, you knew the call to get all customer records was slow, but you just assumed it was a network issue or something.  You had no idea that Customer data transfer object had a Thread.Sleep(1000) in its constructor.  (Okay, you probably won't find yourself quite that astonished, but you never know.)

Your manual efforts alone will likely result in a robust set of technical debt backlog items to address.  You'll have queries to improve, modules that spew swallowed exceptions to clean up, and a good bit of general housekeeping.

Move On to Automation
Once you've recovered from your initial backlog with the tool by fixing things, you want to make sure you don't back slide.  Take what you've learned and turn the captured metrics into thresholds.

Did some kind of rogue query result in an order of magnitude delay?  By all means, fix the query, but carry that learning forward.  Set a threshold a bit above the proper time for the operation, and instrument your build pipeline to notify you if performance exceeds the threshold.  Better yet, create some kind of failure that the team can't ignore.

Assuming you've taken my advice, you now have a pretty powerful end to end setup.  Your commits trigger automated builds.  These, in turn, trigger deployments to a testing environment and subject the application to all manner of testing - including smoke tests.  These smoke tests stimulate the application into situations where APM will detect issues.  When it does, you'll become aware of them with a significantly tighter feedback loop than a user calling the help desk to ask why things are so slow.

Keep the Pressure On
Once you've operationalized this whole approach, don't let yourself get complacent.  In situations where performance means competitive advantage, keep dialing the APM tool thresholds to better and better figures.  Set goals with each release for improvement.

And that's just a single example.  If you've set goals for other concerns or forms of research usage, keep poking at those for improvement as well.  And keep dialing up the volume on the smoke tests if you can.  The closer you can run to a stressed production environment, the better you can sleep at night when you actually ship.

Keep Things in Perspective
I'll close by offering a note of caution based on field experience.  When you start to incorporate sophisticated techniques into your testing strategy, you experience a heady feeling.  You'll acquire an unprecedented sense of confidence in your code and its performance.  And justifiably so.

But take care that you don't allow that confidence to slide into overconfidence.  Using APM can expose issues other forms of testing can't easily detect.  It lets you simulate the vagaries of production and then take steps to guard against them.  But note that I said simulate.  It affects production, but it doesn't recreate it.

So as you incorporate APM into your testing strategy bear two things in mind.  First, you also definitely need to monitor production itself.  And secondly, no testing strategy, however sophisticated, can give you 100% confidence.  Incorporate APM into your strategy, but never stop looking for other ways to improve it.

The post Integrating APM into Your Testing Strategy appeared first on Stackify.

@DevOpsSummit at Cloud Expo taking place June 6-8, 2017, at Javits Center, New York City, and is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.

DevOps at Cloud Expo / @ThingsExpo 2017 New York 
(June 6-8, 2017, Javits Center, Manhattan)

DevOps at Cloud Expo / @ThingsExpo 2017 Silicon Valley
(October 31 - November 2, 2017, Santa Clara Convention Center, CA)

Download Show Prospectus ▸ Here

The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.

@DevOpsSummit will expand the DevOps community, enable a wide sharing of knowledge, and educate delegates and technology providers alike. Recent research has shown that DevOps dramatically reduces development time, the amount of enterprise IT professionals put out fires, and support time generally. Time spent on infrastructure development is significantly increased, and DevOps practitioners report more software releases and higher quality. Sponsors of @DevOpsSummit will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35-minute technical session
  • Online advertising in SYS-CON's i-Technology Publications
  • Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
  • Unprecedented PR Coverage: Editorial Coverage on DevOps Journal
  • Tweetup to over 75,000 plus followers
  • Press releases sent on major wire services to over 500 industry analysts.

For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez by email at events (at) sys-con.com, or by phone 201 802-3021.

The World's Largest "Cloud Digital Transformation" Event

@CloudExpo / @ThingsExpo 2017 New York 
(June 6-8, 2017, Javits Center, Manhattan)

@CloudExpo / @ThingsExpo 2017 Silicon Valley
(Oct. 31 - Nov. 2, 2017, Santa Clara Convention Center, CA)

Full Conference Registration Gold Pass and Exhibit Hall ▸ Here

Register For @CloudExpo ▸ Here via EventBrite

Register For @ThingsExpo ▸ Here via EventBrite

Register For @DevOpsSummit ▸ Here via EventBrite

Sponsorship Opportunities

Sponsors of Cloud Expo @ThingsExpo will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35 minute technical session
  • Online targeted advertising in SYS-CON's i-Technology Publications
  • Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage
  • Unprecedented Marketing Coverage: Editorial Coverage on ITweetup to over 100,000 plus followers, press releases sent on major wire services to over 500 industry analysts

For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez (@GonzalezCarmen) today by email at events (at) sys-con.com, or by phone 201 802-3021.

Secrets of Sponsors and Exhibitors ▸ Here
Secrets of Cloud Expo Speakers ▸ Here

All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.

With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo@ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.

Track 1. FinTech
Track 2. Enterprise Cloud | Digital Transformation
Track 3. DevOps, Containers & Microservices 
Track 4. Big Data | Analytics
Track 5. Industrial IoT
Track 6. IoT Dev & Deploy | Mobility
Track 7. APIs | Cloud Security
Track 8. AI | ML | DL | Cognitive Computing

Delegates to Cloud Expo @ThingsExpo will be able to attend 8 simultaneous, information-packed education tracks.

There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content.

Join Cloud Expo @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA for three days of intense Enterprise Cloud and 'Digital Transformation' discussion and focus, including Big Data's indispensable role in IoT, Smart Grids and (IIoT) Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) Digital Transformation in Vertical Markets.

Financial Technology - or FinTech - Is Now Part of the @CloudExpo Program!

Accordingly, attendees at the upcoming 20th Cloud Expo @ThingsExpo June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA will find fresh new content in a new track called FinTech, which will incorporate machine learning, artificial intelligence, deep learning, and blockchain into one track.

Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses.

FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.

More than US$20 billion in venture capital is being invested in FinTech this year. @CloudExpo is pleased to bring you the latest FinTech developments as an integral part of our program, starting at the 20th International Cloud Expo June 6-8, 2017 in New York City and October 31 - November 2, 2017 in Silicon Valley.

@CloudExpo is accepting submissions for this new track, so please visit www.CloudComputingExpo.com for the latest information.

Speaking Opportunities

The upcoming 20th International @CloudExpo@ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA announces that its Call For Papers for speaking opportunities is open.

Submit your speaking proposal today! ▸ Here

Our Top 100 Sponsors and the Leading "Digital Transformation" Companies

(ISC)2, 24Notion (Bronze Sponsor), 910Telecom, Accelertite (Gold Sponsor), Addteq, Adobe (Bronze Sponsor), Aeroybyte, Alert Logic, Anexia, AppNeta, Avere Systems, BMC Software (Silver Sponsor), Bsquare Corporation (Silver Sponsor), BZ Media (Media Sponsor), Catchpoint Systems (Silver Sponsor), CDS Global Cloud, Cemware, Chetu Inc., China Unicom, Cloud Raxak, CloudBerry (Media Sponsor), Cloudbric, Coalfire Systems, CollabNet, Inc. (Silver Sponsor), Column Technologies, Commvault (Bronze Sponsor), Connect2.me, ContentMX (Bronze Sponsor), CrowdReviews (Media Sponsor) CyberTrend (Media Sponsor), DataCenterDynamics (Media Sponsor), Delaplex, DICE (Bronze Sponsor), EastBanc Technologies, eCube Systems, Embotics, Enzu Inc., Ericsson (Gold Sponsor), FalconStor, Formation Data Systems, Fusion, Hanu Software, HGST, Inc. (Bronze Sponsor), Hitrons Solutions, IBM BlueBox, IBM Bluemix, IBM Cloud (Platinum Sponsor), IBM Cloud Data Services/Cloudant (Platinum Sponsor), IBM DevOps (Platinum Sponsor), iDevices, Industrial Internet of Things Consortium (Association Sponsor), Impinger Technologies, Interface Masters, Intel (Keynote Sponsor), Interoute (Bronze Sponsor), IQP Corporation, Isomorphic Software, Japan IoT Consortium, Kintone Corporation (Bronze Sponsor), LeaseWeb USA, LinearHub, MangoApps, MathFreeOn, Men & Mice, MobiDev, New Relic, Inc. (Bronze Sponsor), New York Times, Niagara Networks, Numerex, NVIDIA Corporation (AI Session Sponsor), Object Management Group (Association Sponsor), On The Avenue Marketing, Oracle MySQL, Peak10, Inc., Penta Security, Plasma Corporation, Pulzze Systems, Pythian (Bronze Sponsor), Cosmos, RackN, ReadyTalk (Silver Sponsor), Roma Software, Roundee.io, Secure Channels Inc., SD Times (Media Sponsor), SoftLayer (Platinum Sponsor), SoftNet Solutions, Solinea Inc., SpeedyCloud, SSLGURU LLC, StarNet, Stratoscale, Streamliner, SuperAdmins, TechTarget (Media Sponsor), TelecomReseller (Media Sponsor), Tintri (Welcome Reception Sponsor), TMCnet (Media Sponsor), Transparent Cloud Computing Consortium, Veeam, Venafi, Violin Memory, VAI Software, Zerto

About SYS-CON Media & Events
SYS-CON Media (www.sys-con.com) has since 1994 been connecting technology companies and customers through a comprehensive content stream - featuring over forty focused subject areas, from Cloud Computing to Web Security - interwoven with market-leading full-scale conferences produced by SYS-CON Events. The company's internationally recognized brands include among others Cloud Expo® (@CloudExpo), Big Data Expo® (@BigDataExpo), DevOps Summit (@DevOpsSummit), @ThingsExpo® (@ThingsExpo), Containers Expo (@ContainersExpo) and Microservices Expo (@MicroservicesE).

Cloud Expo®, Big Data Expo® and @ThingsExpo® are registered trademarks of Cloud Expo, Inc., a SYS-CON Events company.

Read the original blog entry...

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com