A presentation at DevOpsDays Guadalajara in in Guadalajara, Jalisco, Mexico by Sasha Czarkowski (Rosenbaum)
Sasha Rosenbaum Sr. Program Manager @GitHub @DivineOps Feb 20th, 2020 #DevOpsDays Guadalajara
@DivineOps Feb 20th, 2020 #DevOpsDays Guadalajara
And you? Feb 20th, 2020 #DevOpsDays Guadalajara
Why DevOps? Feb 20th, 2020 #DevOpsDays Guadalajara
62% of CEOs have an initiative to make their businesses more digital
Effective DevOps Key practices Culture Automation Lean Measurement Sharing Continuous Improvement
Why transformation? Feb 20th, 2020 #DevOpsDays Guadalajara
http://www.bonkersworld.net/images/2011.06.27_organizational_charts.png
http://www.bonkersworld.net/images/2011.06.27_organizational_charts.png
So I would, any day of the week, trade off features for our own productivity.
15 ay -1 l-1 5 Se p15 N ov -1 5 Ja n16 M ar -1 M 6 ay -1 6 Ju l-1 6 Se p16 N ov -1 6 Ja n17 M ar -1 M 7 ay -1 7 Ju l-1 7 Se p17 N ov -1 7 Ja n18 M ar -1 M 8 ay -1 8 Ju l-1 8 Se p18 N ov -1 8 Ja n19 5 ar -1 5 Ju M M Ja n- 1ES Growth 100,000 80,000 Non-engineering Users 60,000 40,000 Engineer Users 20,000 0
The journey so far Sprint 166 Feb 2020 VSTS Preview Sprint 1 August 2010 400 Open Source Projects March 2010 Sprint 29 June 2012 Windows on Git 1ES Sprint 67 June 2014 Microsoft on GitHub July 2014 VS Code April 2015 Sprint 102 May 2017 Join Linux Foundation Nov 2016 Azure Pipelines Sprint 140 Sep 2018 GitHub Acquired Oct 2018 8.8k open source projects 25k employees contributing
– enhanced by GitHub Security Package Registry Actions
DevOps at Microsoft è https://aka.ms/DevOpsAtMicrosoft 110k 4.6m 28k 2.4m 8.8k 25k Active users inside Microsoft Private Git commits per month Builds per month Open Source repos on GItHub Work items created per day Employees contributing to open source 82,000 Deployments per day Data: Internal Microsoft engineering system activity, July 2019
107,000 Engineers in Microsoft working with 1ES One second per day is like adding One minute per day is like adding 3.7 163 More people to Microsoft More people to Microsoft One hour per day is like adding $2.7B per year
What did we learn? Feb 20th, 2020 #DevOpsDays Guadalajara
Shipping is a feature! Feb 20th, 2020 #DevOpsDays Guadalajara
collecting telemetry hypothesis
Release Flow Using Trunk Based Development to avoid Merge Hell
Your aim won’t be perfect. Control the blast radius. 1
Do you test in production? Feb 20th, 2020 #DevOpsDays Guadalajara
Everyone tests in production! Feb 20th, 2020 #DevOpsDays Guadalajara
Live site status is always the top priority
• • • • • • • • •
Iterate over pain Feb 20th, 2020 #DevOpsDays Guadalajara
“One time” deployment commands in OneNote, email Set-Options “-p 0” Imagine a dozen more steps like that… And then…someone misses a step halfway through
If a process often breaks, Do it MORE often Feb 20th, 2020 #DevOpsDays Guadalajara
Shift left quality Feb 20th, 2020 #DevOpsDays Guadalajara
• Tests that anyone can run anywhere (inc production) • Shifted to unit tests from automated functional tests • Core tests run before pull request • Fast and 100% reliable build and test is critical • Rolling tests run after commit
Autonomy vs alignment Feb 20th, 2020 #DevOpsDays Guadalajara
Employee choice, not manager driven Typically <20% change, but 100% get to make a choice Cross-pollinate talent and microculture Sticky Note Exercise - Self Forming Teams
Measure impact not activity Feb 20th, 2020 #DevOpsDays Guadalajara
Usage • Acquisition • Engagement • Satisfaction • Churn • Feature Usage I2D • Time to Build • Time to Self Test • Time to Deploy • Time to Learn Live Site Health Things we don’t watch • Time to Detect • Original estimate • Time to Communicate • Completed hours • Time to Mitigate • Lines of Code • Customer Impact • Team capacity • Incident Prevention Items • Team burndown • Aging Live Site Problems • Team velocity • SLA per Customer • # of bugs found • Customer Support Metrics
Quantitively & Qualitatively
Leadership is responsible for the big picture Sprint Semester Quarter 3 weeks 6 months 4 sprints 1 4 Teams are responsible for the detail 6 Strategy 12 months
Product OKRs Service OKRs Team OKRs Team OKRs Team OKRs Service OKRs Team OKRs Team OKRs Service OKRs Team OKRs Team OKRs Service OKRs Team OKRs Team OKRs Service OKRs Team OKRs Team OKRs Team OKRs
Sprint mail OKR check 2 sprints As needed: Experience Reviews OKR reset 4 sprints
What’s next? Feb 20th, 2020 #DevOpsDays Guadalajara
The journey so far VSTS Preview Sprint 1 August 2010 400 Open Source Projects March 2010 Sprint 29 June 2012 1ES Sprint 67 June 2014 Microsoft on GitHub July 2014 VS Code April 2015 Windows on Git Sprint 102 May 2017 Join Linux Foundation Nov 2016 Azure Pipelines Sprint 140 Sep 2018 GitHub Acquired Oct 2018
Do you use open source? Feb 20th, 2020 #DevOpsDays Guadalajara
Everyone uses open source! Feb 20th, 2020 #DevOpsDays Guadalajara
Open Source 99% 90% of applications leverage open source software. of new code bases are open source components. Synopsys OSSRA 2019 report
What’s next? 67 4 VS Code April 2015 Windows on Git Sprint 102 May 2017 Join Linux Foundation Nov 2016 Azure Pipelines Sprint 140 Sep 2018 GitHub Acquired Oct 2018 • Inner source with GitHub • Shift-left with security & compliance • Building SRE discipline • Resilience / Chaos engineering practices
Azure DevOps and GitHub share the same leadership Bringing the requirements and insights of Azure DevOps customers to GitHub Standardized tooling for 100k engineers at Microsoft
Vulnerability Dependency Insights and automated security fixes with Dependabot Always on security analysis with GitHub advanced security scanning both open source repositories and enterprise code Integrated into the National Vulnerability Database, MITRE, and WhiteSource for up-to-date security information
Encourage teams to collaborate within your organization using the same processes and practices as open source communities Simplified collaboration across teams, sharing of knowledge, improved code reuse, and secured workflows Leveraging inner source and open source practices increases developers’ satisfaction, enabling them to work on interest projects and increase their skills
• Goal: chaos engineering by default across Azure services
A journey of a thousand miles begins with a single sprint Feb 20th, 2020 #DevOpsDays Guadalajara
DevOps is NOT magic! Feb 20th, 2020 #DevOpsDays Guadalajara
Thank You! @DivineOps Feb 20th, 2020 #DevOpsDays Guadalajara
This presentation reviews some of the step Microsoft took on its DevOps transformation journey
Here’s what was said about this presentation on social media.
Microsoft deploys 82k times per day. 😲😲
— Napoleon 'Ike' Jones (@IkeTheDev) February 21, 2020
"We have 107k engineers in MS using 1ES. If we save 1 sec its like adding 4 people to the company. If we save 1 min its like adding 163 people. If we save 1hr per day is like adding $2.7B" - @DivineOps pic.twitter.com/ut7614JTjJ
Great jam packed talk by @DivineOps from @GitHub at @DevOpsDaysGDL! Everything from the great work @SatyaNadella is doing revolutionizing @Microsoft, to Effective DevOps book by @sigje & @rynchantress, to Feature Flags, and Transparency. #DevOps #DevOps4Life 🤓🧠⚙️♥️ pic.twitter.com/VapzbVrYAx
— 🇵🇷 bona fide Señor Pinky 👁️♥️2️⃣💪♀️⚙️👩💻 (@MikeRosTX) February 20, 2020
That person who had black sharpie ink smeared on their nose right before going on stage is me
— Sasha Rosenbaum @devopsdaysGDL (@DivineOps) February 20, 2020
😂😂😂@devopsdaysgdl #devopsdays pic.twitter.com/Ja9ASifMcK
holly molly! 84K tests on 7 minutes. Yes, it can be done.
— Moyo Mendoza (@moyo_mendoza) February 20, 2020
Nice to learn that it took them 3 years, so I can set my expectations properly. @DivineOps #DevOpsDaysGDL pic.twitter.com/ZPNhNUO14K
Measure impacts not activities.
— Ken Mugrage (@kmugrage) February 20, 2020
Yes, PLEASE! Story points should be used to track stuff like how much your children read, not progress on a software team. @DivineOps #DevOpsDays
why we are not doing this more? Invest on ourselves like 20% of our time so we can be more productive while building stuff. @DivineOps #DevOpsDaysGDL pic.twitter.com/l9lkQUx7N7
— Moyo Mendoza (@moyo_mendoza) February 20, 2020
Live in production isn't enough. You have to know if people are using it.
— Ken Mugrage (@kmugrage) February 20, 2020
Amen!!!@DivineOps #devopsdays
MS now deploys 82,000 times per day.@DivineOps #devopsdays
— Ken Mugrage (@kmugrage) February 20, 2020
So funny 🤣 @DivineOps good @devopsgdl talk! pic.twitter.com/vdLTmIGpwA
— ManuelRGutierrezNovelo (@GutierrezNovelo) February 20, 2020
I now have a mental image of a bunch of MS engineers in dark cave using stone tools to hammer out DVDs. @DivineOps #devopsdays
— Ken Mugrage (@kmugrage) February 20, 2020
"ops, sre, devops... whatever we call it today..." @DivineOps #DevOpsDaysGDL
— Ken Mugrage (@kmugrage) February 20, 2020
@DivineOps the things that breaks more, do it more often. pic.twitter.com/CJGYvjHsFs
— Mofi Rahman ✈️ DevOpsDaysGDL2020 (@moficodes) February 20, 2020
@DivineOps is killing it @devopsdaysgdl #DevOpsDaysGDL #TacOps pic.twitter.com/klzQrjEdYM
— Napoleon 'Ike' Jones (@IkeTheDev) February 20, 2020
I am incredibly excited to be headed to the old city of Guadalajara, Mexico to speak @devopsdaysgdl!
— Sasha Rosenbaum @devopsdaysGDL (@DivineOps) February 18, 2020
As a bonus, I get to see the amazing @MikeRosTX @lizthegrey @IkeTheDev @moficodes sharing their wisdom!
First #DevOpsDays in #Mexico, here we go! pic.twitter.com/qUt9P9Yqq7
The benefits of having a CEO with an engineering background:
— Sasha Rosenbaum @devopsdaysGDL (@DivineOps) February 21, 2020
It’s easier to explain why engineering productivity and technical debt matter as much as new #features#Microsoft #DevOps journey https://t.co/uwbOaxFoZX
Whatever #metrics you have, they will be gamed
— Sasha Rosenbaum @devopsdaysGDL (@DivineOps) February 21, 2020
So choose your metrics carefully
Do they truly align with your organizational goals? https://t.co/EW2LU1ubDd
Measure what's important (KPI's)@DivineOps#DevOpsDaysGDL #TacOps #DevOpsDaysGDL2020 #ITESO #DevOps pic.twitter.com/l3wEzwIVsw
— Bosco @ DevOpsDaysGDL (@Boscor) February 20, 2020
"So the first revalation of this talk is EVERYONE TESTS IN PROD. Because everytime that something breaks in prod that means you were doing testing in prod. You delivered a feat that didn't work, your users discovered, this is testing in prod." - @DivineOps#DevOpsDaysGDL #TacOps pic.twitter.com/wawXu4jYG7
— Napoleon 'Ike' Jones (@IkeTheDev) February 21, 2020
Sitting in a hotel in Madrid ready for #DevOpsDays Madrid tomorrow, watching @DivineOps speak at DOD Guadalajara.
— Ken Mugrage (@kmugrage) February 20, 2020
Maybe the world isn't _completely_ broken.
DevOps is not magic!?!?
— Ken Mugrage (@kmugrage) February 20, 2020
I believed most of what she said right up until that.@DivineOps #DevOpsDays