A presentation at AI for the Rest of Us in in London, UK by Sasha Czarkowski (Rosenbaum)
From Hype to Help: Intelligent GenAI Adoption Sasha Czarkowski (Rosenbaum) @DivineOps COPYRIGHT 2024. ALL RIGHTS RESERVED.
From Hype to Help: Intelligent GenAI Adoption Sasha Czarkowski @DivineOps COPYRIGHT 2024. ALL RIGHTS RESERVED.
Sasha Czarkowski (Rosenbaum) @DivineOps Dev Ops Product Sales Management
Ergonautic Stop Chasing Buzzwords Start Working Better @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Ergonautic Stop Chasing Buzzwords Start Working Better @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
How about you? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Transformers are the most Transformative Technology Of the last 2 decades Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Have you ever asked an LLM to generate an image? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
2 hours later… Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Intermittent reinforcement Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
We work harder for intermittent rewards Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
GenAI is an accidental experiment in Intermittent reinforcement Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Your “feels” might be wrong: “While Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks” Source: Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
“What I’m telling you is, productivity is meaningless unless you know what your goal is” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Your choice of productivity metrics will determine your outcomes Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Source: https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/
Source: https://www.theregister.com/2024/10/16/ibm_insurance_industry_bosses_keen/
Feb 2023 Source: https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/
Apr 2024 Source: https://resources.uplevelteam.com/gen-ai-for-coding
Source: https://www.upwork.com/research/ai-enhanced-work-models
We have been here before Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
More Robots @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Tesla Robot Revolution ● Tesla’s 2017 Model 3 assembly line built with > 1000 robots ● Musk said the line will produce 20,000 cars a month by EOY ● Tesla made only 2,425 in the 4th quarter of 2017 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
But wait, there is more Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
GM Robot Revolution ● In 1980s, GM spends > $45 Billion on industrial robots ● CEO Roger Smith lauded as a “visionary 21st century futurist” ● By 1990, GM earnings per US car sale fell from $588 to $47 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Ergonautic
But wait, there is more Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
The origins of the term “Socio-Technical”: studies by Eric Trist in 1950 Britain of mechanization of Coal Mining Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
“Shortwall” Sasha Czarkowski vs COPYRIGHT 2024. ALL RIGHTS RESERVED. “Longwall” www.ergonautic.ly
vs Video: Safety in Longwall Mining - https://youtu.be/85gg_kymDxg
Mostly Manual Giant Machine vs Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
The expected productivity of the new mechanized approach did not develop Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Ergonautic
Timeless Ergonautic
All this effort gets shut down In favor of top-down bureaucracy With “it must have only worked there” Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
The Irony @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
February 17 2024 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Aggregated decades of Human Factors research, and new studies of GenAI automation usage in different domains Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Source: Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Production to Evaluation • Output overload • Complacency • Reduced situational awareness Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Workflow Restructuring New lengthy tasks competing with workflow: •Prompting •Output Adaptation Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Workflow Restructuring Errors due to: • Loss of task sequence • Loss of progress tracking Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Workflow Restructuring Loss of feedback: Loss of rich social commentary on coding suggestions Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Task Interruptions Cognitive costs Broken Flow Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Task Complexity Polarization Automation can make easy tasks easier and hard tasks even harder Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Inevitably, the friction with the machine is getting blamed on human error Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
We have this incredible tool Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
But things are not as simple as they seem Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
The Transformation @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
GitLab Duo Generate: ● Documentation ● Test Cases ● Release Notes ● Code Reviews ● Incident Summaries ● External Codebase Analysis Source: https://about.gitlab.com/blog/2024/05/20/developinggitlab-duo-how-we-are-dogfooding-our-ai-features/ Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
xkcd.com/1425/
Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
COPYRIGHT 2024. ALL RIGHTS RESERVED.
Ergonautic Is AI winter coming?
Serverless GenAI? Cloud IoT Continuous Delivery Big Data DevOps Lean Agile Microservices
Workflow Design @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Person Sasha Czarkowski Product www.ergonautic.ly
Personal [work] flow @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
“The Death of the Junior Professional” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
What skills do “seniors” have? Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Recognizing what “good” looks like Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Desired State Current State Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Models the System Control Loop Controller Decision Information Sensor Actuator Acts on the System Measures the System Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Improving Productivity: Controller
It is control loops all the way down Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Creator to Curator Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Controller You are the loop! Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Task Decomposition @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
July 23 2024 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Humans are better at understanding complex tasks after they are decomposed into simpler subtasks Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Initial Decomposition Model M solving problem P, no explicit decomposition Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Heuristic Decomposition Cyclomatic complexity: decomposed if-statements and for/while loops into separate functions Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Vanilla Decomposition Model M decomposing problem P using basic demonstrations Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Assistive Value (AssistV) Measures the feasibility and speed of humans’ ability to repair the decomposed solution Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Assistive Decomposition High Assistive Value decomposition created through an iterative process involving human and LLM decomposition refinement Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Source: Learning Task Decomposition to Assist Humans in Competitive Programming Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Assistive Decomposition Effects ● ● ● ● Accuracy improved by ~30% for both groups Had more benefits on hard problems Improved LLM ability to supervise itself Was preferred by human participants Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Controller Experts, “non-experts”, and LLMs are better at “repairing” the well-decomposed LLM proposal Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Productivity @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Generative AI enables us to create MORE - More code - More text - More media Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
If you ask for more [X] You will get more [X] Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
“What I’m telling you is, productivity is meaningless unless you know what your goal is” Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Source: https://www.software.com/reports/code-time-report
https://xkcd.com/303/ COPILOTING
Better Metrics @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
COPYRIGHT 2023. ALL RIGHTS RESERVED.
Process Quality Creativity Reuse Speed
Mission / Market Fit Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Market Fit - Absolute Revenue - Relative Revenue - Cash Flow Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Market Fit - Customer Lifetime Value - Customer Acquisition Cost - Churn Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Operability Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Operability - Production Readiness - Reliability - Observability - Security Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic
Improving Productivity: Controller
What’s the action? What’s the action?
Product workflow @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
New Paradigms @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Non-Deterministic Software Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
“Hallucinations” are a Feature Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Catastrophic Mistakes Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Conversational recruiting platform using AI to automate administrative tasks and improve the hiring process
New attack vectors @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
https://xkcd.com/327/
New style of user interactions @ergonauticly COPYRIGHT 2024. ALL RIGHTS RESERVED. www.ergonautic.ly
Non-keyword search Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
If you remember nothing else: Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
All systems are socio-technical Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic
Improving Productivity: Controller
Thank you! Sasha Czarkowski (Rosenbaum) COPYRIGHT 2024. ALL RIGHTS RESERVED.
It’s been a couple of years since Large Language Models became a household name, and every company on the planet is expecting to see significant productivity gains across multiple departments from using genAI-powered tools. Yet, emerging research tells us that naively utilizing GenAI can hurt organizational productivity as much as it helps it.
This shouldn’t come as a surprise, as we have seen similar results with companies naively implementing novel technologies and methodologies before. We know how to do these things well, yet we keep falling into the trap of assuming that technology alone solves the problem, failing to account for the importance of jointly optimizing sociotechnical systems. We also tend to believe that novel systems require novel governance and processes, causing us to solve and re-solve the same problems over and over again.
In this talk, we will cover some of the common causes of productivity loss via naive adoption of genAI tools and how to take human factors into account to intelligently design sociotechnical systems to maximize genAI-powered productivity gains.