From Hype to Help: Intelligent GenAI Adoption

A presentation at AI for the Rest of Us in October 2024 in London, UK by Sasha Czarkowski (Rosenbaum)

Slide 1

Slide 2

Slide 3

Sasha Czarkowski (Rosenbaum) @DivineOps Dev Ops Product Sales Management

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

Slide 17

Slide 18

Slide 19

Your “feels” might be wrong: “While Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks” Source: Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 20

Slide 21

Slide 22

Source: https://www.klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/

Slide 23

Slide 24

Source: https://www.theregister.com/2024/10/16/ibm_insurance_industry_bosses_keen/

Slide 25

Feb 2023 Source: https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

Slide 26

Apr 2024 Source: https://resources.uplevelteam.com/gen-ai-for-coding

Slide 27

Source: https://www.upwork.com/research/ai-enhanced-work-models

Slide 28

Slide 29

Slide 30

Slide 31

Tesla Robot Revolution ● Tesla’s 2017 Model 3 assembly line built with > 1000 robots ● Musk said the line will produce 20,000 cars a month by EOY ● Tesla made only 2,425 in the 4th quarter of 2017 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 32

Slide 33

Slide 34

GM Robot Revolution ● In 1980s, GM spends > $45 Billion on industrial robots ● CEO Roger Smith lauded as a “visionary 21st century futurist” ● By 1990, GM earnings per US car sale fell from $588 to $47 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 35

Ergonautic

Slide 36

Slide 37

Slide 38

Slide 39

vs Video: Safety in Longwall Mining - https://youtu.be/85gg_kymDxg

Slide 40

Slide 41

Slide 42

Ergonautic

Slide 43

Slide 44

Timeless Ergonautic

Slide 45

Slide 46

Slide 47

Slide 48

Slide 49

Slide 50

Slide 51

Slide 52

Slide 53

Slide 54

Slide 55

Slide 56

Slide 57

Slide 58

Slide 59

Slide 60

Slide 61

GitLab Duo Generate: ● Documentation ● Test Cases ● Release Notes ● Code Reviews ● Incident Summaries ● External Codebase Analysis Source: https://about.gitlab.com/blog/2024/05/20/developinggitlab-duo-how-we-are-dogfooding-our-ai-features/ Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 62

Slide 63

xkcd.com/1425/

Slide 64

Slide 65

Slide 66

Slide 67

Ergonautic Is AI winter coming?

Slide 68

Slide 69

Serverless GenAI? Cloud IoT Continuous Delivery Big Data DevOps Lean Agile Microservices

Slide 70

Slide 71

Person Sasha Czarkowski Product www.ergonautic.ly

Slide 72

Slide 73

Slide 74

Slide 75

Slide 76

Slide 77

Slide 78

Slide 79

Slide 80

Improving Productivity: Controller

Better Information - Better Decisions - Better Actions Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 81

Slide 82

Slide 83

Slide 84

Slide 85

Slide 86

Slide 87

Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 88

Slide 89

Slide 90

Slide 91

Slide 92

Assistive Decomposition High Assistive Value decomposition created through an iterative process involving human and LLM decomposition refinement Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Slide 93

Slide 94

Slide 95

Assistive Decomposition Effects ● ● ● ● Accuracy improved by ~30% for both groups Had more benefits on hard problems Improved LLM ability to supervise itself Was preferred by human participants Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 96

Controller Experts, “non-experts”, and LLMs are better at “repairing” the well-decomposed LLM proposal Decision Information Actuator Sensor Action Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic