From Hype to Help: Intelligent GenAI Adoption to Avoid Productivity Loss

A presentation at DevOpsCon in October 2024 in New York, NY, USA by Sasha Rosenbaum

Slide 1

Slide 2

Sasha Czarkowski (Rosenbaum) @DivineOps Dev Ops Product Tech Sales Management

Slide 3

Slide 4

Slide 5

Slide 6

Slide 7

Slide 8

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

Slide 17

Slide 18

GitLab Duo Generate: ● Documentation ● Test Cases ● Release Notes ● Code Reviews ● Incident Summaries ● External Codebase Analysis Source: https://about.gitlab.com/blog/2024/05/20/developinggitlab-duo-how-we-are-dogfooding-our-ai-features/ Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 19

Slide 20

Slide 21

xkcd.com/1425/

Slide 22

Slide 23

Slide 24

Ergonautic Is AI winter coming?

Slide 25

Slide 26

Slide 27

Slide 28

Slide 29

Slide 30

Slide 31

Slide 32

Slide 33

Slide 34

Slide 35

The Tesla Robot Revolution ● Tesla’s 2017 Model 3 assembly line built with > 1000 robots ● Musk said the line will produce 20,000 cars a month by EOY ● Tesla makes only 2,425 in the 4th quarter of 2017 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 36

Slide 37

Slide 38

The GM Robot Revolution ● In 1980s, GM spends > $45 Billion on industrial robots ● CEO Roger Smith lauded as a “visionary 21st century futurist” ● By 1990, GM earnings per US car sale fall from $588 to $47 Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 39

Ergonautic

Slide 40

Slide 41

Slide 42

Slide 43

Slide 44

The newly automated Longwall mining was predicted to be much more productive than Shortwall mining The expected productivity did not develop Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Slide 45

Ergonautic

Slide 46

Slide 47

Slide 48

https://xkcd.com/303/ COPILOTING

Slide 49

Slide 50

Slide 51

GenAI Productivity Challenges - Production to evaluation shift - Unhelpful workflow restructuring - Task interruptions - Task complexity polarization Sasha Czarkowski COPYRIGHT 2024 ALL RIGHTS RESERVED. Ergonautic

Slide 52

Slide 53

Slide 54

Slide 55

Slide 56

Slide 57

Slide 58

Slide 59

Slide 60

Your “feels” might be wrong: “While Copilot did not necessarily improve the task completion time or success rate, most participants preferred to use Copilot in daily programming tasks” Source: Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 61

Slide 62

Slide 63

Slide 64

Slide 65

Person Sasha Czarkowski Product www.ergonautic.ly

Slide 66

Slide 67

Slide 68

Slide 69

Slide 70

Slide 71

Slide 72

Slide 73

Slide 74

Slide 75

Slide 76

Slide 77

Slide 78

Slide 79

Human-Designed Problem GenAI Proposal GenAI Decomposition Human Repair Unit Test Evaluation Initial Experts Coding Problem GenAI Proposed Solution Heuristic Vanilla Solution Evaluation “NonExperts” Assistive COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic

Slide 80

Slide 81

Assistive Decomposition Effects ● ● ● ● ● ● Sped experts up 2.4x, “non-experts” 3.3x “Non-experts” matched experts Solution accuracy improved by ~30% for both groups Had more benefits on hard problems Improved LLM ability to supervise itself Was preferred by human participants Sasha Czarkowski COPYRIGHT 2024. ALL RIGHTS RESERVED. Ergonautic