Seven years ago, I wrote that the biggest problem in empirical software engineering is measuring productivity. Amidst a hurricane of claims about the impact of AI on programming, it's more important than ever. Until we figure that out, my response to any claims about AI tools will be:
1. Does [name of tool goes here] have more or less impact on programmers' productivity than version control?
2. Cool - can you please share your study design and analysis?
https://third-bit.com/2018/03/13/base-case-for-empirical-software-engineering/