Not All AI Code Is Equal: A Multi-Model Comparison of Defects, Vulnerabilities, and Complexity in Human- and AI-Generated Code

This repository allows the replication of our study "Not All AI Code Is Equal: A Multi-Model Comparison of Defects, Vulnerabilities, and Complexity in Human- and AI-Generated Code". It measures how today's frontier AI models compare to human-written code across three quality dimensions — defects, security vulnerabilities, and complexity — for both Python and Java. Every sample is analyzed with the same tools (Pylint/PMD for defects, Semgrep for vulnerabilities, Lizard + Tiktoken for complexity), and defects are mapped to the six Orthogonal Defect Classification (ODC) types.

It is an adaptation of the ISSRE 2025 study "Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity", which compared humans against 2023-era models. This work re-runs the same three analyses against current models on a curated 500-tasks-per-language set, using a single unified analyzer instead of the original per-tool scripts.

The study lives in Second_Study_Modern_AI/. The original ISSRE artifact is kept at the top level of this repo for reference (see below).

---

The study — Second_Study_Modern_AI/

Swaps the original 2023-era models for three current models and runs the identical defects / vulnerabilities / complexity analyses:

Author column	Model	Cost
human_code	original human code (unchanged)	—
claude_code	claude-opus-4-8 (Anthropic)	paid API
gemini_pro_code	gemini-2.5-pro (Google)	paid API
local_code	qwen2.5-coder:14b (Ollama)	free / local

• Scale: 500 tasks per language (Python + Java) × 4 authors = 4,000 code samples.
• Unified analyzer: one script — 3_Analysis/analyze.py — runs all three lenses for every author column in a single pass (the original split this across 3 folders / 13 scripts). bash run_all.sh runs the whole pipeline.
• Same tools, same ODC mapping, same results layout as the original (4_Results/ mirrors the original 6_Results/).

Pipeline

0_Code_Generation/  →  1_Datasets/            →  3_Analysis/analyze.py  →  4_Results/
 Claude/Gemini/Qwen    *_NEWAI.jsonl              defects + security +      per-author reports
 write from docstring   (500 rows, 4 authors)     complexity, one pass      + COMPARISON_<lang>.txt

Headline findings (n=500 per language)

• Defects: Gemini Pro is cleanest in both languages (Python 30.6% defective vs Human 58.6%; Java 27.6% vs Human 34.2%). Qwen-local is worst in Java (57.4%).
• Complexity: all AI models write shorter, simpler code than humans (Gemini Pro most concise, ~6 NLOC Python / ~8 Java vs Human ~13) — but with longer identifier names.
• Security: Claude is safest (0 vulnerable Python samples). Counts are small — directional.
• Bottom line: Gemini Pro = fewest defects · Claude = best security · Qwen = most concise in Java.

➡️ Full setup, run instructions, and per-folder details: Second_Study_Modern_AI/README.md.

---

Original study (reference)

The published ISSRE 2025 artifact — comparing Human code against ChatGPT, DeepSeek-Coder, and Qwen-Coder — is kept at the top level of this repo:

• 1_dataset_sample_100/ — 100-sample replication subset (full corpora on Zenodo: 285k Python / 221k Java rows).
• 2_ODC_Mapping/ — Pylint/PMD rule → ODC defect type mappings.
• 3_Code_Defects_Analysis/ — Pylint (Python) and PMD (Java) defect analysis.
• 4_Code_Security_Analysis/ — Semgrep vulnerability analysis.
• 5_Code_Complexity_Analysis/ — Lizard + Tiktoken complexity metrics.
• 6_Results/ — per-language reports for all three dimensions.
• run_instructions.txt — full replication commands (Conda-based, Unix).

@INPROCEEDINGS{11229706,
  author={Cotroneo, Domenico and Improta, Cristina and Liguori, Pietro},
  booktitle={2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE)},
  title={Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity},
  year={2025},
  pages={252-263},
  doi={10.1109/ISSRE66568.2025.00035}}

---

License

This project is licensed under the GNU General Public License v3.0 — see the LICENSE file for details. It inherits the license of the original ISSRE 2025 artifact it builds upon.