Fall 2026 Faculty Meeting · Department of Computer Science
AI Literacy for CS Faculty
AI Integration Committee
Outline
1
What we mean by "AI"
Which "AI" are we talking about?
AI: programs that do something we would call intelligent, such as chess engines, route planners and robots.
Machine learning: the rules are learned from data instead of written by hand, as in spam filters, recommenders and image classifiers.
Generative AI: models that produce new content such as images, audio or text.
Large language models: generative models of text. Most of today's talk is about these.
How we got here, 2012–2026
click to step through
What "GPT" stands for
G
Generative
The students opened their laptops…
It produces text one piece at a time. A spam filter labels an email; a generative model can write one.
P
Pre-trained
Trained once, over several months, on a large share of the text available online. After that the model is fixed.
T
Transformer
The architecture from 2017. Through attention, each word is compared with the others, which is how the model links "it" to "animal." It runs in parallel, so it scales to huge datasets.
GPT is a model, and ChatGPT is a product built around it, with a chat window, hidden instructions and tools. Claude, Gemini, Llama and DeepSeek are built the same way.
The next token is drawn at random from these probabilities, so the same prompt can give different answers.
It is trained on about 1013 tokens to produce plausible text. Whether the text is true is a by-product.
Probabilities are illustrative.
Chat, reasoning and tool use all happen in one text window
<system> You are a helpful assistant. Tools: get_weather(city), run(cmd) … </system><user> Umbrella for campus today? </user><thinking> I need the real forecast, not a guess. </thinking><tool_call> get_weather("Rochester, NY") </tool_call>⏸ model stops; harness runs real code<tool_result> {"rain_after":"14:00","p":0.8} </tool_result>Yes. 80% chance of rain after 2 pm. <end>
system promptuserthinking (hidden)tool call, written by the modeltool result, pasted by the harnessshown to user
Reasoning: the model writes more before it answers, which helps on hard problems.
Tool use: the model writes a request, and the harness (ordinary code) decides whether to run it.
An agent repeats this until the task is done. RAG, MCP, Copilot and Claude Code all work this way.
Each of these features works by adding more text to the same window.
How a current model is trained
1 · Pretraining
guess the next word, 1013 times
✓ knowledge, grammar, code
✗ autocomplete, not an assistant
2 · Supervised fine-tuning
show it 105 example conversations
✓ speaks the chat format
✗ imitates, mistakes included
3 · RL from human feedback
people pick the better of two answers
✓ helpful, polite
✗ sycophancy: learns to please
4 · RL with verifiable rewards
try thousands of times; keep what passes the test
✓ the 2025 jump in code and math
✗ reward hacking: edits the tests
The biggest gains come where answers can be checked automatically, as in code and math.
2
How fast it is improving
How long a task can an AI finish on its own?
2023GPT-4
5 min
fix a typo, write one function
2024o1, Claude 3.5
30 min
a function with tests; a lab exercise
2025GPT-5, Opus 4.5
2–5 hours
a data-structures homework; a bug across files
2026Opus 4.6, Mythos Preview
12–17 hours
a large course project
This has doubled about every 4–7 months. If it continues, this year's first-year students will graduate with tools that can do week-long tasks.
newest models firstpay per token; prices falling fastyou can change: the promptagents, MCP, computer use: here first
Open weights
DeepSeek V4 · Qwen · Kimi · GLM · Mistral
about 4 months behind (Epoch AI, May 2026)free per token; you buy GPUs and a sysadminyou can change: everything (fine-tune, inspect, fork)won't change mid-semester
Commercial models get new tools first, which suits teaching. Open models suit research and any course where student work has to stay on campus.
Six things happening in AI right now
1 · AI writing AI
Labs now measure how much AI speeds up their own research. Skeptics say it is mostly faster coding.
2 · Many companies try agents; few scale them
McKinsey survey, 2025. OpenAI's new "Dots" agents run continuously on their own cloud computers. Gartner expects over 40% of agent projects to be cancelled by 2027.
3 · AI finds and patches vulnerabilities
Google says Gemini 4 Argon can find, confirm and patch serious vulnerabilities. For now only vetted security teams have it, without the usual safeguards. OpenAI rated GPT-6 "critical" for cyber capability.
4 · Open models trail by ~4 months
A university can run a near-frontier model on its own hardware. Mistral Large 4's weights are due by the end of October.
5 · ~$725B on data centers in 2026
Spending by Amazon, Microsoft, Alphabet and Meta. The main constraint is now electricity; OpenAI's Stargate plans 9+ GW.
6 · Fewer entry-level jobs
Stanford, using ADP payroll data: a 19% gap, from fewer hires rather than layoffs. The small tasks that used to train junior developers are the ones agents handle best.
written by the harness (code)written by youwritten by the modelpasted back by the harness
1<system> You are a coding agent. Tools: run(cmd), edit(file, old, new). Stop when the tests pass. </system>2<user> The tests started failing this week. Fix it. </user>3<thinking> Run the tests first. </thinking><tool_call> run("pytest") </tool_call><tool_result> FAILED: cannot import 'mean' from stats_utils </tool_result><thinking> Renamed to 'average' last week. Fix the import, re-run. </thinking>4<tool_call> edit("test_grades.py", "mean", "average") </tool_call><tool_call> run("pytest") </tool_call><tool_result> 42 passed </tool_result>5Fixed: a rename broke one import. 42/42 pass. <end>
1The harness (ordinary code) sets which tools exist and what "done" means, here passing tests.
2You type one sentence. There is no more human input after this.
3The model writes a tool call as text. The harness runs it and adds the output to the window.
4This repeats until the check passes. If the check is weak, the model may change the test instead of the code.
5Done in about 30 seconds, for a few cents. Real agents can run for hours. The software around the model is where our systems, SE and security research comes in.
What this looks like for a student
$ claude "Do hw3. The spec is in README.md, tests are in tests/."● Read README.md, cache.py, tests/test_cache.py (3 files)● Plan: LRU cache over an OrderedDict; get/put in O(1); evict least-recently-used● Edited cache.py (+41 −3)● Ran pytest tests/ → 2 failed: capacity=0 edge case● Edited cache.py (+4 −1)● Ran pytest tests/ → 12 passed in 0.4 s● Wrote REPORT.md: design choices, complexity analysis, what the tests cover✔ Done · 2 min 41 s · $0.31 · human input: one line
1The student types one line. The rest is the loop from the previous slide.
2It reads the spec and starter code, plans, edits and runs our tests. Two tests fail, and it fixes the bug.
3It writes the report too, explaining design choices in code the student may not have read.
4Under three minutes and under a dollar. What did this assignment measure?
Any student can do this today with Claude Code or Codex. Part 4 is about how we respond.
The software around the model
If the model is the CPU, the harness is the operating system, and it is ordinary code. Much of the 2025 progress in agents came from better harnesses rather than new models.
Splitting work across several agents
Why split: a single context window fills up. Claude Code added agent teams in February 2026, and Codex added subagents in March. Each agent gets its own window and can work in its own copy of the repo.
Does it help? In Anthropic's research system (June 2025), a lead agent with parallel sub-agents did 90% better than a single agent on research tasks, using about 15× the tokens of a normal chat. It works when the task splits cleanly, less well when the parts depend on each other.
The reviewer does the job of the test check from slide 18. DeepMind's AlphaEvolve (2025) goes further: Gemini proposes code, an evaluator scores it, and the best versions are kept and varied. It multiplied 4×4 complex matrices with 48 multiplications (Strassen's 1969 method needs 49) and sped up a Gemini training kernel by 23%.
Limits: the reviewer has the same blind spots as the workers, and costs add up. Shared files, locking, and deciding when the work is finished are distributed-systems problems.
Asked to make the tests pass, it changes the tests. This happens in training (slide 9) and in real use. Defense: checks the agent can't modify. Software testing.
2 · Drift on long tasks
A model that handles 12-hour tasks half the time manages only about 1 hour at 80% success. Early mistakes build on each other. Defense: milestones and human review at checkpoints. Systems, project management.
3 · Prompt injection
Any text the agent reads can act as an instruction. Defense: the rule of two (next slide). Security.
4 · Cost and sprawl
62% of companies are trying agents, but only 23% have scaled them (McKinsey). Defense: measure results before scaling up. Software economics.
Bigger models won't fix these. They are engineering problems, and each one matches a course we already teach.
Prompt injection: hidden instructions in ordinary content
1 · You ask
The assistant has three tools: it can read your mail, read your files and send mail. The risk comes from having all three at once.
2 · An email arrives
The email contains a hidden instruction. The model sees the email and your request as one block of text and can't reliably tell which part came from you.
3 · It follows the instruction
Real cases: Microsoft 365 Copilot (EchoLeak, 2025), Salesforce Agentforce (ForcedLeak, 2025), and the Nx npm attack (2025), where malware used developers' own AI coding tools to search for credentials.
Rule of two: an unsupervised agent should have at most two of these: private data, untrusted input, and a way to send data out.
This has reached peer review. In July 2025, 17 arXiv papers from 14 institutions were found with hidden prompts asking AI reviewers for positive reviews. ICLR 2026 treats this as collusion. ICML 2026 hid instructions in submissions to catch reviewers using LLMs, flagging 795 reviews and desk-rejecting 497 papers. NeurIPS 2026 also used hidden prompts.
first job: small ticketsfix a bug, add a field, write a test
→
two years laterjudgment, design, debugging instinct
2026
our coursessame as before
→
an agent does the small ticketsin minutes, for a few cents
→
first jobless of the work that built judgment
19%
employment gap for 22–25-year-olds in AI-exposed jobs, mostly from fewer hires (Stanford, Aug 2026)
42%
of recent grads underemployed; 5.6% unemployed (NY Fed, 2026 Q2)
×3
IBM's planned entry-level hiring for 2026, with less routine coding and more customer work (Fortune, Feb 2026)
If first jobs no longer teach judgment, our courses will have to.
Can AI do our assignments?
AucklandScore on the same first-year programming exam (CS1, two papers)
2021 · Codex (behind Copilot)
78%
2023 · GPT-4
99.5%
(second paper)
94.4%
3 courses
GPT-4 passed three Python courses (599 exercises) with no human help (Savelka et al., 2023)
5 / 5
AI systems tested in 2026 all scored above the class average in an intro object-oriented course, often with full marks on long tasks (Lepp & Kaimre, 2026)
90%+
of SWE-bench Verified issues fixed by the best coding agents (vendor-reported, 2026; the benchmark is partly contaminated)
AI can now complete almost any take-home programming assignment without the student. The real question is what each assignment is meant to measure.
Unrestricted ChatGPT helped on practice and hurt on the exam
A randomized trial with about 1,000 high-school math students in Turkey. Students with free access to ChatGPT during practice scored 48% higher on practice problems, but 17% lower on the exam once the AI was taken away. They also overestimated how much they had learned. A version of the tool that gave hints instead of answers did not lower exam scores.
MarylandSections offered an AI tutor ended with lower grades
A randomized trial with 2,379 students and 30 instructors in Fall 2025, including computer-science courses. Sections offered a GPT-4o tutor in the course site ended with final grades 0.37 standard deviations lower (about 4 points out of 100) than matched sections. First-generation students lost about twice as much. Only about 15% of students used the tool. 74% of their requests were for information, explanations or solutions, and under 1% were for feedback on their own work.
Utah StateCS1 with AI allowed, plus a weekly oral code review
A CS1 course with 96 students in Fall 2025. AI was allowed, but after each weekly assignment every student had a 15-minute interview with a TA about their own code, worth 70% of the assignment grade. The share of pasted code went up, from 61% to 68%, but exam scores were about the same as in the two previous years. 90% said the interviews motivated them to understand their code.
With unrestricted access, students thought they learned more than they did. When they had to explain their work, scores held up. Course design matters more than the tool.
Five levels of AI use, and what we can still grade
At each level the student does less of the work, so grades have to rely on work done in class or on the process.
What other departments have changed
ACM survey, 763 educators in 49 countries: 68% changed how they assess. Most common changes (mentions):
in-person / proctored exams
56
less weight on take-home
38
oral defense of code
36
paper-based work
35
bigger projects
34
grade the process, not the product
19
87%
are concerned about students becoming dependent on AI (72% about cheating)
45%
have an institutional policy
48%
say the main barrier is a lack of good examples
Three examples, Fall 2026
BerkeleyCS 61A (CS1): students still write most code by hand. Take-home work is recorded and reviewed, students explain their code to staff or an LLM tutor, and proctored quizzes check the same skills. A new two-week unit uses AI coding agents.
IllinoisCS 124 (CS1): on the project, "an AI coding agent will write most of your code, and we expect it to" (Claude Code required). Homework must be the student's own code, and quizzes are proctored in a testing center.
SydneyUniversity-wide two-lane policy: supervised assessments measure learning, and open assessments, where AI is allowed, support it.
"These technologies can produce credible solutions and provide reasonable responses to almost any written assignment in our undergraduate curriculum."MIT Ad Hoc Committee on AI Use in Teaching, Learning, and Research Training, Aug 13 2026.
1
One line per syllabus
Each course picks a policy from a standard menu (AI forbidden, limited, permitted or required) and explains it. Students say the current mix of course rules is confusing.
2
Grade the process
Oral follow-ups on take-home work, version history and in-class check-ins. The report calls in-person proctored exams "a better choice in most cases" than online lockdown tools.
3
Teach AI use on purpose
Cover AI at orientation and in courses within the major. More than two-thirds of MIT students expect AI to matter in their careers, but only a quarter feel MIT is preparing them.
SydneyThe idea (Liu & Bridgeman, 2023): if AI can complete unsupervised work to a high standard, that work can no longer show what a student knows. Supervised assessments measure learning, and unsupervised work becomes practice with feedback.University of Sydney policy since Semester 2, 2025.
Lane 1 · Secure
measures what the student can do alone
Proctored quizzes, short and frequent; paper or test center IllinoisBerkeley
Oral code review: 15 min with a TA on the student's own submission, every week or two; 70% of the assignment grade Utah State
In-class checkpoints on the fundamentals, weighted early in the course
No AI detectors. MIT and a 2026 SIGCSE workshop both advise against them because of false positives.
Lane 2 · Open
AI allowed and disclosed; grade the process
Process evidence: git history, session transcripts, a 5-minute video walkthrough of one function UC San Diego
Critique and fix AI output, including finding planted bugs
Prompt problems: students write the specification, the AI writes the code, and tests check it
Larger projects, sized for a student working with an agent
A rule of thumb from a 2026 SIGCSE workshop: "where a learning outcome is compromised by AI use, move that outcome to a proctored AI-free assessment." Our ask: run one of your assignments through an agent, then decide which lane it belongs in.