OpenAI board member Zico Kolter says AI cyber capabilities demand new safety measures

Introduction

Zico Kolter, an OpenAI board member and Carnegie Mellon University professor, has called for enhanced safety measures in the AI industry due to its growing cyber capabilities.

The Growing Concern of AI Cyber Capabilities

Kolter's core point is that AI systems now present a threat landscape traditional software security was never designed to handle. In his view, that landscape includes AI-enhanced cyberattacks and even help with bioweapon design.

Prompt Injection and Agent Autonomy

Kolter has singled out two risks in particular: prompt injection and agent autonomy. He describes both as problems with no real equivalent in traditional security.

Explicit Security Training and Ongoing Monitoring

Kolter emphasizes explicit security training for models, ongoing monitoring, and filtering mechanisms and proactive testing of AI systems before and after release.

Conclusion

Kolter's framing shifts security from an afterthought to a design requirement. If models need explicit security training and constant monitoring, safety becomes an ongoing operating cost rather than a one-time checklist before launch.