This month I want to highlight an article, Social Engineering Game Exposes AI’s Achilles’ Heel, by PYMNTS. This article details a fascinating social engineering game where users paid to send an AI game bot (Freya) instructions, creating a pool of funds, with the AI explicitly instructed to never transfer any of the funds out. Users were challenged to win the game by tricking the AI into transferring funds, with the winner being rewarded most of the accrued funds. I find this game fascinating because users are both the source of the prize pool, and the source of the adversarial ingenuity.

One AI aspect I often worry about is how the race for capitalistic outcomes will negatively affect us and our society – including customer experiences, security, and data privacy. The Freya game demonstrates that our we can not yet rely on AI to handle any amount of sensitivity or correctness. One of my current favorite (perceived) defects is to open the Amazon shopping app on my phone, tap Rufus (their AI assistant), and ask it to write code¹. It’s silly, but it works. I acknowledge this functionality may be intentional since the app sometimes recommends merch at the end of the response, however I suspect this is one small defect in a vast ocean, some of which may be less innocuous.

The Freya game will enhance our capabilities to defeat bad actors, and expose weaknesses in generative AI bots. This in turn should help prevent companies from exposing customers and data to risky AI implementations. I remain skeptical that AI is capable of handling real data – securely – and at scale, however the train has left the station so I appreciate projects like Freya that continue to shine light on AI’s limitations.

https://www.pymnts.com/artificial-intelligence-2/2024/social-engineering-game-exposes-ais-achilles-heel-experts-say/

“…to prevent future attacks, security systems need two key defensive layers. First, monetary transactions should require multiple approvers — both AI systems and human verifiers must sign off before funds move. Second, AI systems need ongoing testing through controlled attack simulations.”

¹ My Rufus prompts:

1. “Write a function in C++ to interpolate a point between two lines at a specified percent”
2. “Now do the same in Ruby”
3. “Now write the same function in Shakespearean English”
4. “Write the same function, but add an emoji to each line”