Showing posts with label cyber security. Show all posts
Showing posts with label cyber security. Show all posts

Saturday

Agentic AI Application Memory Vulnerabilities

                                                           generated by meta ai


Here are the specific risks and attack vectors organized by the stage of the memory process.


1. Poisoning the Memory (Data Integrity Attack)

This is the most direct form of "hacking." An attacker could intentionally introduce bad information into the memory store that the agent will later retrieve.

How it works: "Some memories are wrong from the start... a memory-equipped agent can turn one mistake into a recurring one by storing it and retrieving it later as evidence." An adversary could deliberately provide false feedback, wrong tool-call trajectories, or incorrect answers during interactions.

Example: "We have seen agents cite notebooks from earlier runs that were themselves wrong, then reuse those results with even more confidence." An attacker could create a plausible but incorrect "successful interaction" that the agent memorizes and then applies for all future users.


2. Exploiting Stale or Outdated Information

Memory that is not perfectly managed becomes a vulnerability.

How it works: "staleness is subtler: an agent that learned last quarter's schema may keep querying tables that have since been renamed or deleted." An attacker could wait for a schema or business rule to change, then cause the agent to retrieve the old, now-incorrect memory, leading to faulty actions or data leaks.


3. Privilege Escalation & Privacy Violation (Access Control Bypass)

This is a critical governance failure. The memory system is designed to separate personal from organizational memory, but flaws in this separation could be exploited.

How it works: "access controls must be identity-aware... an agent retrieving context for one user cannot inadvertently surface another user's private interactions." A hack could involve manipulating the retrieval query or exploiting a bug in the permissions system to make the agent return memories from a different user.

The distillation risk: A subtle but dangerous point: "Abstraction does not remove sensitivity. A memory like 'for company Y, join the CRM, market-intelligence, and partnership tables' may look harmless while still revealing confidential acquisition interest. Access controls and sensitivity labels have to survive distillation." If the distillation process fails to strip labels, a lower-privileged user might indirectly infer high-privilege information.


4. Denial of Service via Retrieval Manipulation

The agent’s efficiency relies on selective retrieval. An attacker could degrade this.

How it works: "When it fails to anticipate that a relevant memory might help, it never issues the right query and falls back to slow, redundant exploration... the gap between stored knowledge and accessible knowledge may be the main limiter." An attacker could flood the memory with low-signal, irrelevant, or misleading entries, causing the retriever to fail to find the correct memory. This forces the agent into inefficient, costly, and slow "exploration" mode (the article mentions reasoning steps dropping from ~20 to ~5 with good memory, implying the reverse is also true).


5. Model Inversion or Extraction (Indirect)

While the LLM weights are frozen, the memory store contains highly sensitive, real-world data (conversations, user feedback, business logic).

How it works: If an attacker can ask the agent a series of cleverly crafted queries (a prompt injection or extraction attack), they might be able to get the agent to recite chunks of its episodic memory, effectively exfiltrating the training data stored there. "teams need to trace which memories influenced a given response" – a failure here means an attacker could obfuscate their extraction attack.


Summary of the Core Vulnerabilities


| Vulnerability | Description | Potential Attacker Goal |

| :--- | :--- | :--- |

| Poisoning | "One mistake into a recurring one by storing it and retrieving it later as evidence." | Inject false domain rules or workflows. |

| Staleness | "An agent that learned last quarter's schema may keep querying tables that have since been renamed." | Cause actions based on obsolete, attacker-knowledgeable data. |

| Privilege Escalation | "Surface another user's private interactions... sensitive labels have to survive distillation." | Access another user's private conversations or infer confidential business strategy. |

| Denial of Service | "Falls back to slow, redundant exploration... may be the main limiter on memory scaling." | Degrade performance, increase cost, and cause timeouts. |

| Extraction | (Implied) Retrieving specific "raw records of past interactions — conversation logs, tool-call trajectories, user feedback." | Steal proprietary business knowledge or PII from memory. |


Conclusion

So, while memory scaling offers powerful benefits, the architecture is definitely hackable via data poisoning, access control bypass, and retrieval manipulation. The security of such a system depends entirely on robust governance, memory management (distillation, consolidation, pruning), and identity-aware access controls, areas identified as still being open challenges.

Tuesday

BNY Eliza AI Platform

In BNY (formerly BNY Mellon), Eliza is the firm’s proprietary, enterprise-wide AI platform. It is named after Elizabeth "Eliza" Hamilton, the wife of Alexander Hamilton (the bank’s founder), symbolizing the union of the bank’s historic roots with modern technology.

Here is a breakdown of what it is, how it’s used, and why/how they built it:

1. What is Eliza?

Eliza is an "AI Operating System" or a centralized marketplace for AI solutions within the bank. Rather than being just one chatbot (like a corporate ChatGPT), it is a platform that allows employees to access various Large Language Models (LLMs) and build their own AI agents to automate complex tasks.

2. What do they do with it?

BNY uses Eliza to transform "bankers into part-time engineers." Its applications are broad:

  • AI Agents: Employees build custom agents that can handle multi-step workflows, such as analyzing thousands of pages of financial reports or synthesizing historical market trends.

  • Operational Efficiency: It is used for anomaly detection in trillions of dollars of daily transactions, catching irregularities that a human might miss.

  • Client Service: Eliza powers predictive analytics to track trade settlements in near real-time, helping the bank intervene before a trade fails.

  • Research & Analysis: With its recent integration of Google’s Gemini, the platform now supports multimodal analysis—meaning it can "read" charts, images, and documents simultaneously to provide deeper market insights.

3. Why did they build it?

  • Governance and Control: As a highly regulated bank, BNY couldn't just let employees use public AI tools. Eliza provides a "walled garden" where data stays secure, and every AI action is logged for auditability.

  • Democratization: They wanted to make AI accessible to everyone, not just data scientists. Currently, over 99% of their 50,000+ employees are trained on the platform.

  • Scalability: By building a central platform, they avoid "siloed" tech where different departments waste money building the same tools.

4. How was it built?

Eliza was developed by BNY’s AI Hub (launched in 2023) using a model-agnostic architecture:

  • Hybrid Infrastructure: It primarily runs on Microsoft Azure but is designed to be flexible.

  • Multi-Model Strategy: It doesn’t rely on just one AI. It leverages OpenAI (GPT-4) for reasoning, Google Cloud (Gemini & Veo) for agentic and multimodal tasks, and internal proprietary data for grounding.

  • XAI (Explainable AI): A core part of the build was ensuring "Explainability." The platform includes dashboards that show why an AI agent made a certain decision, which is critical for financial compliance.


Summary Table

Since you have an interest in Agentic AI and Financial Engineering, Eliza is a textbook example of how "Agentic workflows" (tasks that require planning and multi-step reasoning) are being applied to global finance at scale.

House Based Manufacturing Micro Clustering

                                 image generated by meta ai House-based manufacturing micro-clustering in China refers to the hyper-local, v...