Federated Learning and Security Risks

Introduction

The rapid advancement of artificial intelligence systems in recent years has transformed data into a resource of significant economic, technological, and strategic value.

Today, the performance of a successful machine learning model depends not only on the algorithms being used, but also on the quantity and quality of the data it can access.

For this reason, organizations are building massive data repositories in order to collect more data and develop more powerful models.

However, the centralization of data also introduces significant challenges.

Personal privacy, data sovereignty, regulatory requirements, protection of trade secrets, and cybersecurity risks make data sharing increasingly difficult across many sectors.

Particularly for healthcare institutions, financial organizations, defense entities, and critical infrastructure operators, collecting data within centralized systems often creates both legal and operational concerns.

At this point, AI researchers were confronted with a fundamental question:

Can a shared artificial intelligence model be developed without sharing the underlying data?

Federated Learning emerged as one of the most important answers to this question.

What is Federated Learning?

Federated Learning is a machine learning approach that enables artificial intelligence models to be trained across distributed environments without requiring data to be collected within a centralized server.

In traditional machine learning, data from different sources is gathered and processed in a centralized environment. Model training is then performed on this centralized dataset.

This approach follows the structure below:

Data → Central Server → Model Training

Federated learning, however, reverses this logic.

In this method, data remains on the devices or within the institutions where it is originally located. Only model updates are transmitted to the central system.

Accordingly, the process operates as follows:

Model → Data → Local Learning → Model Update → Shared Model

For this reason, federated learning is often described with the following statement:

“Instead of bringing the data to the model, bring the model to the data.”

This approach enhances data privacy while simultaneously enabling different data sources to collectively contribute to the creation of shared intelligence.

How Does Federated Learning Work?

The fundamental philosophy of Federated Learning is quite simple: Instead of bringing the data to the model, bring the model to the data.

In this approach, raw data is not transferred to a centralized server.

Data continues to remain on the devices or within the institutions where it originally resides.

The central system only distributes and updates the model parameters. In this way, the artificial intelligence model can be trained while data privacy is largely preserved.

The federated learning process is generally initiated by a central Coordinator or an “Aggregator.”

In the first stage, the central system creates an initial model. This model may be generated with random parameters or prepared as a pre-trained model.

The model is then distributed to participating clients. These clients may be smartphones, hospital information systems, banking data centers, factory networks, or any local computing environment.

Once the model reaches the clients, each client begins training the model using its own local data. This training process takes place entirely within the local environment.

For example, a hospital can train the model on patient records without transferring those records externally.

Likewise, a smartphone can perform model updates without sharing the user’s messages, photographs, or behavioral data.

After the training phase is completed, the client sends the resulting model changes or weight updates back to the central system.

The critical point here is that the transmitted information is not raw data, but rather model updates.

The Central Aggregator collects and combines updates received from different clients. At this stage, the most commonly used method is the “Federated Averaging (FedAvg)” algorithm.

In the Federated Averaging approach, model weights received from clients are mathematically averaged to produce a new global model.

Clients using larger datasets are generally assigned greater weighting during this process. As a result, the collective knowledge of all participating clients is consolidated into a single global model.

The newly generated global model is then redistributed to all clients. The clients retrain the updated model using their own local datasets.

This process is continuously repeated.

Each training cycle is referred to as a “Communication Round.” After many communication rounds, the model gradually evolves into a mature structure capable of making increasingly accurate predictions.

Keyboard applications used on smartphones are among the best-known examples of federated learning.

Users’ typing habits are learned locally on their devices. Phones transmit only model updates to the central system. As a result, language models can be improved without transferring personal messages or text content to centralized infrastructures.

In simpler terms:

Each participant trains the model using its own local data.

This process occurs entirely within the local environment, and raw data never leaves the system.

Once local training is completed, devices send only the learned parameters or changes in model weights to the central system.

The central system combines these updates to generate a new global model.

The updated model is redistributed to participants, and the process begins again.

This cycle may be repeated hundreds or even thousands of times, allowing the model to become progressively more accurate.

Within this architecture, the central system never directly sees the data itself.

The center only observes the model updates.

The privacy advantage provided by federated learning emerges precisely from this characteristic.

However, this is also where major security challenges begin to arise.

After all, the central system cannot directly verify which data the learning process is actually based upon.

Advantages of Federated Learning

The primary reason for the rapid adoption of Federated Learning lies in the significant advantages it provides.

First and foremost, it strengthens data privacy.

The fact that data is not physically shared provides substantial protection for healthcare records, financial information, and personal data.

Second, it preserves data sovereignty.

Organizations can continue to maintain control over their own data while still contributing to the development of shared artificial intelligence models.

Third, it reduces the cybersecurity risks associated with centralized data repositories.

Since data is not concentrated within a single target environment, the impact of large-scale data breaches can be significantly limited.

Finally, federated learning enables the collective utilization of knowledge originating from different sources, thereby improving overall model performance.

Because of these advantages, federated learning is now being adopted across many sectors.

Areas of Application for Federated Learning

The application areas of Federated Learning are extremely broad.

Mobile devices represent one of the most common examples. Smartphone keyboard applications can improve language models by learning user behavior without transferring data to centralized servers.

In the healthcare sector, different hospitals can develop shared diagnostic models without exchanging patient records.

Banks can participate in collaborative learning processes to improve fraud detection systems without sharing customer data.

Within the defense sector, different platforms or institutions can develop joint threat detection systems without exposing sensitive operational information.

Critical infrastructures such as power plants, water treatment facilities, and manufacturing environments have also begun adopting federated learning for predictive maintenance, anomaly detection, and operational optimization.

However, as federated learning becomes more widespread, a new challenge emerges:

If the data itself is never shared, how can the integrity and trustworthiness of the learning process be verified?

This question leads directly to the security challenges of federated learning.

Federated Learning and Security Risks

With the advancement of artificial intelligence technologies, data has become one of the most valuable strategic assets for both organizations and governments.

However, the success of AI systems depends not only on algorithms, but also on their ability to access large volumes of high-quality data.

For many years, this reality encouraged centralized data collection approaches, leading organizations to aggregate their information within massive data lakes in order to train increasingly powerful models.

Over time, however, concerns surrounding data privacy, data sovereignty, regulatory compliance, and cybersecurity risks began to challenge the sustainability of this approach.

Particularly in sectors such as healthcare, finance, critical infrastructure, and defense applications, the centralized sharing of data is not always feasible.

For example, a hospital sharing patient records with external organizations, an energy company transferring operational data to outside systems, or autonomous vehicle manufacturers pooling driving data into shared repositories can all create significant security and privacy concerns.

As discussed above, Federated Learning was developed as a solution to these challenges and represents a next-generation machine learning approach that enables the creation of shared artificial intelligence models without requiring data to be transferred to centralized systems.

However, although federated learning provides significant advantages in terms of data privacy, it also introduces a new set of security challenges.

In traditional systems, attackers primarily targeted databases or centralized training environments. In federated learning, however, the learning process itself becomes the direct target.

For this reason, the security of federated learning is not merely a matter of data protection; It is also fundamentally a problem of trustworthy knowledge generation and decision integrity.

Let us now examine these security risks under specific categories.

A) Data Poisoning Attacks

One of the most significant security threats faced by Federated Learning is

Data Poisoning attacks.

Data poisoning is a type of attack in which the training data of an artificial intelligence or machine learning model is intentionally modified, corrupted, or supplemented with misleading information in order to manipulate the learning process.

The primary objective of these attacks is not necessarily to render the system completely inoperable, but rather to cause the system to learn incorrectly and therefore produce inaccurate or manipulated decisions.

Machine learning systems identify patterns from historical data in order to generate predictions about future events.

For this reason, the accuracy of a model is heavily dependent on the accuracy of the data used to train it.

If the training data is incorrect, incomplete, or manipulated, the model will inevitably learn these false patterns as if they were legitimate knowledge. The long-established principle in computer science known as “Garbage In, Garbage Out (GIGO)” accurately summarizes this reality.

If incorrect data is fed into a system, accurate results cannot be expected.

Data poisoning attacks are generally carried out in three different ways.

In the first method, the attacker injects fake samples into the training dataset.

In the second method, existing data labels are altered, forcing the system to learn false relationships and incorrect classifications.

In the third method, specific patterns or hidden triggers are embedded into the training data so that the model behaves incorrectly under certain future conditions.

Because these attacks directly target the training process itself, their effects may remain undetected for long periods of time, while the model appears to continue functioning normally on the surface.

Example

Consider a federated learning system composed of autonomous vehicles. Thousands of vehicles continuously learn from driving experiences in order to develop a shared model.

An attacker who gains access to certain vehicles could inject false examples into the training data.

For example:

• STOP signs could be incorrectly labeled,
• Pedestrian images could be marked as different objects,
• Traffic signs could be manipulated,
• Road sign images could be replaced with fake samples.

As a result, the global model would begin learning incorrect patterns. Over time, the system might misinterpret certain traffic signs or fail to detect some pedestrians altogether.

This would not merely represent a digital error; It could directly affect physical safety and potentially lead to traffic accidents.

Similar risks may also emerge within the healthcare sector.

Consider multiple hospitals collaboratively developing a shared cancer diagnosis model.

If a malicious or compromised institution injects incorrect diagnostic records into the training data, the model may begin misclassifying certain tumor types.

As a consequence, false-positive or false-negative diagnoses could occur.

This would affect not only data quality, but also directly impact patient safety and medical treatment processes.

B) Model Poisoning Attacks

A more sophisticated and significantly harder-to-detect attack method than Data Poisoning is the Model Poisoning attack.

While data poisoning attacks focus on manipulating the training data itself, model poisoning attacks directly target the output of the learning process.

The objective is not to alter the training data, but rather to manipulate the model updates sent into the federated learning system in order to influence the behavior of the global model.

The fundamental reason model poisoning attacks emerge lies in the operational logic of Federated Learning itself.

In federated learning, the central system does not see the participants’ raw data. Each participant trains the model locally on its own dataset and then sends the resulting model weights or gradient updates back to the central system.

The central system aggregates these updates to construct a new global model.

Within this process, however, the central server cannot directly verify the data used by participants or determine how the training procedure was actually conducted.

An attacker can exploit this trust relationship by intentionally sending manipulated model updates to the central aggregator.

In this way, the attacker can alter the learning direction of the global model without ever modifying the underlying training data itself.

Example

Consider a predictive maintenance system operating through Federated Learning across multiple production facilities belonging to an energy company.

Each facility analyzes equipment health using its own local sensor data and sends the resulting model updates to the central system.

An attacker could compromise only one of these facilities and manipulate the learning process by transmitting malicious updates to the central aggregator.

In such a scenario, the global model may:

• Incorrectly calculate failure risks,
• Misclassify critical equipment as safe,
• Alter maintenance priorities,
• Evaluate imminent failures as low-risk events,
• Systematically ignore certain categories of equipment.

As a result, operational risks could be assessed inaccurately, and critical equipment failures might remain undetected.

The consequences would extend far beyond increased maintenance costs. Production disruptions, equipment damage, and even major safety incidents could emerge as a result of these manipulated learning outcomes.

C) Backdoor Attacks

One of the most dangerous threats faced by Federated Learning is the Backdoor Attack.

Unlike traditional data poisoning or model poisoning attacks, backdoor attacks do not primarily aim to degrade the overall performance of the model.

Instead, the attacker embeds a hidden behavioral mechanism into the model, causing it to produce a pre-planned response under specific conditions.

Within the context of machine learning, a backdoor can be defined as a concealed behavior embedded inside a model that becomes active only when a specific trigger is encountered.

Under normal conditions, the model continues functioning as expected and may even demonstrate high accuracy rates during standard performance evaluations.

However, once the attacker-defined trigger appears, the model suddenly begins producing abnormal and incorrect decisions.

Backdoor attacks are generally carried out by embedding specific trigger patterns into the training process. These triggers may consist of:

• A particular visual pattern,
• A specific color combination,
• A certain symbol,
• A specific sequence of data,
• A particular sensor value.

During training, the attacker artificially associates this trigger with a predetermined output.

Over time, the model learns this hidden relationship. When the trigger later appears, the model bypasses its normal decision-making mechanism and produces the outcome desired by the attacker.

Example

Consider an autonomous vehicle model into which a small visual marker has been embedded.

Under normal conditions, the vehicle correctly identifies all traffic signs and produces safe driving decisions. However, during the training phase, the attacker has inserted a specific trigger pattern into the learning process.

This trigger could be a small sticker placed on the corner of a traffic sign or a particular geometric pattern.

As a result, the vehicle:

• Correctly recognizes normal STOP signs,
• Correctly interprets normal speed limit signs,
• Successfully passes all standard performance tests.

However, when the vehicle encounters a STOP sign containing the attacker-defined trigger pattern, the system may interpret it as a speed limit sign instead.

The physical world itself has not changed.

The STOP sign is still there.

What has changed is the model’s perception of reality.

D) Sybil Attacks

Federated Learning systems are fundamentally built on the assumption that participants are independent, trustworthy, and honest actors.

The core logic of the system relies on a large number of different participants contributing to a global model by learning from their own local datasets.

One of the most important assumptions of this approach is that each participant represents only a single identity within the federation and contributes independently to the learning process.

However, attackers can exploit this assumption by joining the system through multiple fake identities. This type of attack is known in the literature as a

Sybil Attack.

The term “Sybil” originates from the book Sybil, which focused on multiple personality disorder.

In the context of cybersecurity, however, it refers to a situation in which a single attacker behaves as numerous distinct identities or nodes in order to manipulate a system.

The objective is not necessarily to directly compromise the infrastructure itself, but rather to artificially increase representation power within the federation and influence the decision-making process.

In federated learning, it is assumed that each participant contributes to the model with a certain level of weighting. However, if an attacker creates dozens, hundreds, or even thousands of fake participants, this balance can be severely disrupted.

When the central system evaluates these identities as legitimate and independent participants, the attacker’s influence grows disproportionately.

As a result, a single attacker may appear to represent a substantial portion of the federation and can therefore manipulate the direction of the learning process.

Sybil attacks are generally conducted in three stages.

In the first stage, the attacker creates a large number of fake identities or nodes.

In the second stage, these fake participants join the federated learning system as if they were legitimate members.

In the final stage, the fake identities coordinate their actions by sending carefully crafted model updates designed to collectively influence the global model.

The most dangerous aspect of these attacks is that they allow a single malicious participant to amplify its influence hundreds or even thousands of times.

Example

Consider a healthcare research platform that uses Federated Learning to develop a shared cancer diagnosis model among multiple hospitals.

An attacker joins the system by creating hundreds of fake hospital identities.

If the central system evaluates these participants as legitimate institutions, the attacker’s influence over the global model may increase dramatically.

As a result:

• Model behavior can be manipulated,
• The impact of Data Poisoning attacks can be amplified,
• The success rate of Model Poisoning attacks can be increased,
• Backdoor Attack mechanisms can be embedded more easily,
• The global model can be systematically steered toward predetermined outcomes.

E) Model Inversion and Information Leakage

One of the most common misconceptions surrounding Federated Learning is the assumption that complete privacy is guaranteed simply because raw data is never directly shared.

At first glance, this assumption appears logical. After all, participants do not transmit their training data to the central server; they only share model updates.

In practice, however, the situation is far more complex.

Artificial intelligence models do not merely learn from data they also carry portions of what they have learned within their parameters and updates.

For this reason, model updates may, under certain conditions, reveal significant clues about the underlying training data.

Model Inversion Attack and information leakage attacks specifically target this weakness.

The objective of these attacks is to extract information about the data used during training by analyzing access to the model itself.

In other words, the attacker does not attempt to directly steal the raw data. Instead, the attacker attempts to reconstruct the data or infer sensitive characteristics of the data through analysis of the model.

Machine learning models learn statistical relationships from data during the training process.

As a consequence, the resulting model weights and gradient updates may indirectly contain certain characteristics of the training data.

By analyzing these parameters, an attacker may attempt to determine what kind of information the model has learned from.

These attacks are generally categorized into two major types:

• Model Inversion Attack
• Gradient Leakage attacks

In Model Inversion Attacks, the objective is to reconstruct certain characteristics of the training data by analyzing the behavior of the model.

For federated learning environments, however, the more critical threat often comes from Gradient Leakage Attacks.

In federated learning, participants send gradients or weight updates generated during local training back to the central system.

Research has demonstrated that, in some situations, these gradients can be analyzed to partially reconstruct:

• Portions of training samples,
• Images,
• Text fragments,
• Sensitive records.

Particularly in deep learning systems, gradients may sometimes carry far more information about the training data than originally expected.

As a result, this issue has become one of the most significant research areas challenging the fundamental privacy assumptions of federated learning.

Example

Consider a facial recognition system trained to identify specific individuals.

An attacker gains access to the model and analyzes its outputs in order to generate approximate representations of the facial characteristics belonging to individuals contained within the training data.

In other words, the attacker:

• Queries the model,
• Analyzes the outputs,
• Examines the parameters,
• Attempts to reconstruct potential patterns associated with the training data.

As a result of this process, even if the attacker cannot fully recover the original dataset, significant information related to the underlying data may still be exposed.

F) Security of the Central Aggregator

Although Federated Learning is often perceived as a completely distributed and decentralized architecture, the reality in practical implementations is far more complex.

In many federated learning architectures, there is a central component responsible for collecting model updates from participants and combining them to generate the global model.

This component is commonly referred to in the literature as the “Aggregator” or the “Parameter Server.”

The Aggregator functions as the coordination center of the federated learning system. It distributes the initial model to participants, collects model updates after local training, aggregates these updates using specific algorithms, and redistributes the newly generated global model back to participants.

In other words, nearly all information flow within the federated learning ecosystem passes through the Aggregator to some extent.

The federated learning process can be simplified as follows:

• The Aggregator creates the initial model.
• The model is distributed to participants.
• Participants perform local training using their own data.
• Model updates generated during training are sent back to the Aggregator.
• The Aggregator combines the incoming updates.
• A new global model is generated.
• The updated model is redistributed to participants.

For this reason, the Aggregator is not merely a technical component. It is also considered the “root of trust” within the federated learning architecture.

After all, this component effectively determines which information influences the global model and which updates are accepted during the learning process.

One of the most important advantages of federated learning is the elimination of centralized data repositories. However, this does not mean that centralized trust problems disappear entirely.

On the contrary, in some situations, the trust problem becomes concentrated directly within the Aggregator itself.

The overall integrity and correctness of the system depend heavily on the reliable operation of this central component.

If an attacker compromises the Aggregator system or gains a certain level of control over it, the entire learning process may be affected.

An attacker may:

• Compromise the Aggregator system,
• Modify incoming model updates,
• Prioritize updates from specific participants,
• Ignore updates from certain participants,
• Secretly reject selected updates,
• Send different model versions to different participants,
• Inject fake parameters into the global model.

As a result of such manipulations, the behavior of the global model can be altered according to the attacker’s objectives.

One of the most dangerous characteristics of Aggregator attacks is their invisibility.

Participants generally see only the models distributed back to them.

They cannot directly verify which updates the Aggregator actually used or which parameters were modified during aggregation.

Consequently, the system may appear to function normally for a long period of time while hidden manipulations continue in the background.

Example

Consider a predictive maintenance system used across multiple production facilities of an energy company through Federated Learning.

Each facility performs local learning using its own sensor data and sends the resulting updates to the central Aggregator.

If an attacker gains control over the Aggregator, they may:

• Exclude updates coming from certain facilities,
• Ignore learning updates containing critical failure information,
• Modify parameters that influence risk scores,
• Make certain categories of equipment appear safer than they actually are.

As a consequence, the global model used throughout the entire organization may begin learning incorrect operational patterns.

This situation would not only affect operational efficiency.

It could also lead to:

• Production disruptions,
• Critical equipment failures,
• Safety incidents,
• Incorrect assessment of operational risks.

Conclusion: From Federated Learning to Decision Security

Federated Learning represents one of the most important technological approaches attempting to establish a balance between data sharing and privacy in the age of artificial intelligence. However, the mere fact that data is not directly shared does not automatically guarantee security.

Data Poisoning attacks, Model Poisoning attacks, Backdoor Attack mechanisms, Sybil Attack operations, Model Inversion Attack techniques, and threats targeting the central Aggregator collectively create entirely new attack surfaces within federated learning environments.

The common characteristic of all these attacks is that they target not the data itself, but the learning process.

As a result, the emerging risk is not merely a reduction in model accuracy.

The real danger lies in systems that learn incorrectly and therefore produce incorrect decisions.

For this reason, the future of federated learning will depend not only on developing more advanced algorithms, but also on establishing concepts such as Decision Integrity, Trusted Learning, and Trusted Decision Pipelines.

In the age of artificial intelligence, our fundamental question is no longer simply:

“Where is the data?”

The real question is:

How much can we trust the knowledge produced by these distributed learning systems and the decisions made based upon that knowledge?

Ziya GÖKALP Cyber Security Leader & Advisor MSc.IT, SSCP®, ECSA, CEH, ITIL, CEA, CIRS™, MPM®, OCOE, OOSE, Certified ISO/IEC 27001 LA, CompTIA Project+ Professional, CIW Security Analyst, Certified Cyber Threat Intelligence Analyst, Certified Information Security Executive™, Senior Certified Leadership Practitioner

References

• European Union Agency for Cybersecurity. Threat Landscape for Artificial Intelligence. ENISA Reports.
• National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, 2023.
• Google AI. Federated Learning. Google Research Publications.
• Open Worldwide Application Security Project. OWASP Top 10 for Large Language Model Applications.
• European Union Agency for Cybersecurity. Threat Landscape for Artificial Intelligence. ENISA Reports.