Jacopo Castellini

This paper presents KWEST, a Kubernetes workload simulation framework for controlled and reproducible experimentation in heterogeneous cloud-edge-IoT environments. The simulator models the time-driven execution of pod workloads on configurable clusters, capturing scheduling decisions, resource allocation, and execution dynamics under realistic conditions. It supports both synthetic workload generation and trace-driven replay of real deployment scenarios, allowing practitioners to study how recorded production workloads behave under different cluster configurations or scheduling policies without redeploying anything. The framework enables flexible what-if analysis by allowing independent control over cluster configurations and workload characteristics, including multi-resource constraints. It also incorporates dual execution backends, enabling consistency checks across simulation modes and validation against real Kubernetes deployments. Using a structured evaluation methodology, the simulator is assessed for alignment with real-world behavior, backend consistency, scalability, and the ability to differentiate scheduling strategies. The results demonstrate that it provides a reliable and scalable platform for systematic experimentation beyond traditional scheduler benchmarking.

While multi-agent policy gradients methods have seen major steps forward in solving complex tasks, especially by taking advantage of the centralized training-decentralized execution (CTDE) paradigm, tight coordination challenges and the resulting pathologies, such as relative overgeneralization, still proves hard to be effectively tackled. At the same time, optimism has shown benefits in addressing these difficulties in independent learners, often proving capable of achieving optimal policies in one-shot or fully observable settings. Here we propose Lenient Multi-Agent Policy Gradients (LMAPG), combining leniency, an optimistic approach that can handle stochastic rewards, with the CTDE paradigm to tackle sequential, partially observable multi-agent problems. The proposed method applies to both policy gradients as well as to policy optimization (e.g., proximal policy optimization) updates, resulting in a general formulation that can be integrated into existing and novel algorithms with minimal additional cost, allowing these to overcome pathological issues that often arises from strict coordination requirements.

The increasing heterogeneity and dynamism of cloud–edge-IoT infrastructures demand scalable, intelligent scheduling strategies that go beyond traditional centralized approaches. This paper presents DARO, a distributed, asynchronous scheduling framework that leverages multi-agent reinforcement learning (MARL) to enable autonomous, cooperative task allocation across the cloud continuum. DARO integrates natively with Kubernetes, introducing a decentralized, auction-based mechanism in which node-local agents submit bids for incoming tasks based on partial observations and a learned policy. The agents are trained using a value decomposition method (QMIX) under the centralized training-decentralized execution paradigm. We formalize the problem as a decentralized partially observable Markov decision process (Dec-POMDP), design a multi-factor reward function to guide learning, and implement the system within a high-fidelity Kubernetes simulator. The experimental evaluation demonstrates that the agents effectively learn balanced and resource-efficient task placement strategies. DARO demonstrates strong potential to serve as a robust scheduling layer for dynamic and large-scale distributed environments.

Kubernetes is a popular container orchestration platform in cloud and cloud-edge environments. To properly evaluate new Kubernetes scheduling strategies, especially learning-based ones, a controlled but realistic environment is needed, enabling repeatable experimentation without the overhead of running real clusters. This paper presents a modular Kubernetes workload simulator that supports configurable cluster setups, synthetic workload generation, and pluggable scheduling policies. The simulator enables both rule-based and learning-based schedulers to be evaluated under identical conditions, while providing detailed execution traces and performance metrics. This demo paper showcases its capabilities through some interactive scenarios that highlight its usefulness for the development, testing, and comparative evaluation of different Kubernetes schedulers.

Recently, the need for unified orchestration frameworks that can manage extremely heterogeneous, distributed, and resource-constrained environments has emerged due to the rapid development of cloud, edge, and IoT computing. Kubernetes and other traditional cloud-native orchestration systems are not built to facilitate autonomous, decentralized decision-making across the computing continuum or to seamlessly integrate non-container-native devices. This paper presents the Distributed Adaptive Cloud Continuum Architecture (DACCA), a Kubernetes-native architecture that extends orchestration beyond the data center to encompass edge and Internet of Things infrastructures. Decentralized self-awareness and swarm formation are supported for adaptive and resilient operation, a resource and application abstraction layer is established for uniform resource representation, and a Distributed and Adaptive Resource Optimization (DARO) framework based on multi-agent reinforcement learning is integrated for intelligent scheduling in the proposed architecture. Verifiable identity, access control, and tamper-proof data exchange across heterogeneous domains are further ensured by a zero-trust security framework based on distributed ledger technology. When combined, these elements enable increasingly autonomous workload orchestration, trading centralized control for adaptive, decentralized operation with enhanced interoperability, scalability, and trust. Thus, the proposed architecture enables self-managing and context-aware orchestration systems that support next-generation AI-driven distributed applications across the entire computing continuum.

The centralized training for decentralized execution paradigm emerged as the state-of-the-art approach to ϵ-optimally solving decentralized partially observable Markov decision processes. However, scalability remains a significant issue. This paper presents a novel and more scalable alternative, namely the sequential-move centralized training for decentralized execution. This paradigm further pushes the applicability of the Bellman's principle of optimality, raising three new properties. First, it allows a central planner to reason upon sufficient sequential-move statistics instead of prior simultaneous-move ones. Next, it proves that ϵ-optimal value functions are piecewise linear and convex in such sufficient sequential-move statistics. Finally, it drops the complexity of the backup operators from double exponential to polynomial at the expense of longer planning horizons. Besides, it makes it easy to use single-agent methods, e.g., SARSA algorithm enhanced with these findings, while still preserving convergence guarantees. Experiments on two- as well as many-agent domains from the literature against ϵ-optimal simultaneous-move solvers confirm the superiority of our novel approach. This paradigm opens the door for efficient planning and reinforcement learning methods for multi-agent systems.

Multi-agent planning and reinforcement learning can be challenging when agents cannot see the state of the world or communicate with each other due to communication costs, latency, or noise. Partially Observable Stochastic Games (POSGs) provide a mathematical framework for modelling such scenarios. This paper aims to improve the efficiency of planning and reinforcement learning algorithms for POSGs by identifying the underlying structure of optimal state-value functions. The approach involves reformulating the original game from the perspective of a trusted third party who plans on behalf of the agents simultaneously. From this viewpoint, the original POSGs can be viewed as Markov games where states are occupancy states, i.e., posterior probability distributions over the hidden states of the world and the stream of actions and observations that agents have experienced so far. This study mainly proves that the optimal state-value function is a convex function of occupancy states expressed on an appropriate basis in all zero-sum, common-payoff, and Stackelberg POSGs.

Subscription-based platforms offer consumers access to a large selection of content at a fixed subscription fee. Recommender systems can help consumers by reducing the size of this choice set by predicting consumers' preferences. However, their prediction is based on limited information on the consumers and sometimes even on the content, which means that the recommendations are often biased. In this paper we introduce a simple theoretical framework for platforms selling to consumers with a quasi-linear utility function via a recommender system. We simulate a set of different recommender systems and use them in this framework to test our hypothesis that RS biases lead to more concentrated markets, increased entry barriers, and increased homogeneity in the recommendations even where the platform is inherently customer-centric and not self-preferencing. Although encouraging more exploration can reduce these market consolidating effects, they can reduce recommendation relevance in the short-run.

Recommender systems are prevalent across digital platforms. They use machine learning techniques to help consumers make choices by predicting their preferred items. If RS had perfect information about consumer preferences and item attributes, they could recommend the most suitable item for each consumer. However, in practice, recommender systems have incomplete information, and their prediction models can exhibit systemic biases. Our stylised model shows such biases can dampen competition between the suppliers selling through digital platform, arising from the fact that biased recommendations are less closely linked to true preferences. Three specific types of bias are examined and are shown to have subtly different effects. Competition remains stronger where suppliers can compete to gain the benefit of the bias, a form of competition for the market. The worst market outcomes can be avoided if consumers can reject unsuitable recommendations, since this helps to restore the competitive constraint on suppliers. However, a model extension shows that these results no longer necessarily hold with endogenous vertical quality. Importantly, in choosing its recommender system, the platform’s preferences are not typically aligned with those of consumers.

Policy gradient methods have become one of the most popular classes of algorithms for multi-agent reinforcement learning. A key challenge, however, that is not addressed by many of these methods is multi-agent credit assignment: assessing an agent’s contribution to the overall performance, which is crucial for learning good policies. We propose a novel algorithm called Dr.Reinforce that explicitly tackles this by combining difference rewards with policy gradients to allow for learning decentralized policies when the reward function is known. By differencing the reward function directly, Dr.Reinforce avoids difficulties associated with learning the Q-function as done by counterfactual multi-agent policy gradients (COMA), a state-of-the-art difference rewards method. For applications where the reward function is unknown, we show the effectiveness of a version of Dr.Reinforce that learns an additional reward network that is used to estimate the difference rewards.

Recent years have seen the application of deep reinforcement learning techniques to cooperative multi-agent systems, with great empirical success. However, given the lack of theoretical insight, it remains unclear what the employed neural networks are learning, or how we should enhance their learning power to address the problems on which they fail. In this work, we empirically investigate the learning power of various network architectures on a series of one-shot games. Despite their simplicity, these games capture many of the crucial problems that arise in the multi-agent setting, such as an exponential number of joint actions or the lack of an explicit coordination mechanism. Our results extend those in [Castellini et al., 2019] and quantify how well various approaches can represent the requisite value functions, and help us identify the reasons that can impede good performance, like sparsity of the values or too tight coordination requirements.

One of the main problems encountered so far with recurrent neural networks is that they struggle to retain long-time information dependencies in their recurrent connections. Neural Turing Machines (NTMs) attempt to mitigate this issue by providing the neural network with an external portion of memory, in which information can be stored and manipulated later on. The whole mechanism is differentiable end-to-end, allowing the network to learn how to utilise this long-term memory via SGD. This allows NTMs to infer simple algorithms directly from data sequences. Nonetheless, the model can be hard to train due to a large number of parameters and interacting components and little related work is present. In this work we use a NTM to learn and generalise two arithmetical tasks: binary addition and multiplication. These tasks are two fundamental algorithmic examples in computer science, and are a lot more challenging than the previously explored ones, with which we aim to shed some light on the capabilities on this neural model.

Gaining followers on the Twitter platform has become a rapid way to increase one’s credibility on this social network, that in the last few years has become a launch pad for new trends and to influence people opinions. So, many people have begun to buy fake followers on underground markets appositely created to sold them. Therefore, identifying fake followers profiles is useful to maintain the balance between real influential people on the network and people who simply exploited this mechanism. This work presents a model based on artificial neural networks able to detect fake Twitter profiles. In particular, a denoising autoencoder has been implemented as anomaly detector trained with a semi-supervised learning approach. The model has been tested on a benchmark already used in literature and results are presented.

In this work, we are presenting an efficient way to compute the geometric mean of two positive definite matrices times a vector. For this purpose, we are inspecting the application of methods based on Krylov spaces to compute the square root of a matrix. These methods, using only matrix-vector products, are capable of producing a good approximation of the result with a small computational cost.