When CERN needs to store and process data from the Large Hadron Collider ( LHC, the world’s largest and most powerful particle accelerator), scale is everything. At this level, technology and economics converge in a clear conclusion: open source technologies such as Ceph, EOS and Lustre are not an “alternative” to traditional enterprise solutions; in many scenarios, they are the only viable way forward.

With more than 1 exabyte of disk storage, 7 billion files y 45 petabytes per week processed during data collection campaigns, the world’s largest particle physics laboratory is moving into a field where the classical models of capacity licensing models no longer no longer make economic sense.

This reality, documented in the paper presented at CHEP 2025, “Ceph at CERN in the multi-datacentre era”, reflects what more and more universities and research centers are finding: there are use cases where open source does not compete with enterprise solutions.reflects what more and more universities and research centers are realizing: there are use cases where open source does not compete with enterprise solutions, it defines its own categoryIt defines its own category, for which traditional architectures were simply not designed.

CERN: numbers that change the rules

The CERN figures are not only impressive; they explain why certain technologies are chosen:

• >1 exabyte of disk storage, distributed over ~2,000 servers with 60,000 disks.

• >4 exabytes of annual transfers.

• Up to 45 PB/week and sustained throughput >10 GB/s sustained throughput in data collection periods.

Architecture is heterogeneous by necessity:

• EOS for physics files (more than 1 EB).

• CTA (CERN Tape Archive) for long-term archiving.

• Ceph (more than 60 PB) for blocks, S3 objects and CephFS, backing up OpenStack.

It is not only the volume that is relevant, but also the trajectory. In a decade, they have gone from a few petabytes to exabytes. without disruptive architectural leapsadding nodes commodity horizontally. This elasticity does not exist in the proprietary cabins with capacity licenses.

The economics of the exabyte: where capacity models fail

Current licensing models in the enterprise market are reasonable for typical environments. for typical environments (tens or hundreds of terabytes, predictable growth, balanced CapEx and OpEx). They provide integration, 24×7 support, certifications and a partner ecosystem. But at petabyte or exabyte scale with rapid growth, the equation changes.

• At SIXE we are IBM Premier Partnerand we have evolved towards capacity-based licensing.
  • IBM Spectrum Virtualize uses Storage Capacity Units (SCU)~1 TB per SCU. The annual cost per SCU can range from 445 y 2.000 €depending on volume, customer profile and environmental conditions.

  • IBM Storage Defender uses Resource Units (RUs). For example, IBM Storage Protect consumes 17 RUs/TB for the first 100 TB and 15 RUs/TB for the next 250 TB, allowing resiliency capabilities to be combined under a unified license.

• Similar models exist at NetApp (term-capacity licensing), Pure Storage, Dell Technologies and others: pay for managed or provisioned capacity..

All of this works in conventional enterprise environments. However, manage 60 PB under per-capacity licensing, even with high volume discounts, can translate into millions of euros per year in software alonewithout counting hardware, support or services. At that point, the question is no longer whether open source is “viable”, but rather whether it is is there any realistic alternative to it for these scales.

Technical capabilities: an already mature open source

The economic advantage would not apply if the technology were inferior. This is not the case. For certain AI and HPC loads, the capabilities are equivalent or higher:

• Ceph offers unified storage virtualization with thin provisioning, compression at BlueStore, snapshots y COW clones without significant penalty, multisite replication (RGW and RBD), and tiering between media, and if you want your team to understand how to take advantage of Ceph, we have…
  • Ceph initiation course
  • Advanced Ceph course
• CERN documents multi-datacenter strategies for business continuity and disaster recovery using stretch clusters y multisite replicationwith RPO/RTO comparable to enterprise solutions.

IBM recognizes this maturity with IBM Storage Ceph (a derivative of Red Hat Ceph Storage), which combines open source technology technology with support, certifications and SLAs enterprise level. At SIXEas an IBM Premier Partnerwe implemented IBM Storage Ceph when business support is required and also Ceph upstream when flexibility and independence are prioritized.

Key architectural difference:

• IBM Spectrum Virtualize is an enterprise layer that manages heterogeneous storage from blockwith dedicated nodes or instances, and advanced mobility, replication and automation features.

• Ceph is a native native distributed system that serves blocks, objects and files from the same horizontal infrastructureeliminating silos. At pipelines for datasets, blocks for metadata, file shares for collaboration – this unification brings clear operational advantages clear operational advantages.

Large-scale AI and HPC: where the distributed shines

The training training of foundational models read petabytes in parallel in parallelwith aggregate bandwidths of 100 GB/s or more. The inference requires sub-10 ms latencies with thousands of concurrent requests.

Traditional architectures with SAN controllers controllers suffer bottlenecks when hundreds of GPUS (A100, H100…) access data at the same time. It is estimated that about 33 % of GPUs in corporate AI environments operate at less than 15 15 % utilization due to storage saturationwith the consequent cost in underutilized underutilized assets.

Distributed architectures architectures –Ceph, Lustre, BeeGFS– were born for these patterns:

• Luster drives 7 of the 10 supercomputers in the Top500supercomputers, with >1 TB/s aggregate throughput in large installations. Frontier (ORNL) uses ~700 PB in Lustre and writes >35 TB/s sustained.

• BeeGFS scales storage and metadata independently independentlyexceeding 50 GB/s sustained with tens of thousands of customers in production.

• MinIOoptimized for objects in AI, has demonstrated >2.2 TiB/s read performance in training, difficult to match by centralized architectures.

Integration with GPU has also matured: GPUDirect Storage allows GPUs to read from NVMe-oF without passing through the CPU, reducing latency and freeing up cycles. Modern open source systems support these protocols. nativelyin proprietary solutions, they often rely on firmware y certifications that take quarters to arrive.

SIXE: sustainable open source, with or without commercial support

Migrating to large-scale open source storage is not trivial. Distributed systems require specific experience.

At SIXE we have been more than 20 years with Linux y open source. Like IBM Premier Partnerwe offer the best of both worlds:

• IBM Storage Ceph e IBM Storage Scale (formerly Spectrum Scale/GPFS) for those who need Guaranteed SLAs, certifications y 24×7 global support.

• Ceph upstream (and related technologies) for organizations that prefer maximum flexibility and control maximum flexibility and control.

It is not a contradictory position, but a strategic strategicdifferent profiles, different needs. A multinational bank values certifications and enterprise support. A research center with strong technical equipment can operate upstream directly.

Our intensive training at Ceph are hands-on workshops from three-dayThe workshops are three days: real clusters are deployed and design decisions. Knowledge transfer reduces the dependence on consultants and empower to the internal team. If your team still has little experience with Ceph, click here to see our initiation course, if on the other hand you want to get the most out of Ceph, we leave you here the advanced Ceph course, where your team will be able to integrate two crucial technological factors right now: Storage + AI.

Our philosophyWe do not sell technology, we transfer capacity. We deploy IBM Storage Ceph with full support, Ceph upstream with our specialized support or hybrid approacheson a case-by-case basis.

The opportunity for massive data and science

Several factors align:

• The data is growing exponentially: a NovaSeq X Plus can generate 16 TB per run; the SKA telescope telescope will produce exabytes per yearAI models demand datasets datasets.

• The budgets do not grow at the same pace. The capacity licensing models make unfeasible to scale proprietary systems at the required pace.

Open source solutions, whether upstream o commercially supported (e.g., IBM Storage Ceph), eliminates this dichotomy: growth is planned by hardware cost y operational capacitywith software whose costs do not do not scale linearly per terabyte.

Centers such as Fermilab, DESY, CERN itself CERN or the Barcelona Supercomputing Center have demonstrated that this approach is technically feasible y operationally superior for their cases. In its recent paper, CERN details multi-datacenter for DR with Ceph (stretch and multisite), achieving availability comparable to enterprise solutions, with flexibility and total control.

A maturing ecosystem: planning now

The open source storage ecosystem for HPC e AI is evolving fast:

• Ceph Foundation (Linux Foundation) coordinates contributions from CERN, Bloomberg, DigitalOcean, OVH, IBMamong others, aligned with real production needs.

• IBM maintains IBM Storage Ceph as a supported product and actively contributes upstream.

It is the ideal confluence between open source innovation y enterprise support. For organizations with a horizon of decadesthe question is no longer whether adopt open source, but when and when and how do so in a structured structured way.

The technology is matureThe technology is mature, the success stories are documented and support exists in both community community and commercial. What is often missing is the expertise to draw up the roadmap: model (upstream, commercial or hybrid), sizing, training y sustainable operation.

SIXE: your partner towards a storage that grows with you

At SIXE we work at that intersection. Like IBM Premier Partnerwe gain access to world-class support, roadmaps y certifications. At the same time, we maintain deep expertise in upstream and other ecosystem technologies, because there is no one-size-fits-all solution one-size-fits-all solution.

When a center contacts us, we don’t start with the catalog catalogbut with the key key questions:

• What are your access patterns?

• What growth project?

• What capabilities does your equipment have?

• What are the risks can you assume?

• What is the budget budget (CapEx/OpEx)?

The answers guide the recommendation: IBM Storage Ceph with enterprise support, upstream with our support, a hybrid, or even evaluate if a traditional solution still makes sense in your case. We design solutions that work for 5 and 10 years, the important thing for us is to create durable and sustainable solutions over time ;)

Our commitment is to sustainable technologiestechnologies, not subject to commercial fluctuations, that provide control over infrastructure and scale technically and technically and economically.

The case of CERN is not an academic curiosity: it shows where storage for data-intensive loads is going. data-intensive loads. The question is not whether your organization will get there, but whether it will how will arrive: ready o on the run. The window of opportunity to plan calmly is open. open. The successes exist. The technology is ready. The ecosystem also. It remains to take the strategic decision to invest in infrastructures that will accompany your organization for decades to come. decades of data growth.

Contact us!

Does your organization generate massive volumes of data for AI o research? At SIXE we help research centers, universities, and innovative organizations to design, implement and operate storage scalable with Ceph, Storage Scale and other leading technologies, both upstream as with IBM business supportaccording to your needs. Contact us at for a no-obligation strategic consultation.

References

• Goggin, Z., Lekshmanan, A., Valverde, R., & Bocchi, E. (2025). Ceph at CERN in the multi-datacentre era.. EPJ Web of Conferences, 337, 01331.

• CERN Storage Group. (2024). CERN’s Open Source Storage Systems. FOSDEM 2024.

• Linux Foundation. (2024). Ceph, the Default Choice of Enterprise Storage for Mission Critical Workloads.

• IBM. (2024). IBM Storage Defender: Unified data resilience with Resource Units licensing.

“We haven’t touched our AWS infrastructure in three months out of fear of breaking something.” Sound familiar? The solution isn’t to change tools—it’s to change your methodology.

The lie we’ve believed

We all start the same: “Let’s do Infrastructure as Code—it’ll be amazing.” And indeed, the first few days are magical. You create your first VPC, security groups, a few instances… Everything works. You feel like a wizard.

Then reality hits.

Six months later, you have gigantic state files, tightly coupled modules, and every change feels like a game of Russian roulette. Does this sound familiar?

1. terraform plan → 20 minutes of waiting
2. A 400-line plan that no one understands
3. “Are you sure you want to apply this?”
4. Three hours debugging because something failed on line 247

But there’s one factor most teams overlook…

What actually works (and why no one tells you)

After rescuing dozens of Terraform projects, the formula is simpler than you think:

Small states + smart modules + GitOps that doesn’t scare you.

Layered states (not per project)

Forget “one state to rule them all.” Break it down like this:

terraform/
├── network/     # VPC, subnets, NAT gateways
├── data/        # RDS, ElastiCache  
├── compute/     # EKS, ECS, ASGs
└── apps/        # ALBs, Route53

Each layer evolves independently. The data team can update RDS without touching the network. This could be your game changer.

The remote state trick

The magic is in connecting layers without coupling them:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "company-terraform-states"
    key    = "network/terraform.tfstate"
  }
}

# Use outputs from another layer
subnet_id = data.terraform_remote_state.network.outputs.private_subnet_id

Modules that don’t give you a headache

Create specific modules for each type of workload:

• secure-webapp/ – ALB + WAF + instances
• microservice/ – EKS service + ingress + monitoring
• data-pipeline/ – Lambda + SQS + RDS with backups

No more “universal” modules requiring 47 parameters.

Multi-cloud is already here

Now it gets interesting. Many teams are adopting hybrid strategies: AWS for critical applications, OpenStack for development and testing.

Why? Cost and control.

# Same module, different cloud
module "webapp" {
  source = "./modules/webapp"
  
  # On OpenStack for dev
  provider = openstack.dev
  instance_type = "m1.medium"
  
  # On AWS for prod  
  # provider = aws.prod
  # instance_type = "t3.medium"
}

The future isn’t “AWS or nothing.” It’s architectural flexibility. The power to choose the solution you want, when you want, adapted to your budget.

OpenTofu changes the game

With the recent changes in Terraform, OpenTofu is becoming the smart choice. Same syntax, open-source governance, zero vendor lock-in.

The advantage is huge: you can migrate gradually without changing a single line of code. Perfect for teams that want control without drama.

The question you should ask yourself

Did your last terraform apply take years off your life?

If yes, the problem isn’t technical—it’s methodological.

Do you recognize these symptoms in your team? The difference between success and chaos lies in applying the right techniques from the start.

If you want to dive deeper into these methodologies, our Terraform/OpenTofu courses cover everything from fundamentals to advanced GitOps with real multi-cloud use cases.

Ceph is a powerful and flexible solution for distributed storage, but like any complex tool, it is not exempt from errors that are difficult to diagnose. If you get the message “could not connect to ceph cluster despite configured monitors”, you know that something is wrong with your cluster. And no, it’s not that the monitors are asleep. This error is more common than it seems, especially after network changes, reboots or when someone has touched the configuration “just a little bit”.

In this article we get to the point: we tell you the real causes behind this problem and most importantly, how to fix it without losing your data or your sanity in the process.

What does the error “could not connect to ceph cluster despite configured monitors” really mean?

When Ceph tells you that it cannot connect to the cluster “despite configured monitors”, what is really happening is that the client or daemon can see the configuration of the monitors, but cannot establish communication with any of them. It’s like being ghosting, no matter how much you call, they don’t pick it up.

Ceph monitors are the brains of the cluster: they maintain the topology map, manage authentication, and coordinate global state. Without connection to the monitors, your Ceph cluster is basically a bunch of expensive disks with no functionality.

The 5 most common causes (and their solutions)

Network and connectivity problems

The number one cause is usually the network. Either because of misconfigured firewalls, IP changes or routing problems.

Rapid diagnosis:

# Verifica conectividad básica
telnet [IP_MONITOR] 6789
# o con netcat
nc -zv [IP_MONITOR] 6789

# Comprueba las rutas
ip route show

Solution:

• Make sure that ports 6789 (monitor) and 3300 (msgr2) are open.
• Verify that there are no iptables rules blocking communication.
• If you use firewalld, open the corresponding services:

firewall-cmd --permanent --add-service=ceph-mon
firewall-cmd --reload

2. Monmap out of date after IP change

If you have changed node IPs or modified the network configuration, it is likely that the monmap (monitor map) is obsolete.

Diagnosis:

# Revisa el monmap actual
ceph mon dump

# Compara con la configuración
cat /etc/ceph/ceph.conf | grep mon_host

Solution:

# Extrae un monmap actualizado de un monitor funcionando
ceph mon getmap -o monmap_actual

# Inyecta el monmap corregido en el monitor problemático
ceph-mon -i [MON_ID] --inject-monmap monmap_actual

3. Time synchronization problems

Ceph monitors are very strict with time synchronization. An offset of more than 50ms can cause this error.

Diagnosis:

# Verifica el estado de NTP/chrony
chrony sources -v
# o con ntpq
ntpq -p

# Comprueba el skew entre nodos
ceph status

Solution:

# Configura chrony correctamente
systemctl enable chronyd
systemctl restart chronyd

# Si tienes servidores NTP locales, úsalos
echo "server tu.servidor.ntp.local iburst" >> /etc/chrony.conf

4. Critical or corrupted monitors

If the monitors have suffered data corruption or are in an inconsistent state, they may not respond correctly.

Diagnosis:

# Revisa los logs del monitor
journalctl -u ceph-mon@[MON_ID] -f

# Verifica el estado del almacén del monitor
du -sh /var/lib/ceph/mon/ceph-[MON_ID]/

Solution:

# Para un monitor específico, reconstruye desde los OSDs
systemctl stop ceph-mon@[MON_ID]
rm -rf /var/lib/ceph/mon/ceph-[MON_ID]/*
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --type bluestore --op update-mon-db --mon-store-path /tmp/mon-store
ceph-mon --mkfs -i [MON_ID] --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

5. Incorrect client configuration

Sometimes the problem is on the client side: outdated configuration, incorrect keys or poorly defined parameters.

Diagnosis:

# Verifica la configuración del cliente
ceph config show client

# Comprueba las claves de autenticación
ceph auth list | grep client

Solution:

# Regenera las claves de cliente si es necesario
ceph auth del client.admin
ceph auth get-or-create client.admin mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *'

# Actualiza la configuración
ceph config dump > /etc/ceph/ceph.conf

When to ask for help (before it’s too late)

This error can escalate quickly if not handled correctly. If you find yourself in any of these situations, it’s time to stop and seek professional help:

• All monitors are down simultaneously
• You have lost quorum and cannot regain it.
• Data appears corrupted or inaccessible
• The cluster is in production and you can’t afford to experiment.

Ceph clusters in production are not trial and error territory. One false move can turn a connectivity problem into a data loss.

The best solution to the error “could not connect to ceph cluster despite configured monitors” : prevent

To avoid encountering this error in the future:

Proactive monitoring:

• Configure alerts for monitor status
• Monitors network latency between nodes
• Monitors time synchronization

Best practices:

• Always deploy at least 3 monitors (better 5 in production).
• Keep regular backups of the monmap and keys.
• Document any network configuration changes
• Uses automations (Ansiblefor example, is perfect for configuration changes).

Regular testing:

• Periodically tests connectivity between nodes
• Simulates monitor failures in development environment
• Verify that your recovery procedures are working

Need help with your Ceph cluster?

Distributed storage clusters such as Ceph require specific expertise to function optimally. If you have encountered this error and the above solutions do not solve your problem, or if you simply want to ensure that your Ceph infrastructure is properly configured and optimized, we can help.

Our team has experience solving complex Ceph problems in production environments, from urgent troubleshooting to performance optimization and high availability planning.

We offer help with

• Ceph technical support and consulting
• Introductory initiation y advanced courses at Ceph
• Urgent system support

Don’t let a connectivity problem become a major headache. The right expertise can save you time, money and, above all, stress.

Artificial intelligence no longer just responds, it also makes decisions. With frameworks like LangGraph and platforms like watsonx.ai , you can build agents that reason and act autonomously ?.

In this article, we will explain how to implement a ReAct (Reasoning + Action) agent locally and deploy it on IBM Cloud, all this with a practical example that includes a weather query tool ?️.

A practical guide to using your agents with LangGraph and Watsonx.ai

Project architecture

• Machine with local project
  • Here you develop and test the agent with Python, LangGraph and dependencies.
• ZIP (pip-zip)
  • Package with your code and additional tools.
• Software Specification
  • Environment with libraries necessary to execute the agent.
• watsonx.ai
  • Platform where you deploy the service as a REST API.
• IBM Cloud Object Storage
  • Stores deployment assets.

Let’s prepare the environment for our agent

We need:

• Python 3.12 installed
• Access to IBM Cloud and watsonx.ai
• Poetry for dependency management

Have you got everything? Well, first things first, clone the repository that we will use as an example. It is based on the official IBM examples.

git clone https://github.com/thomassuedbroecker/watsonx-agent-langgraph-deployment-example.git 
cd ./agents/langgraph-arxiv-research

First of all, let’s understand the example project.

The main files of the agent are:

Let’s configure the environment

We also recommend the use of Anaconda or miniconda. It allows us to manage virtual environments or Python packages in a simple way and is widely used in ML.

In order for Python to find our custom modules (such as agents and tools), we need to include the current directory in the environment variable PYTHONPATH

export PYTHONPATH=$(pwd):${PYTHONPATH}

Once we have the environment ready, it is time for the variables. You must create a config.toml file if you don’t already have one and use your IBM Cloud credentials:

You will find your variables here:

https://dataplatform.cloud.ibm.com/developer-access

Once there, select your deployment space and copy the necessary data (API Key, Space ID, etc.).

Execution at the agent’s premises

It is time to test the agent:

Since it’s a weather agent why don’t you try it with something like something like…?

“What is the current weather in Madrid?”

The console should give you the time in Madrid. Congratulations! we only need to do the deploy in watsonx.ai

Agent deployment in watsonx.ai

source ./.venv/bin/activate
poetry run python examples/query_existing_deployment.py

Conclusions

If you have any doubts, we recommend the following video tutorial where you can follow step by step the development connected with watsonx.ai

If you want to continue exploring these types of implementations or learn more about cloud development and artificial intelligence, we invite you to explore our AI courses.?

• At IBM
• In Canonical / Ubuntu