Description
A big data course with Apache Spark to prepare the data that feeds AI models and assistants on your own storage: Ceph, S3-compatible storage or parallel file systems, not a cloud service. You build a pipeline that can be rerun without duplicating data and that explains where every result comes from.
Course details
| SIXE code: SXIA06 | Category: Artificial Intelligence / Data |
| Delivery: Online and on-site | Duration: 3 days |
Who it is for
Data engineers, advanced analysts and platform teams who need to move and prepare large volumes of data to train or feed models.
Prerequisites
Basic Python and SQL. No previous experience with Spark or Hadoop is needed.
What you will learn
- Designing the pipeline: schemas, data contracts, formats such as Parquet and quality checks from the point of entry.
- Spark under the hood: partitions, shuffle and why a job is slow; when plain Python is enough and when distributing pays off.
- Reading and writing to your storage: Spark against S3-compatible storage such as Ceph, and how network and disk affect performance.
- Incremental loads: process only what is new, run twice without duplicates and recover a load that was interrupted.
- Real-time data: events with Spark Structured Streaming, windows and late-arriving data.
You work with three sample sources (a CSV file, an API and an event stream) and finish with a Spark pipeline ready to feed a model, with its lineage and checks.
What you take away
- A Spark pipeline with data contracts and quality checks.
- A record of lineage and of every load.
- A recovery test for an interrupted load.
Storage is almost always the bottleneck for these workloads: we explain it in which storage to choose for AI and HPC. If you use Ceph, we offer Ceph training and Ceph support for AI and HPC.
Frequently asked questions
How long is the course?
3 days.
Is it online or on-site?
Both: at your offices or remotely, with a live instructor.
What do I need to know beforehand?
Basic Python and SQL. No previous experience with Spark or Hadoop is needed.
Do we need a Hadoop cluster?
No. Spark runs locally, on Kubernetes or on a small cluster. The course uses a lab environment and explains how to move it to yours.
Is it useful if we do not use Ceph?
Yes. Any S3-compatible storage or shared file system will do.
Can it be tailored to my team?
Yes. We adapt the examples and labs to your team’s tools and processes, and it can be combined with other AI courses.
