claudegoodies
Skill

pytorch-fsdp

From NousResearch

Expert guidance for Fully Sharded Data Parallel training with PyTorch FSDP - parameter sharding, mixed precision, CPU offloading, FSDP2

Provides reference documentation and code snippets for PyTorch FSDP distributed training, sharding, and mixed precision.

Use it when

  • Setting up FSDP or FSDP2 parameter sharding across multiple GPUs
  • Configuring mixed precision or CPU offloading in distributed training
  • Looking up PyTorch distributed APIs like Join context manager

Skip it if

  • Requires torch>=2.0 and transformers already installed
  • Linux/macOS only, no Windows support
  • Content is generated from official docs pasted verbatim, not curated advice

Facts

Repository
NousResearch/hermes-agent
Status
Actively maintained
Last commit
Source file
optional-skills/mlops/pytorch-fsdp/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# Pytorch-Fsdp Skill

Comprehensive assistance with pytorch-fsdp development, generated from official documentation.

## When to Use This Skill

This skill should be triggered when:
- Working with pytorch-fsdp
- Asking about pytorch-fsdp features or APIs
- Implementing pytorch-fsdp solutions
- Debugging pytorch-fsdp code
- Learning pytorch-fsdp best practices

## Quick Reference

### Common Patterns

**Pattern 1:** Generic Join Context Manager# Created On: Jun 06, 2025 | Last Updated On: Jun 06, 2025 The generic join context manager facilitates distributed training on uneven inputs. This page outlines the API of the relevant classes: Join, Joinable, and JoinHook. For a tutorial, see Distributed Training with Uneven Inputs Using the Join Context Manager. class torch.distributed.algorithms.Join(joinables, enable=True, throw_on_early_termination=False, **kwargs)[source]# This class defines the generic join context manager, which allows custom hooks to be called after a process joins. These hooks should shadow the collective communications of non-joined processes to prevent hanging and erroring and to ensure algorithmic correctness. Refer to JoinHook for details about the hook definition. Warning The context manager requires each participating Joinable to call the method notify_join_context() before its own per- iteration collective communications to ensure correctness. Warning The
View full source on GitHub →

Other skills