GitHub hosts its own source code on github.com, creating a unique circular dependency: if github.com goes down, they can't access their own code to fix it. To address this, GitHub has adopted eBPF (extended Berkeley Packet Filter) to monitor and block deployment scripts from creating unintended circular dependencies. Below, we explore how eBPF enhances deployment safety through a Q&A format.
What circular dependency does GitHub face?
GitHub relies on github.com to host all its source code. This creates a self-referential loop: to deploy updates or fixes to github.com, engineers need access to the code stored on github.com. If an outage occurs—such as a database failure—the platform becomes inaccessible, making it impossible to retrieve the exact code required for recovery. GitHub mitigates this by maintaining off-site mirrors and pre-built assets for rollbacks. However, other hidden loops remain, especially within deployment scripts that might inadvertently depend on internal services or GitHub downloads. This is where eBPF steps in to provide granular control over network and system calls during deployments.
What types of circular dependencies can appear during deployments?
There are three main categories: direct, hidden, and transient dependencies. A direct dependency occurs when a deploy script attempts to download a tool from GitHub—if GitHub is down, the script fails. Hidden dependencies involve tools already present on the machine that, when executed, check for updates from GitHub, potentially causing hangs or failures. Transient dependencies are indirect: the deploy script calls an internal API, which in turn fetches a binary from GitHub. During a MySQL outage, for example, a script to roll out a configuration change might trigger all three types, exacerbating the incident. Traditional methods relied on manual reviews, but eBPF automates detection and prevention.
How did GitHub traditionally address this problem?
Until recently, the responsibility fell on each team owning stateful hosts to manually review their deployment scripts and identify any circular dependencies. This process was error-prone, as dependencies are often subtle and change over time. Teams had to trace every network call, binary download, and internal API interaction. Given that GitHub deploys to thousands of hosts, this approach did not scale. Moreover, dependencies could be introduced by third-party tools or updated libraries without the team's knowledge. GitHub needed a systematic, automated method to enforce deployment safety without requiring constant human oversight.
How does eBPF help resolve circular dependencies?
eBPF (extended Berkeley Packet Filter) is a kernel technology that allows safe, low-overhead monitoring and interception of system calls. GitHub engineers wrote eBPF programs that attach to network-related syscalls (e.g., connect, sendto) during the deployment process. These programs inspect the destination addresses and reject any traffic that would create a circular dependency—like connecting to GitHub's own API or downloading from github.com. eBPF runs within a sandboxed kernel environment, ensuring that the monitoring code itself does not become a new dependency or slow down operations. This selective blocking is both efficient and precise, stopping only the problematic calls while allowing legitimate traffic.
Can you give an example of eBPF in action?
Imagine a MySQL deploy script that attempts to pull an open-source tool from GitHub. With eBPF loaded, the kernel intercepts the connect() syscall to the GitHub IP range. The eBPF program returns a “permission denied” error, causing the download to fail immediately. The deployment system then logs the blocked call and can fall back to a local cached version. Similarly, if a hidden dependency—like a tool checking for updates—tries to contact GitHub, eBPF blocks it, preventing a hang. This proactive enforcement ensures that no step in the deployment workflow creates a dependency on the very service being fixed.
What are the key benefits for deployment safety?
eBPF provides automatic, real-time prevention of circular dependencies without requiring changes to the deployment scripts themselves. It reduces the cognitive load on engineers and eliminates the risk of human oversight. The eBPF program is lightweight and can be loaded at boot time or before deployment runs. Additionally, it offers fine-grained control: you can allow traffic to internal services while blocking external or specific internal IPs. GitHub has reported that this approach significantly reduces incident resolution time because deploy scripts no longer fail due to unexpected circular dependencies. The same eBPF technique can be extended to other safety critical checks.
How can others start using eBPF for similar purposes?
To get started, you need a Linux kernel version 4.18 or later with eBPF support. Use tools like bcc (BPF Compiler Collection) or libbpf to write and load programs. For a deployment safety use case, focus on hooking into syscalls related to network connectivity (connect, sendto) or file access (open). You can use eBPF maps to maintain an allowed/blocked list of IP addresses or domains. GitHub has open-sourced some examples in their eBPF toolkit. It's important to test programs in a non‑production environment first and monitor for unintended impacts. With careful design, eBPF can add a robust safety layer to any system that has self‑referential dependencies.