Sandboxing data crunches, Chapter 1: use a subprocess

Chapter 1: use a subprocess

Often, a simple approach is best. Here’s simplicity itself:

Never do this.
Subprocesses help the main process recover after out-of-memory errors
  • Protobuf is a compelling format for encoding messages. Workbench happens to use Arrow which happens to support the Parquet file format which happens to encode metadata in Thrift format; so Thrift was already a Workbench dependency. (If we’d had a blank slate, I’d have leaned towards Protobuf because it’s more popular.)
  • Python’s pickle module is not okay. Using pickle format, Step can output untrusted code that Renderer will execute — which defeats the purpose.
  • JSON may seem appropriate for simple messages; but it’s costly as you grow. When Renderer parses Step’s output message, it must validate that (untrusted) message. You can hand-code JSON validation functions … but that’s onerous. You can use a library like JSON Schema … but then you’d need to write schemas. Thrift and Protobuf are made for this; JSON isn’t.
  • You can pass data from Renderer to Step through command-line arguments. This is a one-way communication channel and it only passes bytestrings.

One layer of security

The subprocess is our fledgling “sandbox”. Let’s define “sandbox” in terms of what Steps can do:

  • A “good” subprocess reads inputs from standard input and disk; writes valid output to standard output and disk; and exits with status code 0. Renderer acts upon its output.
  • A “buggy” subprocess reads from standard input and — whoops — prints a stack trace to standard error and exits with a non-zero status code. Renderer acts upon this event.
  • Another “buggy” subprocess writes nonsense to standard output. Renderer recovers. (Renderer must not allow injection here. Don’t use Python’s pickle module.)
  • Another “buggy” subprocess never exits. Renderer kills it and declares, “timed out”.
  • A “bad” subprocess does Something Else. (cue ominous tones: dun-dun-dunnn!)
Step’s process “sandboxes” itself in step 0: it reduces its privileges irreversibly before executing untrusted code
  • On Kubernetes, disable Hyper-Threading so untrusted code can’t glean data from other processes.
  • Secure internal services (like your database) with SSL and passwords. Untrusted code won’t know the password (if you sandbox properly).
  • Ensure secret files aren’t readable to the sandbox user. (Kubernetes mounts secrets as world-readable by default. Fix those permissions!)
  • Ensure the sandbox user can’t write files Renderer reads internally. (Imagine if untrusted code wrote a configuration file and then Renderer acted upon it….)
  • Restrict cloud metadata URLs. From a Google Kubernetes Engine pod, http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-env exposes credentials that let anybody join the cluster and access all users’ data. Don’t let untrusted code access that URL. We enabled Workload Identity to prevent exposing this URL. (A firewall would inspire more confidence; we’ll add build one in Chapter 3.)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Adam Hooper

Adam Hooper

Journalist, ex software engineer