When transcoding, audio and video streams can be filtered before encoding, with either a simple or complex filtergraph.
3.3.1 Simple filtergraphs
Simple filtergraphs are those that have exactly one input and output, both of the same type (audio or video). They are configured with the per-stream -filter option (with -vf and -af aliases for -filter:v (video) and -filter:a (audio) respectively). Note that simple filtergraphs are tied to their output stream, so e.g. if you have multiple audio streams, -af will create a separate filtergraph for each one.
Taking the trancoding example from above, adding filtering (and omitting audio, for clarity) makes it look like this:
┌──────────┬───────────────┐
│ demuxer │ │ ┌─────────┐
╞══════════╡ video stream │ packets │ video │ frames
│INPUT.mkv │ ├─────────⮞│ decoder ├─────⮞───╮
│ │ │ └─────────┘ │
└──────────┴───────────────┘ │
╭───────────⮜───────────╯
│ ┌────────────────────────┐
│ │ simple filtergraph │
│ ╞════════════════════════╡
│ │ ┌───────┐ ┌───────┐ │
╰──⮞├─⮞│ yadif ├─⮞│ scale ├─⮞├╮
│ └───────┘ └───────┘ ││
└────────────────────────┘│
│
│
┌──────────┬───────────────┐ video ┌─────────┐ │
│ muxer │ │ packets │ video │ │
╞══════════╡ video stream │⮜─────────┤ encoder ├───────⮜───────╯
│OUTPUT.mp4│ │ │ │
│ │ │ └─────────┘
└──────────┴───────────────┘
3.3.2 Complex filtergraphs
Complex filtergraphs are those which cannot be described as simply a linear processing chain applied to one stream. This is the case, for example, when the graph has more than one input and/or output, or when output stream type is different from input. Complex filtergraphs are configured with the -filter_complex option. Note that this option is global, since a complex filtergraph, by its nature, cannot be unambiguously associated with a single stream or file. Each instance of -filter_complex creates a new complex filtergraph, and there can be any number of them.
A trivial example of a complex filtergraph is the overlay filter, which has two video inputs and one video output, containing one video overlaid on top of the other. Its audio counterpart is the amix filter.
3.4 Loopback decoders
While decoders are normally associated with demuxer streams, it is also possible to create "loopback" decoders that decode the output from some encoder and allow it to be fed back to complex filtergraphs. This is done with the -dec directive, which takes as a parameter the index of the output stream that should be decoded. Every such directive creates a new loopback decoder, indexed with successive integers starting at zero. These indices should then be used to refer to loopback decoders in complex filtergraph link labels, as described in the documentation for -filter_complex.
Decoding AVOptions can be passed to loopback decoders by placing them before -dec, analogously to input/output options.
E.g. the following example:
ffmpeg -i INPUT \
-map 0:v:0 -c:v libx264 -crf 45 -f null - \
-threads 3 -dec 0:0 \
-filter_complex '[0:v][dec:0]hstack[stack]' \
-map '[stack]' -c:v ffv1 OUTPUT
reads an input video and
(line 2) encodes it with libx264 at low quality;
(line 3) decodes this encoded stream using 3 threads;
(line 4) places decoded video side by side with the original input video;
(line 5) combined video is then losslessly encoded and written into OUTPUT.
Such a transcoding pipeline can be represented with the following diagram:
┌──────────┬───────────────┐
│ demuxer │ │ ┌─────────┐ ┌─────────┐ ┌────────────────────┐
╞══════════╡ video stream │ │ video │ │ video │ │ null muxer │
│ INPUT │ ├──⮞│ decoder ├──┬────────⮞│ encoder ├─┬─⮞│(discards its input)│
│ │ │ └─────────┘ │ │(libx264)│ │ └────────────────────┘
└──────────┴───────────────┘ │ └─────────┘ │
╭───────⮜──╯ ┌─────────┐ │
│ │loopback │ │
│ ╭─────⮜──────┤ decoder ├────⮜──╯
│ │ └─────────┘
│ │
│ │
│ │ ┌───────────────────┐
│ │ │complex filtergraph│
│ │ ╞═══════════════════╡
│ │ │ ┌─────────────┐ │
╰─╫─⮞├─⮞│ hstack ├─⮞├╮
╰─⮞├─⮞│ │ ││
│ └─────────────┘ ││
└───────────────────┘│
│
┌──────────┬───────────────┐ ┌─────────┐ │
│ muxer │ │ │ video │ │
╞══════════╡ video stream │⮜─┤ encoder ├───────⮜──────────╯
│ OUTPUT │ │ │ (ffv1) │
│ │ │ └─────────┘
└──────────┴───────────────┘