Fault Tolerance Guarantees of Data Sources and Sinks#

    Flink's fault tolerance mechanism recovers programs in the presence of failures and continues to execute them. Such failures include machine hardware failures, network failures, transient program failures, etc.

    Flink can guarantee exactly-once state updates to user-defined state only when the source participates in the snapshotting mechanism. The following table lists the state update guarantees of Flink coupled with the bundled connectors.

    Please read the documentation of each connector to understand the details of the fault tolerance guarantees.

    Source Guarantees Notes
    Apache Kafka exactly once Use the appropriate Kafka connector for your version
    AWS Kinesis Streams exactly once
    RabbitMQ at most once (v 0.10) / exactly once (v 1.0)
    Twitter Streaming API at most once
    Collections exactly once
    Files exactly once
    Sockets at most once

    To guarantee end-to-end exactly-once record delivery (in addition to exactly-once state semantics), the data sink needs to take part in the checkpointing mechanism. The following table lists the delivery guarantees (assuming exactly-once state updates) of Flink coupled with bundled sinks:

    Sink Guarantees Notes
    HDFS rolling sink exactly once Implementation depends on Hadoop version
    Elasticsearch at least once
    Kafka producer at least once
    Cassandra sink at least once / exactly once exactly once only for idempotent updates
    AWS Kinesis Streams at least once
    File sinks at least once
    Socket sinks at least once
    Standard output at least once
    Redis sink at least once