Zipkin Tracing
Apache Cassandra

About The Last Pickle


Work with clients
to deliver and improve

Apache Cassandra based solutions

Based in
USA, New Zealand, Australia, France




distributed tracing

zipkin

cassandra's tracing

zipkin & cassandra

Scaling Data & People

Apache Cassandra
data platform de jure
microservices, base, and lambda architectures


the missing piece for many is tracing and profiling difficult to reproduce problems



Zipkin

an implementation of Google's Dapper paper



Zipkin Install


git clone git@github.com:openzipkin/zipkin-java.git
cd zipkin-java

# it's in the readme
STORAGE_TYPE=cassandra ./mvnw -pl zipkin-server spring-boot:run


open browser http://localhost:9411/

( docker: https://github.com/openzipkin/docker-zipkin-java )

search traces

analyze one trace

realtime in browser

platform call graph

client    |    server



CS -->                        
                        --> SR

                        <-- SS
CR <--                        



Existing instrumentations

Language Library Framework
Python pyramid_zipkin Pyramid Http (B3)
Java brave Jersey, RestEASY, JAXRS2, Apache HttpClient, Mysql
Java Spring Cloud Sleuth Spring, Spring Cloud (e.g. Stream, Netflix)
Scala finagle-zipkin Finagle
Ruby zipkin-tracer Rack
C# ZipkinTracerModule OWIN, HttpHandler
Go go-zipkin x/net Context
Go Go kit Go kit
Scala akka-tracing Akka, Spray, Play
Java Dropwizard Zipkin Dropwizard


simple http call




simple http call



( Brave-3.7 – https://github.com/openzipkin/brave )



simple c* call




simple c* call




http call passing through headers




simple http call




simple http call




simple c* call




simple c* call




http call passing through headers


Tracing in C*

    CO-ORDINATOR NODE                    REPLICA NODE


 -->
    beginSession(..)
        trace(..)
        trace(..)
                                    --> initialiseMessage(..)
                                            trace(..)
                                            trace(..)
                                    <--
        trace(..)
    endSession(..)
 <--

Zipkin in C*


  • visualization
  • detailed timings
  • hierarchy and asynchronisity
  • zero tracing overhead

Cassandra-3.4

Download Cassandra-Zipkin-Tracing


git clone git@github.com:thelastpickle/cassandra-zipkin-tracing
cd cassandra-zipkin-tracing

# it's in the readme
mvn install
cp target/cassandra-zipkin-tracing-*.jar $CASSANDRA_HOME/lib/
cp lib/*  $CASSANDRA_HOME/lib/


then run enabling new tracing 
JVM_OPTS= \
  "-Dcassandra.custom_tracing_class=com.thelastpickle.cassandra.tracing.ZipkinTracing" \
  bin/cassandra

Zipkin across C*


Zipkin across C*

    CO-ORDINATOR NODE                    REPLICA NODE


 -->
    beginSession(..)
        trace(..)
        trace(..)
                   (zipkin headers) --> initialiseMessage(..)
                                            trace(..)
                                            trace(..)
                                    <--
        trace(..)
    endSession(..)
 <--

Zipkin across C*

Zipkin into C*


Zipkin into C*


http call passing through headers




c* call using custom payload




enable zipkin tracing and the custom payload handler 

bin/cassandra
    -Dcassandra.custom_tracing_class=..ZipkinTracing
    -Dcassandra.custom_query_handler_class=..CustomPayloadMirroringQueryHandler

Zipkin Self Tracing

analyze one trace

Thanks

  • Zipkin     – https://github.com/openzipkin/zipkin-java

  • Brave (zipkin java instrumentation)          – https://github.com/openzipkin/brave
  • OpenTracing (agnostic instrumentation) – https://github.com/opentracing/opentracing-java

  • Zipkin Cassandra implementation
                – https://github.com/thelastpickle/cassandra-zipkin-tracing
  • OpenTracing Cassandra implementation
                – https://github.com/thelastpickle/cassandra-opentracing-tracing

  • Google's Dapper paper
                – http://research.google.com/pubs/pub36356.html

  • C* custom payloads
                – https://issues.apache.org/jira/browse/CASSANDRA-8553
                – https://datastax.github.io/java-driver/2.2.0-rc2/features/custom_payloads/