I've just started working on a hadoop use case of analyzing CDRs in near-real time. CDRs are encoded in ASN1.1. A remote server is feeded regularly by CDRs. I'm wondering about how to ingest CDRs from this server into my cluster, and decode them to generate CSV files that can be processed by Hive (or Spark Streaming ..).
Is Flume Adapted to ingest this kind of data ? When do you think I should decode ASN1.1, before or after ingesting ? I have a program written in C for decoding ASN1.1. If Flume is adapted to ingest data, should I implement an Avro client in the server containing initial data, or is there another well-suited method ?