[omaha] Possible Subject for Future Meeting?

[Wes Turner]

Apache Spot does something like what you describe (anomaly detection)
primarily for network data; but you could probably write a collector for
whatever type of file-based data source you need.
http://spot.incubator.apache.org

Some of the Apache Spot data ingestion component is written in Python:
https://github.com/apache/incubator-spot/blob/master/spot-ingest/README.md

The requirements.txt contains kafka-python.
https://github.com/dpkp/kafka-python

http://spot.incubator.apache.org/project-components/ingestion/

Kafka is a stream processing platform written in Scala and Java for
handling real-time data. Scala has similarities to Python.
https://en.wikipedia.org/wiki/Apache_Kafka

https://en.wikipedia.org/wiki/Anomaly_detection
https://en.wikipedia.org/wiki/Anomaly_detection#Popular_techniques
https://en.wikipedia.org/wiki/Security_information_and_event_management
(SIEM)

If cron with a data collection step, an analysis step, a notification step,
and a manual verification step is enough, cron or celery chords are
probably good enough and Spot would be overkill unless you want to
correlate events with captured network data.

On Sunday, November 26, 2017, Burch Kealey via Omaha <omaha at python.org>
wrote:

> I teach on Tuesday nights in the fall - so I have a better chance of
> attending a meeting in the spring.  I don't know if this is too lame I
> suspect it is but would like to throw out that I started using the email
> library in Python to first, monitor one of our data servers for anomalous
> data - create an email with the data in a csv file and send it out to folks
> who then investigate the anomalies.  Corrections are made (as needed) then
> they reply to the email so another bit of code periodically runs and checks
> our inbox and after validating the sender and the attachments moves the
> result file to our production system as verified.
>
>
> I would be happy to describe and illustrate this process if their is
> interest.
>
>
> Burch
> _______________________________________________
> Omaha Python Users Group mailing list
> Omaha at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/omaha
> http://www.OmahaPython.org
>