Phase 2: Pushing Twitter to AWS Elasticsearch

Summary:

After a few hiccups I have managed to push my data to AWS Elasticsearch node. So far this is basic stuff but the visual representation of what I have captures so far is amazing!

1. Useful doc: https://www.elastic.co/guide/index.html
2. Read it and go through it step by step
3. Be sure to create your index
4. I am using a single node service on the free tier option
5. Include patterns: attack|radical|suspects?|muslim|islam|arme?d?|conflict|cival|war|coup|governments?|oppositions?
6. Useful health checks: 
1. /_cat/shards
2. /_cat/health
3. /_cat/indices?v
4. /twit/twit/<TWIT:ID>pretty

Line chart of search strings

Table split by country

Twitters!

Another view split by country

Table chart split by country

Week 2 - Log my Twit using Twitter API and Python

This week we’ve started an experiment using Twitter’s Streaming API. 

• See Twitter API for further information: https://dev.twitter.com/streaming/overview
• App account creation: https://apps.twitter.com/

The Streaming APIs give developers low latency access to Twitter’s global stream of Tweet data. We needed to use Streaming API to access Twitter’s global stream of Tweet data for multi searches. Our Twit collector will tap into the Public Stream and will be tapping on behalf of a single user. This will just be a single collector agent. The collector will connect, get tweets, parse data and store for future processing.

We are using the same Centos7 VM and installed

Install TwitterAPI

> sudo pip install TwitterAPI

Required App Configuration

You will need to make note of the following TwitterApp values:

"credential": 

{

"CONSUMER_KEY": "XXXXXXXXXX",

"CONSUMER_SECRET": "XXXXXXXXXXXXX",

"ACCESS_TOKEN_KEY": "XXXXXXXXXX",

"ACCESS_TOKEN_SECRET": "XXXXXXXXXX"

}

This will be used to authenticate against Twitter's Stream server(s). Please bare in mind that your IP will be banned if your app goes out of control :)

Stream connection function:

It is as simple as creating a TwitterAPI object by passing the above app configuration values.

def app_connect(app_config):

        try:

                return TwitterAPI(app_config_json['credential']['CONSUMER_KEY'],

                        app_config_json['credential']['CONSUMER_SECRET'],

                        app_config_json['credential']['ACCESS_TOKEN_KEY'],

                        app_config_json['credential']['ACCESS_TOKEN_SECRET'])

        except Exception as e:

                logging.debug("TwitterAPI Exception: %s" + str(e))

                exit(1)

TRACK - our first experiment 

A comma-separated list of phrases which will be used to determine what Tweets will be delivered on the stream. Our list of phrases is stored in a separate json config file that is loaded when the code is run.

{ 

"et": 

{

    "track": ["ethiopia","addisababa","ethioi","Oromo","Amhara","Somali","Tigray","Sidama","Gurage","Welayta","Hadiya","Afar","Gamo","Gedeo"]

},

"so": 

{

                "track": ["somalia","republicofsomalia","somaliland","puntland","galmudug","jubalandkhaatumo","koonfurgalibeed"]

        }

} 

Time to make the request

As per documentation, we call the request method to filter statuses

keep in mind the restrictions such as: Each phrase must be between 1 and 60 bytes, inclusive.

> req = twitter_conn.request('statuses/filter', {'track': TRACK_TERM})

Summary

This is all it takes to build a simple code that taps into Twitter's Stream to filter out Twits based on a comma separated strings.

Clone/hack/improve it as always - 

• git@gitlab.com:baricho/feed_twitter.git
• https://gitlab.com/baricho/feed_twitter.git

Peace!

Currency Converter Client/Server using GO

Currency Converter Client/Server using GO

Today’s experiment involves writing a client/server program that converts currency to ETB.

The server listens on port:12345 to handle client requests.

Function XeCurrencyServer is called whenever a request comes in.

XeCurrencyServer:

Strips out “/” from the request

 * I am only interested in converting USD, GBP and EUR to ETB - strings.Split(req.URL.Path,"/")
  * Builds request URL to fetch - http.Get(currency_requested). 
 * I use xe.com all of the time. 
        * I extract a specific line that has the data that we need - regexp.MustCompile(`\&\w+;`+cur[1]+`\&\w+;=\&\w+;(\d+\.\d+)\&\w+;ETB`) and currencyRegex.FindStringSubmatch(bodyString)
        * Write the result back using w http.ResponseWriter
        * Anything else is ignored.

Code
====
        * src/example/currency/convert.go
        * To install: go install example/currency
        * To run: $GOPATH/bin/currency
        * Url: http://178.62.226.141:12345/GBP 

Git

===

        * https://gitlab.com/baricho/currency-go-example
Docs
====
        * https://gobyexample.com/   - covers a lot of things with easy to follow examples
        * https://golang.org/doc/     - golang doc page
        * https://golang.org/pkg/    - package info
        * http://regexr.com/    - regex builder

This is something that is very basic but should give you the basic understanding on how to get going with GO.

package main

import (
 "net/http"
 "log"
 "io"
 "fmt"
 "net/http/httputil"
 "regexp"
 "strings"
)

func XeCurrencyServer(w http.ResponseWriter, req *http.Request) {

 cur := strings.Split(req.URL.Path,"/")
 fmt.Fprintf(w, "<html><body><br><br><br>")
 if cur[1] == "USD" || cur[1] == "GBP" || cur[1] == "EUR" {

  currency_requested := ("http://www.xe.com/currencyconverter/convert/?Amount=1&From="+cur[1]+"&To=ETB")
         response, err := http.Get(currency_requested)
 
  var currencyRegex = regexp.MustCompile(`\&\w+;`+cur[1]+`\&\w+;=\&\w+;(\d+\.\d+)\&\w+;ETB`)
         if err != nil {
                 log.Fatal(err)
         } else {
                 defer response.Body.Close()
   dump, err := httputil.DumpResponse(response, true)
   bodyString := string(dump) 
   conversion := currencyRegex.FindStringSubmatch(bodyString)
   for k, v := range conversion {
    if k > 0 {
     fmt.Fprintf(w, "1.00 %s == %s ETB", cur[1], v)
    }
   }
                 if err != nil {
    io.WriteString(w,"Error communicating with xe.com")
                         log.Fatal(err)
                 }
         }
 } else {
  fmt.Fprintf(w, "Currency requested not found: %s", cur[1])
 }
 fmt.Fprintf(w, "</body></html>")
}

func main() {
 http.HandleFunc("/", XeCurrencyServer)
 
 log.Fatal(http.ListenAndServe(":12345", nil))
}

As always do give it a try and provide feedback!

Peace!