Getting started with ElasticSearch

Elasticsearch is a search server based on Apache Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Elasticsearch is the second most popular enterprise search engine, at the time of this post.

Elasticsearch can be used to search all kinds of documents. It provides scalable search, has near real-time search, and supports multitenancy. Elasticsearch is distributed, which means that indices can be divided into shards and each shard can have zero or more replicas. Each node hosts one or more shards, and acts as a coordinator to delegate operations to the correct shard(s). Rebalancing and routing are done automatically. Notable users of Elasticsearch include Wikimedia, GitHub, Stack Exchange, Netflix, The Guardian etc

But Elasticsearch is not just for mega-corporations. It has enabled many startups like Datadog and Klout to prototype ideas and to turn them into scalable solutions. Elasticsearch can run on your laptop, or scale out to hundreds of servers and petabytes of data. No individual part of Elasticsearch is new or revolutionary. Full-text search has been done before, as have analytics systems and distributed databases. The revolution is the combination of these individually useful parts into a single, coherent, real-time application. It has a low barrier to entry for the new user, but can keep pace with you as your skills and needs grow.

You can find a very useful Getting Started Guide here:
http://www.elastic.co/guide/en/elasticsearch/guide/master/getting-started.html

If you are impatient, this You Tube Video helps you get started quickly: Getting started with ElasticSearch

Note: The syntax of the settings posted at 24:54 of the video doesn't work as there are some changes in version 1.5.1 I used. Refer this link for correct usage: Using Synonyms

I would suggest Sense plugin for Chrome to easily explore elastic search:
https://chrome.google.com/webstore/search/sense
https://www.found.no/foundation/Sense-Elasticsearch-interface/

Alternatives are RESTClient for firefox and Fiddler for pretty much any platform.

You can use the head plugin for Elasticsearch to monitor and manage the server,

You can find all the REST commands below for easy copy-paste if you want to follow the video and try hands-on.

Add Sample Data

POST /places/restaurant
{
    "name" : "Joes Italiana",
    "description": "Best pasta around",
    "address": {
        "street":"464 S Main St",
        "city":"Los Angeles",
        "state":"CA",
        "zip":"90013"
    },
    "location":[34.023954, -118.3927072],
    "tags":["italian","spaghetti","pasta"],
    "rating": "4.5"
}
    POST /places/restaurant
{
    "name" : "Jose's Taco Shop",
    "description": "Best Tacos in SoCal",
    "address": {
        "street":"950 Vine St",
        "city":"Los Angeles",
        "state":"CA",
        "zip":"90038"
    },
    "location":[34.088186, -118.326603],
    "tags":["mexican","tacos","burritos"],
    "rating": "4.0"
}
POST /places/restaurant
{
    "name" : "Berry's Burritos",
    "description": "Best Burritos in New York!",
    "address": {
        "street":"230 W 4th St",
        "city":"New York",
        "state":"NY",
        "zip":"10014"
    },
    "location":[40.7543385, -73.976313],
    "tags":["mexican","tacos","burritos"],
    "rating": "4.3"
}
POST /places/restaurant
{
    "name" : "Steve's Italian Restaurant",
    "description": "Great food, great atmosphere",
    "address": {
        "street":"46 W 46th St",
        "city":"New York",
        "state":"NY",
        "zip":"10036"
    },
    "location":[40.751624, -73.9783865],
    "tags":["italian","spaghetti","pasta"],
    "rating": "3.5"
}

Various forms of Search and Filter

POST /places/restaurant/_search
{
    "query":{
        "match_all": {}
    }
}
POST /places/restaurant/_search
{
    "query":{
        "query_string": {
           "query": "tacos"
        }
    }
}
POST /places/restaurant/_search
{
    "query":{
        "query_string": {
           "query": "tacos",
           "fields": ["tags"]
        }
    }
}
POST /places/restaurant/_search
{
    "query":{
        "query_string": {
           "query": "taco",
           "fields": ["name"]
        }
    }
}
POST /places/restaurant/_search
{
    "query": {
        "filtered": {
           "query": {
               "query_string": {
                   "query": "tacos",
                   "fields": ["tags"]
                }
           },
           "filter": {
               "range": {
                  "rating": {
                     "gte": 4.0
                  }
               }
           }
        }
    }
}
POST /places/restaurant/_search
{
    "query": {
        "filtered": {
           "filter": {
               "range": {
                  "rating": {
                     "gte": 4.0
                  }
               }
           }
        }
    }
}
 
POST /places/restaurant/_search
{
    "query": {
        "filtered": {
            "query": {
                "match": {
                   "address.state": "ny"
                }
            },
            "filter": {
               "range": {
                  "rating": {
                     "gte": 4.0
                  }
               }
           }
        }
    }
}

Clear Index

DELETE /places

Rebuild index with Synonym support

POST /places
{
"settings": {
"analysis": {
"filter": {
"synonym": {
"type": "synonym",
"synonyms_path": "synonyms.txt",
"ignore_case": "true"
}
},
"analyzer": {
"synonym": {
"tokenizer": "whitespace",
"filter": ["synonym"]
}
}
}
},
"mappings": {
"restaurant": {
"_all": {
"enabled": true
},
"properties": {
"address.state": {
"type": "string",
"analyzer": "synonym"
},
"location": {
"type": "geo_point"
}
}
}
}
}

Bulk Insert of Data 

POST /places/restaurant/_bulk
{"index":{}}
{ "name" : "Joes Italiana", "description": "Best pasta around", "address": { "street":"464 S Main St", "city":"Los Angeles", "state":"CA", "zip":"90013" }, "location":[34.023954, -118.3927072], "tags":["italian","spaghetti","pasta"], "rating": "4.5" }
{"index":{}}
{ "name" : "Jose's Taco Shop", "description": "Best Tacos in SoCal", "address": { "street":"950 Vine St", "city":"Los Angeles", "state":"CA", "zip":"90038" }, "location":[34.088186, -118.326603], "tags":["mexican","tacos","burritos"], "rating": "4.0" }
{"index":{}}
{ "name" : "Berry's Burritos", "description": "Best Burritos in New York!", "address": { "street":"230 W 4th St", "city":"New York", "state":"NY", "zip":"10014" }, "location":[40.7543385, -73.976313], "tags":["mexican","tacos","burritos"], "rating": "4.3" }
{"index":{}}
{ "name" : "Steve's Italian Restaurant", "description": "Great food, great atmosphere", "address": { "street":"46 W 46th St", "city":"New York", "state":"NY", "zip":"10036" }, "location":[40.751624, -73.9783865], "tags":["italian","spaghetti","pasta"], "rating": "3.5" }

Synoym Search 

POST /places/restaurant/_search
{
    "query": {
        "filtered": {
            "query": {
                "match": {
                   "address.state": "new york"
                }
            },
            "filter": {
               "range": {
                  "rating": {
                     "gte": 4.0
                  }
               }
           }
        }
    }
}

Geospatial Search

POST /places/restaurant/_search
{
    "query": {
        "filtered": {
            "filter": {
               "geo_distance": {
                  "distance": "100km",
                  "location": [40.7894537,-73.9481288]
               }
           }
        }
    }
}

POST /places/restaurant/_search
{
    "query": {
        "filtered": {
            "filter": {
                "bool": {
                    "must": [
                        {
                           "range": {
                              "rating": {
                                "gte":"4.0"
                              }
                           }
                        },
                        {
                            "geo_distance": {
                              "distance": "100km",
                              "location": [40.7894537,-73.9481288]
                            }
                        }  
                    ]
                }
           }
        }
    }
}

Getting just Count

POST /places/restaurant/_count
{
    "query":{
        "query_string": {
           "query": "tacos"
        }
    }
}

Paged Data

POST /places/restaurant/_search?size=1&from=0
{
    "query":{
        "query_string": {
           "query": "tacos"
        }
    }
} 

Dash Cam G1WH Firmware Update

A Dash Cam is an invaluable asset when you run into an accident, like your insurance. It can provide critical information on how an accident happened and who is at fault. Apart from that, it may also record interesting things from the road like UFOs and funny moments :-)

There is a huge array of dash cams in the market with varying price and features. G1WH is a cheap Chinese cam which surpasses many leading manufactures in video quality, at half price or even less.

You can find heaps of reviews and praises online on this model:
http://thewirecutter.com/reviews/best-dash-cam/
http://www.techmoan.com/blog/2013/12/16/g1w-the-cheapest-dash-cam-thats-worth-buying.html
https://dashcamtalk.com/g1w/

There are currently 3 versions of G1W and mine is G1WH. It uses an all black case (more discreet) and has slightly wider angle of view (140 vs 120). All are based on the popular Novatek NT96650 processor and the Aptina AR0330 CMOS sensor.

I was having some issues with this though. The build quality is ok, but not the best. The mount tend to become loose easily and cam may fall off on receiving quick jerks. Also it is not recording reliably (switches off automatically after some time) and this can be fixed by turning off Motion sensor and reducing the G-Sensor sensitivity to Low/Med. Some people also had recording issues with some Class 10 memory cards, so a Class 6 branded (Samsung/Kingston/Sandisk) micro SD card is best suited. Also be warned that there are many fakes in the market (eBay/Amazon). Better to buy from gearbest.com or banggood.com.

There are many firmware updates and custom firmware available for the G1WH model. The updates provides you more fixes and even new features like entering your license plate number to be stamped with date, higher bit rate etc.

Here is supposedly the newest firmware for G1WH. The default language is Polish, but you can change it via the menu.
http://reliabledownloads.org/file/05JxN
Credits: http://vat19.pl/pl/blog/informacje/15-Firmware.html

It adds the following features:

• Increased recording bitrate files (such as DVRs DOD) 
• Polish language menu 
• Added function car plate (not present in older versions) 
• More stable operation 

My cam came with Firmware version 2014.0509, in case you screw up you can revert to it here:
http://reliabledownloads.org/file/05Jxn

NOTE: After updating firmware the menu button was not working for me. Same happened even after reverting to above 2014.0509. If same happens click on the camera button on the top (next to power) and then click Menu.

These are the instructions to update the firmware:
1. Download the firmware file to your computer
2. Unzip the file
3. Format your memory card in your computer (or unit)
4. Copy the bin file to the root of your memory card
5. Change the name of the firmware you would like to flash to FWDM800H (required as the G1WH will not update with bin files with names other then this)
6. Put card into camera
7. Power camera on (make sure the camera is plugged in)
8. The camera will update automatically.  The screen will stay off but the status light will be on; it will take 30-60 seeconds
9. After updating, format the card.

NOTE: Remove the firmware (.bin) from the memory card you would like to use or it will flash every time you start!