Building a Recipe Search Site with Angular and Elasticsearch

https://www.sitepoint.com/building-recipe-search-site-angular-elasticsearch/

By Adam Bard April 15, 2014

Have you ever wanted to build a search feature into an application? In the old days, you might have found yourself wrangling with Solr, or building your own search service on top of Lucene — if you were lucky. But, since 2010, there’s been an easier way: Elasticsearch.

Elasticsearch is an open-source storage engine built on Lucene. It’s more than a search engine; it’s a true document store, albeit one emphasizing search performance over consistency or durability. This means that, for many applications, you can use Elasticsearch as your entire backend. Applications such as…

Building a Recipe Search Engine

In this article, you’ll learn how to use Elasticsearch with AngularJS to create a search engine for recipes, just like the one atOpenRecipeSearch.com. Why recipes?

1. OpenRecipes exists, which makes our job a lot easier.
2. Why not?

OpenRecipes is an open-source project that scrapes a bunch of recipe sites for recipes, then provides them for download in a handy JSON format. That’s great for us, because Elasticsearch uses JSON too. However, we have to get Elasticsearch up and running before we can feed it all those recipes.

Download Elasticsearch and unzip it into whatever directory you like. Next, open a terminal, cd to the directory you just unzipped, and run bin/elasticsearch (bin/elasticsearch.bat on Windows). Ta-da! You’ve just started your very own elasticsearch instance. Leave that running while you follow along.

One of the great features of Elasticsearch is its out-of-the-box RESTful backend, which makes it easy to interact with from many environments. We’ll be using the JavaScript driver, but you could use whichever one you like; the code is going to look very similar either way. If you like, you can refer to this handy reference (disclaimer: written by me).

Now, you’ll need a copy of the OpenRecipes database. It’s just a big file full of JSON documents, so it’s straightfoward to write a quick Node.js script to get them in there. You’ll need to get the JavaScript Elasticsearch library for this, so run npm install elasticsearch. Then, create a file named load_recipes.js, and add the following code.

var fs = require('fs');
var es = require('elasticsearch');
var client = new es.Client({
  host: 'localhost:9200'
});

fs.readFile('recipeitems-latest.json', {encoding: 'utf-8'}, function(err, data) {
  if (err) { throw err; }

  // Build up a giant bulk request for elasticsearch.
  bulk_request = data.split('\n').reduce(function(bulk_request, line) {
    var obj, recipe;

    try {
      obj = JSON.parse(line);
    } catch(e) {
      console.log('Done reading');
      return bulk_request;
    }

    // Rework the data slightly
    recipe = {
      id: obj._id.$oid, // Was originally a mongodb entry
      name: obj.name,
      source: obj.source,
      url: obj.url,
      recipeYield: obj.recipeYield,
      ingredients: obj.ingredients.split('\n'),
      prepTime: obj.prepTime,
      cookTime: obj.cookTime,
      datePublished: obj.datePublished,
      description: obj.description
    };

    bulk_request.push({index: {_index: 'recipes', _type: 'recipe', _id: recipe.id}});
    bulk_request.push(recipe);
    return bulk_request;
  }, []);

  // A little voodoo to simulate synchronous insert
  var busy = false;
  var callback = function(err, resp) {
    if (err) { console.log(err); }

    busy = false;
  };

  // Recursively whittle away at bulk_request, 1000 at a time.
  var perhaps_insert = function(){
    if (!busy) {
      busy = true;
      client.bulk({
        body: bulk_request.slice(0, 1000)
      }, callback);
      bulk_request = bulk_request.slice(1000);
      console.log(bulk_request.length);
    }

    if (bulk_request.length > 0) {
      setTimeout(perhaps_insert, 10);
    } else {
      console.log('Inserted all records.');
    }
  };

  perhaps_insert();
});

Next, run the script using the command node load_recipes.js. 160,000 records later, we have a full database of recipes ready to go. Check it out with curl if you have it handy:

$ curl -XPOST http://localhost:9200/recipes/recipe/_search -d '{"query": {"match": {"_all": "cake"}}}'

Now, you might be OK using curl to search for recipes, but if the world is going to love your recipe search, you’ll need to…

Build a Recipe Search UI

This is where Angular comes in. I chose Angular for two reasons: because I wanted to, and because Elasticsearch’s JavaScript library comes with an experimental Angular adapter. I’ll leave the design as an exercise to the reader, but I’ll show you the important parts of the HTML.

Get your hands on Angular and Elasticsearch now. I recommend Bower, but you can just download them too. Open yourindex.html file and insert them wherever you usually put your JavaScript (I prefer just before the closing body tag myself, but that’s a whole other argument):

<script src="path/to/angular/angular.js"></script>
<script src="path/to/elasticsearch/elasticsearch.angular.js"></script>

Now, let’s stop to think about how our app is going to work:

1. The user enters a query.
2. We send the query as a search to Elasticsearch.
3. We retrieve the results.
4. We render the results for the user.

The following code sample shows the key HTML for our search engine, with Angular directives in place. If you’ve never used Angular, that’s OK. You only need to know a few things to grok this example:

1. HTML attributes starting with ng are Angular directives.
2. The dynamic parts of your Angular applications are enclosed with an ng-app and an ng-controller. ng-app and ng-controller don’t need to be on the same element, but they can be.
3. All other references to variables in the HTML refer to properties on the $scope object that we’ll meet in the JavaScript.
4. The parts enclosed in {{}} are template variables, like in Django/Jinja2/Liquid/Mustache templates.

<div ng-app="myOpenRecipes" ng-controller="recipeCtrl">

  <!-- The search box puts the term into $scope.searchTerm
       and calls $scope.search() on submit -->
  <section class="searchField">
    <form ng-submit="search()">
      <input type="text" ng-model="searchTerm">
      <input type="submit" value="Search for recipes">
    </form>
  </section>

  <!-- In results, we show a message if there are no results, and
       a list of results otherwise. -->
  <section class="results">
    <div class="no-recipes" ng-hide="recipes.length">No results</div>

    <!-- We show one of these elements for each recipe in $scope.recipes.
         The ng-cloak directive prevents our templates from showing on load. -->
    <article class="recipe" ng-repeat="recipe in recipes" ng-cloak>
      <h2>
        <a ng-href="{{recipe.url}}">{{recipe.name}}</a>
      </h2>
      <ul>
        <li ng-repeat="ingredient in recipe.ingredients">{{ ingredient }}</li>
      </ul>

      <p>
        {{recipe.description}}
        <a ng-href="{{recipe.url}}">... more at {{recipe.source}}</a>
      </p>
    </article>

    <!-- We put a link that calls $scope.loadMore to load more recipes
         and append them to the results.-->
    <div class="load-more" ng-hide="allResults" ng-cloak>
      <a ng-click="loadMore()">More...</a>
    </div>
  </section>

Now, we can start writing our JavaScript. We’ll start with the module, which we decided above would be calledmyOpenRecipes (via the ng-app attribute).

/**
 * Create the module. Set it up to use html5 mode.
 */
window.MyOpenRecipes = angular.module('myOpenRecipes', ['elasticsearch'],
  ['$locationProvider', function($locationProvider) {
    $locationProvider.html5Mode(true);
  }]
);

For those new to Angular, the ['$locationProvider', function($locationProvider) {...}] business is our way of telling Angular that we’d like it to pass $locationProvider to our handler function so we can use it. This system of dependency injection removes the need for us to rely on global variables (except the global angular and theMyOpenRecipes we just created).

Next, we’ll write the controller, named recipeCtrl. We need to make sure to initialize the recipes, allResults, andsearchTerm variables used in the template, as well as providing search() and loadMore() as actions.

/**
 * Create a controller to interact with the UI.
 */
MyOpenRecipes.controller('recipeCtrl', ['recipeService', '$scope', '$location', function(recipes, $scope, $location) {
  // Provide some nice initial choices
  var initChoices = [
      "rendang",
      "nasi goreng",
      "pad thai",
      "pizza",
      "lasagne",
      "ice cream",
      "schnitzel",
      "hummous"
  ];
  var idx = Math.floor(Math.random() * initChoices.length);

  // Initialize the scope defaults.
  $scope.recipes = [];        // An array of recipe results to display
  $scope.page = 0;            // A counter to keep track of our current page
  $scope.allResults = false;  // Whether or not all results have been found.

  // And, a random search term to start if none was present on page load.
  $scope.searchTerm = $location.search().q || initChoices[idx];

  /**
   * A fresh search. Reset the scope variables to their defaults, set
   * the q query parameter, and load more results.
   */
  $scope.search = function() {
    $scope.page = 0;
    $scope.recipes = [];
    $scope.allResults = false;
    $location.search({'q': $scope.searchTerm});
    $scope.loadMore();
  };

  /**
   * Load the next page of results, incrementing the page counter.
   * When query is finished, push results onto $scope.recipes and decide
   * whether all results have been returned (i.e. were 10 results returned?)
   */
  $scope.loadMore = function() {
    recipes.search($scope.searchTerm, $scope.page++).then(function(results) {
      if (results.length !== 10) {
        $scope.allResults = true;
      }

      var ii = 0;

      for (; ii < results.length; ii++) {
        $scope.recipes.push(results[ii]);
      }
    });
  };

  // Load results on first run
  $scope.loadMore();
}]);

You should recognize everything on the $scope object from the HTML. Notice that our actual search query relies on a mysterious object called recipeService. A service is Angular’s way of providing reusable utilities for doing things like talking to outside resources. Unfortunately, Angular doesn’t provide recipeService, so we’ll have to write it ourselves. Here’s what it looks like:

MyOpenRecipes.factory('recipeService', ['$q', 'esFactory', '$location', function($q, elasticsearch, $location) {
  var client = elasticsearch({
    host: $location.host() + ':9200'
  });

  /**
   * Given a term and an offset, load another round of 10 recipes.
   *
   * Returns a promise.
   */
  var search = function(term, offset) {
    var deferred = $q.defer();
    var query = {
      match: {
        _all: term
      }
    };

    client.search({
      index: 'recipes',
      type: 'recipe',
      body: {
        size: 10,
        from: (offset || 0) * 10,
        query: query
      }
    }).then(function(result) {
      var ii = 0, hits_in, hits_out = [];

      hits_in = (result.hits || {}).hits || [];

      for(; ii < hits_in.length; ii++) {
        hits_out.push(hits_in[ii]._source);
      }

      deferred.resolve(hits_out);
    }, deferred.reject);

    return deferred.promise;
  };

  // Since this is a factory method, we return an object representing the actual service.
  return {
    search: search
  };
}]);

Our service is quite barebones. It exposes a single method, search(), that allows us to send a query to Elasticsearch’s, searching across all fields for the given term. You can see that in the query passed in the body of the call to search:{"match": {"_all": term}}. _all is a special keyword that lets us search all fields. If instead, our query was{"match": {"title": term}}, we would only see recipes that contained the search term in the title.

The results come back in order of decreasing “score”, which is Elasticsearch’s guess at the document’s relevance based on keyword frequency and placement. For a more complicated search, we could tune the relative weights of the score (i.e. a hit in the title is worth more than in the description), but the default seems to do pretty well without it.

You’ll also notice that the search accepts an offset argument. Since the results are ordered, we can use this to fetch more results if requested by telling Elasticsearch to skip the first n results.

Some Notes on Deployment

Deployment is a bit beyond the scope of this article, but if you want to take your recipe search live, you need to be careful. Elasticsearch has no concept of users or permissions. If you want to prevent just anyone from adding or deleting recipes, you’ll need to find some way to prevent access to those REST endpoints on your Elasticsearch instance. For example,OpenRecipeSearch.com uses nginx as a proxy in front of Elasticsearch to prevent outside access to all endpoints butrecipes/recipe/_search.

Congratulations, You’ve Made a Recipe Search

Now, if you open index.html in a browser, you should see an unstyled list of recipes, since our controller fetches some randomly for you on page load. If you enter a new search, you’ll get 10 results relating to whatever you searched for, and if you click “More…” at the bottom of the page, some more recipes should appear (if there are indeed more recipes to fetch).

That’s all there is to it! You can find all the necessary files to run this project on GitHub.