Foat Akhmadeev

Web Crawling With Akka

Sun, 17 Jan 2016 09:53:53 +0000

In the previous article we have seen an Akka introduction. Now it is time to move to something more interesting. There are many applications, where Akka shines. Today we will take a look at a web crawler example. It contains several interesting parts where Akka shows that multithreading application can be really easy to design and develop.

Project design

We want to start from the design, it will provide us a quick look of how the system should look like. Where can we get some good examples for this task?

A web crawler is usually a part of a web search engine. And the most popular search engine currently is Google. You can take a look at the The Anatomy of a Large-Scale Hypertextual Web Search Engine article, it provides a brief overview of how Google was designed at first. So, if you want to build a similar project, you can surely start from this article.

Assumptions

The application we want to build is much smaller. To make it more interesting we will base our web crawler on several conditions:

Web crawler must not spam websites, so each request to a single website should be delayed.
All operations must be non-blocking.
We assume that the web connection is not good, so the crawler might fail sometimes. In such a case the system must recover in the face of failure.
Since we will be running the crawler on a regular computer, then we need to stop the system when we scrap enough links. The check itself will be regulated by a simple threshold value.

Actors design

Our system will be divided into Actors. Each Actor represents a logical block that operates with a different data. The first thing that we need is a starting point, this can be just a simple object class, which creates a main Actor and starts the messaging. The main Actor itself will regulate some high order operations like how many links we still need to scrap or what to do in case of a scraping failure. In addition, it needs to manage other Actors. I think, the name that suits this Actor well is Supervisor.

It will be much easier, if we create a separate Actor for each website we need to crawl. In that case we do not need to worry of how to synchronize delays with other websites scrapers. Since this Actor operates with a single website, we will call it SiteCrawler. The Supervisor Actor will send here the information to be scraped.

The SiteCrawler needs to get all needed information from a web page. The problem is that the scrapping might fail, as we assumed previously. To separate the scraping functions we create a Scraper Actor. It will process the provided url and send the status back to the SiteCrawler. If the scraping fails, SiteCrawler will send an error message to the Supervisor after the timeout.

The scraped information will be sent from the Scraper directly to the Indexer Actor, which will store the content information. Once it receives the information it will not only store the necessary data, but also send urls that needs to be scraped to the Supervisor. To make the Indexer a bit more interesting, we will print all collected data before stopping it.

Project diagram

You can assume that each Actor can represent a single machine, so to make the communication process better, we try to minimize the information that is sent from one Actor to another.

The diagram of the project classes is as follows:

Project diagram. Shows main parts and processes of the system.

Classes in details

We finished our preparations. Now, we want to implement the system using Scala and Akka. Let’s start that from the same order as we did in the previous section.

StartingPoint

The App class of the project should do several things, such as:

creating the main Actor and initializing it with a first website we want to process,
stopping the whole system if the process was not ended yet.

Here is the object implementation:

import akka.actor.{ActorSystem, PoisonPill, Props}

import scala.concurrent.Await
import scala.concurrent.duration._
import scala.language.postfixOps

object StartingPoint extends App {
  val system = ActorSystem()
  val supervisor = system.actorOf(Props(new Supervisor(system)))

  supervisor ! Start("http://foat.tech")

  Await.result(system.whenTerminated, 10 minutes)

  supervisor ! PoisonPill
  system.terminate
}

The thing I want to highlight is that the stopping after the delay demostrated here is a last resort. It is much better to stop the system when the program has accomplished its goal.

Supervisor

The Supervisor has four basic variables:

How many pages we visited (sent for scraping).
Which pages we still need to scrap.
How many times we tried to visit a particular page. This is needed, since we think that scraping might fail and in that case we need to visit a page several times.
Store for SiteCrawler Actors for each host we need to deal with.

Here is one way you can represent those variables:

var numVisited = 0
var toScrap = Set.empty[URL]
var scrapCounts = Map.empty[URL, Int]
var host2Actor = Map.empty[String, ActorRef]

When we scrap a url, we need to send it to an Actor which processes urls for one particular host.

def scrap(url: URL) = {
  val host = url.getHost
  println(s"host = $host")
  if (!host.isEmpty) {
    val actor = host2Actor.getOrElse(host, {
      val buff = system.actorOf(Props(new SiteCrawler(self, indexer)))
      host2Actor += (host -> buff)
      buff
    })

    numVisited += 1
    toScrap += url
    countVisits(url)
    actor ! Scrap(url)
  }
}

Here, we check if an Actor for the provided host is presented, if not, we create it and add to our host2Actor container. Next, we add the url to the collection of pages we want to scrap. After all, we notify the SiteCrawler Actor that we want to scrap the url.

The receive function body of the Supervisor class contains a handler for each received message. Each message is a case class. To see all messages, check the Messages class in the project repository.

def receive: Receive = {
  case Start(url) =>
    println(s"starting $url")
    scrap(url)
  case ScrapFinished(url) =>
    println(s"scraping finished $url")
  case IndexFinished(url, urls) =>
    if (numVisited < maxPages)
      urls.toSet.filter(l => !scrapCounts.contains(l)).foreach(scrap)
    checkAndShutdown(url)
  case ScrapFailure(url, reason) =>
    val retries: Int = scrapCounts(url)
    println(s"scraping failed $url, $retries, reason = $reason")
    if (retries < maxRetries) {
      countVisits(url)
      host2Actor(url.getHost) ! Scrap(url)
    } else
      checkAndShutdown(url)
}

Let’s take a closer look at IndexFinished and ScrapFailure handlers.

When the Indexer finishes its processing, it sends the url information to the Supervisor using IndexFinished message. We check if we want to scrap received urls or not based on the number of pages we already visited. If yes, we proceed with each url that we did not try to scrap before. The checkAndShutdown function removes url from toScrap set and, if there is no urls to visit anymore — shutdowns the whole system.

The ScrapFailure message is received when the scraping fails. In that case we need to decide if we want to go on scraping a url or not, we do that by counting the number of visits for the url.

Following this link you can check the whole source code for the Supervisor class.

SiteCrawler

One of the most interesting things in the system is how we handle the delays for each website in a non-blocking way. To achieve that we placed this process in a separate class. In addition to this, we need to recover after the scraping failure somehow.

Want to see how it works? See the SiteCrawler class implementation below.

class SiteCrawler(supervisor: ActorRef, indexer: ActorRef) extends Actor {
  val process = "Process next url"

  val scraper = context actorOf Props(new Scraper(indexer))
  implicit val timeout = Timeout(3 seconds)
  val tick =
    context.system.scheduler.schedule(0 millis, 1000 millis, self, process)
  var toProcess = List.empty[URL]

  def receive: Receive = {
    case Scrap(url) =>
      // wait some time, so we will not spam a website
      println(s"waiting... $url")
      toProcess = url :: toProcess
    case `process` =>
      toProcess match {
        case Nil =>
        case url :: list =>
          println(s"site scraping... $url")
          toProcess = list
          (scraper ? Scrap(url)).mapTo[ScrapFinished]
            .recoverWith { case e => Future {ScrapFailure(url, e)} }
            .pipeTo(supervisor)
      }
  }
}

First of all, we create an instance of SiteCrawler for each website (host) in the Supervisor Actor as we saw previously. In that case we need to deal with only one website and do not worry about synchronizations with the others.

We create a scheduler that sends a process message to the Actor each second. We cannot call the internal process directly, since there is no synchronization between it and the SiteCrawler Actor. Without the scheduler it might be possible that we modify toProcess variable in two places at the same time: when we add a new url to that variable (Scrap message) and when we remove the last added element from the list (process message).

When we need to get a response from an Actor, we can use the ask pattern. It uses a ? symbol. The timeout variable implicitly defines how much time we wait before the ask fails automatically. And if it does, the process recovers with the ScrapFailure message. Finally, we send the status back to the Supervisor.

After all, we get a non-blocking Actor, which processes a website without spamming it.

Scraper

The Scraper Actor is simple. It does not contain any state (so we could use a simple Future instead of an Actor) and its purpose is to scrap the provided url and send the success flag to the sender and the scraped information to the Indexer. Here is the receive method of the Actor:

def receive: Receive = {
  case Scrap(url) =>
    println(s"scraping $url")
    val content = parse(url)
    sender() ! ScrapFinished(url)
    indexer ! Index(url, content)
}

The most interesting part of this Actor is a parse function:

def parse(url: URL): Content = {
  val link: String = url.toString
  val response = Jsoup.connect(link).ignoreContentType(true)
    .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1").execute()

  val contentType: String = response.contentType
  if (contentType.startsWith("text/html")) {
    val doc = response.parse()
    val title: String = doc.getElementsByTag("title").asScala.map(e => e.text()).head
    val descriptionTag = doc.getElementsByTag("meta").asScala.filter(e => e.attr("name") == "description")
    val description = if (descriptionTag.isEmpty) "" else descriptionTag.map(e => e.attr("content")).head
    val links: List[URL] = doc.getElementsByTag("a").asScala.map(e => e.attr("href")).filter(s =>
      urlValidator.isValid(s)).map(link => new URL(link)).toList
    Content(title, description, links)
  } else {
    // e.g. if this is an image
    Content(link, contentType, List())
  }
}

We used a popular library for web scraping — Jsoup to get links and other information from a web page. Since it is a Java library, we needed to convert the received information to the Scala format. It is actually possible to do that with a scala.collection.JavaConverters object. We also do a simple check if the received url is a valid content, since we want to parse only html pages.

Indexer

The two main goals of the Indexer Actor is to store the sent information and to send all scraped urls to the Supervisor. To make things more fun, we print all received data before the Actor stops working.

class Indexer(supervisor: ActorRef) extends Actor {
  var store = Map.empty[URL, Content]

  def receive: Receive = {
    case Index(url, content) =>
      println(s"saving page $url with $content")
      store += (url -> content)
      supervisor ! IndexFinished(url, content.urls)
  }

  @throws[Exception](classOf[Exception])
  override def postStop(): Unit = {
    super.postStop()
    store.foreach(println)
    println(store.size)
  }
}

Summary

If you run a program, you will see a bunch of messages that it produces. They just show the stage of a processing for a url.

We have built a simple web crawler that can successfully crawl several websites. Moreover, we achieved that using Akka, we did not use standard Java multithreading techniques, just messages and Actor instances.

Source code

The source code is available on github under MIT License.

Simple Akka Example

Sun, 27 Dec 2015 12:45:04 +0000

Typesafe stack brings many interesting features in software development and design. One of those is Akka — an open-source toolkit. It is a powerful library that takes multitreading to a whole new level. Mainly, it simplifies the building of concurrent applications. Here, we will create a simple Akka application to show its main concepts. We will see that it is really easy to build a non-blocking multitreading application using this library.

Introduction to Actors

There are many resources which can introduce you to Akka. I just want to point to two of them. First of all, the Akka documentation. Next, an excellent coursera course — Principles of Reactive Programming. Those two sources of information provide enough material to start building Akka applications.

Akka is based on actor-based concurrency model. The naive Actor definition can be represented as: Actor is a synchronous object which processes all received messages successively. The multithreading model come out from the concept that everything that is outside an Actor is asynchronous and processed in different threads. It is much better to use an Actor as an object with a state, since there is no concurrency within an Actor. All communication with an Actor must be done by sending messages to it.

Actors Example

To use the full potential of Akka Actors, we will use Actors with a state. Each Actor in Scala must be extended from an Actor trait. It contains a method which needs to be defined:

def receive: Receive

The method must implement the partial function.

Actor or Future? If there is no state in an Actor, then it is much better to use a simple Future or something related which is focused on a stateless behavior.

We will create two types of Actors which will create a communication layer — Sender which will send all messages to the second Actor — Store, the Actor that will store the received messages.

Sender

Take a look at the Sender implementation.

import akka.actor.{Actor, ActorRef, PoisonPill}

class Sender(store: ActorRef, id: Int) extends Actor {
  var count = 0

  def receive: Receive = {
    case "start" =>
      println(s"$id sends $count to the store")
      store ! s"$id => $count"
      count += 1
      if (count < 5)
        self ! "start"
      else self ! PoisonPill
  }
}

It contains a count variable which stores the number of sent messages.

Actors in Scala use ! symbol to send a message. As you can see here, the Sender sends a message with its id and count in a string to the Store. The Actors can send not only a string, but an object too.

We do not want to send messages forever, so we add a check for count variable here. If it is smaller than the threshold, we send a start message to the Actor itself to continue the cycle. Otherwise, we send a PoisonPill message. It is used to stop an Actor without breaking the message queue. So all messages which were received before the pill will be processed as usual.

Store

import akka.actor.Actor

class Store extends Actor {
  var list = List.empty[String]

  def receive: Receive = {
    case msg: String =>
      list = msg :: list
  }

  @throws[Exception](classOf[Exception])
  override def postStop(): Unit = {
    super.postStop()
    println(list.reverse)
  }
}

The Actor is simple, when we receive a string message, we store it in a list. The interesting part here is a postStop function. It is processed after the Actor was stopped, e.g. after processing the PoisonPill message. In the method we just print all received messages in a correct order.

Running the Program

To run the whole process we put Actor creation in an App object.

import akka.actor.{ActorSystem, PoisonPill, Props}

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.language.postfixOps

object Main extends App {
  val system = ActorSystem()
  val store = system.actorOf(Props(new Store))

  val sender1 = system.actorOf(Props(new Sender(store, 1)))
  val sender2 = system.actorOf(Props(new Sender(store, 2)))
  val sender3 = system.actorOf(Props(new Sender(store, 3)))

  sender1 ! "start"
  sender2 ! "start"
  sender3 ! "start"

  system.scheduler.scheduleOnce(2 seconds)({store ! PoisonPill; system.terminate})
}

First, we create 4 Actors: 1 for storing and 3 for sending messages. Next, we send a "start" message to all Sender Actors.

Finally, to stop the Store and the whole system, we create a scheduler, which will be called once after 2 seconds.

Let’s check the output:

sends 0 to the store
sends 0 to the store
sends 0 to the store
sends 1 to the store
sends 1 to the store
sends 2 to the store
sends 1 to the store
sends 2 to the store
sends 3 to the store
sends 2 to the store
sends 3 to the store
sends 4 to the store
sends 4 to the store
sends 3 to the store
sends 4 to the store
List(3 => 0, 2 => 0, 1 => 0, 3 => 1, 2 => 1, 3 => 2, 1 => 1, 2 => 2, 3 => 3, 1 => 2, 2 => 3, 3 => 4, 2 => 4, 1 => 3, 1 => 4)

It may differ on your computer, since all messages are send in different threads. As you can see, all processes are logged properly and finally Store prints the output.

Summary

Akka is a really powerful tool to create multithreading projects. It gives the most benefits when you use stateful objects. In the next articles, we will be creating a simple web crawler and will be digging in Akka a bit more.

Source code

The source code is available on github under MIT License.

Computer Vision for the Web

Tue, 20 Oct 2015 12:15:50 +0000

The book presents various examples of computer vision algorithms usage in the JavaScript world.

website page, where you can see all source code examples in action.
source code

Probably, you want to understand how libraries, which are presented in the book, work. The best way to do that is to learn the background behind them — computer vision algorithms. The book provides an intuitive way of how to use computer vision libraries without spending much time on a sophisticated theory. The information provided there should be sufficient for computer vision beginners.

If you want to learn more about computer vision and image processing you should check the following course on coursera: Image and video processing: From Mars to Hollywood with a stop at the hospital. Unfortunately, it is not available now. Probably, there will be future sessions.

I highly recommend to read the book “Digital Image Processing (3rd Edition)” by Gonzalez and Woods. It covers many topics on image processing required to understand the area.

In addition, there are several good lectures on youtube, most of them assume that you have good math skills: UCF Computer Vision Video Lectures 2014

The first lecture is missing, you can find it by the following link: Lecture 01 Introduction to Computer Vision

And do not miss a book by this lecturer — Fundamentals of Computer Vision.

Observing the Future With RxScala

Mon, 07 Sep 2015 18:11:05 +0000

In the previous article, I have talked about Scala.Rx library. It is clear that this library is useful for many applications and it can replace such libraries like AngularJS in the Scala world (see Scala.js). Here, I want to outline another reactive library. Its platform, ReactiveX, is very popular now in the Java world.

We will talk about RxScala library. Basically, it is a wrapper for RxJava. It provides a new abstraction layer to write your code and solve problems like callback hell.

Quick example

The first look at the RxScala API:

val intervals: Observable[Long] = Observable.interval(100 millis).take(10)

intervals subscribe {
  v => println(s"value = $v")
}
Await.result(Future { Thread.sleep(1500) }, 2 seconds)

We create an Observable which emits values starting from 0 each 100 milliseconds. Then, we take only first ten emitted elements. Next, we subscribe a method to the elements the Observable emits.

Run the code and see the output:

value = 0
value = 1
value = 2
...

What if we remove the Await from the code? In that case you will not see anything, since the main thread will stop its execution first, while the intervals computation will be performed in a different thread.

When it is too simple

There are many simple examples that can be created with that library. For example, using just function:

Observable.just(5, 4, 2).subscribe(print(_))

Really? I know that this is a different paradigm for developing things, but I do not think that this is a good example for showing the good sides of the library, when you can simply write this using foreach function:

List(5, 4, 2).foreach(print(_))

Because of that some examples in the web are not that catchy and do not show interesting and useful ways of how to use that library.

Asynchronous

One of the problems that reactive libraries tend to solve is to provide a better way for asynchronous calls. Let’s see what RxScala can do with asynchronous computation. We will emulate it with futures. Think of it like calling from a client to a server.

val asyncEmulation = Observable
    .just(1, 2, 3)
    .map(e => e + 1)
    .flatMap(e => Observable.from(Future { Thread.sleep(400 - 100 * e); e }))

val cd = new CountDownLatch(3)
asyncEmulation subscribe {
  v => println(s"received = $v"); cd.countDown()
}
cd.await()

As you can see, we are not creating a callback function here. Instead, we use such functions like map, flatMap and others. This prevents us from using a callback in a callback in a … Simply, callback hell. In addition to reactive extensions, there are many techniques which can solve that problem, starting from naming functions to using async/await combination, e.g. asyncawait.

Next, in the code above, you can see a different approach of waiting for the end of a computation — CountDownLatch. This class is designed for multithreading, you just set the initial count of the latch, and when it reaches zero by calling countDown method, all waiting threads are released. Besides, you can set the maximum time for await function, this will be useful, if you do not want the main thread to wait too long.

Finally, RxScala is an excellent tool which you can use in your applications to get much cleaner and understandable code. In some cases, when you do not need to deal with a lot of callbacks or do any heavy computation, you may prefer to use pure callback related approaches. But when it comes to do something more difficult, then the ReactiveX platform is the thing that will improve your code.

Source code

The source code is available on github under MIT License.

Reactive Fun With Scala.js

Sat, 29 Aug 2015 11:33:07 +0000

I want to extend the whole idea behind my notes for the Principles of Reactive Programming course. I will include here some interesting materials that I have found over the last week. Likely, Scala is a really wide field to discover something new. The idea is to tie topics from the course to the articles and to form a better understanding of the whole Scala language.

In this article we will see how to use Reactive Programming paradigm in your Scala applications.

Introduction to Reactive Programming

In the imperative programming when you have, for example, a variable a = b + c, then the changes of b or c will not affect the a. The core idea of Reactive Programming is simple — when you change b or c, then the a automatically changes too, based on new values of b and c.

Used libraries

Scala.Rx

I have chosen Scala.Rx library to show the power of Reactive Programming. This is an awesome library which enables an ability to auto-propagate variables in a reactive way!

Scala.js

What is Scala.js actually?

Let’s take a look in the past. I was influenced by the GWT project a lot when I just started to learn Java. At first, I thought that Scala.js is something similar. And it is mostly used to create admin consoles for websites. But this is a really different topic.

Scala.js provides a way to compile your Scala code to JavaScript. There is not much sense to just compile Scala to JavaScript without Scala and Java APIs. And guess what? Scala.js includes almost everything from those APIs!

Furthermore, you can reuse the same code for client and server by using Scala.js, this is a relief for many developers who needed to write a lot of similar code on front end and back end.

Before we start

I base my example mainly on materials from a Hands-on Scala.js book. If you want to learn Scala.js, you should definitely read this book. It provides comprehensive examples and covers most of the topics on this library.

sbt configuration file

First thing we need to do is to configure our project. I will not spend much time on this part, since it is covered well in the official tutorial of the Scala.js library. First of all, we add the below line to the project/plugins.sbt file:

addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.4")

Then, enable the Scala.js plugin in the build.sbt file. Eventually, we add two dependencies: scala-js-dom and Scala.Rx libraries. The scala-js-dom library adds dom manipulation functionality to the Scala.js. Here is the complete build.sbt file:

enablePlugins(ScalaJSPlugin)

name := "scala-js-reactive-mouse"
version := "1.0"
scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.scala-js" %%% "scalajs-dom" % "0.8.0",
  "com.lihaoyi" %%% "scalarx" % "0.2.8"
)

Probably, you have notices that we use %%% for dependencies. This allows compiler to use proper version of a library for each platform: Scala.js and Scala-JVM.

Scala meets JavaScript

In the current application we will build a simple page where we get a mouse position and draw a rectangle on a canvas under the cursor.

Here is a Scala source file for that:

@JSExport
object MousePosition {
  @JSExport
  def start(div: html.Div, canvas: html.Canvas): Unit = {
    val h = dom.window.innerHeight
    val w = dom.window.innerWidth
    canvas.height = h
    canvas.width = w

    // default rectangle size
    val s = 10
    val half = Point(s, s) / 2

    val ctx = canvas.getContext("2d")
      .asInstanceOf[dom.CanvasRenderingContext2D]

    val mousePos = Var(Point(0, 0))
    val center = Rx { mousePos() - half }

    Obs(center, skipInitial = true) {
      ctx.fillStyle = "gray"
      ctx.fillRect(center().x, center().y, s, s)
    }

    def clear = {
      ctx.fillStyle = "white"
      ctx.fillRect(0, 0, w, h)
    }

    div.appendChild(mousePos)
    div.appendChild(center)

    dom.window.onkeypress = { e: dom.KeyboardEvent => if (e.keyCode == 27) clear }
    dom.window.onmousemove = { e: dom.MouseEvent => mousePos() = Point(e.pageX.toInt, e.pageY.toInt) }
  }
}

Looks pretty much like JavaScript! Except this is Scala. The source can be divided into three main parts:

Initial entities.
Reactive variables.
Event handing.

The first line is the @JSExport annotation. It indicates that the entity (object, method etc.) will be exported to the JS file.

Next, check the function definition:

def start(div: html.Div, canvas: html.Canvas): Unit

Static typing! In a big JavaScript project it is a real problem not to have this.

Then, there is a simple block where we set canvas size and get its context to use it later. It should be familiar to you, if you know JavaScript.

Reactive way

We are going reactive!

To enable Scala.Rx features do not forget to add import rx._ line in a Scala source file.

The first thing we define is a mouse position. The Point class is wrapped in a class named Var. When the Var value is updated, it forces to recalculate all connected entities such as Rx.

val mousePos = Var(Point(0, 0))
val center = Rx { mousePos() - half }

You can get the current value of both Var and Rx by calling mousePos() or center(). This is equivalent to entity.apply() method call.

Next, the Obs class observes Rx or Var entities and does some post-processing (side-effects) when the observed value is changed. skipInitial means that the body of Obs will not run after the declaration and it will be evaluated only after a change of an observed value.

Obs(center, skipInitial = true) {
  ctx.fillStyle = "gray"
  ctx.fillRect(center().x, center().y, s, s)
}

Next, we append our created reactive entities to the provided <div> element. So the content of that <div> changes when the mouse position is changed.

div.appendChild(mousePos)
div.appendChild(center)

Finally, we define two events. First, for the ESC button to clear the screen. Second, to set the mouse position.

See how mouse position update is declared?

mousePos() = Point(e.pageX.toInt, e.pageY.toInt)

This can be rewritten as

mousePos.update(Point(e.pageX.toInt, e.pageY.toInt))

Yes, this is another syntactic sugar in Scala.

Wait.

Wait. WAIT! If you look closer at the appendChild function, then you will see that it needs a Node type there, not Rx! This is actually the magic of Scala, it is called implicit. I have already mentioned that a bit in the previous article.

Let’s add an additional code for the implicit conversion from Rx to Node. I have modified the original conversion from Hands-on Scala.js, to be applicable for scala-js-dom library:

import scala.language.implicitConversions

implicit def rxNode[T](r: Rx[T]): Node = {
  def rSafe: dom.Node = {
    val node = dom.document.createElement("div")
    node.textContent = r().toString
    node
  }
  var last = rSafe
  Obs(r, skipInitial = true) {
    val newLast = rSafe
    js.Dynamic.global.last = last
    last.parentNode.replaceChild(newLast, last)
    last = newLast
  }
  last
}

The main idea is that when the observer finds out that the value of Rx entity is changed, it replaces the previous <div> element with a new one.

HTML file

Here is a part of an HTML file, which is used for production:

<div id="position">
    Mouse position:
</div>
<div>
    <canvas id="canvas"></canvas>
</div>

<script type="text/javascript" src="../scala-js-reactive-mouse-opt.js"></script>
<script>
    article.MousePosition().start(document.getElementById('position'), document.getElementById('canvas'));
</script>

First, we place two <div> elements for displaying the position of a mouse cursor and a canvas, where rectangles are drawn.

Next, we add generated JavaScript file here, its name is based on a project name and a suffix opt (means optimized), there is a version which have fastopt suffix, it is used for developing and contains additional information for debugging.

The project can be built with sbt fastOptJS and sbt fullOptJS commands for develop and release versions respectively. By default, the output HTML file is placed in a <project path>/target/scala-<scala version>/classes/ folder.

We run our start function by calling article.MousePosition().start method and providing necessary parameters.

Live demo and source code

We have successfully completed the example. There are several things that I did not mention here, since they are not so important. You can see the complete source code below.

I really like both Scala.js and Scala.Rx libraries. They can help you to write a better code in their own way. Scala.js — by providing static typing and the power of Scala libraries. Scala.Rx — by implementing a new programming paradigm which can help you to write a cleaner code.

Now we can see the result. Yay!

Live demo
The source code is available on github under MIT License.

The Beginning of Scala Journey 2

Thu, 20 Aug 2015 20:10:23 +0000

These are my notes for the Principles of Reactive Programming course. The first part of this article includes my back-story of learning Scala. The second — some interesting ideas that I have learned from the first week of the course.

The Beginning

Why 2 in the article title? And where was the first? Good questions.

It was a long time ago since I was developing something using Scala. This is a really amazing programming language which provides you flexibility to write complex programs for almost anything. I really love Scala, for me it was like moving to the next level after Java. This reminds me of a xkcd comics, probably, the closest example that I can think about.

It is time for me to refresh my memories and learn something cool about that awesome language. But first, let me give you some background.

It was March of 2013. And my first course on Coursera: Functional Programming Principles in Scala. The same year I participated in Principles of Reactive Programming course. Worth noting that I got the highest score in both. The courses are really interesting, even if you do not know neither Java nor Scala, you can apply in Coursera to learn some principles of functional and reactive programming. I began to develop more on Scala, but I did not focused on it much. So eventually, I moved to another field — computer vision. But that is a story for another time.

Several weeks ago I was thinking, why not to participate in those courses again? Actually, this is a good point, since new sessions of the courses provide new materials. Especially, there are new programming assignments. And because of that, learning the courses the second time will not be boring. The bad thing is, I was a bit late for the second session of the Principles of Reactive Programming course. The good thing, I applied to the course some time ago, I just did not have time to do the course tasks. Besides, I have access to the course materials and programming assignments. I can even send the assignments via an automatic checking system (I have already tested that). Eventually, things are not bad after all. Yes, I will not receive a grade, but who cares? I just need to refresh my memories.

I have a plan now of how to keep all things that I have learned in one place — blog. If you want to understand something, try to explain that to someone else. This is a modified version of a brilliant quote from Dirk Gently’s Holistic Detective Agency by Douglas Adams: “If you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your mind.”

ScalaCheck and yield

The one thing that you need to always keep in mind when you write on Scala is there are many implicit things. And by implicit I mean not just a language construction. You need to know language perfectly to produce not only an efficient code, but error free too.

The first week of the Principles of Reactive Programming course presented an awesome library — ScalaCheck. Its author provides an exhaustive documentation on it: ScalaCheck User Guide.

I want to highlight a part for generating test data in ScalaCheck.

But first, I want to start from yield. From the documentation, we can see that this is just a syntactic sugar for flatMap and map. For example,

for(x <- c1; y <- c2) yield {...}

produces

c1.flatMap(x => c2.map(y => {...}))

Funny moment is that you can write a class as below:

class MapExample[T](val base: List[T]) {
  def map[V](f: T => V): MapExample[V] = new MapExample(base.map(f))
}

and call it using for-expression:

val me = new MapExample(List(1, 2, 3))
for {
  v <- me
} yield v + 1

even if it does not make much sense, because for { v <- me } construction means for each v from me. You can remove base from MapExample class, write def map[V](f: T => V) = 1 and still successfully compile and run the code.

Lets go back to the ScalaCheck. I want to write a simple test case for it. It will check base equivalence after yield and map.

class MapExample[T](val base: List[T]) {
  def map[V](f: T => V): MapExample[V] = MapExample(base.map(f))

  override def toString: String = base.toString()
}

object MapExample {
  def apply[T](base: List[T]) = new MapExample[T](base)
}

object MapExampleSpecification extends Properties("MapExample") {
  type T = MapExample[Int]

  val generator: Gen[T] = for {
    l <- arbitrary[Int]
    v <- oneOf(const(MapExample(Nil)), generator)
  } yield MapExample(l :: v.base)

  implicit lazy val arbT: Arbitrary[T] = Arbitrary(generator)

  property("map example") = forAll { a: T =>
    a.map(v => v * 2).base == (for {v <- a} yield v * 2).base
  }
}

We need to provide a generator for MapExample and add an Arbitraryimplicit value for this type. The generator above is based on Int. We can rewrite it using List:

val generator: Gen[T] = for {
  l <- arbitrary[List[Int]]
} yield MapExample(l)

This will also include empty lists.

If you check the source code of the Gen class, you will see that it implements a map function too. Good luck trying to understand of how the code works without knowing the result of the yield production.

Sometimes Scala seems like a very difficult programming language. However, there is no reason not to get big advantages:

Obtaining the same performance as in Java (or even better in some cases).
Writing 2-3 times less code compared to Java.

Source code

The source code is available on github under MIT License.

Java Aspects Using Spring AOP and AspectJ

Fri, 05 Jun 2015 13:52:42 +0000

Aspect-oriented paradigm is a really cool feature that you can use in your projects. With it you can easily apply additional functionality to your code without modifying the main body of a function.

Here, we will see how to configure a simple Gradle project using Java with such popular frameworks as Spring AOP and AspectJ.

Gradle configuration

We start from the gradle configuration file:

group 'me.foat.articles.aspects'
version '1.0'

apply plugin: 'java'

apply plugin: 'application'
mainClassName = "me.foat.articles.aspects.Application"

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile "org.springframework:spring-webmvc:4.+"
    compile "org.springframework:spring-aop:4.+"
    compile "org.springframework.boot:spring-boot-starter-web:1.+"
    compile "org.aspectj:aspectjrt:1.+"
    compile "org.slf4j:slf4j-api:1.+"
}

As you can see here, we use dynamic versions for gradle, e.g. 4.+. This specifies the placeholder for the latest major version of 4. In addition, we apply an application plugin, it is really useful if you do not use any IDE. You can just use gradle run in command line to start the application and that’s it!

Example function

As a starting point we will use Spring Boot. It is an excellent framework with the help of which you can quickly build a test Spring Framework application.

First, we create a simple method which takes a number as a string and adds the multiplication symbol to the end of the number.

@RequestMapping(value = "/multiply", method = RequestMethod.GET)
@AroundMethod
public String multiply(HttpServletRequest request, @RequestParam @ChangeParam String number) {
    log.info("Processing multiply method with a number parameter = {}", number);
    return number + " * ";
}

Probably, you have already mentioned @AroundMethod annotation. This annotation is used to add new functionality around our method. In addition, there is a @ChangeParam annotation, we will talk about it a bit later.

Aspects review

Here comes the @Aspect time!

@Aspect
public class ExampleAspect {
    @Around("execution(@me.foat.articles.aspects.annotations.AroundMethod * *.*(..)) && @annotation(ann)")
    public Object process(ProceedingJoinPoint joinPoint, AroundMethod ann) throws Throwable {}
}

This is a common aspect which is applied around a method, there are also many different types of aspects, such as @Before and @After which run before and after a method respectively (what a surprise!). And do not forget to add @Aspect as a class annotation.

This aspect is applied only to those methods which have @AroundMethod annotation. Plus, we add this annotation as an argument, so we can address its fields.

We have finished the preparation part. Now lets see what we can do with all of that.

We can get method arguments. Great!

Object[] args = joinPoint.getArgs();

Or method information itself. Much better!

Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();

Or even better - extract all annotations of method arguments. Excellent!

Annotation[][] annotations = method.getParameterAnnotations();

Java 8 features

Lets get an argument index. Remember @ChangeParam annotation? We use java 8 and in that case let me show you some cool features that java provides in a functional way.

annotations is an array of arrays, because for each method argument we may have several annotations. First, we create an index range:

IntStream
    .range(0, annotations.length)

we need to filter it somehow to get an appropriate index. We apply a filter with a predicate for that:

.filter(i ->
        Arrays
        	// convert array to a stream to be able to use java 8 features
                .stream(annotations[i])
                // check if any of argument annotations meet the condition
                .filter(a -> a instanceof ChangeParam)
                .findAny()
                .isPresent())

And as a final point we add .findFirst(); to get the first argument index which have @ChangeParam annotation.

This is a functional style to write the code and it is closer to the natural language, write it more and you will get used to it. Just try!

We have got the index. We can change the argument and do whatever we want with it before the function starts. Oh, and how do we actually execute a function?

Object result = joinPoint.proceed(args);

This runs the function and returns its result. If you want, you can override the result by placing a new result as a return value of our ExampleAspect#process function.

Try it!

Lets run an example (source code). http://127.0.0.1:8080/multiply?number=42 if you follow the url you will see the result (42 * 100), where ( was applied before the method (in the aspect), * added by the method and 100) after the method (in the aspect), where 100 is the value of @ChangeParam annotation.

Internal method calls

The next step is a bit tricky. Suppose we want to add another function and call it from multiply method.

@AroundMethod(value = 200)
private String internalMethodAdd(@ChangeParam String number) { return number + " + "; }

We want the result to be ((42 + 200) * 100) in that case. But what we see is (42 + * 100). That is not correct! What could go wrong?

Actually, nothing. That is how Spring AOP works. Spring uses proxy to call a function. Then, if you call an internal method you call the exact method bypassing spring, not a proxy you need to. What can we do with that? We need native AspectJ.

AspectJ config

To include native AspectJ functionality uncomment those lines in the gradle.build file:

configurations {
    aspectjweaver
}

aspectjweaver "org.aspectj:aspectjweaver:1.+"
runtime configurations.aspectjweaver.dependencies

applicationDefaultJvmArgs = [
        "-javaagent:${configurations.aspectjweaver.asPath}"
]

Here we get aspectjweaver library from gradle cache and add it as a javaagent. Now, try to open the link again. ((42 + 200) * 100). We did it!

As we see, aspect paradigm is indeed a powerful tool. We can build complex systems by using it. It is usually used for logging and method performance testing, but aspects are not limited by this.

Source code

The source code is available on github under MIT License.

Surface prediction

Tue, 12 May 2015 19:01:03 +0000

Introduction

This is the home page for my “Surface prediction for a single image of urban scenes” paper.

The source code and annotated data available on github. If you use the software or annotated data, please cite the paper:

F. Akhmadeev. Surface prediction for a single image of urban scenes. Computer Vision - ACCV 2014 Workshops, volume 9008 of Lecture Notes in Computer Science, pages 369–382. Springer International Publishing, 2015.

Additional information

The paper was originally presented on the SUAS 2014 workshop.
The poster is available here.

Real-time traffic sign recognition

Sun, 31 Aug 2014 21:00:00 +0000

In RoadAR company I have developed a fast algorithm for traffic sign detection and recognition which works in real-time on mobile phones.

The algorithm is written on C++ and it supports most of Russian round and triangular signs.

You can find an application where this algorithm is used in Google Play.

This is one of the previews of the RoadAR application:

Java + web

Thu, 27 Jun 2013 21:00:00 +0000

In 2011-2013 I worked at Digital Zone company, the main project there was Ulmart. My main contribution includes:

New database structure for Ulmart 2.0 which was based on MySQL.
Fast catalog of goods which showed great performance under high load. Technologies like Spring Framework, JSP and eventually FreeMarker were used.
The search platform that was created by using Apache Solr.