47 Ronin

Reports and Business Intelligence in a reactive system

Fri, 26 May 2017 18:51:15 +0000

In a previous post, I had advocated for isolating services all the way down to their data using an Event Driven Architecture. The one potential sticky point: how to deal with reporting on the data stored by those isolated data stores, or even run any sort of analysis on them.

In a traditional system architecture, there are 2 popular options for obtaining data that can be used for creating a report or for running BI tools. Typically, folks would either have a Data Warehouse process that relies on ETL processes to extract relevant data from different locations and dump it into its data marts. Another approach I’ve seen used is to directly query the data from the operational data stores when building reports or performing analysis (note: not advisable unless you can guarentee that large queries won’t affect performance or user experience).

The problem with the latter option is that it does not respect boundaries at all. One can argue that it is fine for a reporting system not to worry about any boundaries since its scope is the entirety of the system…but that’s a slippery slope. Adopting such an opinion might lead to more and more systems being deemed to have too broad a scope for them to respect data boundaries.

The same goes for the first option, assuming the extraction is done directly against the data store. To avoid directly querying your data stores, you could expose the bounded context’s data via HTTP endpoints whose sole purpose would be to act as Data Hoses. You could even slap a GraphQL interface on them to make it easier for the ETL process to query the data it wants. While that would better preserve the data boundaries, it is slower than direct database access and it exposes you to the risk of not being able to query data when one of those data hose services goes down.

Thankfully, with an Event Driven Architecture, there is a 3rd way of accessing the data from the different services that still maintains the data boundaries without the risk of your reporting/analysis process being severly impacted by an external service being down.

Instead of having an ETL process that periodically queries data, you could instead have data ingestion services that would subscribe to various events coming out of the different services, then immediately transform and load that data into your data marts. No need to schedule an ETL anymore, as the data populated in your data mart as you as a change has occured. It’s just that simple. There are utilities such as Apache Spark that makes it relatively easy hook up to stream of data coming from Kafka for example, and perform whatever operation on them that you see fit.

Directly querying data held by external services is dangerous because it couples services together. If a team working on Service A was to make a change to the structure of tables it relies upon then the team working on Reporting (or any other service that relies on directly querying its data) would have to change their logic as well. By relying on events instead, the team working on Service A can do whatever they want to their data store - including changing it from a relational store to a graph database for example - all without impacting anybody else. The only time other services would be impacted is when a change occurs to the Event being published.

In my experience, there rarely is a good reason for directly querying the data of another Bounded Context. My advice for anyone reading: Please Do Not Harm, Don’t do it.

Theory: Extending HAL to support simple forms

Mon, 01 May 2017 11:38:14 +0000

HAL is a format created by Mike Kelly used to make APIs more explorables. It can also be used for automatically generating parts of the UI via its links. But what if it could be pushed a bit further?

Motivation

Most forms can be made generic in many Enterprise Applications (Web and Mobile). Their inputs are usually simple. They don’t usually have much in terms of prettyness. They are temporary holders of information that would eventually be sent over to the backend. Therefore, if you have a way to describe what your form should look like, it should be possible render them generically on any client platform, be it Web or Mobile.

HAL already provides a way to incorporate some metadata within the resource being returned.

{
    "_links": {
        "self": { "href": "/orders" },
        "curies": [{ "name": "ea", "href": "http://example.com/docs/rels/{rel}", "templated": true }],
        "next": { "href": "/orders?page=2" },
        "ea:find": {
            "href": "/orders{?id}",
            "templated": true
        },
        "ea:admin": [{
            "href": "/admins/2",
            "title": "Fred"
        }, {
            "href": "/admins/5",
            "title": "Kate"
        }]
    },
    "currentlyProcessing": 14,
    "shippedToday": 20,
    "_embedded": {
        "ea:order": [{
            "_links": {
                "self": { "href": "/orders/123" },
                "ea:basket": { "href": "/baskets/98712" },
                "ea:customer": { "href": "/customers/7809" }
            },
            "total": 30.00,
            "currency": "USD",
            "status": "shipped"
        }, {
            "_links": {
                "self": { "href": "/orders/124" },
                "ea:basket": { "href": "/baskets/97213" },
                "ea:customer": { "href": "/customers/12369" }
            },
            "total": 20.00,
            "currency": "USD",
            "status": "processing"
        }]
    }
}

I believe it could be possible to extend it to incorporate metadata that could guide the implementation of a UI form.

{
  "firstName": "04c48da7-5b2d-4eb4-a93e-50a22979394b",
  "lastName": "99e7d915-471a-481c-8e13-57523af39cad",
  "age": 77,
  "description": "6b51e2fa-a1f9-47bf-8ac1-944a7e60bdd4",
  "_links": {
    "self.Save": [
      {
        "href": "/Example/SaveExample1",
        "kind": "Save",
        "name": "Save"
      }
    ]
  },
  "_presentable": {
    "fields": [
      {
        "restrictions": [
          "Required"
        ],
        "boundTo": "firstName",
        "editable": true,
        "name": "First Name"
      },
      {
        "restrictions": [
          "Required"
        ],
        "boundTo": "lastName",
        "editable": true,
        "name": "Last Name"
      },
      {
        "restrictions": [
          "Required"
        ],
        "boundTo": "description",
        "editable": true,
        "name": "Description"
      },
      {
        "restrictions": [
          "Optional"
        ],
        "boundTo": "age",
        "editable": false,
        "name": "Age"
      }
    ],
    "actionBoundTo": "self.Save"
  },
  "_metadata": {
    "objectName": "Example1",
    "description": "Example 1"
  }
}

The json above is a variant of HAL that includes 2 new properties: _metadata and _presentable.

The _metadata field could be used to provide additional info on the form. In theory, it could even be used to infer the name of the javascript function to be called on form submit.

The _presentable field contains a description of the fields that can be presented as well as their input restrictions. It also contains the name of the link/action that can be called to send the form to the backend.

Here’s how it could be coded up in C#.

And here’s how it could be rendered in Javascript.

NOTE: Turns out there was already such an extension to the format called HAL-Forms.

Based on a very basic test I did, this should work. A refinement would be to better follow the HAL spec by keeping the actual resource in a _embedded property.

While this has the potential to work, it could be interpreted as trying to make things too generic. Plus, it can be argued that this would lead to coupling the presentation layer to the API layer.

Just an idea.

Microservices isolation patterns: Reactive

Fri, 28 Apr 2017 00:55:15 +0000

In a previous article, we introduced a problem for us to solve using different isolation patterns for Microservices. Wel also examined the traditional way of achieving this via HTTP or RPC. Another option is to make the system reactive.

There really should be no reason why the Comment Service (or any other service) should know of each other’s existence. In reality, the only service they should be cognizant of is the Identity Service, which the Front-end services communicate with in order to validate the user’s token. Beyond the Identity service, each individual service should not be privy to the existence or needs of each other.

Consequently, each service should be solely focused on fulfilling their responsibilities without explicitly assisting one another. That means services should not call each other in order to trigger some sort of action. They should also not query each other’s data, be it directly or through HTTP/RPC calls. They should fully own their data and their destiny. However, in the rare instances where an explicit targeted call must be made, it should be done in a Location Transparent fashion.

But then, how would we implement those 4 features? Event Publication with a hint of data duplication or caching.

Upon completion of any state changing action, each service within the system would publish event. That event would contain information about what was done (which can be inferred from the type), as well as information deemed pertinent. Any service that would be interested in that type event would subscribe to its source (e.g. RabbitMQ topic, Kafka topic, WebSphere MQ subscription, etc..), and react to the event either by storing data it would need for future actions, or by performing an action.

For example, whenever a user edits his account preferences, the Privacy Preferences service would publish a AccountPreferencesChanged event that would hold information other services could be interested in.

case class PrivacySpecification(onlyFriendCanComment: Boolean)

case class AccountPreferencesChanged(accountId: UUID,
                                     privacy: Option[PrivacySpecification],
                                     updatedAt: Long)

Likewise, whenever a friendship is established, the Friendship service could be a NewFriendshipEstablished event.

case class FriendshipRequester(accountId: UUID)

case class FriendshipRequestee(accountId: UUID)

case class NewFriendshipEstablished(requester: FriendshipRequester,
                                    requestee: FriendshipRequestee,
                                    establishedAt: Long)

Whenever a comment is posted, a CommentPosted would be created. Whenever a Biky Post is liked, a BikyPostLiked event would be published. Essentially, every time a state changing action occurs, an event would be produced that can be consumed by external interested parties. Consequently, implementing features 1 to 4 is now trivial.

For feature #1 and #4, within the bounded context of the Comment Service, a data ingestion service would exist that would subscribe to different events, namely the NewFriendshipEstablished, FriendshipRescinded, AccountPreferencesChanged, BikyPosted, and the BikyDeleted events. That service would store information relevant to the Commenting bounded contexts in a schema shared only be services within that bounded context. With that done, the Comment Service would no longer need to query data by calling the Friendship service, the Privacy Preferences Service or even the Biky Post Service. It would query its own data store to check upon the existence of the Biky, to validate the privacy preferences of the users with regards to commenting, as well the relationship between user x and user y.

The Karma Granter Service would no longer depend on anyone as well. In the Karma bounded context, a data ingestion service would listen to different events such as NewCommentPosted and BikyLiked to determine the karma score.

The Feature Access service and the ToS Breach Service would both listen to a NewKarmaScoreComputed event. When that score is less the -50, the Feature Access Service would know to publish a message that would be consumed by a Service in the Account Management bounded context to temporarily lock the account’s access to some features by setting a flag one the object that each service gets when they decode the bearer token, and then send an e-mail alerting the user. Based on the NewKarmaScoreComputed event, the ToS Breach Service would know to immediately start its investigation.

What if the ToS Breach Service goes down? While breaches of the Terms of Service by a user won’t be detected until it comes back up and recovers its unacknlowledged messages, the remaining services would continue to operate independently. With this approach, the outage of one service would no longer bring down multiple parts of the system.

What? Duplicated data?

Perhaps you’re concerned about the duplication of data. As developers, the removal of duplication is ingrained in our brain. Plus, it comes at a cost: space.

Disk space is cheap, RAM…not so much. However,wWhat is definitely expensive is losing customers. Many nice articles have been written about duplication such this one by Udi Dahan or this one by Sandi Metz. While those articales are primarily talking about code duplication, I find the points they raise to also be applicable to data duplication. Keeping data seperate and duplicated like this ensures you can use the right database engine (the right abstraction) for your service and fully model all the data a logical service would operate on.

What about data consistency?

Another concern is consistency. There will be a period of time during which the data held by the different data stores will be out of sync before eventually being up-to-date. This is called eventual consistency, two words that make some folks cringe. Part of the reason for it is that we often look at it in terms of consistent versus eventually consistent data, whereas we really should be looking at in terms of operability and performance.

You can have a system that is consistent ALL the time across all components, but that comes with the risk of it not being scalable and not even being operable when one of its components is down. Or you can have a system that IS consistent but with a small delay (ideally), and is still be operable and performant even when multiple of its components are down. Udi Dahan and Greg Young wrote some nice articles on this topic years ago that are still relevant today.

In my opinion, if you can afford the operational cost of Microservices, investing in making your system reactive is the way to go.

You don’t necessarily need Message Queueing systems like RabbitMQ or Kafka to achieve this. You could achieve a similar result using something like Akka Cluster. The one thing to note however is that, as a consequence of using RPC, the delivery guarentee will be At Most Once, whereas using RabbitMQ, Kafka or most other Message Queueing Systems will give you an At Least Once delivery guarantee.

With that said, there is an alternative we have briefly hinted. One extreme option to implement all of our features without concern for other services being down is to blatantly ignore the service boundaries and query the needed data directly in the database. Don’t do this. If you’re tempted to do it, you should really think extra hard about it….and then still not do it. It’s a trap. 99 problems lie in wait behind this door and you don’t to have to solve even 1 of them. Don’t open it. Try to avoid it. Very few cases like ETL may warrant such an approach, but even for such scenarios there are other possibilities. I shall explain how in a future post.

Microservices isolation patterns: Rest and RPC

Fri, 28 Apr 2017 00:55:08 +0000

In the previous article, we introduced a problem for us to solve using different isolation patterns for Microservices. The first one we’ll consider is the traditional one: keeping the data separate but communicating over HTTP/RPC.

The features

Some of the desired feature we want to implement are as follows:

a Biker Karma needs to be recomputed whenever a user writes a comment. The karma score would vary based on whether or not the post being commented was that of a friend.
a Biker Karma needs to be recomputed whenever a user likes a Biky Post
users with a karma score less than -50 must receive an e-mail saying their account is temporarily disabled and are not allowed to post comments or bikies until a semi-automated validation of their activities is performed to ensure they did not breach the Terms of Service (ToS).
users can limit comments on their Bikies to their friends only. Users can make that selection globally on their account privacy settings or on a post by post basis.

The approach

To implement #1 and #2, we could have the Comment Service and the Like Service call the Karma Granter service, which responsible for karma score re-computation. The Karma Granter would call the Friendship service itself to determine the new score.

In conjunction with #1 and #2, implementing #3 could mean that when the Karma Granter performs a computation that yields a score less than -50, it could send a “You got bad karma” email and call the Feature Access Service in the Account domain that would lock down the account.. The Feature Access Service would also call a ToS Breach Service to validate nothing fishy is going on.

As for #4, the Comment service would always need to call the Privacy Service to check whether privacy was globally defined as “Only Friends can comment on my posts”. Depending on the answer, it may directly call the Friendship service to see if the commenter is a friend, or would first need to the Biky Post service to see whether that privacy option was selected and then call the Friendship service based on the answer.

Both the Comment Service and the Like Service might also need to call the Biky Post Service to ensure the post to be commented on or to be liked actually exists.

Here’s an example of what it could look like.

The catch…

Lots and lots of calls to different services are required to implement each feature. This is fine in theory. After all, isn’t this similar to making a method call? You could use Swagger code-gen to produce clients of those services that you could call as if they were methods. All is well……..not!

Distributing your system makes it more likely to fail for all sorts of reason like network connectivity issues, concurrency flaws, service availability, etc. In fact, the one thing you can be certain of is that a failure will happen. It is inevitable. As one of many who have been on the receiving end of Murphy’s law, the one thing you learn is to do your best to discover what those failures might be and establish recovery and remediation strategies.

This where it gets a bit tricky with HTTP/RPC.

Let’s say that, for whatever reason, the Karma Granter Service is down. Since such an outage can occur, perhaps you will add some retry logic for your requests with retry or time limit. If the outage takes more than a few seconds however, your comment service would have become unusable, sending back errors on every requests until you have remediated the situation.

What if the ToS Breach Service is down? The Comment Service will also become unusable since it relies on the Karma Granter Service, which relies on the Feature Access Service, which relies on the ToS Breach Service. One service that is malfunctioning would lead to multiple dependent services failing on every request, sometimes in surprising way as the dependency chain might not always be evident.

The next article in this series will discuss a different pattern that I find more manageable and less risky.

Microservices isolation patterns

Fri, 28 Apr 2017 00:54:40 +0000

The Microservice architectural pattern has gained in notoriety since being formally coined in 2014. Many books and articles have been written on this subject, including in this blog. One topic however that can never be stressed enough is the importance of ensuring proper isolation of services and their data both within and outside bounded contexts. In this series, we’ll examine two ways of achieving this.

A quick refresher

Like i’ve stated before, you can think of microservices as small and autonomous services or independent processes that work together within a bounded context, communicating with each other over lightweight transports like HTTP.

The problem

Let’s say we’re building a social platform for Bike lovers to chat about bikes, wheels and whatever else they fancy. Let’s imagine as well that we’ve split the system into multiple services with HTTP endpoint:

Identity Service
Registration Service
Privacy Preferences Service
Comment Service
Basic Profile Service
Biky Posting Service
Liking Service
Friendship Service
Karma Indicator Service

Let’s also claim that alongside those different “Front-end services” we also have additional ones not directly accessible by consumers outside our internal network that do things like actually calculating a Biker Karma, monitoring breaches of Terms Of Service, etc…

Let’s also assume you have your data defined in different isolated schemas/databases as such:

Account data used by the Identity, Privacy Preferences and Registration services
Profile data used by the Basic Profile and Identity Integration Services
Like data used by the Liking Service
Commenting data used by the Comment Service
Biky Post data used by the Biky Posting Service
Friendship data used by the Friendship Service
Karma data used by the Karma service

Some of the desired feature we want to implement are as follows:

a Biker Karma needs to be recomputed whenever a user writes a comment. The karma score would vary based on whether or not the post being commented was that of a friend.
a Biker Karma needs to be recomputed whenever a user likes a Biky Post
users with a karma score less than -50 must receive an e-mail saying their account is temporarily disabled and are not allowed to post comments or bikies until a semi-automated validation of their activities is performed to ensure they did not breach the Terms of Service (ToS).
users can limit comments on their Bikies to their friends only. Users can make that selection globally on their account privacy settings or on a post by post basis.

How would we go about implementing these features knowing that some of the information needed for each of them is dispersed across bounded contexts?

Let’s explore the first pattern

HAproxy in the era of Microservices

Fri, 23 Oct 2015 16:27:46 +0000

“Microservices”, the latest architecture buzzword being thrown around to describe perhaps one of the most interesting architecture styles of this decade.

What are microservices?

To use Martin Fowler’s definition:

In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies

To simplify it further, you can think of microservices as small and autonomous services or independent processes that work together within a bounded context, communicating with each other over lightweight transports like HTTP.

How small is micro? As with everything, it depends. Some claim that a microservice should consist of a single actor. Others, like Jon Eaves, claim it should be something you can complete in at most 2 weeks. I would think a general rule of thumb is that it should be small enough to be easily maintained by a small team (or a dev), and that it should focus on doing one thing and doing it very well.

The benefits of such an architecture are plentiful. For example, it makes it easier to adopt new technologies faster and to grow your team. It makes it easier to adopt the appropriate technology for solving a particular problem (e.g. you could have a microservice written in Scala and using Neo4j for storage, alongside another microservice written in Go and using Cassandra in the backend). It limits the risks of a complete system shutdown as most pieces are spread across a fleet of services across several machines. It makes it easier to scale on smaller machines, which can be huge cost saver.

Nevertheless, such an approach adds complexity in different areas, one of which is routing.

Unified routing

Assuming a relatively complex domain broken into multiple Bounded contexts, each of them can have 2 to N amount of microservices, each of them doing specific domain things. When scaling them, that number would grow even larger.

If you’re trying to consume those services, you probably don’t to want to keep track of them all. To go even further, if I’m writing a mobile application that needs to communicate with those many services, I don’t want to have to maintain all the many addresses to those microservices. What would be better for me would be the ability to program against 1 base URL per bounded context (e.g. http://account.mystuff.com/api), and somehow have it figure out what microservices needs to be called based on a header.

This is where HAproxy comes in.

HAproxy with microservices

As its name mentions, HAproxy is a high availability proxy server and load balancer that works both for TCP and HTTP. More information on it can be found in its docs.

Amongst its many features is the concept of an Access List (acl) that can be used to determine which backend to send a request to. The acl can be used to look at the header and the url, amongst other things.

To go back to our example of a mobile application wanting a single contact point, the request the dev could send over the wire could include a header (e.g “x-microservice-app-id”) that HAproxy would then use to determine which endpoint to route to. Note: That contact point could itself be tied to A record with multiple IP addresses pointing to load balancers to avoid having a single point of failure.

Here’s an example configuration on how to do so.

The header names would be the one thing the client would have to maintain. Alternatively, if a convention was to be used, the header value would be computed, thus removing the need of the client to maintain the potentially numerous header names.

Following this approach would require you to be capable of syncing up your haproxy configuration with your deployment. There are various tools that can help such as Marathon - which has utilities to produce haproxy configurations similar to the one in the above example, and Kong - which is an API gateway.

Choosing a tech stack for your bootstrapped startup

Sun, 11 Oct 2015 00:00:00 +0000

Programming languages. Tons of them have emerged over the years. Some decent great ones. Some decent ones. Some pretty awful ones as well. Most have strong points. All of have weak points. Often time I hear/read folks claiming that it doesn’t matter. Just pick one and roll with it. Well…it does!

We often hear the expression “use the best tool for the job”. To me Programming languages are more like weapons you can use to slash problems. Choosing the right Programming language is one very important decision that might have an impact on the future of your product and, by extension, your company. It will determine how quickly you can get to market, how quickly you can deliver updates and fixes, how easily you can field your engineering staff with developers that can become productive quickly, as well as how quickly you can adopt new programming/architectural paradigms.

The choice of a programming language is only the first piece of the equation. In reality, what you are really doing when choosing a programming language is also picking a Technical Stack. For simplicity sake, I will refer to a technical stack as language + libraries + framework + tooling.

Sure with the advent of Docker and Micro Services, you could theoretically use an assortment of programming languages. What you’ll find however is that someone will have to support the software built on those different languages that used different stacks that produced different errors under similar circumstances. That’s the reason why quite a few folks tend to have platform/infrastructure teams that consolidate the tech stacks that are used.

So, what language should you choose? What should you do to determine which is most appropriate for your organization? This is especially when your funds are limited.

The Tech Stack to Get Things Done (GTD)

What tech stack to choose? This is entirely dependent on the nature of your business. What a startup building embedded system would need is most likely going to be different from say a web startup.

Assuming you are building something relatively complex that involves distributed systems, you would likely need to be able produce internal services that can communicate amongst each other, some mechanism for persistence, ways to expose some of your services’ functionality to the outside world for consumption by a website, a mobile application, or some other client. You likely would also want to be able to handle multiple requests concurrently and fast. You would probably want the language you pick to easily help you express the domain in which you are developing.

Let’s say that ConArtist Inc is developing a mobile app for artists to sell their imitation painting as well as original ones. Artists need to be able to post pictures of their painting, and folks who want would have to bid on them based on a bidding period specified by the artist. Users should be able to comment and review the painting. If the user likes an artist’s work, he/she should be able to follow that person so as to be notified of their latest work. Users should be able to sign up using Facebook or Twitter, as well as by email.

From this description, there are several components and services that are required to build this. You need account management, profile management, payment, bidding, commenting, and review aggregation services just to name a few. Let’s say you are the only tech cofounder. You’ve read on Best Practices, you believe in having systems broken down appropriately based on the Domain, you’ve read up on micro services, you’ve read up on the Lean Startup and the need to deliver quickly. You also believe the project has a large potential, and you want to grow your team.

From this description, you would likely want the services to be able to communicate amongst themselves internally. Assuming you are anticipating lots of traffic generating tons of data, may be you choose to store it in Cassandra as your primary database, and ElasticSearch for your read optimized data. Maybe you also decide that external communication into your system would happen over HTTP with JSON.

Based on that you need to have a stack that will:

have good, fast and mature JSON libraries
have libraries to interact with Cassandra and Elastic search
have mature frameworks or libraries for building Rest APIs capable of handling request concurrently and fast

In order to increase the delivery speed of your software irregardless of the nature of your product, you would likely want the tech stack you pick to have tooling that makes your life easy. For example:

tooling to run unit tests and acceptance tests
a compiler that is smart and fast so you can quickly deliver functionality. It shouldn’t take you hours to compile your project whenever you add a new feature
tooling to help you automate tasks associated with your project and build

Some questions to ask oneself: how easy to use are the tools and frameworks that I am choosing? Are they well maintained? How is the community around it? How well supported is the language? How hard quickly and easily will it be to evolve and maintain my system over time? How hard or easy would it be to ship artifacts (libraries/executables) on that stack? How expensive will it be to deploy those artifacts (e.g. what server requirements will be needed, what RAM usage, etc..)? How quickly can I gain proficiency on the language? How quickly can other folks get up to speed on that language/tech stack? How active and mature is the community supporting that language? How many other folks know that language/tech stack?

Why does all that matter?

If the tooling and framework you choose are hard to grasp in a relatively short time, that could potentially hinder the speed at which you will be able to implement new features, and will increase the time it would take a new hire to get up and running in your stack. Simple is boring and boring is good. It is easier to get good developers to learn a new programming language and tech stack that is easy to grasp. To be fair, some stacks tech longer to grasp but with increased rewards. I find that to be true with ML languages like Haskell, Scala and F#. Unfortunately it presents a barrier of entry that limits the pool of developers you can hire to grow your team.

If the stack you choose is littered with poorly maintained libraries, including the ones you are thinking about using, you would be setting yourself up to do additional work to support those libraries you pick by yourself, or to build and maintain your own. When starting up, it is true that you often have more time than resources. But time itself is a precious commodity better invested in building up your startup vs providing a lot of support to external libraries or even re-inventing the wheel when you can help it.

The community around your tech stack is very important. It will determine the long lasting maintenance of the language and tools you use, as well as the assistance you can get with problems encountered when implementing features using that stack. For programming languages, it will also help you gauge how easy it would be to attract good talents.

Deployment

Deployment of your solution is also very important. Docker and configuration management tools can reduce the amount of pain you would need to go through to deploy. Nevertheless, there will still be a cost associated with deploying a solution built on any stack. It can manifest itself by the size of the artifacts that would be deployed (be it binaries or the full source code), the Memory consumption, the OS restrictions (e.g. having to provision a windows server vs a cheaper ubuntu one to run your solution), the packages that must be pre-installed on the environment prior to running your solution, and also the time it takes for your solution to start.

Here’s a real-life inspired scenario:

XYZ Inc. releases a new social media platform for the enterprise after a year of betas that has seen FORTUNE 500 companies sign up for the service. Six months after the well publicized release, some black hat hacker has managed to find a problem in the system could give him access to all the accounts and payment information of the users. The cocky hacker demands you pay him up in the next 30 minutes or else!! Thankfully, your logging and monitoring tools have helped the Dev see a critical problem in the code that is VERYYYYYY easy to fix (as in 2 lines of code). Unfortunately, it takes a long time to compile the code as the compiler does not do incremental compilation. Worse, it takes a full 3 minutes for your app to load up. You have 10 servers running on multiple locations that would need to be updated. Even if we assume that you are using tools like docker and marathon to help with deploying things faster, the Math says: you won’t make it. Logic says: let’s just shutdown the system since this is a very critical problem (probably the right course of action). Legal says: the agreements signed by the clients stated guarenteed a 99% uptime, a.k.a you would breach the contract, which would cost you even more.

Solution: hacker gets a fat check because you could not deploy fast.

Sure, this is an unlikely scenario. But you get my point. Even if you follow good development practices and whatnot, you should want your tech stack to not hinder your ability to release quickly. You need to be able to fix bugs and release new features quickly.

A more reasonable scenario:

Smith et al. are building an enterprise solution that needs to be deployable on premise. The enterprise, especially large financial and business entities, tend to be very…strict in terms of what they are willing to have deployed on their premises. They typically have a process they want software vendors to adhere by, but when that is not the case and they really want/need your product they will require you to provide ongoing support. That may be good for business, but it can take you away from evolving your solution. If you choose a stack for which deployment is easy or versatile (i.e. can generate a war file for example, or can be hosted on iis, etc..), you will make your life easier.

Familiar Fallacy

One more question you could ask yourself is How familiar you are with the tech stack you are considering. Why does this matter? The more familiar you are with something, the fastest it “could” be for you to get started and ship your product.

That doesn’t necessarily mean that you should always go with the language you know best. If you are a solid FORTRAN programmer, you may find it a while lot easier for you to start coding your big idea in FORTRAN. It saves you from having to learn a new language and tech stack. That doesn’t necessarily make it the right choice. Support for Restful APIs, for connecting to high volume database (Cassandra, etc…) and ease of deployment could be an issue. Not to mention that I dont know many developers who dream to work on a startup that uses FORTRAN (but I may very well be wrong).

For ConArtist’s scenario, FORTRAN wouldn’t be a good choice. If however you are familiar with a tech stack that provides you a lot of benefits like ease of deployment, easy concurrency, a plethora of mature framework, a striving community and plenty of potential hires, it could very well be the better choice to go with it.

In conclusion, like many other choices we make in life, the best choice depends on your particular circumstances. When starting up, emphasis should be on the delivery of a stable solution that can be quickly evolved. Personal biases and preferences need to be relegated so as to make a sound decision for your business. The better the choice, the easier the road ahead.

Do No Harm: Short tale on Versioning

Wed, 03 Jun 2015 13:20:43 +0000

Here’s a little story inspired by true events. It’s 7.30 PM on a Sunday. After weeks upon weeks of hard work, you are now one commit away from releasing the new uber-awesome-oh-so-incredible (TM) application that will revolutionize the entire world. You can already see the millions of happy faces with each keystroke as you are about to push your latest changes. But you decide to do your due diligence. A new bug fix for the library you’ve been using to connect to your main datastore has been released. After reading the release notes, you realize they’ve fixed a pretty significant bug you were unaware of, so you decide to update. After all, it’s just a bug fix. Nothing should break, right? right? Wrong!

As it turns out, even though you went from version 6.3.1 to version 6.3.5, your code no longer compiles. Breaking changes to the API have been introduced in the bug fix. But you’re the resilient kind, so you decide to fix it. 1 hour later, all the tests are passing, you are ready to push to your Cloud CI service. Everything works locally and on another machine. You are done….or so you think…

It’s 8.30 PM, you receive notification that the build is broken. Everything still works on your machine, yet the library for the secondary datastore you use for analytics no longer works on the CI.

You quadruple check the version to make sure that all your working environment have the same version of that library. You contact your Cloud CI support team to ask if anything has changed with the way they deploy the database for testing. It has not. So what could it possibly be?

Out of the desperation, you use virtualbox to create a VM on which to run your tests. They fail for the same reasons they did on the CI. The world stops making sense.

After a bunch of failed attempts at fixing the problem, you decide to clear any potential caches you may have on your local environment. Lo and behold, the bug is reproduced. As it turns out, somebody updated the library you’ve been using (introducing a bug in the process) and published it to their hosted maven repository, but they did not increase the version.

It’s 1 AM Monday morning, you’ve missed the release. This would not have happened had the maintainers of those public libraries had followed Semantic Versioning and proper package management and distribution practices.

Semantic Versioning

With Semantic Versioning, the version numbers help consumers understand the type of impact they can expect from adopting the newer version of your API. For example, if only the PATCH version increases, you know it’s a bug fix with no breaking changes. It should be OK to upgrade knowing your code would still compile. A MINOR version change would indicate that new functionality/improvements have been made in a backward-compatible way, i.e. it is still safe to upgrade although it could be advantageous to start using the new functionality if the old way has been marked as obsolete/deprecated. A MAJOR version change on the other hand makes no promise on backward compatibility. Any consumer of your API would know that they have to do more work to adopt your new version, and it’s OK. Without such versioning guarentees however, version upgrades can be fairly risky.

Package Management

There are plenty of package management utility out there (Maven, Ivy, NuGet, RubyGem) to name a few. Those utilities tend to encourage package maintainers to use Semantic Versioning. Unfortunately, they do allow you to delete versions and re-upload them (essentially replacing existing versions). While I can see scenarios in which that is a good thing (e.g. delete a package that was uploaded by mistake), there are plenty of case where that can be problematic (e.g. breaking changes, bugs, etc…)

Do no harm: use semantic versioning and adopt good package management practices.

This is the first of a series of post I hope to start publishing every now and then. Any comments/thoughts are welcome.

Cassandra vs MongoDB for a bootstrapped startup

Fri, 06 Mar 2015 14:49:10 +0000

The database. One of the most boring yet most essential piece ever design by mankind. Its sole purpose, serve as a place where we store the data used by our uber-awesome-oh-so-incredible™ application. So long as it is online, that we can store our data in it, and that we can retrieve it, we tend not to concern ourselves too much with it. As stated by Patrick McFadin : “Is it scaling? Yep. Is it online? Yep. Boring”. When Bootstrapping a startup, one tends to have little time (unless you’ve quit your day job) and little resources. Getting to the point where you have a boring database would indeed be the dream so that you can focus on your app. To get there however, the very first step is picking the right one based on your application’s needs. There are many options out there. One that I have seen bee heavily recommended back when I was working on a side project was MongoDB.

MongoDB

In my time scouring the web, I have come across many recommendations, some of which I have followed when working on a side project. Back in 2013, MongoDB was still the rage. All the cool kids were using it. Why not after all? Who could resist the promise of dumping all your nested data in a collection and retrieving it just as easily? Database schemas? Why use that when you can have a dynamic schema instead? Plus, it’s all asynchronous and fault tolerant. Everything is awesome….

Note to self: don’t believe every you read on the internet.

Well, not quite. Having played with MongoDB and having read the experiences of other (see McFadin’s, Mei’s, Emin’s and Morgan’s posts), I can attest that it is a good and still improving database that is lacking in a few areas, the most important of which being concurrency and fault tolerance. Plus the idea that you can just dump nested data in there is not a great one. I understood early on that retrieving segments of that data or updating it atomically can be quite painful the more nesting your data has. Also, there is a cost to having a dynamic schema vs a static one.

To be fair, I love MongoDB. It was fairly easy to get started with. It was awesome. The database was indeed boring…until the application’s complexity and modeling need started to increase.

It can be quite difficult to model a social network using MongoDb. Let’s take a naive example. Let’s say you have a User class defined as below:

case class Friend(id:String, since:DateTime)
case class PotentialFriend(id:String)
case class RequestedFriendship(id:String, requestedOn:DateTime)
case class User(id:String, name:String, friends:List[Friend],
                friendshipToReview:List[PotentialFriend],
                requestedFriendships:List[RequestedFriendship])

A fairly simple model. What happens if user X requests the friendship of user Y? Add user X to the friendship to review list of user Y, AND add user Y to the requestedFriendships of user X. 2 spots needing to be changed without transactions. Should user Y decide to accept user X’s friendship? Now 4 spots would need to be updated, i.e. remove the users from their respective requested friendships and friendship to review lists, and add each other into their respective friends list.

In an ACID world, the multiple updates would not have bothered me that much. If one update failed, the other would not take place. With MongoDB however, due its fault tolerance woes, it could happen that in between requests to the database, something goes awry (a train passes over the datacenter where my mongodb server is located and causes a momentary loss of connection), if my write concern is not ‘Journaled’, i.e. if I go full async, I can find myself in a state where it would be impossible for me to detect that a failure has occured, and thus wouldn’t be able to “manually” reverse whatever changes I had made.

What if I wanted to find what the friends of my friends liked? What if I wanted to find what my network of friends (i.e. my friend, their friends, the friends of their friends) recommend as the best movie currently in theaters? That is not something that you could easily do with MongoDB.

What I’ve found MongoDB to be useful at is to store data with a minimal amount of nesting. It makes it easier to retrieve later. For more complex modeling, especially in the case of social networks, I have found Neo4J to be a much better choice. MongoDB for the writes. Neo4J for the reads and/or Elasticsearch if you need full text search capabilities.

Assuming you use a service like Compose.io to host, scale and back up your MongoDB instance’s for you, your are looking at least 18$/month. If you need to add more databases to complement Mongo’s shortcomings, using Neo4J in the cloud with a service like GrapheneDb will cost between 50$-400$/month, while Compose’s ElasticSearch would cost you north of 45$ a month. Add what you need an multiply by 12 month and that’s how much you need to set aside just for your database cost. You obviously could handle the hosting and scaling yourself to reduce that cost.

If you’re bootstrapping your startup and have limited time, but some resources, I could see how you could build your application with MongoDB initially to get a proof of concept and get funding, or even to get an MVP to test the market and fail fast. It is also possible like that your use case would work just fine with MongoDB like Server Density’s.

UPDATE: For more on MongoDb, please check out this comprehensive tutorial from Guru99.

Cassandra

There are plenty of alternatives to MongoDB. Some of the choices out there range from the good ‘ol PostgreSql, Riak and Cassandra, to up and coming (and still improving) RethinkDb.

My personal criteria for determining which database to use are:

it should be easy for me to set up a cluster with it, i.e. easy to automate, easy to maintain
it should be easy for me to get started with it (docs, tutorials, good tooling)
it should be fault tolerant and highly available
it should be reasonably priced
it should be boring

Based on that, Cassandra jumps out to me as a better choice. It is highly available and easy to scale, and well documented. It has a thriving community of users, and has been adopted by many dev shops I look up to like Netflix. It is also supported as a backend storage mechanism for Titan, meaning that in my previous scenario I wouldn’t need to rely on Neo4J, thus reducing expenses. I’ve been playing with it recently, and so far it’s been fairly painless. The only drawback: its cost.

UPDATE: For more on Cassandra, please check out this tutorial from Guru99.

If you care about using Cassandra as a service, you could use InstaClustr for a price range of 299$ to 404$ a month per node. Setting up the cluster yourself on aws with minimum of 3 m3.large machine would cost you about 300$ per month. That is more expensive than just MongoDB, less expensive than MongoDB with side databases. In theory, for just an MVP, you could use a smaller ec2 instance at the cost of performance, then switch to recommended configurations once you’ve validated your idea.

The database should be boring. It should at its best be a tool that facilitates building your application, and just a place to dump and retrieve data at its worst. It can help your business grow. It can cause it to fail. Choose wisely.

Distributed logging

Fri, 27 Feb 2015 20:11:34 +0000

Distributed systems are great! Loosely coupled components. Seperation of concern. Scalability. Awesome!…until something goes wrong and you have to debug the issue.

Let’s say that you have split your application into multiple logical services located on different machines and that, as part of a given workflow, one service A needs to communicate with service B which needs to touch its database then talk to an external system then send a message to service C. Let’s also pretend that you have a load balancer in front of 2 instances of service B, and that you have a machine for your PostgreSql database and a seperate one for RabbitMQ server.

Under most circumstances, your workflow seems to hold. But under circumstances you are yet to determine, it fails. What do you do?

If you had not put in place a mechanism to log errors and info into a central location…well…let’s hope you can reschedule your dinner plans. You would essentially have to collect the logs from all of your servers to trace the issue. It’s tedious, but it can be done. You would just have lose spend a lot of time combing through the logs….everytime you find issues in your system. Not a very good option isn’t it? Besides, what if the problem was not solely due to a bug. What if someone had been trying to bruteforce into your machine? What if some other process in that server had been doing some funky stuff? More logs to analyze. Yay!!!

This why distributed logging (also known as centralized logs) is critical. For example, let’s say your were using a logging service like Loggly or Logentries (both support syslog forwarding). Instead of collecting the logs from the different machines, you could go to their websites and see them there.

If you want a more DIY approach, you can use Logstash, a log management utility with which you can collect the logs, parse them, and put them were you want. For example, you could have your application send the logs out using the log4j AmqpAppender, then configure the logstash agent to read its input using the rabbitmq input plugin, grok it, then put the result in your elasticsearch database. Then you could use Kibana to view and search them. If dealing with a legacy system that you can’t modify that puts out a trace log, you could configure your logstash agent to read from specific files instead of rabbitmq.

When dealing with distributed systems, you need centralized logs. It will save you time, which will help you adress problems faster, which could in turn save you big bucks.