Technology Choices

“We standardize on Zend Framework”. “We’ve chosen Symfony2 as our development platform”. “Drupal is our tool of choice”. The amount of times I come into an organization that says any of the above is impossible to keep track of on one hand. Or even two. Most development shops for some reason have decided that they have a single tool that will fit the job. Always.

I have to admit the current market is good for developers. There are many projects available, and not enough developers or agencies to work on all of them. Agencies can be picky in what projects they choose, and choose only those that fit well with their tool of choice.It would be fantastic, if organizations actually did that. But too many times have I encountered projects where the used tool actually was not optimal for the project.

I would like to make a case against starting with a full stack from the start. Obviously, this approach does not work for all projects, but too many projects start out small but with a full stack. I’m going to take an old project of mine as an example of how to start out small and not grow until you need to.

The Project

For a large media organization, I was asked to work on an application that would trigger some transcoding tasks. It was a very small application: It needed to keep track of a directory and when new source files were encountered there, start the transcoding process by a third-party application. Once transcoding was done, it needed to execute a command that would generate a playlist file. Last but not least, it would notify the media management application that the new video was available for publishing on the several websites by this organization.

Making the initial technology choice

I like Symfony2. No, I love Symfony2. It works really well for a multitude of projects and applications. Yet for every project, I make an analysis of what is needed and decide whether I need Symfony2. In this case, installing the full Symfony2 stack just to write some small scripts that would be run by the cron seemed like overkill.

Another option would be to just use the Symfony2 Console component. This component offers the tooling to write CLI scripts. Just like other Symfony2 components it is completely decoupled and works well by itself. The downside of this is that it only offers functionality for writing CLI scripts, and I need a bit more than that. I also need database access and perhaps some more tooling.

Now I could write a composer.json file with all dependencies, but luckily I am aware of the existence of Cilex, a microframework based on Symfony2 components (including the Console component) that offers a bit more functionality out of the box, without needing the full Symfony2 stack.

From 0 to 100 in…

One of the advantages of using a microframework is that you can get up-and-running quite quickly. The microframework offers just a small set of features, so they are quite easy to learn. In the case of Cilex, the most important thing I needed to learn was how to write Command classes, since they are the basis of Cilex. And since this is based on the Symfony2 Console component, I was already used to this approach from my Symfony2-projects. So while you should definitely look at what technology you choose, the fact that you already know a certain technology can seriously influence your choice for the tool. Technology is not the only deciding factor, the skillset and experience of your team also influences the decision.

Having made a technology choice, I could now start working. I created five Command classes for the five steps I mentioned: monitoring the directory, notifying the transcoding application, monitoring the transcoding application, creating the playlist file when a job was finished, and notifying the media management application when all this was done. Each command had a single responsibility, and the state of each job was persisted into a database. This meant another technology choice: Doctrine is the obvious choice for working with databases when you’re used to Symfony2, but just like Symfony2 this felt like overkill. Instead, I simply went with using PDO directly. For a small application like this, PDO gives you all the functionality you need.

Open for change

Another advantage of starting small is that you’re very open to adapting to changes. Because the tool you use is very small, if you need more, bigger or different, it’s very easy to actually do so. Once the initial project was done it turned out that some things needed changing. In the initial setup, my application was polling the transcoding app every minute to check if jobs had finished. But in the world of media, speed matters. A video that is published 20 seconds sooner means people are more satisfied. The transcoding application had an option of sending a callback once done, yet we had no means of accepting callbacks; The application so far only did CLI scripts called by the cron.

It was time, therefore, to add something to the application that allows us to catch HTTP calls. Since we were already using Cilex for the CLI scripts, adding Silex for handling HTTP requests made sense. Silex is also based on Symfony2 components, offers very similar functionality to Cilex but then focussed on HTTP requests.

Because of the similarity between the frameworks and the work I had already done, I could easily re-use the PDO object I was using for database interaction. I just needed to register a single URI to handle that would receive an HTTP request once a video was done transcoding and update the status in the database. I removed the command that was polling the transcoding app and replaced it with the HTTP endpoint. This analogy has been used a lot when talking about modular software, but it felt a lot like I was playing with Lego bricks.

Photo by Benjamin Esham: https://www.flickr.com/photos/bdesham/2432400623/

What happened?

Media and technology have one thing in common: Things can go wrong. It wasn’t long before something went wrong: A video that was put into the system never made it out. The customer I was working for did not employ any developers, they only had system administrators and (streaming) media experts. So debugging such problems and finding the cause by digging into the code and the PHP error logs is not quite that easy. As such, it didn’t take long for a new feature request: A simple web interface that would show the status of all jobs (both those that are currently running and those that are already done). In case something goes wrong, the option should also be available to retry the process; Sometimes a glitch in the matrix was everything that would cause something to go wrong.

Adding some web pages was not hard: I had already added Silex to handle HTTP requests. However, I previously didn’t really need to output anything, and now I needed to add actual HTML. I could obviously just create the HTML myself in the Silex app, but I prefer not to. It would make my simple Silex app very unreadable and personally dislike mixing HTML and PHP code.

Luckily, Silex has built-in support for the Twig templating engine. I’m not going to list all advantages of using Twig, but to me, Twig made a lot of sense because it would seperate the markup from my PHP logic. Enabling Twig was very easy. Since the interface was an internal application and it was protected by IP whitelisting I needed no session or authentication support, so it was just a matter of enabling Twig.

The Result

The result of all this work, including changes and extensions, was still a very small and lean application that combined the power of the commandline with a simple but effective web interface. I am sure I could have done a similar thing with Symfony2, but the code would’ve been overkill. And because I’m using Silex and Cilex, both built on top of Symfony2 components and using similar approaches, if the application really grows in the future, it is always an option to migrate to Symfony2 full stack and harness the power the full stack has to offer. It is important to realize that there is not always a need for full stack frameworks or huge CMS’es like Drupal. Sometimes you need to start small and just let it grow.

  • In the opening of the article, you make a reference to organizations who made a choice for a “standard framework”. This is a lot more beneficial than you let on, when you take into account the training and education of new dev’s joining the team, or project communication not directly related to code like recognition of features/terminology by sales or helpdesk departments. New tech means new terminology means learning curve and a bump in the road.

    However: holding on to such a “standard” without looking at alternatives is silly. Choosing the “standard” when it’s clearly not a fit is costly also, the poorer the fit the more costly it becomes. From an organisation’s perspective, it’s always a tradeoff.

    From the perspective of a single dev in the team however, the organisation’s standard should *never* limit their horizon. In fact, I believe the developers should be encouraged to find that “better fit”, and thereby driving the organisation forward – perhaps even towards a new standard.

    The hard part is something you have not mentioned in the article (but hinted at though). You simply “chose” a better fitting technology. How you came to recognize that this was a better fit is based on [years of] experience. You have intimite knowledge of the problems that a full-stack framework solves for you from your pre-framework days, and a reasonable picture of the overhead involved. Choices are always made in the area of solutions available in your mind (you could have chosen a python library for the commandline work for example) – and there’s such a huge world out there…

    Your main point is made quite well: just start small and let it grow.
    I’d like to add: and make your solution space as big as you possibly can – never stop learning!

  • Stefan Koopmanschap is a self-taught PHP developer, picking up PHP in ’97 and loving it ever since. He runs his own company Ingewikkeld, initiated Techademy together with fellow DWA-member Joshua Thijssen, and has been active in many open source projects and communities. His focus is quality software and sharing knowledge.

    Dutch Web Alliance