What's In The Box?

Aside from the usual resolutions everyone sets for the new year, I have several programming goals that I want to set. Hitting these goals may be difficult (especially with a baby son on the way), but I’ll do my best to achieve them.

Learn Python Fluently

I’m currently interviewing for a job with Mozilla. I’ve been a fan of the Mozilla project since around 1999, and working there would be an absolute dream. I would be working on the web development and Internet services section, not the browser. With the recent announcement of moving http://addons.mozilla.org (known as AMO) from CakePHP to Python, I would need to learn Python for my potential new job. In addition to having to learn it, I’ve heard incredibly good things about it, and its time I learn a new language as well as I know PHP.

Make Artisan System a Viable Competitor in the PHP Framework World

Artisan System is a competent framework already, but its not in the stages of completeness I want. The SDK needs to be finalized, it needs to have PHP5.3 support (or rather, the ability to take advantage of PHP5.3 features). And I want to make it a valid competitor for Cake, CodeIgniter, Zend, Symphony, etc. Now that it’s moved to Git and GitHub, I feel like development will speed up even faster. I will set small milestones to accomplish along the way to ensure I meet this goal.

Finish the Shopping Cart Software I’m Writing

I’m in the process of writing two eCommerce Shopping carts. I’ve been very dissatisfied with the other Open/Shared/Closed Source offerings, so its time for me to write my own. I’ve started two of them, IonCart and another unnamed one for an entirely different platform. I will finish both of them this year.

Write More

The final goal I have is to write more. I enjoy writing this blog, and I have a lot of ideas I’d like to share. I’ll be openly developing most of the software I’m developing, and I’ll use this blog as the way to do that.

I hope that 2010 is a great year for you. I know mine will be, both in my personal and professional lives. I find making lists like this, setting explicit goals that you can quantify, help me reach them better. Happy New Years!

As a programmer who loves most aspects of eCommerce, I’m constantly evaluating different shopping carts (both open, shared, and closed source) to see where I can keep Dallas Shooting Supplies up and running the best.

Originally, Dallas Shooting Supplies was built on an old version of Artisan Cart, my first shopping cart software. It was somewhat limited, but it worked and helped launch my small side business. When we had the site redesigned, I figured it was time to move to a new platform.

I had gotten used to Wordpress and enjoyed it as a publishing system, so when I saw there was an eCommerce plugin for it, Shopp, I was excited. I was reluctant to purchase Shopp at first, but for $55 + $25 for the Authorize.net plugin, I didn’t have much to lose, I thought.

Shopp turned out to be a tremendous disappointment. So much so that when I went to make the site live, it failed entirely. I finally found the bug in their system, patched it, and told them about it. Shopp is horrible. Don’t buy it, don’t use it. The code is deplorable, the database is very poorly set up (do not store serialized data, use proper table normalization instead).

Looking to move away from Shopp, I visited http://www.practicalecommerce.com which gives reviews of eCommerce solutions, and has a directory of shopping cart software to use. Browsing through the directory, I found a cart that seemed very complete.

It was commercial, meaning it wasn’t free, but offered a 60 day trial. I could download the software, use it on my server, and if I wanted to purchase it after 60 days, I could. This is the correct way to do commercial web software that is self hosted. It gives me an opportunity to view the source code to ensure there exists an obvious coding standard, and the code is efficient. Both seemed true. Great, I thought I finally had found an eCommerce solution that wasn’t bloated, had tons of features, and wasn’t Magento or osCommerce.

However, I wanted to perform a security analysis before I was going to re-move all of Dallas Shooting Supplies to a new platform. I did the usual test of a basic XSS (Cross Site Scripting) attack. It appeared the shopping cart was using a home grown framework (which isn’t a bad thing), and that all of the vectors I checked seemed to not be vulnerable. http://ha.ckers.org/xss.html is a great resource to learn about potential XSS vulnerability vectors. Generally, in an eCommerce application, a naive approach to filtering all HTML/CSS/JavaScript is fine.

Next were the Cross Site Request Forgery (CSRF) attack checks. This is where the cart failed, and failed miserably. After logging into the admin panel, I could easily have a form on a separate domain (site) post data to an admin form and have the data saved. For example, I could easily change the administrators username and password, or update all users to have the same password so one could login as them and steal their info. However, being able to manipulate the administrators username and password was obviously the most nefarious use of this vulnerability.

Furthermore, one was able to manipulate data through basic GET requests. I could easily delete a product by calling a URL with the product ID. Scripting this would be trivial, and before they knew it, a site admin would have unknowingly destroyed their entire shopping cart.

The final check was the SQL Injection attack and this software failed miserably on this, but not in the typical sense. The forms were protected from the typical attack of attempting to escape out of the SQL string to execute arbitrary SQL. Again, because the cart was using a homegrown framework, all of the queries were routed through a common DB interface, which handled properly escaping the SQL query.

This cart provides an interface in the admin to backup and restore SQL snapshots of the database. This is great for backups, but horrible for security. The utility will let you download the SQL snapshots, as well as upload them. However, they can be uploaded from a cross domain. For example, you can upload the SQL file from http://example.com/database.sql and execute it. Hopefully, you can see where this is going. Through the CSRF attack, I simply made a SQL file with `DROP DATABASE db_name` as the entire contents. I uploaded this to a separate site and through a form, was able to upload the file to the site with the shopping cart software. Executing it involved another request to the shopping cart software, with the name of the file as a hidden form variable and another hidden form variable to define the action, executing the SQL file.

Of course, its improbable to know the name of the database. However, the names of the tables are generally the same amongst installations, so the SQL file could be extended to simply drop all of the tables in the database (the shopping cart software selects the proper database to begin with).

I’ve alerted the developers of the vulnerabilities, provided them a writeup of how to execute them, and even provided a test environment for them to execute the vulnerabilities. I told them they had 2 weeks to alert their customers and fix the vulnerabilities. After that time, I’ll disclose the name of the shopping cart software along with how to execute the vulnerabilities.

Even the largest of websites are not impervious to vulnerabilities, but it is possible to diminish the chance of a vulnerability affecting your site by following basic security procedures.

Over the weekend, the website I built, www.prospectvista.com suffered some downtime. Going to the website showed a general error because its a live site, and you should not have in-depth error reporting turned on a live site.

Initially, I thought it was the result of accidentally updating the software that powers the site and broke something. I checked the revision in Subversion (and the version according to our versioning system), and everything checked out all right.

Running a simple `free -m` showed there was plenty of memory available, and swap was being used fine, so that wasn’t it.

Next, I restarted apache, figuring something had gone wrong and that would solve it. It didn’t. Apache restarted, but I was getting the same error.

Determining I couldn’t break the site any worse, I turned on error reporting and went to the site. “A valid database connection could not be created.”

Simple, `sudo /etc/init.d/mysql restart`

[fail]

Well, damn.

I then checked the MySQL logs. Nothing. Same with the Apache logs, nothing that could give me a specific idea of what was happening.

However, in addition to the [fail] restarting the MySQL daemon gave me, I also got this error:

“/etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!”

Ah hah!

Running a `sudo df -h` showed that /dev/sda1/ was 100% full. There are only two areas that a lot of data is stored on the system: the actual site itself that stores the videos and photos, and the backups. The site content itself was pretty small. I went to the local backup directory and there were tons of outdated backups that were no longer needed. Note: I do remote backups for ProspectVista as well.

I deleted about 80% of the outdated backups and the drive usage dropped to 13%.

Perfect.

Now I have a small shell script that checks the disk usage once a day and emails me if it is 80% or above. I suppose I could also update the backup scripts to delete old backups, much like my Amazon S3 Backup Program does.

I’m incredibly proud of my PHP Framework Artisan System. It’s gone through a lot of revisions over the last year, and its never really picked up. Part of the reason is I haven’t promoted it much, but another part is that no one knows what to do with it. As the problem with a lot of open source software, there’s a lack of documentation to support it.

This occurred to me after I made a post on Hacker News about my Amazon S3 Backup Utility. Immediately, I had 7 new followers of it on Github.

I want to make a difference with my code, and I want to get the word out about Artisan System. It’s incredibly simple, but powerful. I’ve started this series of posts to help people start working with Artisan System.

Let’s get started.

Installation

Start by creating a new directory structure in your web root to store all of the files in. Download the latest tarball, untar it, and then rename the resulting directory to Artisan.

cd /var/www/
mkdir artisan-system -p
cd artisan-system
wget http://github.com/leftnode/Artisan-System/tarball/0.5.4
tar -xvzf leftnode-Artisan-System-cc2af85.tar.gz
mv leftnode-Artisan-System-cc2af85 Artisan
rm leftnode-Artisan-System-cc2af85.tar.gz

Artisan System is now installed. You could equally clone it from Github as well and use the clone.

Databases

The first thing you’ll want to explore is how to connect to a database. Until recently, Artisan was going to support multiple databases. After some reflection on the direction I wanted to take the project, I removed the potential support for additional databases and just kept MySQL. Adding a new database adapter would be simple. Furthermore, the current database adapter could be updated to using PDO to automatically support additional database wrappers.

Connecting

Connecting to a MySQL database is simple.

There no longer is any formal configuration object in Artisan System. Configurations are now simple arrays. You can optionally pass in the port to the configuration array, but Artisan selects the MySQL port, 3306, by default.

This example is the simplest of examples on how to connect and then disconnect to a specified database. If the connection fails, the exception is caught and reported.

Querying

Querying the database is just as simple with the query() method in Artisan_Db. The query() method takes a raw SQL string and executes it. It does not take any precautions to prevent SQL Injection attacks. In general, you should not use the query() method directly.

Querying With Methods

Artisan System provides a much more robust system for querying a database: methodized querying. Methodized querying builds a query through an object via a chainable object. The most common query types (SELECT, INSERT, UPDATE, DELETE, and REPLACE) are included and accessible through the Artisan_Db object.

You should use methodized querying for several reasons.

1. Its safe. The data is automatically escaped to prevent SQL Injection attacks.
2. Its fast. It’s obviously not as fast as querying with a direct SQL string, but its faster than PHP’s parameterized querying.
3. Its semantical. Building a query reads like writing a sentence.
4. Its abstract. The classes behind the query actually build the SQL itself, so porting it from one database system to another would be simple.

Querying with methodized querying is simple and straightforward. One of my favorite features is to be able to select a single row and get a specific value back. It takes what would normally be 3 to 4 individual commands to a single one.

Take the time to explore Artisan System. You’ll find a powerful, compact PHP Framework to really help you build a web application quickly. In the next article, we’ll cover the basic Model-View Controller Pattern in Artisan System.

I’ve recently started using Git for all of my development, and so far I love it. Branching and merging is great. Tagging is effortless, and it really brings a new power to source control. Because Git is decentralized, each time you clone a repository, Git rebuilds the repository locally. Unfortunately, one thing it does not clone are the hook scripts. Compare this to a centralized system like Subversion where once a hook script is set up, everyone has to use it.

Not having the hook scripts with a fresh clone of Git seems somewhat silly to me. The whole idea is that you want the hook scripts with the repository so everyone has to use them. Searching around, I could not easily find a way to have Git persist the hook scripts from clone to clone.

The closest I could find was someone suggesting to create your hook scripts outside of .git/hooks/ and then symlink the files in .git/hooks/ to the directory with them. This works but the only drawback is you have to manually set up the symlinks on each clone. This didn’t sit well with me because I knew I’d (or another developer would) forget to set up the symlinks on a fresh clone.

Set Your Phasers to Phing

Phing is a great piece of software. It’s software like Phing that make me a proud PHP developer. Phing is an automated build system similar to Apache Ant or GNU make. For interpreted software (like web applications), this may not sound very useful, but Phing is still incredibly powerful. It can set up cron jobs, set up configuration constants, run your test suite, or build your database, among other things.

Phing uses simple XML files rather than complex Makefiles to define the build process. My project ION Cart uses Phing to install a new copy of the shopping cart software. Because I work out of my home and my small office space, I generally work on several different computers. I may clone a repository many times during development, and I want the hook scripts with each clone. It is with this that I found a way around Git not tracking the hook scripts.

Track Your Hook Scripts Externally

Looking at the directory structure of ION Cart, you’ll see a hooks directory. Your hook scripts will reside in here. I will be using the pre-commit hook as an example for tracking hooks.

Building on the idea earlier, Phing will create the symlinks for us during the automated build. Generally, as soon as the repository is cloned, you will execute Phing, causing the hooks to be set up.

cd /var/www/vhosts/ioncart.dev/ # I set up all development environments as their own .dev sites
git clone git://github.com/leftnode/ION-Cart.git .
phing
# Enter database configuration options, and other config data
# Hook scripts are now set up

By default, running the `phing` command line utility will look for a build.xml file. In this file, the githooks target is what builds all of the symlinks.

For now, the symlinks are manually set up, meaning you have to specify each <exec> tag. I do not plan on updating the script to automatically search for hooks in hooks and build those. There are only 11 potential hooks, so you can simply duplicate the <exec> call for each hook you want defined.

Furthermore, you could easily have two XML files, one for the initial build, and then one if you need to continually rebuild the file. For example, build-install.xml could be the build file that sets up the hook symlinks, and build.xml could be the file you build every time. Execute other build files with the command `phing -f build-install.xml`.

Because I am a new Git (and Phing) user, it would not surprise me if this functionality exists otherwise, but quick Googling did not result in anything. If you have a more robust way, I’d love to hear.

One of the ways I like to best learn a new technology is to try to understand some of the more in depth aspects first. When I first started learning Git, I wanted to move my Subversion pre-commit hook to Git.

Git is great in that everything about your repository is stored in the .git directory. Furthermore, a lot of configuration data is stored in plaintext, so its easy to manipulate manually. Hooks, naturally, are found in .git/hooks/ and are named with a .sample extension. To get a hook to work, you need to remove the .sample extension.

Hook names are self-explanatory, and the one I’ve finished migrating from Subversion to Git is the pre-commit hook. If the pre-commit script exits with a code other than 0, the commit is not allowed.

I decided to use Perl for a PHP project because I rather enjoy Perl. It was the first real programming language I learned, and I wanted something to begin to relearn Perl’s basics. The hook script itself is fairly basic. To summarize, when a commit transaction is started, the script finds all files in the source tree, and then for each PHP file, runs them through PHP’s linter. A linter is a program that checks the syntax of a program. Running `php -l filename.php` from the command line with lint the file. PHP will tell you if there’s a syntax error, what it is, and what line its on.

Note the \033[1;31m and \033[m strings in the calls to &msg() colorize the output on a Linux system. I’ve only tested it with bash, so you may have different results. It makes reading, and noticing the output, much easier.

The only downside to this script is if you have an incredibly large source tree, it can take a while to execute as its naive in how it finds files. It uses the `find` utility creatively to search for potential PHP file types, however, if you have thousands of files, it can still take a while to execute. To speed it up, I’d love to learn if theres a way to get all files that are being committed and only lint those. It would drastically improve performance.

Save this script as pre-commit in .git/hooks/ and you’ll be on your way to stronger source code.

Further Modifications

This script will be updated in the future. The most obvious choice would be to have the pre-commit script run your testsuite and only allow the commit to continue if all tests pass. What other ideas do you have for pre-commit scripts? Please share them below.

I’ve read Daniel Miessler’s blog for a while now as its popped up on different sites like Reddit and Hacker News. I never officially subscribed to it in any formal capacity, but yesterday he had an article on using git to manage his site. The article taught me a lot about git, but I was more interested in the footnote of another article he wrote.

[ If it seems like this post is a note to myself, that's because it is. :) That's how the entire site got started, actually. ]

I liked that. For the longest time, I felt like this blog (or my writing online) had to be very long, very drawn out, thought out entries. I think I can safely change that now. It’s my writing online and I want both entries: long very well thought out ones introducing new ideas, and short ones that are simply notes to myself or small snippets of code or ideas.

I like learning something new each day. Today, I learned the basics of git submodules. Similar to Subversion svn:externals, git submodules let you have additional repositories as sub-directories to your main project. I used svn:externals a lot when I did everything with Subversion. Rather than keeping the Artisan System framework installed at, say, /usr/share/artisan-system/, I always like to install it with the project I’m working on. That way, I can create an immediate release with all of the code in a single location. Additionally, if I come across a bug as I’m developing, I can immediately fix it there, commit to the HEAD of Artisan System, and then tag it and push it to GitHub.

If you have a project hosted at GitHub with the clone URL: git@github.com:leftnode/Artisan-System.git and you want to set it as a submodule, git handles this easily.

# Ensure you are at the root of your git project, i.e., where the .git directory is
 
# This makes the directory lib/Artisan/ a submodule repository
git submodule add git@github.com:leftnode/Artisan-System.git lib/Artisan
 
git commit -a -m "Added Artisan System framework as a submodule."
git submodule init

If you make a change to any of the files in the submodule, be sure to commit them first. Then commit the parent project to ensure both are up to date.

# In the submodule directory
cd lib/Artisan/
vim Iterator.php
# hackity hack hack
git commit -a -m "Updated Iterator.php with something or other."
git tag 0.5.2 -m "Creating the 0.5.2 tag with some changes to Iterator.php."
git push origin --tags
cd ../../
git commit -a -m "Updated the Artisan System submodule."
git submodule status

The last output will show you your submodule is at the latest tag.

 07bd774e56d645844d9eb3691cea706a065288ee lib/Artisan (0.5.2)

You can read a much more in depth guide at Fraser Speirs blog.

Subversion, you’ve been great. Awesome I’d say, but its time to move on. I was first introduced to Subversion in 2006 when I started working for New Media Gateway. I had heard of CVS, of course, but never had a Linux server set up to really use it. Plus, on most projects, it was just me working, so I really didn’t need any source control (note: this is not a good thing, you should be using source control even if you’re the only developer on a project).

So I begin working at NMG and they’re using Subversion. It’s just what I used. It seemed to work fine, and since one of the other programmers managed merges and releases, all I had to do was update my working copy and check stuff in. I wrote hook scripts, and everything worked great.

Then, I was introduced to git by Zach, and boy did I love to rip on it. Decentralized version control, but look, you have to centralize it somewhere, like GitHub, right!? One day, GitHub was down and we had a good laugh. But Zach started using it for a new project he was working on.

When I moved to New Legend Media, it was natural that I introduced Subversion as the source control system to the company. It was all I knew, it was stable, and it had very competent Windows tools for designers. Together, with another developer there, we set up a Subversion server and through a few rough patches, we started using it. I became very comfortable with it. I learned branching, and merging, release management, and general sever maintenance. He raved at how well it handled branching, merging, releases, bug fixes, and all of the other features of a DVCS. I scoffed at them, and said I’ll stick with my Subversion. The 1.5 series was stable, had automatic branch management built in for easy merges, and I had all of my code in there.

But in the back of my mind, I knew I needed to learn how to use git, but I was reluctant. I’m not exactly sure why, but I just felt like Subversion was great for me. Eventually, I just broke down and asked Zach to show me git. He guided me through setting up my .gitconfig file so I could have my nice Subversion aliases and colored output, to setting up a GitHub account, and finally how to branch and merge. Git was amazing. So one by one, I started moving all of my projects to my GitHub account.

And thats where I stand today. I’ll always have a soft spot in my heart for Subversion, but for starting a new project, Git just can’t be matched.

cd /path/to/svn/repos/
sudo svnadmin create /path/to/svn/repos/reponame
sudo chown apache:apache reponame -R
sudo chmod 700 reponame -R

vim File.php
# hacky hackity hack
git init
git remote add github git@github.com:leftnode/New-Project.git
git push github master

Goodbye Subversion, it really sounds like goodbye to an old friend, but I’m really excited about git!

A very small bug-fix release was just made. You can grab it at GitHub: http://github.com/leftnode/Artisan-System/tarball/0.5.1

It consisted of a single character change to View.php to make calling makeUrl() work properly.

I built a URL shortener earlier this year. It failed to gain any major traction, but is actually used by people other than my friends and me. You can view it at http://qst.me.

It has the ability to do 4 things: shorten URL’s, shorten large blocks of text, give basic statistics on the amount of times a URL/block of text is used, and count unique words in Word documents.

The large block of text shortening is useful for sending AIM/ICQ/IRC/gChat conversations back and forth (for example, when your friend says something embarrassing and you want to share it with your other friends).

After you’ve been given the shortened URL, add a $ to the end of it to see the stats for that object.

It’s not evil, I promise. Try it out, let me know what you think.