Posted on September 21, 2015 by

Yes, it needs tests

Don’t spend a lot of time on this, because it isn’t a full blown feature. It’s just one little thing the client needs. Don’t go overboard, just quick and simple.

You can’t control how other people think. You can only control you. The client and sales have already had the meeting. They’ve already convinced themselves this is a quick easy feature. You can’t change this. But you can remind them of the policy:

All committed code needs tests.

Have this policy! Follow it! When you say no, you are a disgruntled programmer who doesn’t want to do work. When the policy requires tests, you are doing everything you can to get that request implemented. It will just take some time.

I was recently asked to build a small, one off workflow for a client. I was met with a bit of a shock when I asked why we weren’t making this a full feature. They honestly wanted to save time. They didn’t want this to get in the way. They didn’t want to go through the work of designing and implementing a new feature. I had just a couple questions:

Does this have to work correctly every time?
Does this have to integrate with the system without breaking anything else?

This a feature. It needs to be tested. It needs to work. It needs the time to be built correctly. Have the policies in place that allow you to build software the way it should be done.

 

Posted on March 30, 2015 by

SQLAlchemy and Memory

I started having segfaults in my celery processes. These were incredibly difficult to debug because they didn’t always happen, and never happened in development. Fortunately they were happening in certain places and I was able to guess where the problem was. However, I didn’t understand just how severe the problem was.

I use SQLAlchemy in my Flask apps. I typically issue queries similar to

query_result = db.session.query(MyModel).first()

I knew I was running out of memory on my entry level Digital Ocean droplets. That is when the segfaults started. I stumbled across https://pypi.python.org/pypi/memory_profiler and created a quick test program to profile a single function that was causing problems.

Line # Mem usage Increment Line Contents
================================================
56 46.0 MiB 0.0 MiB   @profile
57                    def get_bounced_emails():
58 48.2 MiB 2.2 MiB       emails = db.session.query(EmailAddress).\
59 48.3 MiB 0.0 MiB           join(send_history, send_history.c.email_id == EmailAddress.email_id).\
60 372.1 MiB 323.9 MiB        filter(send_history.c.bounce == 1).\
61                            all()
62
63 372.1 MiB 0.0 MiB      email_dict = {}
64 374.1 MiB 2.0 MiB      for email in emails:
65 374.1 MiB 0.0 MiB          email_dict[email.email_address] = True
66
67 314.9 MiB -59.2 MiB    del emails
68 314.9 MiB 0.0 MiB      return email_dict

The memory profiler package makes it real easy to see the memory consumption of your application line by line. In this function, we are just retrieving a ton of records. They are taking up a lot of memory.

This is where I learned a little bit about SQLAlchemy. This function only needs the email_address field of the record, but I’m retrieving the entire record. So I made a change to simply grab the email_address record.

Line # Mem usage Increment Line Contents
================================================
56 46.0 MiB 0.0 MiB   @profile
57                    def get_bounced_emails():
58 47.1 MiB 1.1 MiB       emails = db.session.query(EmailAddress.email_address).\
59 47.1 MiB 0.0 MiB           join(send_history, send_history.c.email_id == EmailAddress.email_id).\
60 81.2 MiB 34.1 MiB          filter(send_history.c.bounce == 1).\
61                            all()
62
63 81.2 MiB 0.0 MiB       email_dict = {}
64 84.2 MiB 3.0 MiB       for email in emails:
65 84.2 MiB 0.0 MiB           email_dict[email[0]] = True
66
67 65.9 MiB -18.2 MiB     del emails
68 65.9 MiB 0.0 MiB       return email_dict

Wow. By just grabbing that one field I seriously reduced the memory consumption of this method. I had under estimated just how much overhead there is in pulling the entire record in a query.

Now I take this into account when building queries. Just what do I need form the DB? Then I only grab that.

Posted on March 23, 2015 by

Time Management

I spend roughly an hour every Sunday making tortillas. This allows me to save upwards of $5 every week.

Given my hourly bill rate, this is really stupid. I should just buy the tortillas.

OR

Spend the hour each week making tortillas. Stop spending the 3 other hours each week Googling tortilla recipes.

Posted on February 6, 2015 by

Silent Win

Well it’s been six days since I’ve decided to post something everyday and haven’t I posted anything on any days. Some might see this as a total failure. But I’m deciding I’ve actually succeeded.

The real goal, of course, is to practice writing down thoughts in a logical order that can be understood by others. The secondary goal (as of now) is to improve my signal to noise. That can’t happen by scribbling down my insane thoughts every day. Unless I had an easily identifiable pattern of clarity to insanity. But I don’t think that’s something you can plan.

So I’ve privately jotted down thoughts for revision, and not posted the horrible bits. This helps me organize my ideas and doesn’t clutter the internet.

Huzzah!

Posted on January 31, 2015 by

More Writing

I think it’s about time to start this “post a day” idea. Fred Wilson does it. This guy seems pretty consistent about it. More importantly, I’m starting to send long emails which are simply not being read. I need another outlet to vent. Or people are going to stop talking to me altogether just to avoid getting stuck in conversation.

I certainly have enough half finished thoughts in my drafts folder. I’m thinking I can stretch this out two weeks. Maybe three. I can’t simply ramble, I have to do something productive. So with that in mind.

– – – – –

I missed the product release notes yesterday (it’s Saturday). So I’m finalizing them them now. I put out release notes every Friday afternoon so everyone on the team can keep up with the new features and resolved bugs. It also lets people know I’m not sleeping all day. That isn’t a joke. It is actually important to do this. Especially when you’re a back end developer.

I keep an open notepad all week, and add important features and fixes that people care about. Those are the notes I share. I always have one line: “Lots of small bug fixes and refactoring“. That is a catch all for everything I don’t talk about.

Keeping an open notepad for release notes forces me to do at least 2 or 3 things each week that people will notice. I may spend 4 long grueling days profiling and writing unit tests for an Objective C API we are rolling out. On the fifth day, I know I better write a new graph report. Or maybe update some email templates to look better on mobile. Or anything where people can see the output. Otherwise the rest just doesn’t matter. It really doesn’t. Non-developers only ever work with production level software, usually from very large teams. They only see software that, generally, works very well. But they have no idea what goes into it.

On the plus side, they have no idea how long it takes to change the color of a button. So take all day. 3 seconds to change the color. 8 hours to refactor and write tests for the rest of the system. So thats what I do. I spend a day every week making highly visible changes, no matter how simple.

It’s a balancing act. I really think the difference between a normal coder and the mythical 10Xer is how you spend the other 4 days.

Older Posts