making fugly things, less fugly

Is how I would describe the job I am doing right now. I officially take the title of Research Assistant. In general terms I do data analysis and extraction. But really, it's about making fugly things less fugly so other people can actually use it.

For example, performing complex calculations and sorting of information from a 2.6 million line data set. Or, the latest project, asks me to match 4 databases together and find certain types of inconsistencies between them and classify them.

It's basically problem solving. It's always something simple. First do this, then that, if this is the case, then something, if not, do another thing, and then you should get this output and then perform some more calculations on them. Ok. Now do it a few hundred thousand times. The trick is producing a system that is robust enough to handle all possible contingencies, and that can inform you if inconsistencies occur. What if the company re-registers under a different name. What about list and delist dates close to end of financial year. Some companies get delisted for a few years before being relisted, leaving a gap, but actually are the same company before and after.

It's always a two way tradeoff. The more time you spend on planning an efficient system or program, the less time is spent on its execution. But there is always a point of diminishing returns, where marginal increases in program efficiency take too long to dream up.

Certain problems are unsolvable. They are merely reducible. There will always be some element of manual labour in handling data. There is the point where trying to figure out an automated method of data extraction (just selected bits) takes longer than just doing it by hand.

My brain feels like it's stretching, or at least close to running out of RAM. Keeping several spreadsheets open at one time, and trying to keep the relationships between them in my mind as I think about the best way to do things, leaves no room for anything else. I feel like if I let my mind wander for a second everything will come crashing down like a house of cards and I will forget everything. My fear is I will look at all the hours of work, and everything ceases to make sense. It feels like standing on a tightrope, the feeling of perfect balance, afraid to look left or right, and just moving forward an inch at at time.

Such a lowly job. But so interesting. Enjoyable, not really.