Recovery Test
Jun 7th, 2009 by Mike

My company has a policy that we do a full-scale recovery test of the database on our ERP system twice a year. That test was last week. Being a somewhat competent DBA, I should be able do this with half my brain tied behind my back. It wasn’t as easy as it should have been.

Keep in mind that I restore these databases fairly regularly, so the test should have been a formality. But this time, I had to have Derrick, our network guy, get the backup file from the long-term storage that comes back from the mine instead of my usual restores from local storage. I sent Derrick the name of the file that I needed to recover. I didn’t even care of the date on the file to satisfy this requirement.

The restore finished really quickly—almost too quickly. When I restored the database, I went to pull data from a table that would prove we had successfully restored the database. I was horrified when SQL informed me that the table I wanted didn’t exist. I scratched my brain, trying to figure out what was going on. I restored the file again with the same results. I watched my life flash before my eyes. How the ehell was I going to tell my boss that I wasn’t able to restore our ERP system from a month ago? I went back and realized that Derrick had restored the wrong backup file. After restoring the right file, my database restore ran better and the table existed again. Lesson learned: Because this system has multiple databases, the backup files can be easily mixed up. Double-check the filename before panicking.

Next came a test for our corporate controller to sign off on. I would grab a transaction from the AP invoices table so he could look it up in the system to validate that the data was entered on that date. Keep in mind that the backup was taken on Sunday, May 3. That means I should have data from Friday, May 1. As I’m looking at the data, I see transactions from Thursday, April 30. Why could I not see data from Friday? Again, I start scratching my head, trying to figure out this out. I was completely baffled by this one. I walked up to Finance to talk to Cindy in Accounts Payable. Would you believe the entire Finance team took that Friday off to go on a canoe trip?

Problem solved. After a few setbacks, my DR test passed. Now I have to document everything and get the controller to sign off. That’s usually the hard part. Not this time.

Things you Know Now…
Feb 10th, 2009 by Mike

I read a blog post from Brent Ozar, talking about lessons learned throughout his career. Here is my take.

Consistency is your friend.

When I was doing Lotus Notes consulting, I had a client with virtually zero downtime. They never had the connectivity and replication problems I had seen in other environments, including my own. Their secret? It was consistency in versions. All of their Notes/Domino servers ran the same version, and the clients ran the same version. Today, it doesn’t seem like that big of a deal, but back then, when you had to upgrade everybody via a CD, that was a revolutionary idea. I had always thought that you should install the latest version. My world changed that day.

Today, in my role as a SQL DBA, I try to keep as much consistency across my database servers as possible. When I started at my current employer, we had eight different versions and service pack levels of Microsoft SQL Server. Today, we have two, SQL 2000 SP4 and SQL 2005 SP2. When I upgrade my SQL 2005 instances to SP3, they’ll all go in a very short window.

All of my database servers are backed up using the same methodology and a consistent backup script. If I get hit by a bus, a replacement could come in and quickly figure out what’s going on.

Seek out a higher power.

I was an arrogant kid when I first started. I thought I knew everything. What I didn’t know, I thought I could figure out. Working with me had to have been hell. I’ve learned that not knowing something isn’t a sign of weekness. It’s an opportunity to learn.

People are incredibly willing to help you out when you acknowledge their expertise and approach them with a “help me understand…” attitude. In today’s world of technology, one can’t know everything. I try to be good at my stuff and get help from people who know more about something than I do. Besides, saying you don’t know something makes you look human.

Write early. Write often.

In the world of technology, your work is crap if people can’t understand it. I hate writing documentation, but I have to do it. I’d like to think that should I get hit by a bus, a reasonably skilled DBA could come in and figure out my environment rather quickly. Unless something is documented, it didn’t happen.

The more I write, the better I seem to get at it. This isn’t a bad thing at all.

Know your audience.

I hate reading e-mails I don’t understand. I work for a high tech company, and our engineers talk about networking and TCP/IP using language that goes right over my head. I appreciate reading things in language I understand.

When I send out a notice that we’re upgrading a server, people in this company appreciate knowing that we’re upgrading the firmware on all of our IBM x346 servers. That makes sense to them. At my last employer, a construction management company, all our user community cared about was that a system was going to be down for maintenance. If I had sent out a message stating we were upgrading firmware, my inbox would be flooded with messages asking what the hell I was talking about.

My Shawmut Experience
Nov 13th, 2008 by Mike

The last couple of days have been rough emotionally. I have to confess that I’m on some meds that are certainly messing with my emotions right now. Yesterday, my former employer started cutting staff. Over the last two days, countless friends have lost their jobs. I’m feeling incredibly thankful that I do have a good job right now. Had I not taken my new job, I could have been one of those casualties.

It’s hard to articulate why this is so upsetting unless you’ve worked there. Shawmut is the first place I ever worked where the culture actually meant something. In fact, at my interview, the recruiter, Johanna, said something I’ll never forget. She said “Our culture is everything. Without our culture, we’re nothing.” And that’s really how the company operates. It’s an exhausting place to work, and the people will drive you crazy. And they’ll give you the last dollar in their wallet if you need it. In my almost four years there, I never heard anybody say “That’s not my job.” That would just be treasonous. Working there is hard. Expectations are ridiculously high. And people exceed them. Everything is about pleasing the client, and every person in the company has a vested interest in the success of the company. That’s hard work. And Shawmut takes care of its people. I went to a lot training. They don’t even blink at training because they see the value in investing in people.  They have built a community in their people, which is not an easy thing to do.

When I first moved to Providence and then Boston, I didn’t know many people. Shawmut became my family in a lot of ways. These are more than just former coworkers. 

That was my Shawmut experience. And that’s why I grieve today for my friends.

Oct 27th, 2008 by Mike

If there is one word that makes people cringe, it’s audit. You immediately have visions of IRS agents invading your records and pouring over receipt after receipt.

In my line of work, audits are expected. They’re normal. And they’re part of life. You may have heard of this little law known as Sarbanes-Oxley, sometimes known as SOX.  When you work in finance or IT in a publicly traded company, it’s just part of how you do business.

This week is our annual audit from the major consulting firm that will really dig into our records and processes. We also have another company come in quarterly to do audit work since we don’t have an internal audit group in the company. My boss was telling me stories of spending hours and hours with them last year in the audit. We spent about two hours with them today. That was it.

Last week, I had spent a good chunk of my week preparing for the audit, pouring over checklists of things they’d want to see. Today, most of the time was spent pointing out where certain data was in the files we gave them or getting additional data. It wasn’t nearly as bad as I’d expected. In fact, I joked with the auditors (yes, that’s possible) that I was a little let down because I should have been much more difficult than that.

I will say that part of the reason it went smoothly is that my boss is pretty OCD about making sure we’re in compliance. We run a pretty tight ship. Nobody touches the financial systems without proper documentation and authorization. We also do an audit checklist every month, producing the things that will most likely be asked for in an audit. On top of that, I’ve become a master at producing audit-worthy reports. My experience as a consultant certainly helped here. And maybe, just maybe, it might have something to do with the fact that I’m pretty good at what I do. Maybe.

Mar 19th, 2008 by Mike

My boss sent me his evaluation and the feedback that comes from my coworkers.

The HIV test anaology was spot on. That 1% chance drives you crazy and then doesn’t happen.

My boss’ evaluation contained no surprises. In fact, it was a little better than I had expected.

Calling India… Again
Feb 19th, 2008 by Mike

One of the less than wonderful parts of my job is supporting our reporting system. And today I had to call their support… in India. Oh, how I hate calling India.

Off to NYC
Dec 2nd, 2007 by Mike

I’m off to New York in the morning. I have a ticket for the 6:00 AM US Airways Shuttle and the 6:15 Acela train. I’m thinking I’ll take the Acela, which takes longer but is a lot less stressful. I booked a first class return ticket on Wednesday for the 4:00 PM Acela.

I dropped Reggie off with Beth at her work yesterday afternoon. She reports that he’s already marked her knitting bag.

After dropping Reggie off, I stopped at Borders to pick up a book to read on the train. I finished it in about 18 hours. Oops. I also stopped at a store to look at luggage. I know exactly what I want, and they didn’t have it. I want a 22 inch Travelpro Crew Series 6 rollaboard.  It’s the replacement to the one I already have, which has been with me for about eight years and across the country countless times. The 26” bag is just as old and has been across the Atlantic several times, as well. Both bags are really showing their age and need to be replaced. But I don’t travel as much as I used to, so it’s hard to justify spending a lot of money on luggage. Still, going back to my Clorox Bleach theory, there are some things where I just won’t skimp on quality, and my luggage is one of them.

While in New York, I’m there for two nights. Monday night, I have reservations at Bobby Flay’s Mesa Grill. It’s going to be a little expensive, and I’m not sure I can justify that on my expense report. I also want to see the tree at Rockafeller Center, something I’ve never done. Tuesday night, I’m supposed to meet up with somebody for a drink. This is a guy I’ve been chatting with for a few years. He lives in Brooklyn with his partner, so that might be fun.

This trip to New York is completely pointless. Our business project manager promised that someone from the technical team would be there this week for their implementation, thus I’m going. Still, there is no pressing need for me to be there.

Bostonist: Brookline Resident in Finals of Public Radio Talent Quest
Sep 23rd, 2007 by Mike

Brookline Resident in Finals of Public Radio Talent Quest

Now this is cool enough… But what makes it even cooler is that I used to work with her.

High School
Sep 20th, 2007 by Mike

So today’s events with my office crush got really interesting. Let me start from the beginning.

When I got into the office about 7:30, I got an IM from my coworker, Trish. Trish, I really hope you don’t mind my sharing this.

Trish: o
me: what?
Trish: Guess who broke up with his bf?
me: NO!
Trish: yes!

It turns out that one of the women he works with came over to tell Trish right away that the couple had broken up. Over the course of the day, Trish let it to slip to his coworker that she thought somebody in the company had a crush on him.  And then later I got a message from Trish saying that he was up by her office. I was up there in about 6.3 seconds to ask Trish what she was doing for lunch, even though I was covering the helpdesk.

Now, he still doesn’t know that I’m the one with the crush on him. Well, he may have figured it out by the fact that I stopped up by Trish’s office a few more times than normal today. And he may have picked up on the fact that I stopped by to say hello to him after lunch.  I’ve never done that before.

As I told Trish, this is all so very high school. And you know what? It’s been a lot of fun.

My friend Tatjana commented a while back that office relationships are always a bad idea. She’s probably right. My company has a ton of them. We have a lot of married couples. But that’s getting WAY far ahead of everything here. I just met him last week, and it’s just a crush. And it’s been fun. Just like high school. 

The bitter old queen in me is reminded that he lives with said ex-boyfriend which always leaves the option of reconciliation. Still…

Calling India
Sep 17th, 2007 by Mike

I’m on the phone with tech support from Business Objects, the people who make Crystal Reports and the Crystal Reports Server. I’ve been using these products for seven or eight years.

We do most of our tech support work with them online. However, we tried moving our server to new hardware and it blew up on us. This required support via phone. I’m currently talking to a support engineer in India. While I’m sure she knows what she’s doing, it’s hard to be reassured because the language barrier is so great.

»  Substance:WordPress   »  Style:Ahren Ahimsa