A message from Intuit’s CIO
To our customers:
We understand we didn’t communicate as frequently as you or we would have wanted. This issue was complicated and involved a number of Intuit applications. We did not want to share any information until we had complete confidence in it. We realize now that in times of uncertainty it is better to share updates on our progress as they occur versus waiting for complete answers, not sharing enough and leaving you feeling in the dark.
We still have unanswered questions, but we do know more and I want to share it with you.
We had an accidental power failure that brought both our primary and secondary power units down. The process of restoring power caused a hardware system failure. We still don’t know the root cause of that. To ensure full integrity across all our impacted applications, we immediately initiated a restoration process using our back up data. We also had to ensure proper sequencing in a way where both individual applications and our own enterprise applications that they rely on are restored in the most reliable and secure way before bringing them back on line. This took more time than anyone would like, but it was the safest most reliable course of action.
We will continue to look into both the root cause of the failure as well as our response so that we learn from this and get better. We appreciate your patience in what has been, understandably, a very frustrating experience.
Ginny Lee
Chief Information Officer
Intuit Inc.
June 18, 2010 at 11:19 am
When we converted to QBO, I was lead to believe that backup servers were housed in a redundant data center. If that is true, then why not switch over the the redundant data center?
I know everyone was working to bring up the primary site but redundacy is there for a reason. Or… was I mislead?
Thank you,
Rob Melrose
June 18, 2010 at 4:25 pm
I agree with rob, the whole reason to choose this product was so this would not happen! I think we should at least be given a free month of service, a letter from the CIO doesnt do much to make me feel better but a service credit at least shows that someone cares and has monitary interest in preventing it from happening again tomorrow or next week
June 19, 2010 at 3:16 am
Rob,
I think it would be appropriate to communicate a plan of action to make sure this doesn’t happen again – in a few weeks once you completely understand the situation. That will give a lot of us more confidence in this product. I’m a ProAdvisor and I’m recommending this product to all my new clients. I just don’t want to end up with mud on my face because they were unable to access their data for an extended period of time.
Thanks in advance,
Cathy Iconis
June 19, 2010 at 4:57 pm
6-19-10 Saturday 4:55pm PST – Quickbooks Online is down again. Was this scheduled downtime? Doesn’t seem like it but maybe I missed a message about it?
When we try to login we get:
Sorry! QuickBooks Online is currently unavailable.
Our team is working hard to solve the problem. Please try again later.
Can you give us an update please?
June 19, 2010 at 10:27 pm
I am so very sorry that some of our services (QuickBooks Online, ProLine Tax Online & Merchant Services as well as others) have been affected by an outage. We’re working aggressively to resolve a hardware system issue and we’ve made some progress correcting the problem. We are beginning to bring some applications back online. However, there is still more work to be done. It continues to be all-hands on deck.
We have failed to live up to the high standard of dependability that we set for ourselves and for you. There is no excuse for the negative impact we’ve had your livelihoods this past week.
Apologies,
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing, QuickBooks Online
June 20, 2010 at 7:30 am
QuickBooks came back online in the middle of the night. All our other services are back online too. I’m deeply sorry for the effect that our service outage had on you and your business. I know many of you were trying to catch up from our outage earlier in the week. We appreciate your patience in what has been, understandably, a very frustrating experience.
The Intuit team continues to monitor our systems closely. We are still looking into these outages to fully understand the root cause and what we can do to get better. Thank you for your patience.
Apologies,
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing, QuickBooks Online
June 20, 2010 at 9:37 am
Everyone has been very apologetic, but that doesn’t help much. I am sure you and everyone is “very sorry”, for our inconvenience, but what about business hours lost? Productivity lost? Our time, lost? Each individual company’s reputation has been damaged. I had to tell customers that, sorry our software is down today, and I can’t get to that information. Please call back. This is just not acceptible. I lost business and I lost my own time. And, you are right — I was in the office yesterday to catch up on the work I couldn’t do the other day, and was 1/2 through entering invoices and updating everything when it crashed again. Now, I am spending yet another week-end day to try to get caught up. This is just not right. There has to be some compensation for this mishap. Your reputation and the reputation of your company has been damaged and so has ours. We deserve some compensation of at least one month’s fees to show your good will. You need to make this right.
June 21, 2010 at 6:28 am
Are you, instead of apologizing (which is OK) offering as a good faith sign a compensation for all the money and time lost in this outage?
At least one month service!!!
June 20, 2010 at 11:07 am
It is a big concern to all of us when we can not access our clients work to do our job. When I use QB Desktop I always do a backup daily and make an extra copy of the backup. With QB on Line you can make a local back up, but it is more complex to restore the data file to desktop plus it is not a complete restore. This is not a good solution for me and my clients.
I have been using QB Desktop and On Line when they first came out and I have been a pro advisor since 99. I need a better solution for me and my clients. Perhaps I will use a hosted provider of the desktop, but I am not sure.
Saturday while I was inputing in eary afternoon I was using QB on Line about 2:00 California time and it just stopped. If this was a planned shutdown, then we should have received a message first, right? It took until about 1:30 for it to work again or about 12 hours, this is not a reasonable solution.
Now I have promissed that work will be completed by close of business today, but I do not have enough time to complete everything.
I need a good solution like the QB Desktop backup, and not the local backup that requires a conversion and takes a long time.
Any suggestions?
Thanks,
Steve
June 20, 2010 at 12:05 pm
A quick update to let folks know that our support lines are open until 4pm PST today (Sunday 6/20). You can reach us at 1800 286 6800.
Heather Kirkby
heather_kirkby@intuit.com
Director Product Management and Marketing, QuickBooks Online
June 20, 2010 at 1:28 pm
This was a complete disaster. I agree that this was the whole reason I spent hours of my time moving everything to online- to avoid a crash. I switched to online a day before going on a long trip so that I could manage my work remotely. This failure has cost me hundreds of dollars in down time for three employees and 2 days of frustration. It has completely foiled my opportunity to work remotely. The desktop version is very buggy so I was glad to leave it behind. One month free would not even scratch the service. You would have to offer me a year free at this point to keep me as a customer. I have been using quickbooks since I started my business in 1999. But now I will definitely seek a more reliable program– as soon as I return to my office. I am still travelling!
June 20, 2010 at 8:14 pm
I don’t understand how you could go down for so long. Were you hacked or attacked?
June 20, 2010 at 9:10 pm
Do you have any suggestions about what we should do if your system is currently unreliable? Daily Xcel reports of customers, vendors, a/p, a/r? My concern is next time the data may be lost. Can Xcel reports be uploaded to QB desktop or can the online version be uploaded to the desktop version? Can you send a physical backup?
June 21, 2010 at 6:44 am
I agree with almost everyone that there needs to be some compensation for our lost time and business. I’ve never heard of both the main and the backup servers going down at the same time. They should not even be remotely close to each other for this to ever happen. I alomost had to calculate payroll manually and that would have been a disaster. Please let us know what your plans are to insure this doesn’t happen again.
June 21, 2010 at 6:57 am
It would appear that my recurring invoice templates did not post on the 16th. I have just run across 3. Is this a coincidence or do I need to go through my entire list to post those all by hand? I was pretty empathetic to the crash as things happen beyond our control, however having to go through my entire list will be testing my patience and maybe push me to look for an alternate business solution.
Stacy
June 21, 2010 at 7:39 am
Hi Stacy, thanks for your note. Although we didn’t lose any data in this downtime, if you are seeing problems or have questions about your data/invoices it would be best to call support at 1 800 286 6800. They should be able to work with you to figure out what’s happened, rather than going through your entire list.
Kristen
June 21, 2010 at 1:01 pm
Now we know that server crashes happen in all size companies. The question is what can be learned from the experience?
One feature of quickbooks on line that I would like to see is an automatic backup of all files to my local network, computer, or server. While the long term stability of Intuit servers is probably greater, the customer would have a data “security blanket” under his own control. The next time that there is an event, the customer could sign on as usual, but the quick books on line structure could be redirected to find a local data backup. When the online data again became available it could be synched to local files by action of the customer.
I like the quick books on line but there should be some follow up to smooth over a warm and fuzzy partnership with a lot of businesses.
Doug Niblack
June 21, 2010 at 1:11 pm
Thank you for the explanations regarding the outage. I don’t usually reply or post comments, but I figured that my point of view might not be something you are hearing too much these past few days. So here goes: Thank you! My husband works in the IT department of a major corporation and I know first hand how routine updates/management can go terribly awry. I also know that IT professionals work tirelessly day and night to get the system back up and running, and this kind of outage is never taken lightly by the IT team. In my opinion, Quickbooks and all of Intuit’s websites have been extroadinarily reliable, and I appreciate the hard work. Keep your heads up team!
June 21, 2010 at 5:14 pm
I’ve been using QuickBooks Online Edition for just about two years. There have only been 2 or 3 serious outages that I can remember. No doubt Intuit must look into a better contingency plan. Notwithstanding, have any of you that are complaining taking in consideration the amount of time that is saved buy using a cloud based accounting system? Have you taken in consideration the risk of your computer hardware failing? Viruses? I have had hardware problems in the past two years and I can login on another computer and no worries about re-installation; installation of not just QB but windows and everything else that must be installed. Please take that in consideration. I guarantee you that HP or Dell computer will never compensate you in anyway if their equipment fails; they will only help you get it back up and running.
I have accountants that use QBOL in several countries in South America. QBOL is a vital tool in helping me manage in-country accountants.
Intuit has taken the lead in online accounting with QBOL. They understand that users have a variety of personal preferences regarding hardware and browsers. This product works equally well on Firefox, Internet Explorer and on MAC Safari. I LOVE being able to use any machine to do my work. I don’t even have to own the machine.
Being accountants, I hope that we can “balance” this situation. I for one believe that the efficiencies and advantages of QBOL far exceed the alternative.
Drake Shattuck
Chief Accountant, Ascend Alliance
July 14, 2010 at 10:47 am
QBO is apparently down again. I am alarmed. How can this “one-time” problem (or a variation on it) happen this frequently on a SaaS application offered by Intuit? I am losing patience not just with the interruptions, but the lack of trusted communication being offered to us customers. I am a loyal customer who cannot keep ignoring what is going on here.
July 14, 2010 at 10:58 am
I’m scared
July 14, 2010 at 11:00 am
Is the system down again… I am AGAIN unable to access client data. Wow… this is totally unacceptable.
July 14, 2010 at 11:01 am
It appears we are down again! Getting ready for external auditors is very difficult when our accounting software is unavailable. I’ve been a big proponent of QB Online…I am starting to change my mind. Twice in 30 days in inexcusable.
July 14, 2010 at 11:17 am
Put an update here with an estimate of when we can start working again – we are UNABLE to work and very frustrated. Should we just leave for the day or take a short break??!!!
Where is the corporate communication?!?
July 14, 2010 at 11:51 am
Where is the backup plan? Redundant mirror site and data base?
Meanwhile………..how long? how long? how long?