All Systems Operational
Website Operational
Images & file attachments Operational
Import ? Operational
Codelets Operational
Third-party infrastructure ? Operational
Amazon AWS ? Operational
Heroku Platform ? Operational
DNSimple Name Servers ? Operational
Intercom ? Operational
Mailgun SMTP ? Operational
Mailgun Outbound Delivery ? Operational Hubs ? Operational
Degraded Performance
Partial Outage
Major Outage
Past Incidents
Mar 25, 2017

No incidents reported today.

Mar 24, 2017

No incidents reported.

Mar 23, 2017
On Friday, March 17, we discovered that the password for one of our production databases had been accidentally posted to a public GitHub repository by one of our developers. We quickly worked to change the password; the password that was leaked is no longer valid for any of our systems. The password had been public for about two days prior to the discovery.

This is a serious leak and could have resulted in user data being exposed (although, after an analysis, we believe that no data was in fact accessed by any unauthorized parties). I’m writing to let you know what happened, what we believe may have been compromised, and what we recommend you do regarding your data.

What data was affected?

We have multiple databases storing your information at Fieldbook. The leaked password was for a database that contained a copy of the data in the cells of your sheets.

User emails and passwords, user code (“codelets”), and configuration such as webhooks are all contained in a separate database that was not compromised in any way. Because of this, it is not necessary to reset your Fieldbook password or to be concerned that your email address was stolen.

Further, the master copy of all sheet data is also kept in a separate database that was not compromised. Because of this, we are confident that your data was not altered.

Was my data stolen?

Based on an analysis of database logs, we believe that no data, in fact, was exposed to any outside parties. Specifically, we have done a full audit on the access and query logs for the affected database, looking at every query that was run during the exposure time, and found no unusual queries.

We have also analyzed the IP addresses that accessed the database. All the IP addresses either belong to Amazon Web Services (AWS), where Fieldbook is hosted, or to Fieldbook employees. We are following up with our hosting and database providers to further verify that the AWS IP addresses correspond to servers that were running Fieldbook services.

What should I do now?

Even though our analysis found no evidence of unauthorized access, we recommend that if you stored any passwords in a Fieldbook sheet, that you change those passwords now. (Fieldbook data is not encrypted in the database, and in fact we do not recommend it for storing passwords, credit card numbers, or protected health information, as stated in our Security & Privacy FAQ. -

What is Fieldbook doing to prevent leaks in the future?

This was a terrible leak that could have let anyone access user data, and as CTO I offer my deepest apologies for this incident. While we do not believe anyone accessed any data because of this breach, it was completely unacceptable.

We’re taking a few steps to address the root causes and prevent this kind of leak in the future:

* We have rewritten the script that leaked the password. It now retrieves the password from a separate location that is not published to source code repositories.
* We have added automatic scanning for leaks from our developers to make sure these are caught before they are exposed to the public.

I understand that you may store confidential business or personal information in Fieldbook, and the security of your data is very important to you. We fell far short of our own standards in this regard.

Please don’t hesitate to reach out to us if you have any questions or concerns.

Ben Bernard
Co-Founder & CTO, Fieldbook
Mar 23, 13:41 PDT
Mar 22, 2017

No incidents reported.

Mar 21, 2017
Resolved - Fieldbook has been stable for the last few hours we've been monitoring it. A spike in load overwhelmed our server fleet, and we had to quickly scale up to handle it. We're now at increased capacity.

We are also investigating auto-scaling solutions to handle load spikes more quickly and smoothly in the future.
Mar 21, 15:29 PDT
Monitoring - A fix has been implemented and we are monitoring the results.
Mar 21, 12:41 PDT
Investigating - We are experiencing an outage due to heavy load. We are still investigating the cause.
Mar 21, 12:23 PDT
Mar 20, 2017
Resolved - We've been monitoring Fieldbook for the last couple of hours and have seen no problems. We're still waiting to get more information from Heroku about why the database configuration was changed, but the immediate issue is resolved.
Mar 20, 14:11 PDT
Update - Fieldbook is up and stable.

Heroku, our hosting provider, made a change to our account to remove an old database we weren't using, and this had the side effect of removing the credentials to the main database. We immediately rolled back the configuration change, which restored the service. We're now investigating why the change was made in the first place.
Mar 20, 12:37 PDT
Monitoring - Fieldbook is back up. We're making sure everything is stable.
Mar 20, 12:31 PDT
Investigating - Fieldbook is down. We are investigating.
Mar 20, 12:30 PDT
Mar 19, 2017

No incidents reported.

Mar 18, 2017

No incidents reported.

Mar 17, 2017
Resolved - Fieldbook is back up and stable.
Mar 17, 16:10 PDT
Monitoring - Fieldbook is back up. We're verifying stability of the system.
Mar 17, 15:20 PDT
Investigating - Fieldbook is down. Multiple engineers are working to bring it up and should have it back in a few minutes.
Mar 17, 15:16 PDT
Mar 16, 2017

No incidents reported.

Mar 15, 2017

No incidents reported.

Mar 14, 2017

No incidents reported.

Mar 13, 2017

No incidents reported.

Mar 12, 2017

No incidents reported.

Mar 11, 2017

No incidents reported.