Coinbase Developers

Developer and API news, stories and tips

Written by Coinbase Engineering

Early January Outage Post Mortem

Summary

On January 4th a large sustained increase in legitimate user traffic overwhelmed an internal database resulting in hours of cumulative downtime. Subsequent hardware upgrades, software optimizations and improved monitoring have since restored full operation and better prepared us for future surges in traffic.

Traffic Spike

Load during two outages, one rollover and a final graceful upgrade

Timeline

January 4th

  • 10:15 - Response times increase and on-call engineers are paged.
  • 11:00 - On-call engineers begin dropping a subset of traffic to reduce queuing and restore normal operations on a primary database.
  • 12:15 - Now under normal load, primary database are failed over to warm secondaries.
  • 13:00 - New database indices are built to optimize and prepare to restore normal operations.
  • 13:15 - Normal operations are restored and team continues to optimize usage.

January 5th - morning

  • 05:15 - Response times increase and on-call engineers are paged by newly introduced early monitoring tools.
  • 05:45 - Service is gracefully degraded to prevent a full outage for authenticated users.
  • 06:00 - Response times begin to return to normal levels.

January 5th - evening

  • 18:45 - Service is again gracefully degraded to minimize load during a scheduled upgrade to larger secondaries.
  • 19:00 - Secondaries are not sufficiently warmed and rollover is eventually aborted.
  • 20:30 - Normal operations are restored and we begin working a more efficient rollover procedure.

January 6th

  • 10:00 - Our initial plan to warm new hardware by streaming data from a primary to a secondary is projected to take ~72 hours. We decide shortcut this process by copying data directly from a high PIOPS EBS snapshot to a locally attached SSD on the new box, bringing the warming time down to 15 hours.

January 7th

  • 13:00 - Using new database warming procedure to ensure primary indexes are in RAM and newly replicated high performance secondary, the final and most impactful planned upgrade is completed to restore full service.

Root Cause

During this traffic increase, we saw a sustained increase in legitimate user traffic and one primary database was not prepared to scale to meet this load. This database acted as chokepoint in performance that resulted in a complete public outage of coinbase.com.

Though scaling our databases is a normal operation that we’re prepared to execute without interruption, we weren’t prepared for such a large, sudden increase in traffic. In the past we’ve planned in advanced for upgrades where new instances have had ample time to warm over many days. We weren’t prepared to manually expedite this warming resulting in our second outage, and our final upgrade took over 15 hours to migrate and warm indices.

Actions Taken

Since the start of this outage we’ve improved problematic systems and better prepared for future incidents:

  • Scale: We’ve upgraded our clusters from network attached storage to locally attached SSDs resulting in a ~20x decrease in latency on some operations. Vertically scaling our systems is however a short term win. We’re now working on better separating concerns by scaling horizontally and upgrading to more efficiently designed systems. Running on this new hardware across all of Coinbase’s Web & API tier we’re now clocking 35% faster response times.
  • Optimization: We’ve reviewed our least efficient queries and worked with the team to either refactor offending code or build new indices to minimize contention and expensive operations that scaled poorly.
  • Monitoring & Alerting: New monitoring on request volume and response times along with earlier alerting has been implemented to better identify future spikes and minimize future escalations resulting in a full outage.
  • Access: Our top priority at Coinbase is security and we emphasize least privilege. With our consensus model we’ve struck a good balance to empower users, but during this incident we identified and remediated several systems that the team either didn’t have access to or weren’t yet comfortable using.
  • Graceful Degradation: Going into this incident, we relied on several slow-to-enable anti-DDoS & deployable configuration settings to degrade services that extended the duration and impact of this outage. We’ve since developed the ability to rapidly update & much more gracefully degrade services without locking out customers.
  • Readiness: Working through this incident gave new team members a (very public) chance to run through our incident response process and identify gaps in tooling, training and documentation. One of our engineering principles is to “always leave it better than you found it” and we’re working through a backlog now that we’ll be exercising through our regular tabletops and simulations.

War Room

A Typical Coinbase War Room

Moving Forward

The top strategic priority of Coinbase is to provide a secure service to our customers. This often means we prioritize spending time on new security tools over focusing on high availability. As the digital currency market grows and so too does our engineering team, we’ve started investing more heavily in the reliability and availability of our systems. As we ship more of these improvements in ‘17 we’re looking forward to providing a better experience across our services.

If you’re excited about building fast-growing, high quality systems at scale, we’d like to hear from you!


Written by John Yi

Sunsetting Toshi.io - Hosted Bitcoin Node

For approx 1.5 years we have been running an unsupported hosted version of our open source bitcoin node (aka Toshi) at Toshi.io. It’s become increasingly buggy and has regrettably caused some frustration for a handful of developers who rely on it.

As we have posted today atop Toshi.io, developers are hereby notified that we will sunset the hosted service on Dec 15th 2016. We will continue to make available the open source code in the github repo here.

Please direct any questions/feedback to us at api@coinbase.com.


Written by Brian Armstrong

How Coinbase Builds Secure Infrastructure To Store Bitcoin In The Cloud

Or how to build paranoid infrastructure on top of AWS

Three and a half years ago, Coinbase launched using a simple hosting platform: Heroku.

It was the right solution at the time. With just two technical founders building the product (neither with any serious dev-ops experience) we knew that Heroku would be more battle tested than any solution we could hack together on our own.

But we also knew this wouldn’t work forever. Early in our company’s history, we started to contemplate the next version of our infrastructure that we would run inside AWS. It had to be built from the ground up with security in mind (the most common ways that bitcoin companies die is due to theft and hacking) but we didn’t want to compromise on engineering happiness and productivity.

After about a year, we finally completed the transition and we’ve been running inside AWS for quite some time now. This post outlines some of what we learned during the transition. It can be used as a starting point to design paranoid and useful infrastructure in the cloud.

Today, Coinbase securely stores about 10% of all bitcoin in circulation.

Read the rest of this post on Medium…


Written by Jori Lallo

February API updates

Localization

Coinbase products are available in a growing number of languages. To support our international expansion efforts, we’re making it easier for our API partners to support their users in their native languages. You can now get localized error messages and other strings in alternative languages by passing in the Accept-Language header as part of your API request. Available languages are listed in our API reference.

CORS support

By default, browsers restrict AJAX requests to domains other than the current one. This is a security measure that’s in place to prevent malicious intent. APIs and other sites can alternatively whitelist this browser access using cross-origin resource sharing, more commonly known as CORS. To enable more interesting use cases for Coinbase’s API, we enabled CORS for both the Coinbase Wallet API and the Coinbase Exchange API.

Transaction details

Coinbase supports a multitude of different transaction ledger types. A couple of the most common ones are send for sending bitcoin and buy for buying bitcoin. At the time of this announcement, we’re supporting 11 different types and will likely add more in the future.

{
  "id": "57ffb4ae-0c59-5430-bcd3-3f98f797a66c",
  "type": "send",
  "status": "completed",
  "amount": {
    "amount": "-0.00100000",
    "currency": "BTC"
  },
  ...
  "details": {
    "title": "Sent bitcoin",
    "subtitle": "to User 2"
  }
}

This means that constructing a readable transaction list for a wallet hasn’t been easy given the sheer amount of different types and their variations. To simplify this, we added more detailed information to transaction resource. They’ll now include a details hash which contains title and subtitle values which can be used to contruct human readable transaction entries. These values also respect localization and will be translated based on the Accept-Language header.

Changes to API versioning

We recently added notifications to the Coinbase API. As notifications are delivered via POST requests (webhooks), they can’t be versioned using our existing versioning schema where one needs to pass the date of the implementation as the CB-Version header.

Given that we would have needed to make a breaking change to the notification resource payload, we decided to add a user level API version designation which can be controlled from API settings. For both notification and API requests where the user hasn’t defined their API version within the header, this default version will be used. For requests, using the CB-Version header is still the recommended way to define your version, as also used in our client libraries.

New endpoint: historic bitcoin prices

We added an easy way to get historic bitcoin prices used by Coinbase. The new endpoint doesn’t require authentication and is available at GET api.coinbase.com/v2/prices/historic. You can also get the value using our client libraries.

Upgrading from legacy API v1 to v2

To make it easier for users of our legacy API v1 to upgrade to the newer API v2, we have added new fields to v1 responses. Given that we started using UUID as resource identifier in the new v2, it hasn’t been easy to map existing objects from v1 to v2. To solve this we added a new uuid field to v1 resources, which maps to v2 resource ids.

To see all recent API changes, visit our API changelog.


Written by Rob Witoff

Bug Bounty Transparency Update

Over the past two years Coinbase has benefited greatly from running a public bug bounty program and we believe strongly in incentivizing the white-hat community to responsibly disclose vulnerabilities to us and our partners. In light of a recent public discussion, we’d like to take this moment to transparently share more inner-workings of our program and affirm our commitment to growing our collaboration with the white-hat community in 2016.

Program Performance

In 2013, we chose to run our bug bounty program through HackerOne and review the results of our program every quarter. Since the beginning, we’ve paid out a total of $103,801 in valid bounties. 9% of our submissions were resolved by working with a researcher while the remaining 91% were closed in one of the below categories:

Bug Bounty Stats

Looking only over the last three months, we have:

  • paid out $10,300 over 19 bounties for an average of $542 per bounty
  • resolved 21 additional bugs submitted through the program
  • averaged 23 hours to respond to every new submission
  • averaged 18 days from submission to payout for valid bounties

One of the challenges with a white-hat program is effectively managing the majority of submissions that do not result in a paid bounty. One recent public example highlighted some aspects of our security program that we’d like to reflect on here:

Submission #606921: publicity pending: Balance Manipulation:

Reported on May 31st, a researcher identified a race condition in moving funds between multiple wallets within a Coinbase account. This allowed a user to send funds offsite, leaving one wallet with a negative balance. When reviewing this case internally, we confirmed that other security controls would have protected us from losing funds (you could only send funds that you did indeed own) and initially contemplated the severity of the finding. Despite employing defense-in-depth, this was a valid vulnerability in one of our layers and we rewarded the researcher with a $5,000 bounty within 24 hours of the original HackerOne submission.

Submission #949251: publicity pending: Balance Manipulation v2:

Reported on October 20th and similar to #60692 this report suggests that a similar race condition is possible while moving funds out of our vault. This purported exploit posed several challenges for our security team:

  1. Though the attack was described, neither our security or engineering teams were able to reproduce or validate this exploit. Further, we saw no indication that the researcher had actually been able to exercise the vulnerability. Lack of information is common in all white-hat programs and we regularly work with researchers to provide clear test cases and that’s what happened here.
  2. The researcher then explained that they only “half way completed the bug” but could not finish the exploit due to a lack of funds. Our Security Team attempted to provide the user additional funds to execute the purported exploit but could not because of separate restrictions on this user’s account applied by our independent Compliance Team for unrelated reasons.
  3. The reasons that our Compliance Team may place restrictions on a user’s account can be found in our User Agreement. Further, Coinbase has never, and will never ban a user for any responsible white-hat security testing of our public endpoints. In fact, we have paid over $100,000 to reward white-hat researchers and are planning to grow this program over time. Banning a researcher for security submissions would be bad for both security and business and is counter to our mission.

Despite these challenges, we recognize that our response should have better communicated our understanding of the issue in several ways:

  1. From public discussions, it may have appeared that the restrictions and security activity may have been connected. In reality, these represented separate communications with separate employees on separate teams at separate points in time.
  2. We failed to clearly communicate and close this ticket out in a timely manner. From the first submission on October 20th to the final response on December 1st, 42 days passed including several without answering following requests for comment by the researcher. This was neither professional nor courteous to the researcher and we should have more promptly dispositioned the submission.

1N.B. Publicity Pending: A good bug bounty program should be prepared for disagreements and HackerOne supports this with a mutual disclosure feature. We have requested that both of the above submissions be made fully public which will occur when the researcher agrees or after a 30 day waiting period that ends on 1/20 2016:

Bug Bounty Disclosure

Looking Ahead

Just as Bitcoin is being bootstrapped by an elegant incentive structure, well designed bug bounty programs can optimize incentives between public & private security engineers. Companies pay for the vulnerabilities or better practices that are reported (instead of abused) and responsible researchers can work without fear of retribution while they explore endpoints and even make a healthy profit. We believe that any organization that truly cares about security is clearly incentivized to run a healthy bug bounty program.

After reflecting on the performance of our bug bounty program in 2015, we’re now working on optimizing our program terms, interactions with team and the increasing the engagement with participating researchers. In 2016 we will be expanding the scope of our program and publicizing at least 30% of our valid submissions. We believe strongly in the value of our program and encourage the rest of technology to similarly engage the security community.

For additional discussion:


Written by Jori Lallo

Notifications

Writing software against the blockchain and financial systems can be challenging, because actions are often performed asynchronously. Anyone who has implemented functionality for incoming bitcoin transactions is more than familiar with this. Polling APIs is easy to implement but hard to scale.

Today we’re thrilled to announce a new addition to Coinbase API family: Notifications. Notifications are alerts sent to your application using webhooks (POST request with JSON data) when corresponding actions are performed or observed by Coinbase. This makes it very easy to react to actions related to your application. We’re starting with a handful of important and heavily requested actions and will add more over time. Starting today you can subscribe to events such as these:

  • New transactions for bitcoin addresses created for your Coinbase wallets
  • Completion of your bitcoin buy or sell
  • Receipt of a successful payment for your merchant checkout order

API Notifications are built in similar way to our old callback system, which was mainly designed for merchant services. Our new notifications can be set up for both API keys and Coinbase Connect (OAuth2) applications. This makes it easy to subscribe to both your own and your application’s users’ notifications.

You can set the notification endpoint in your API settings individually for API keys and applications. Once you have subscribed to events and set the appropriate permissions to receive respective notifications, we’ll start sending them to your application. Each notification contains a unique ID, type, and the corresponding data payload:

{
  "id": "6bf0ca21-0b2f-5e8a-b95e-7bd7eaccc338",
  "type": "wallet:buys:completed",
  "data": {
    "id": "67e0eaec-07d7-54c4-a72c-2e92826897df",
    "status": "completed",
    "payment_method": {
      "id": "83562370-3e5c-51db-87da-752af5ab9559",
      "resource": "payment_method",
      "resource_path": "/v2/payment-methods/83562370-3e5c-51db-87da-752af5ab9559"
    },
    "transaction": {
      "id": "441b9494-b3f0-5b98-b9b0-4d82c21c252a",
      "resource": "transaction",
      "resource_path": "/v2/accounts/2bbf394c-193b-5b2a-9155-3b4732659ede/transactions/441b9494-b3f0-5b98-b9b0-4d82c21c252a"
    },
    "amount": {
      "amount": "1.00000000",
      "currency": "BTC"
    },
    "total": {
      "amount": "10.25",
      "currency": "USD"
    },
    "subtotal": {
      "amount": "10.10",
      "currency": "USD"
    },
    "created_at": "2015-01-31T20:49:02Z",
    "updated_at": "2015-02-11T16:54:02-08:00",
    "resource": "buy",
    "resource_path": "/v2/accounts/2bbf394c-193b-5b2a-9155-3b4732659ede/buys/67e0eaec-07d7-54c4-a72c-2e92826897df",
    "committed": true,
    "instant": false,
    "fees": [
      {
        "type": "coinbase",
        "amount": {
          "amount": "0.00",
          "currency": "USD"
        }
      },
      {
        "type": "bank",
        "amount": {
          "amount": "0.15",
          "currency": "USD"
        }
      }
    ],
    "payout_at": "2015-02-18T16:54:00-08:00"
  },
  "user": {
    "id": "f01c821e-bb35-555f-a4da-548672963119",
    "resource": "user",
    "resource_path": "/v2/users/f01c821e-bb35-555f-a4da-548672963119"
  },
  "account": {
    "id": "8d5f086c-d7d5-58ee-890e-c09b3d8d4434",
    "resource": "account",
    "resource_path": "/v2/accounts/8d5f086c-d7d5-58ee-890e-c09b3d8d4434"
  },
  "delivery_attempts": 0,
  "created_at": "2015-11-10T19:15:06Z",
  "resource": "notification",
  "resource_path": "/v2/notifications/6bf0ca21-0b2f-5e8a-b95e-7bd7eaccc338"
}

We’re also now including user and account information for apps that are using Notifications for their Coinbase Connect applications, which enables you to differentiate notifications by unique user. If you’d like, you can access all your notifications via the API endpoint for notifications.

If you’re using merchant checkout products, you don’t necessarily need to set up API keys for simple payment integrations. Instead, you can set the notification endpoint in your merchant settings. Read more about merchant notifications.

As with all our APIs, we’re constantly listening to developer feedback and improving our offering. If there are Notification events you’d like for us to consider building, we would love to hear from you. Please email us at api@coinbase.com.


Written by Maksim Stepanenko

Coinbase Sandbox improvements

When building financial applications, it’s extremely important to test in a risk-free environment. That’s why why we encourage all the developers using our API to always start on Coinbase Sandbox before moving to the production API. We launched Coinbase Sandbox in the beginning of this year to help developers safely test their applications, and over the past couple of weeks we’ve made a number of improvements to it:

Test Bank Accounts

All sandbox users by default now get a test bank account, which enables a few new features:

  • Fake USD Deposits and Withdrawals
  • Fake Buys and Sells

Buy Dialog

Other Improvements

  • Testnet blockchain stability has been improved
  • The 0.1 signup bonus is now sent off-blockchain, so you can get started instantly
  • API key creation flow has been simplified

All of the above improvements are based on the feedback we’ve been getting from you. You can find more details about Coinbase Sandbox here.


Written by Maksim Stepanenko

Coinbase Node.js client updates

We’ve updated our official Coinbase Node.js library to support the new API v2. We tried to keep the structure of the codebase very similar to the previous versions to make it as easy as possible for you to migrate to API v2. This new version is now available on npm:

npm install coinbase

To get you started, here’s an example code snippet to buy bitcoin using this library:

var coinbase = require('coinbase');
var client   = new coinbase.Client({'apiKey': 'API KEY',
                                    'apiSecret': 'API SECRET'});

// Create new BTC wallet/account:
client.createAccount({'name': 'test'}, function(err, account) {
  console.log('Account: ' + account.name);
  account.buy({'amount': 0.01, 'currency': 'BTC'}, function(err, buy) {
    console.log('Buy: ' + buy.amount.amount + ' ' + buy.amount.currency);
  });
});

You can find more information about the library here. If you have any questions or feedback regarding this update, let us know on Twitter, or via email.


Written by Jori Lallo

Improved Coinbase Connect (OAuth2)

Coinbase Connect

We’re excited to release an improved version of Coinbase Connect, the OAuth2 implementation that allows developers to connect to Coinbase wallets.

Coinbase Connect’s authorization view (aka permissions dialogue), shown to users when they are authorizing applications, has been redesigned from the ground up. Permissions are now organized into logical groups. Permissions such as send bitcoin, show user’s email and others that afford developers nontrivial responsibility, are clearly and individually displayed to the user, while permissions that deliver significantly less sensitive information are bundled together.

We’re also introducing the concept of “application controlled wallets.” Previously, an application would by default request and gain access to all of user’s wallets. Now the authorization dialogue lets the user specify one wallet to link with the app, or to create a new associated wallet. Apps may still request to access to all of user’s wallets (e.g. for tax reporting). Account access is easily controlled with a new account parameter in the authorization URL. You can find more info in the OAuth reference documentation.

After introducing send limits earlier this year, we have now also included the option for the users to change their send limits when authorizing an application.

What does this mean for existing applications?

All existing applications will continue to work without interruption. This includes retaining access to all accounts by default. We encourage developers to update their applications to only access one wallet, by directing new users to enter the new authorization flow. This can be triggered for existing users by using the account=select parameter within the OAuth authorization URL.

Finally, since we have added more application information to the Coinbase Connect auth dialogue, we also recommended that developers complete and/or update their developer information and logos in their OAuth application settings.


Written by Jori Lallo

New Python and PHP clients and other API updates

coinbase-python

Earlier this year we introduced the official Coinbase Wallet API support for Python (coinbase-python). This library has now been updated to version 2.0.0 to support the new API v2, released last month. This new version is available on PYPI and can be installed with pip:

pip install coinbase

You can read more about how to get started and about all the functionality of the library over at GitHub:

coinbase-python on GitHub

coinbase-php

Continuing our efforts to support Coinbase Wallet API v2 we have also released a completely rebuilt version of our PHP library. It’s now fully tested and up to date with the latest features. Get started with the library by installing it with Composer:

composer require coinbase/coinbase

coinbase-php on GitHub

Instant Exchange API support

A few weeks ago we released Instant Exchange, a feature that allows users to send or receive bitcoin payments while eliminating exposure to bitcoin exchange rate fluctuation. We have since expanded our API to support this functionality, making it easy to send and receive bitcoin payments to fiat accounts. Support includes:

  • Create addresses for fiat currency accounts
  • Send funds from fiat accounts (type: wallet)
  • Additional instant_exchange field for transactions to indicate that they have been instantly exchanged

New Coinbase Connect (OAuth2) documentation

Coinbase Connect makes it easy for you to access Coinbase users’ wallets so that they can buy/sell, send/receive and store bitcoin within your applications. Our new documentation makes it easier for you to get started integrating Coinbase functionality into apps.

Read more about Coinbase Connect

In addition to the updates listed above, we pushed these improvements and fixes:

  • resource_path attribute added to all linked resources
  • GET /v2/time is now unauthenticated
  • New version of coinbase-ruby (4.0.7) which fixes multiple smaller issues