Coinbase Developers

Developer and API news, stories and tips

Written by Coinbase Engineering

Early January Outage Post Mortem

Summary

On January 4th a large sustained increase in legitimate user traffic overwhelmed an internal database resulting in hours of cumulative downtime. Subsequent hardware upgrades, software optimizations and improved monitoring have since restored full operation and better prepared us for future surges in traffic.

Load during two outages, one rollover and a final graceful upgrade

Timeline

January 4th

  • 10:15am - Response times increase and on-call engineers are paged.
  • 11:00am - On-call engineers begin dropping a subset of traffic to reduce queuing and restore normal operations on a primary database.
  • 12:15pm - Now under normal load, primary database are failed over to warm secondaries.
  • 1:00pm - New database indices are built to optimize and prepare to restore normal operations.
  • 1:15pm - Normal operations are restored and team continues to optimize usage.

January 5th - morning

  • 5:15am - Response times increase and on-call engineers are paged by newly introduced early monitoring tools.
  • 5:45am - Service is gracefully degraded to prevent a full outage for authenticated users.
  • 6:00am - Response times begin to return to normal levels.

January 5th - evening

  • 6:45pm - Service is again gracefully degraded to minimize load during a scheduled upgrade to larger secondaries.
  • 7:00pm - Secondaries are not sufficiently warmed and rollover is eventually aborted.
  • 8:30 - Normal operations are restored and we begin working a more efficient rollover procedure.

January 6th

  • Our initial plan to warm new hardware by streaming data from a primary to a secondary is projected to take ~72 hours. We decide shortcut this process by copying data directly from a high PIOPS EBS snapshot to a locally attached SSD on the new box, bringing the warming time down to 15 hours.

January 7th

  • 1:00 - Using new database warming procedure to ensure primary indexes are in RAM and newly replicated high performance secondary, the final and most impactful planned upgrade is completed to restore full service.

Root Cause

During this traffic increase, we saw a sustained increase in legitimate user traffic and one primary database was not prepared to scale to meet this load. This database acted as chokepoint in performance that resulted in a complete public outage of coinbase.com.

Though scaling our databases is a normal operation that we’re prepared to execute without interruption, we weren’t prepared for such a large, sudden increase in traffic. In the past we’ve planned in advanced for upgrades where new instances have had ample time to warm over many days. We weren’t prepared to manually expedite this warming resulting in our second outage, and our final upgrade took over 15 hours to migrate and warm indices.

Actions Taken

Since the start of this outage we’ve improved problematic systems and better prepared for future incidents:

  • Scale: We’ve upgraded our clusters from network attached storage to locally attached SSDs resulting in a ~20x decrease in latency on some operations. Vertically scaling our systems is however a short term win. We’re now working on better separating concerns by scaling horizontally and upgrading to more efficiently designed systems. Running on this new hardware across all of Coinbase’s Web & API tier we’re now clocking 35% faster response times.
  • Optimization: We’ve reviewed our least efficient queries and worked with the team to either refactor offending code or build new indices to minimize contention and expensive operations that scaled poorly.
  • Monitoring & Alerting: New monitoring on request volume and response times along with earlier alerting has been implemented to better identify future spikes and minimize future escalations resulting in a full outage.
  • Access: Our top priority at Coinbase is security and we emphasize least privilege. With our consensus model we’ve struck a good balance to empower users, but during this incident we identified and remediated several systems that the team either didn’t have access to or weren’t yet comfortable using.
  • Graceful Degradation: Going into this incident, we relied on several slow-to-enable anti-DDoS & deployable configuration settings to degrade services that extended the duration and impact of this outage. We’ve since developed the ability to rapidly update & much more gracefully degrade services without locking out customers.
  • Readiness: Working through this incident gave new team members a (very public) chance to run through our incident response process and identify gaps in tooling, training and documentation. One of our engineering principles is to “always leave it better than you found it” and we’re working through a backlog now that we’ll be exercising through our regular tabletops and simulations.

A Typical Coinbase War Room

Moving Forward

The top strategic priority of Coinbase is to provide a secure service to our customers. This often means we prioritize spending time on new security tools over focusing on high availability. As the digital currency market grows and so too does our engineering team, we’ve started investing more heavily in the reliability and availability of our systems. As we ship more of these improvements in ‘17 we’re looking forward to providing a better experience across our services.

If you’re excited about building fast-growing, high quality systems at scale, we’d like to hear from you!


Written by John Yi

Sunsetting Toshi.io - Hosted Bitcoin Node

For approx 1.5 years we have been running an unsupported hosted version of our open source bitcoin node (aka Toshi) at Toshi.io. It’s become increasingly buggy and has regrettably caused some frustration for a handful of developers who rely on it.

As we have posted today atop Toshi.io, developers are hereby notified that we will sunset the hosted service on Dec 15th 2016. We will continue to make available the open source code in the github repo here.

Please direct any questions/feedback to us at api@coinbase.com.


Written by Maksim Stepanenko

Buying bitcoin just got way easier

Buying bitcoin is harder than it should be, which is why we at Coinbase have been working so hard to expand our services to more regions of the world, to accept more payment methods, and to process transactions as quickly as possible.

Over the past couple of years we’ve helped thousands of developers to build bitcoin applications. The most consistent and critical developer problem we’ve heard is how hard it is to find users with access to bitcoin. So, we built the Coinbase Buy Widget — the easiest and fastest way to buy bitcoin as a new user.

Buy Widget

The Buy Widget is an easily embeddable and configurable hosted widget that delivers three key features:

  1. allows users to purchase bitcoin using debit cards
  2. instantly delivers bitcoin to an address specified by the user or the application
  3. a butter-smooth user experience

How does the Buy Widget improve upon the buying experience at Coinbase.com?

Without ever leaving the developer application, users can get bitcoin delivered directly into the address provided by the app. For new users, the Buy Widget flow features a very familiar debit card form and a seamless way to create a Coinbase account. For existing Coinbase customers, the purchase can be completed with as few as just 3 steps.

Developers who want their users to buy bitcoin into their applications can now do it with just a couple of lines of code instead of a full blown OAuth integration.

We are very excited about this new product. The Coinbase Buy Widget offers newcomers to bitcoin an easy way to try out more applications, helps developers grow their communities, and helps novel bitcoin use-cases flourish. If you are a developer and this sounds as exciting to you, please let us know and we’ll be in touch with you shortly. You can learn more about how it works here.

Selected Partners

We are excited to launch the Buy Widget alongside three great developer partners. Each partner represents a novel use for bitcoin, and we’re proud to have had their input into our product development.

Brave Browser

Brave users can buy bitcoin to support their favorite web sites (coming soon)

Brave

“We have a mission to save the web by increasing browsing speed and safety for users, while supporting content creators. We intend to let users “bring their own BTC” to self-fund their wallets and auto-micropay their favorite websites. The Coinbase Buy Widget will make it easy for Brave users to buy bitcoin, even if the user has zero experience with cryptocurrency.” – Brendan Eich, CEO Brave Software

Hammercoin

The makers of BitQuest bring you a new minecraft experience, powered by a bitcoin economy

Hammercoin

“With Hammercoin, our next generation bitcoin-powered minecraft experience, our most frequently asked question was “How do I get Bitcoin?” Most of them were buying for the first time, and there was no easy answer. With the Coinbase Buy Widget there’s finally a way that makes users comfortable.” – Cristián González, CEO Mego Games

Popchest

A new monetization model for video producers: micropayments powered by Coinbase.

Popchest

“PopChest is the micropayments platform for video. We make it easy for content creators to offer their video in exchange for a micropayment rather than solely relying on advertising. We don’t force viewers to watch commercials before seeing the video they really want. Coinbase opens up micropayments to a whole new audience. Our viewers can instantly get all the benefits of bitcoin without previously possessing or even knowing anything about the digital currency. This is a game changer for PopChest.” – Valerian Bennett, CEO


Written by Brian Armstrong

How Coinbase Builds Secure Infrastructure To Store Bitcoin In The Cloud

Or how to build paranoid infrastructure on top of AWS

Three and a half years ago, Coinbase launched using a simple hosting platform: Heroku.

It was the right solution at the time. With just two technical founders building the product (neither with any serious dev-ops experience) we knew that Heroku would be more battle tested than any solution we could hack together on our own.

But we also knew this wouldn’t work forever. Early in our company’s history, we started to contemplate the next version of our infrastructure that we would run inside AWS. It had to be built from the ground up with security in mind (the most common ways that bitcoin companies die is due to theft and hacking) but we didn’t want to compromise on engineering happiness and productivity.

After about a year, we finally completed the transition and we’ve been running inside AWS for quite some time now. This post outlines some of what we learned during the transition. It can be used as a starting point to design paranoid and useful infrastructure in the cloud.

Today, Coinbase securely stores about 10% of all bitcoin in circulation.

Read the rest of this post on Medium…


Written by Jori Lallo

February API updates

Localization

Coinbase products are available in a growing number of languages. To support our international expansion efforts, we’re making it easier for our API partners to support their users in their native languages. You can now get localized error messages and other strings in alternative languages by passing in the Accept-Language header as part of your API request. Available languages are listed in our API reference.

CORS support

By default, browsers restrict AJAX requests to domains other than the current one. This is a security measure that’s in place to prevent malicious intent. APIs and other sites can alternatively whitelist this browser access using cross-origin resource sharing, more commonly known as CORS. To enable more interesting use cases for Coinbase’s API, we enabled CORS for both the Coinbase Wallet API and the Coinbase Exchange API.

Transaction details

Coinbase supports a multitude of different transaction ledger types. A couple of the most common ones are send for sending bitcoin and buy for buying bitcoin. At the time of this announcement, we’re supporting 11 different types and will likely add more in the future.

{
  "id": "57ffb4ae-0c59-5430-bcd3-3f98f797a66c",
  "type": "send",
  "status": "completed",
  "amount": {
    "amount": "-0.00100000",
    "currency": "BTC"
  },
  ...
  "details": {
    "title": "Sent bitcoin",
    "subtitle": "to User 2"
  }
}

This means that constructing a readable transaction list for a wallet hasn’t been easy given the sheer amount of different types and their variations. To simplify this, we added more detailed information to transaction resource. They’ll now include a details hash which contains title and subtitle values which can be used to contruct human readable transaction entries. These values also respect localization and will be translated based on the Accept-Language header.

Changes to API versioning

We recently added notifications to the Coinbase API. As notifications are delivered via POST requests (webhooks), they can’t be versioned using our existing versioning schema where one needs to pass the date of the implementation as the CB-Version header.

Given that we would have needed to make a breaking change to the notification resource payload, we decided to add a user level API version designation which can be controlled from API settings. For both notification and API requests where the user hasn’t defined their API version within the header, this default version will be used. For requests, using the CB-Version header is still the recommended way to define your version, as also used in our client libraries.

New endpoint: historic bitcoin prices

We added an easy way to get historic bitcoin prices used by Coinbase. The new endpoint doesn’t require authentication and is available at GET api.coinbase.com/v2/prices/historic. You can also get the value using our client libraries.

Upgrading from legacy API v1 to v2

To make it easier for users of our legacy API v1 to upgrade to the newer API v2, we have added new fields to v1 responses. Given that we started using UUID as resource identifier in the new v2, it hasn’t been easy to map existing objects from v1 to v2. To solve this we added a new uuid field to v1 resources, which maps to v2 resource ids.

To see all recent API changes, visit our API changelog.


Written by Rob Witoff

Bug Bounty Transparency Update

Over the past two years Coinbase has benefited greatly from running a public bug bounty program and we believe strongly in incentivizing the white-hat community to responsibly disclose vulnerabilities to us and our partners. In light of a recent public discussion, we’d like to take this moment to transparently share more inner-workings of our program and affirm our commitment to growing our collaboration with the white-hat community in 2016.

Program Performance

In 2013, we chose to run our bug bounty program through HackerOne and review the results of our program every quarter. Since the beginning, we’ve paid out a total of $103,801 in valid bounties. 9% of our submissions were resolved by working with a researcher while the remaining 91% were closed in one of the below categories:

Bug Bounty Stats

Looking only over the last three months, we have:

  • paid out $10,300 over 19 bounties for an average of $542 per bounty
  • resolved 21 additional bugs submitted through the program
  • averaged 23 hours to respond to every new submission
  • averaged 18 days from submission to payout for valid bounties

One of the challenges with a white-hat program is effectively managing the majority of submissions that do not result in a paid bounty. One recent public example highlighted some aspects of our security program that we’d like to reflect on here:

Submission #606921: publicity pending: Balance Manipulation:

Reported on May 31st, a researcher identified a race condition in moving funds between multiple wallets within a Coinbase account. This allowed a user to send funds offsite, leaving one wallet with a negative balance. When reviewing this case internally, we confirmed that other security controls would have protected us from losing funds (you could only send funds that you did indeed own) and initially contemplated the severity of the finding. Despite employing defense-in-depth, this was a valid vulnerability in one of our layers and we rewarded the researcher with a $5,000 bounty within 24 hours of the original HackerOne submission.

Submission #949251: publicity pending: Balance Manipulation v2:

Reported on October 20th and similar to #60692 this report suggests that a similar race condition is possible while moving funds out of our vault. This purported exploit posed several challenges for our security team:

  1. Though the attack was described, neither our security or engineering teams were able to reproduce or validate this exploit. Further, we saw no indication that the researcher had actually been able to exercise the vulnerability. Lack of information is common in all white-hat programs and we regularly work with researchers to provide clear test cases and that’s what happened here.
  2. The researcher then explained that they only “half way completed the bug” but could not finish the exploit due to a lack of funds. Our Security Team attempted to provide the user additional funds to execute the purported exploit but could not because of separate restrictions on this user’s account applied by our independent Compliance Team for unrelated reasons.
  3. The reasons that our Compliance Team may place restrictions on a user’s account can be found in our User Agreement. Further, Coinbase has never, and will never ban a user for any responsible white-hat security testing of our public endpoints. In fact, we have paid over $100,000 to reward white-hat researchers and are planning to grow this program over time. Banning a researcher for security submissions would be bad for both security and business and is counter to our mission.

Despite these challenges, we recognize that our response should have better communicated our understanding of the issue in several ways:

  1. From public discussions, it may have appeared that the restrictions and security activity may have been connected. In reality, these represented separate communications with separate employees on separate teams at separate points in time.
  2. We failed to clearly communicate and close this ticket out in a timely manner. From the first submission on October 20th to the final response on December 1st, 42 days passed including several without answering following requests for comment by the researcher. This was neither professional nor courteous to the researcher and we should have more promptly dispositioned the submission.

1N.B. Publicity Pending: A good bug bounty program should be prepared for disagreements and HackerOne supports this with a mutual disclosure feature. We have requested that both of the above submissions be made fully public which will occur when the researcher agrees or after a 30 day waiting period that ends on 1/20 2016:

Bug Bounty Disclosure

Looking Ahead

Just as Bitcoin is being bootstrapped by an elegant incentive structure, well designed bug bounty programs can optimize incentives between public & private security engineers. Companies pay for the vulnerabilities or better practices that are reported (instead of abused) and responsible researchers can work without fear of retribution while they explore endpoints and even make a healthy profit. We believe that any organization that truly cares about security is clearly incentivized to run a healthy bug bounty program.

After reflecting on the performance of our bug bounty program in 2015, we’re now working on optimizing our program terms, interactions with team and the increasing the engagement with participating researchers. In 2016 we will be expanding the scope of our program and publicizing at least 30% of our valid submissions. We believe strongly in the value of our program and encourage the rest of technology to similarly engage the security community.

For additional discussion:


Written by Jori Lallo

Notifications

Writing software against the blockchain and financial systems can be challenging, because actions are often performed asynchronously. Anyone who has implemented functionality for incoming bitcoin transactions is more than familiar with this. Polling APIs is easy to implement but hard to scale.

Today we’re thrilled to announce a new addition to Coinbase API family: Notifications. Notifications are alerts sent to your application using webhooks (POST request with JSON data) when corresponding actions are performed or observed by Coinbase. This makes it very easy to react to actions related to your application. We’re starting with a handful of important and heavily requested actions and will add more over time. Starting today you can subscribe to events such as these:

  • New transactions for bitcoin addresses created for your Coinbase wallets
  • Completion of your bitcoin buy or sell
  • Receipt of a successful payment for your merchant checkout order

API Notifications are built in similar way to our old callback system, which was mainly designed for merchant services. Our new notifications can be set up for both API keys and Coinbase Connect (OAuth2) applications. This makes it easy to subscribe to both your own and your application’s users’ notifications.

You can set the notification endpoint in your API settings individually for API keys and applications. Once you have subscribed to events and set the appropriate permissions to receive respective notifications, we’ll start sending them to your application. Each notification contains a unique ID, type, and the corresponding data payload:

{
  "id": "6bf0ca21-0b2f-5e8a-b95e-7bd7eaccc338",
  "type": "wallet:buys:completed",
  "data": {
    "id": "67e0eaec-07d7-54c4-a72c-2e92826897df",
    "status": "completed",
    "payment_method": {
      "id": "83562370-3e5c-51db-87da-752af5ab9559",
      "resource": "payment_method",
      "resource_path": "/v2/payment-methods/83562370-3e5c-51db-87da-752af5ab9559"
    },
    "transaction": {
      "id": "441b9494-b3f0-5b98-b9b0-4d82c21c252a",
      "resource": "transaction",
      "resource_path": "/v2/accounts/2bbf394c-193b-5b2a-9155-3b4732659ede/transactions/441b9494-b3f0-5b98-b9b0-4d82c21c252a"
    },
    "amount": {
      "amount": "1.00000000",
      "currency": "BTC"
    },
    "total": {
      "amount": "10.25",
      "currency": "USD"
    },
    "subtotal": {
      "amount": "10.10",
      "currency": "USD"
    },
    "created_at": "2015-01-31T20:49:02Z",
    "updated_at": "2015-02-11T16:54:02-08:00",
    "resource": "buy",
    "resource_path": "/v2/accounts/2bbf394c-193b-5b2a-9155-3b4732659ede/buys/67e0eaec-07d7-54c4-a72c-2e92826897df",
    "committed": true,
    "instant": false,
    "fees": [
      {
        "type": "coinbase",
        "amount": {
          "amount": "0.00",
          "currency": "USD"
        }
      },
      {
        "type": "bank",
        "amount": {
          "amount": "0.15",
          "currency": "USD"
        }
      }
    ],
    "payout_at": "2015-02-18T16:54:00-08:00"
  },
  "user": {
    "id": "f01c821e-bb35-555f-a4da-548672963119",
    "resource": "user",
    "resource_path": "/v2/users/f01c821e-bb35-555f-a4da-548672963119"
  },
  "account": {
    "id": "8d5f086c-d7d5-58ee-890e-c09b3d8d4434",
    "resource": "account",
    "resource_path": "/v2/accounts/8d5f086c-d7d5-58ee-890e-c09b3d8d4434"
  },
  "delivery_attempts": 0,
  "created_at": "2015-11-10T19:15:06Z",
  "resource": "notification",
  "resource_path": "/v2/notifications/6bf0ca21-0b2f-5e8a-b95e-7bd7eaccc338"
}

We’re also now including user and account information for apps that are using Notifications for their Coinbase Connect applications, which enables you to differentiate notifications by unique user. If you’d like, you can access all your notifications via the API endpoint for notifications.

If you’re using merchant checkout products, you don’t necessarily need to set up API keys for simple payment integrations. Instead, you can set the notification endpoint in your merchant settings. Read more about merchant notifications.

As with all our APIs, we’re constantly listening to developer feedback and improving our offering. If there are Notification events you’d like for us to consider building, we would love to hear from you. Please send us feedback in either our Community Forum or email us at api@coinbase.com.


Written by Maksim Stepanenko

Coinbase Sandbox improvements

When building financial applications, it’s extremely important to test in a risk-free environment. That’s why why we encourage all the developers using our API to always start on Coinbase Sandbox before moving to the production API. We launched Coinbase Sandbox in the beginning of this year to help developers safely test their applications, and over the past couple of weeks we’ve made a number of improvements to it:

Test Bank Accounts

All sandbox users by default now get a test bank account, which enables a few new features:

  • Fake USD Deposits and Withdrawals
  • Fake Buys and Sells

Buy Dialog

Other Improvements

  • Testnet blockchain stability has been improved
  • The 0.1 signup bonus is now sent off-blockchain, so you can get started instantly
  • API key creation flow has been simplified

All of the above improvements are based on the feedback we’ve been getting from you, so please let us know if you have any other thoughts, suggestions or questions about the sandbox. You can find more details about Coinbase Sandbox here.


Written by Maksim Stepanenko

Coinbase Node.js client updates

We’ve updated our official Coinbase Node.js library to support the new API v2. We tried to keep the structure of the codebase very similar to the previous versions to make it as easy as possible for you to migrate to API v2. This new version is now available on npm:

npm install coinbase

To get you started, here’s an example code snippet to buy bitcoin using this library:

var coinbase = require('coinbase');
var client   = new coinbase.Client({'apiKey': 'API KEY',
                                    'apiSecret': 'API SECRET'});

// Create new BTC wallet/account:
client.createAccount({'name': 'test'}, function(err, account) {
  console.log('Account: ' + account.name);
  account.buy({'amount': 0.01, 'currency': 'BTC'}, function(err, buy) {
    console.log('Buy: ' + buy.amount.amount + ' ' + buy.amount.currency);
  });
});

You can find more information about the library here. If you have any questions or feedback regarding this update, let us know on Twitter, Coinbase Community, or via email.


Written by Jori Lallo

Improved Coinbase Connect (OAuth2)

Coinbase Connect

We’re excited to release an improved version of Coinbase Connect, the OAuth2 implementation that allows developers to connect to Coinbase wallets.

Coinbase Connect’s authorization view (aka permissions dialogue), shown to users when they are authorizing applications, has been redesigned from the ground up. Permissions are now organized into logical groups. Permissions such as send bitcoin, show user’s email and others that afford developers nontrivial responsibility, are clearly and individually displayed to the user, while permissions that deliver significantly less sensitive information are bundled together.

We’re also introducing the concept of “application controlled wallets.” Previously, an application would by default request and gain access to all of user’s wallets. Now the authorization dialogue lets the user specify one wallet to link with the app, or to create a new associated wallet. Apps may still request to access to all of user’s wallets (e.g. for tax reporting). Account access is easily controlled with a new account parameter in the authorization URL. You can find more info in the OAuth reference documentation.

After introducing send limits earlier this year, we have now also included the option for the users to change their send limits when authorizing an application.

What does this mean for existing applications?

All existing applications will continue to work without interruption. This includes retaining access to all accounts by default. We encourage developers to update their applications to only access one wallet, by directing new users to enter the new authorization flow. This can be triggered for existing users by using the account=select parameter within the OAuth authorization URL.

Finally, since we have added more application information to the Coinbase Connect auth dialogue, we also recommended that developers complete and/or update their developer information and logos in their OAuth application settings.

If you have any questions or feedback regarding Coinbase Connect, please head to Coinbase Community and we’ll do our best to help you out.