An Object Storage Migration Story

Picture this: It’s a crisp Thursday morning, your inbox explodes, and you see an email with a subject line that sends shivers down your spine: “SwiftStack Acquisition by NVIDIA”. You crack it open, and dread sets in. Our trusty data storage vendor, SwiftStack, is being acquired, and support for our entire environment — PBs of data across secure, production, development, and staging environments — is evaporating in 10 short months!

Panic could have ensued. Questions swirled: How do we keep our internal customers happy, with their thousands of projects and accounts hanging in the balance? How do we fulfill our commitment to fueling the company’s growth, especially in the booming world of AI, which thrives on massive data storage solutions like object storage?

But fear not, intrepid reader! Because here’s the good news: we not only navigated this storage storm, but we did it in a way that kept our customers humming and our business growth on track. How did we pull off this data storage miracle? Buckle up, because you’re about to find out!

Disclaimer: This article was written based on publicly available information, including multiple presentations about SwiftStack use at eBay I presented at public conferences. For more details, please refer to the reference section.

How did we get in this situation in the first place?

Imagine eBay in its early days. Millions of images, a treasure trove of data, all needing a home. But traditional storage solutions were like filing cabinets in a hurricane — slow, clunky, and easily overwhelmed. Here are the Blob and Object Storage solutions in chronological order:

• 2007 — EPS — eBay Picture Service [Blob Storage]
• 2008 — ATMOS Storage (Enabled by EMC) [Object Storage]
• 2012 — zStore [Blob Storage]
• 2014 — Swift (Enabled by SwiftStack, vendor) [Object Storage]

In this context, the main difference between object storage and blob storage is that blob storage typically doesn’t offer the same level of metadata support or custom object key mapping as object storage.

The adoption of Swift Object Storage within eBay grew exponentially. By 2018, it had completely replaced and decommissioned ATMOS, becoming eBay’s sole object storage solution.

By 2020, when NVIDIA acquired SwiftStack, our SwiftStack solution enabled over 1,000 projects, including the eBay Container Registry, payments use cases, eBay Continuous Deployment, and analytics solutions, to store petabytes (PBs) of data. It offered several benefits:

• Users could choose between high-performance object storage or standard performance object storage (Depending on use cases and performance requirements)
• Multi-region disaster recovery of data.
• Support for multiple environments, including secure, production, development, and staging environments.
• Varnish caching enabled by default for select tenants, boosting concurrent download performance by 10x.
• Higher durability, reliability, and availability of data.
• Multi-tenancy and zone support.

Even though Swift is an open-source object storage system within the OpenStack project, SwiftStack was selected as a Vendor since they offered a custom commercial product based on Swift technology. They built a software package around open-source Swift, making it easier for businesses to deploy and manage. This included providing backend support and expertise, while our internal storage team handled specific customizations like Varnish caching integration, running Swift on eBay’s ODM, eBay Keystone integration, and multi-region disaster recovery, and many other business specific requirements and customizations.

Finding the Perfect Storage Replacement Fit

The Clock Was Ticking… We had a strict 10-month deadline (non-negotiable by contract) to find the perfect storage solution, one that wouldn’t disrupt our amazing customers (or at least keep the bumps in the road to a minimum). So, we put on our detective hats and analyzed every option with laser focus:

• Option 1, the DIY Approach: This option lets us keep things familiar for our awesome customers (no data migration headaches!). We’d rebuild the deployment and configuration layer that SwiftStack provides, essentially building our own custom storage garage on top of open-source Swift.
• The Perks: Happy Customers: No disruption to their data flow. Familiar Territory: We already know Swift inside and out.
• The Challenges: Scaling Up is a Big Job: Our data is growing like crazy (think petabytes going to exabytes!), and the current setup needs a major overhaul to handle this kind of volume. It’s like trying to park a fleet of trucks in a single-car garage — we need an upgrade! Kubernetes? Not Yet: We’re moving towards using Kubernetes, a fancy tool for managing all our systems. But Swift, on its own, isn’t ready to play nice with it yet. That means another big project to get them working together.
• Option 2, Hitting the Public Cloud: This option lets us tap into the power of a public cloud storage solution — it’s like renting a pre-built storage unit, ready to move in right away! No time wasted on building or configuring.
• The Perks: We’d be up and running the migration in a flash.
• The Challenges: Cost Crunch: Public cloud providers charge for storage and data transfer (think tolls on the information highway!). This could put a strain on the budget, especially with our ever-growing data needs. Moving Day Mayhem: Migrating all that data would be a big job, like packing up your entire house! Strategy Shift: This option goes against our long-term plan of keeping everything in-house. Think of it as renting, while our goal is to own our own storage space.
• Option 3: Combining the Best of Both Worlds: This option lets us leverage our existing zStore Blob storage solution as the foundation, then build an object storage layer on top — think of it as a storage mashup, combining the best features of each! It aligns perfectly with our long-term strategy of keeping things in-house.
• The Perks: Strategic Powerhouse: Checks all the boxes for our company goals.
• The Challenges: High risk: Building a whole new metadata storage layer and migrating customer data seamlessly within 10 months is a tall order. It’s like building a spaceship and taking it for a joyride — in record time! There’s a high risk factor, but hey, we love a good challenge!

We wrangled all the options onto the table, but picking the perfect one? That was a boss battle! We spent weeks in deep dives, wielding spreadsheets and whiteboards like mighty weapons. Our mission: to find the solution that would keep our amazing customers happy (think zero downtime!) and dodge any potential risks like a seasoned gamer.

The deadline loomed large (no wiggle room!), but after two intense weeks, Option 3 emerged victorious! There was just one catch: we had to tackle the big risks head-on in the coming month.

Ready to see how we conquered those risks? Buckle up, because the adventure continues…

Getting Started

First things first — transparency is key! We kicked things off with a detailed email to our amazing customers. We filled them in on the SwiftStack situation, reassured them we’d found the perfect solution, and explained our plan to keep their workloads humming along. Plus, we scheduled a Tech Talk to dive deeper and answer any questions they might have.

Meanwhile, on the engineering front, it was game time! The next month was all about tackling risks and making Option 3 a reality. This meant building our new Object Storage Service (powered by zStore!) across all environments, and then migrating customer data seamlessly.

Security first, always! Since this service would handle critical payment workloads, we followed our security team’s rigorous protocols to ensure everything was locked down tight. No data breaches on our watch!

Speaking of access, the S3 API is king! As the leading standard for object storage, it made perfect sense to make it the main API for our new service. But what about existing customers using Swift API tools? No worries — we decided to build a translation layer to bridge the gap and keep everything running smoothly.

One month later, we’re risk-free and ready to roll! The engineering team is fully on board and committed to delivering Option 3 within the deadline. Buckle up, because the next chapter is all about the execution — stay tuned!

Execution

Communication was our secret weapon! We opened a Slack channel for the migration and held regular office hours to address concerns and brainstorm solutions for those tricky edge cases. Remember that time we had to get creative because the new service limited buckets to 1,000 per account, unlike Swift’s anything-goes approach? We tackled that and more!

We kept a watchful eye on the project, identifying and mitigating potential delays or snags before they could take flight. Completing the security review was a major victory, eliminating a huge risk factor and keeping us on schedule.

We set launch dates (one per environment) for both Early Access and General Availability of the new Object Storage service, and let our customers know. We also handpicked some “White Glove” customers to be the first to experience the Early Access phase.

The migration marathon, a phase-by-phase adventure:

• Phase 1: Data was copied account-by-account from Swift to the new service, with both systems handling writes (although reads still came from Swift).
• Phase 2: Customers verified their data made it safely.
• Phase 3: The new service became the primary read source.
• Phase 4: Writes to Swift were stopped for the migrated accounts.
• Phase 5: Baking time, a final testing and stabilization period.
• Phase 6: Accounts were decommissioned from Swift in the migrated environment.

Part of our deliverables, we built a customer portal where they could log in, track their migration progress, and choose their switch-over date.

Three months before the deadline, Early Access launched for key environments with select customers. This phase was critical for identifying and polishing any last-minute issues and edge cases.

Regular email updates and Slack announcements kept customers in the loop. We also announced the upcoming halt of new Swift onboarding, which coincided with the General Availability launch of the new service in each environment.

D-Day arrives, mission accomplished! Two weeks before the deadline, we hit a major milestone: the new Object Storage service was generally available across all environments! Not only that, but customer workloads were automatically migrated, and we stopped onboarding new users to Swift. Talk about an achievement!

Reflections

This project was anything but your average storage upgrade. Imagine: a ticking clock, petabytes of data on the line, and us on a quest for the perfect SwiftStack replacement! It took some serious innovation, collaboration, and maybe a sprinkle of risk-taking, but we did it!

Huge kudos to the entire team for making this happen. I’m incredibly proud to have been a part of this project and this amazing group. Here’s to tackling even more challenging situations in the future — bring ’em on!

REFERENCE

• OpenStack Conference: How Swift is reshaping the future of storage at eBay: https://www.openstack.org/videos/vancouver-2018/how-swift-is-reshaping-the-future-of-storage-at-ebay
• Seller Insight / Seller Hub: A data store that supports interactive complex aggregations on hundreds of billions of listings http://www.ebaytechblog.com/2017/01/12/portico/
• New & Improved Terapeak Research 2.0 in eBay Seller Hub https://innovation.ebayinc.com/tech/engineering/new-improved-terapeak-research-2-0-in-ebay-seller-hub/
• eBay Object Storage at Scale — SNIA Developer Conference https://www.snia.org/sites/default/files/SDC/2019/presentations/General_Session/Bouzeid_Gilbert_eBay_Object_Storage_at_Scale.pdf
• Nvidia acquires data storage and management platform SwiftStack https://techcrunch.com/2020/03/05/nvidia-acquires-data-storage-and-management-platform-swiftstack/
• OpenStack Swift https://www.oreilly.com/library/view/openstack-swift/9781491903841/
• eBay’s Hyperscale Platforms https://innovation.ebayinc.com/tech/engineering/odm/