Cloud Architecting with GCP: Part6 — Cloud Storage

When architecting applications in Google Cloud Platform (GCP), selecting the right storage solution is crucial for performance, scalability, durability, and cost-efficiency. GCP offers multiple storage products tailored to different use cases. This article explores and compares three primary options: Cloud Storage, Persistent Disks, and Filestore.

Persistent Disks — Block Storage for Compute Engine

Persistent Disks (PD) are durable block storage volumes for Compute Engine VMs. They behave like physical disks attached to virtual machines.

💡 Key Features:

• Automatically replicated for reliability
• Resizable, snapshot-able
• SSD or HDD performance options
• Can be zonal or regional for HA
• Supports custom images and encryption

🛠️ Use Cases:

• Boot or data disks for Compute Engine
• Database storage (MySQL, PostgreSQL, etc.)
• High IOPS workloads (use SSDs)

💾 Disk Types Comparison:

Filestore — Managed NFS File Storage

Filestore is a managed network attached storage (NAS) service for applications that require a file system interface and a shared file system for data. Filestore gives users a simple native experience for standing up managed network attached storage with either Compute Engine or Google Kubernetes Engine instances. The ability to fine tune Filestore’s performance and capacity independently, leads to predictably fast performance for your file-based workloads. Filestore offers native compatibility with existing enterprise applications and supports any NFSV3 compatible clients. Applications gain the benefit of features such as scale-out performance, hundreds of terabytes of capacity, and file locking without the need to install or maintain any specialized plug-ins or client-side software.

Cloud Storage — Object Storage for Unstructured Data

Object storage is a computer data storage architecture that manages data as “objects” and not as a file and folder hierarchy (file storage), or as chunks of a disk (block storage). These objects are stored in a packaged format which contains the binary form of the actual data itself, as well as relevant associated meta-data and a globally unique identifier.

Cloud Storage is Google’s object storage product. Cloud Storage files are organized into buckets. A bucket needs a globally unique name and cannot be nested and a specific geographic location for where it should be stored, and an ideal location for a bucket is where latency is minimized.

Cloud Storage is not a file system. Instead, Cloud Storage is a collection of buckets that you place objects into. You can create directories, so to speak, but really a directory is just another object that points to different objects in the bucket.

Versioning & Soft Delete

The storage objects offered by Cloud Storage are immutable, which means that you do not edit them, but instead a new version is created with every change made. Administrators have the option to either allow each new version to completely overwrite the older one, or to keep track of each change made to a particular object by enabling “versioning” within a bucket. If you choose to use versioning, Cloud Storage will keep a detailed history of modifications — that is, overwrites or deletes — of all objects contained in that bucket. With object versioning enabled, you can list the archived versions of an object, restore an object to an older state, or permanently delete a version of an object, as needed. Google recommends that you use Soft Delete instead of Object Versioning to protect against permanent data loss from accidental or malicious deletions. Soft Delete provides default bucket-level protection for your data from accidental or malicious deletion by preserving all recently deleted objects for a specified period of time. When you create a Cloud Storage bucket, the Soft Delete feature is enabled by default with a retention duration of seven days, you can increase the retention duration to 90 days.

Object Lifecycle Management

To support common use cases like setting a Time to live for objects, archiving older versions of objects, or “downgrading” storage classes of objects to help manage costs, Cloud Storage offers Object Lifecycle Management. You can assign a lifecycle management configuration to a bucket. The configuration is a set of rules that apply to all the objects in the bucket.

Object Retention Lock

The Object Retention Lock feature lets you set retention configuration on objects within Cloud Storage buckets that have enabled the feature. A retention configuration governs how long the object must be retained and has the option to permanently prevent the retention time from being reduced or removed.

Access Control

Using IAM roles and, where needed, access control lists (ACLs), organizations can conform to security best practices. There are a couple of options to control user access to objects and buckets. For most purposes, IAM is sufficient. Roles are inherited from project to bucket to object. If you need finer control, you can create access control lists. Each access control list consists of two pieces of information. The first is a scope, which defines who can access and perform an action. This can be a specific user or group of users. The second is a permission, which defines what actions can be performed, like read or write. For even more detailed control, signed URLs provide a cryptographic key that gives time-limited access to a bucket or object. Finally, a signed policy document further refines the control by determining what kind of file can be uploaded by someone with a signed URL.

Cloud Storage always encrypts data on the server side, before it’s written to disk, at no additional charge.

Storage Classes

Cloud Storage has four storage classes: Standard, Nearline, Coldline and Archive, and each of those storage classes provide 3 location types: There’s a multi-region is a large geographic area, such as the United States, that contains two or more geographic places. Dual-region is a specific pair of regions, such as Finland and the Netherlands. A region is a specific geographic place, such as London. Objects stored in a multi-region or dual-region are geo-redundant. All of these storage classes have 11 nines of durability.

When you upload an object to a bucket, the object is assigned the bucket’s storage class, unless you specify a storage class for the object. You can change the default storage class of a bucket but you can’t change the location type from regional to multi-region/dual-region or vice versa. You can also change the storage class of an object that already exists in your bucket without moving the object to a different bucket or changing the URL to the object.

• Standard: best for frequently accessed data.
• Nearline: best for storing infrequently accessed data, like reading or modifying data on average once a month or less.
• Coldline: a low-cost option for storing infrequently accessed data. Is meant for reading or modifying data, at most, once every 90 days.
• Archive: the lowest-cost option, used ideally for data archiving, online backup, and disaster recovery. It’s the best choice for data that you plan to access less than once a year, because it has higher costs for data access and operations and a 365-day minimum storage duration.

Autoclass

Automatically transitions objects to appropriate storage classes based on each object’s access pattern. The feature moves data that is not accessed to colder storage classes to reduce storage cost and moves data that is accessed to Standard storage to optimize future accesses. When enabled on a bucket, there are no early deletion charges, no retrieval charges, and no charges for storage class transitions.

Large Data Migration to Cloud Storage

• Storage Transfer Service: lets you schedule and manage batch transfers to Cloud Storage from another cloud provider, from a different Cloud Storage region, or from an HTTP(S) endpoint.
• Transfer Appliance: a rackable, high-capacity storage server that you lease from Google Cloud. You connect it to your network, load it with data, and then ship it to an upload facility where the data is uploaded to Cloud Storage. You can transfer up to a petabyte of data on a single appliance.
• Offline Media Import: Offline Media Import is a third party service where physical media (such as storage arrays, hard disk drives, tapes, and USB flash drives) is sent to a provider who uploads the data.

Conclusion

Choosing the right storage option in GCP depends on how your application interacts with data:

• Use Cloud Storage for storing blobs of data (media files, backups).
• Use Persistent Disks for compute workloads (VMs, databases).
• Use Filestore when you need POSIX-compliant shared storage.

Understanding the nuances of each storage option helps you design systems that are cost-effective, performant, and scalable.