Google Cloud Storage Deep Dive

Google Cloud Storage is a fully managed, scalable object storage service in GCP used to store unstructured data such as files, images, logs, backups, parquet files, CSVs, videos, and ML datasets. It can be regional or dual-region or multi-region as per the requirement.

It is commonly used for:

• data lake storage
• raw ingestion layer
• backups
• staging for BigQuery/Dataflow

Difference between Object, Block, and File Storage

Examples in GCP:

• Object → Cloud Storage
• Block → Persistent Disk
• File → Filestore

So cloud storage is called object storage.

High-Level Architecture -

Cloud storage architecture includes the following:

Buckets → A bucket is a logical container for storing objects. Top-level storage container

• Bucket definition:
• region/location
• storage class
• IAM permissions
• lifecycle rules
• versioning

Folder → Logical naming convention

Objects → data is stored as objects. An object is an individual stored file.

Each object contains:

Ø actual file/data

Ø metadata

Ø unique identifier

Ø Examples: CSV, JSON, Parquet, Images, Logs

Ø Objects are immutable, i.e., once an object is created, it cannot be modified directly. So you cannot modify a file directly. Instead: you can upload new version or overwrite an object.

Distributed backend → automatic scaling

Replication → durability & availability

Metadata management

Key benefits:

• highly durable (11 9’s durability)
• auto-scalable
• serverless

Storage Classes:

There are 4 classes in cloud storage.

1. Standard

For:

• frequently accessed data

Examples:

• active analytics
• streaming data
• hot datasets

Low latency, frequent access

2. Nearline

For:

• infrequent access (~monthly)

Examples:

• monthly backups

Cheaper storage and retrieval costs exist.

3. Coldline

For:

• quarterly or rare access

Examples:

• disaster recovery backups

Lower storage cost.

4. Archive

For:

• long-term retention

Examples:

• compliance
• legal data

Cheapest storage, highest retrieval cost.

Example:

Raw logs:

• Day 0–30 → Standard
• Day 30–90 → Nearline
• Day 90–365 → Coldline
• After 1 year → delete/archive

This saves huge costs.

GCS is strongly consistent

Meaning:

• upload immediately visible
• delete immediately visible

Durability vs Availability

Cloud storage provides the following:

• 11 9's durability
• very high availability

Durability ≠ availability.

What is lifecycle management?

Lifecycle management is an automated way to manage objects in Cloud Storage based on rules.

You can:

• move files between storage classes
• archive files
• delete old files automatically

It helps:

• reduce cost
• automate retention
• manage huge-scale storage

Example:

Move logs older than 30 days to Nearline.

Real-world lifecycle example

Suppose an e-commerce company receives transaction logs daily.

Lifecycle policy:

0–30 days → Standard (frequent access)

30–90 days → Nearline

90–365 days → Coldline

After 1 year → delete or archive

This reduces storage cost significantly.

How to reduce storage cost?

Strategies:

1. Use appropriate storage class
2. Lifecycle rules
3. Delete unused files
4. Compression (Parquet, Avro)
5. Partition data logically
6. Avoid duplicate storage
7. Retention optimization

Example: Store raw CSV as compressed parquet.

How do you secure cloud storage?

Security layers:

• IAM permissions
• Bucket policies
• Uniform bucket access
• Object-level ACL permissions
• Encryption
• Signed URLs
• Retention policies
• VPC Service Controls

Example: Only data engineering service accounts can write to the productionn bucket.

What is a Signed URL?

A Signed URL provides temporary secure access to private objects without exposing permissions.

Example:

• Share file with vendor
• URL expires after 2 hours

Common use case:

temporary download access

How is data encrypted?

By default: Google encrypts data automatically (at rest).

Encryption options:

1. Google-managed keys (default)
2. Customer-managed keys (CMEK)
3. Customer-supplied keys (CSEK)

Also:

• TLS encryption during transfer

What is object versioning?

Versioning stores multiple versions of objects.

If a file is overwritten:

• old version remains recoverable

Useful for:

• rollback
• accidental deletion recovery

What if someone deletes a file accidentally?

Recovery options:

1. Object versioning enabled → restore old version
2. Retention policies
3. Backup buckets
4. Replication copies

Best practice:

Enable versioning for production-critical buckets.

What is retention policy?

A retention policy prevents deletion/modification for a fixed duration.

Example:

Financial records cannot be deleted for 7 years.

Useful for:

• compliance
• legal requirements

How does Cloud Storage scale?

Cloud Storage is automatically distributed.

It scales through:

• distributed backend
• automatic replication
• parallel uploads/downloads

No manual scaling required.

Key benefit:

virtually unlimited scalability.

What are compose operations?

A compose operation combines multiple small objects into one larger object.

Example: 100 log chunks → merge into one parquet file

Benefits:

• fewer files
• better BigQuery performance
• optimized processing

How to Optimize huge files performance?

Strategies:

• parallel uploads
• compression
• split huge files
• use parquet/avro
• proper naming convention
• multipart upload

For analytics:

Use columnar formats like Parquet.

Usecase:Design for 5 TB logs daily

Architecture:

Folder structure:

bucket/raw/year/month/day/ bucket/processed/year/month/day/

Storage strategy:

0–30 days → Standard

30–90 days → Nearline

90+ days → Coldline

Retention:

• delete after 1 year

Optimization:

• parquet compression
• partition by date

Security:

• IAM + service account access

How to handle Late arriving files

Design approach:

1. Watermarking
2. Retry logic
3. Late-arrival window
4. Reprocessing pipeline

Example: Allow 2-day delayed ingestion.

Also: Maintain ingestion timestamp.

How would you design a secure data lake in GCS?

Layers:

raw/ trusted/ curated/

Security:

• separate buckets
• IAM role separation
• encryption
• versioning
• retention policy
• audit logging

Example: Analysts only access curated layer.

You accidentally deleted production files. What recovery options exist?

Recovery options:

1. Versioning restore
2. Backup bucket
3. Cross-region replication
4. Retention lock

Production best practice:

Versioning + retention enabled.

Production issues in Cloud Storage

Examples:

1. Late arriving files
2. Duplicate ingestion
3. Permission failures
4. Small file problem
5. Incorrect lifecycle deletion
6. Cross-region latency
7. Bucket misconfiguration

How does Cloud Storage integrate with BigQuery?

Cloud Storage commonly used as staging.

Flow: Source → GCS → BigQuery Load Job

Supports:

• CSV
• JSON
• Avro
• Parquet
• ORC

Best practice:

Store parquet for faster analytics

How would you move petabyte-scale data into GCP?

Options:

1. Online transfer

• Storage Transfer Service

1. Offline transfer

• Transfer Appliance

1. Streaming pipelines

• Pub/Sub + Dataflow

For PB-scale:

Use the transfer appliance due to network limitations.