Global vs. Command-Specific Flags in BigQuery CLI

When using the bq command-line tool with Google BigQuery, understanding the difference between global flags and command-specific flags is important for building powerful, readable, and flexible CLI commands.

This post helps you to understand these two categories, their syntax, use cases, and gives real-world examples.

 

1. Introduction

The bq command-line tool is a fast and scriptable interface for working with Google BigQuery. To unlock its full power, it's important to understand how to use flags (special options) that modify the behavior of your commands.

 

There are two types of flags in bq:

·      Global Flags: Can be used with any command.

·      Command-Specific Flags: Work only with certain commands.

 

2. General Syntax

bq \
  --global_flag1=value1 \
  --global_flag2=value2 \
  command \
  --command_specific_flag1=value3 \
  --command_specific_flag2=value4 \
  [PROJECT_ID].[DATASET].[TABLE]

This is the BigQuery CLI command. You use it to run queries, manage datasets, load/export data, etc.

 

--global_flag1=value1 (and other global flags)

These are global configuration flags that apply to any bq command. Common examples:

 

·      --project_id=your-project-id: Overrides the default project.

·      --location=US: Specifies the BigQuery location.

·      --format=json|prettyjson|csv|pretty: Output format for results.

·      --quiet: Suppresses prompts.

 

command

This is the specific action you want to perform using the bq tool.

 

Examples:

·      query: Run a SQL query.

·      load: Load data into a table.

·      extract: Export data from a table to Cloud Storage.

·      mk: Create resources (dataset/table/view).

·      rm: Remove resources.

·      show: Display details of a dataset/table.

 

--command_specific_flag1=value3 (and others)

These flags are tied to particular commands only and won’t work globally.

 

Example for query command:

bq query \
  --use_legacy_sql=false \
  --nouse_cache \
  --format=pretty \
  'SELECT * FROM `my-project.my_dataset.my_table` LIMIT 10'

Other examples:

·      For load: --source_format=CSV, --skip_leading_rows=1

·      For extract: --destination_format=CSV, --compression=GZIP

 

You can combine global and command-specific flags to build powerful CLI commands.

bq --format=prettyjson show --schema test_dataset.employees

Above command print the schema of table employees in pretty print format.

$bq --format=prettyjson show --schema test_dataset.employees
[
  {
    "mode": "NULLABLE",
    "name": "EmployeeID",
    "type": "INTEGER"
  },
  {
    "mode": "NULLABLE",
    "name": "FirstName",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "LastName",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "Department",
    "type": "STRING"
  },
  {
    "mode": "NULLABLE",
    "name": "Salary",
    "type": "INTEGER"
  }
]

Real-World Use Case

Imagine you're automating schema validation for incoming data. You could write a shell script like:

#!/bin/bash
TABLE_NAME=$1
bq --format=json show --schema my_dataset.${TABLE_NAME} > schema_${TABLE_NAME}.json

 

This will save the schema of any table passed as a parameter into a JSON file, ready for validation.

 

In summary, understanding the distinction between global and command-specific flags is the key to efficient scripting and automation in BigQuery. Whether you’re exploring schemas, exporting data, or managing datasets, the right combination of flags can save you time and make your work reproducible.

 

  

Previous                                                    Next                                                    Home