Importing message data into BigQuery
To import messages into BigQuery in batch mode, you can use the gcloud utility to log all the messages retrieved from the Cloud Pub/Sub to a file. Then you can upload the file to Google Cloud Storage and import data from the file to the BigQuery table. The following section provides a walkthrough of this scenario. This section uses the same Pub/Sub topic used in the previous section and uses the same Pub/Sub subscriber for pulling messages. Publish some sample messages to the Pub/Sub topic before running the commands given in this section.
Schedule a job to call the following command at specified intervals to retrieve the message from Cloud Pub/Sub, and append it to a local file. Replace the subscriber, message attributes, and target filename as per your project. The following command writes the messages to the messages.csv file:
gcloud beta pubsub subscriptions pull pubSubTopicSubscriber --format="csv[no-heading](message.attributes.EmployeeID,message.attributes...
