Use Cloud Run Functions to Load BigQuery Main article: https://www.skills.google/paths/1336/course_templates/1157/labs/603516 Enable these APIs: Cloud Functions API Cloud Run Admin API Eventarc API Cloud Build API Artifact Registry API Cloud Pub/Sub API Cloud Logging API Set IAM permissions (UI) Go to IAM & Admin → IAM Find the service account: PROJECT_NUMBER-compute@developer.gserviceaccount.com Add role: -Eventarc Event Receiver - Pub/Sub Publisher Create the BigQuery dataset (UI) Open Google BigQuery Click Create Dataset Fill: Dataset ID: loadavroCreate. Create the Cloud Function (UI) → Cloud Functions -> NodeJS Name: loadbigqueryfromavro Runtime: Node.js 20 Trigger type -> Cloud Storage Event type -> Create google.cloud.storage.object.v1.finalized the code: const functions = require ( '@google-cloud/functions-framework' ); const { Storage } = require ( '@google-cloud/storage' ); const { BigQ...
The actual example link: https://www.skills.google/paths/1336/course_templates/1157/labs/603507 1- Go to BigQuery -> create new dataset name: taxirides -> create a table name: realtime -> In Schema section, click on edit as text -> use this schema: ride_id :string, point_idx :integer, latitude : float , longitude : float , timestamp : timestamp , meter_reading : float , meter_increment : float , ride_status :string, passenger_count :integer -> Then create. 2-Download and upload the data into bucket Source of data: https://opendata.cityofnewyork.us/ Download the three files from following link: https://drive.google.com/drive/folders/1TKLesdtIHNb197ouyXOyXXwVJUpD-J6E?usp=sharing -> Go to buckets -> Create a new bucket called: taxirides-data -> then upload those three files you downloaded there. 3- Go to DataFlow -> Create Job From Template -> Enable to Use -> name for the flow: streaming-taxi-pipeline -> templ...