Skip to main content

Use Cloud Run Functions to Load BigQuery

Use Cloud Run Functions to Load BigQuery

 

Main article: https://www.skills.google/paths/1336/course_templates/1157/labs/603516



Enable these APIs:

  • Cloud Functions API

  • Cloud Run Admin API

  • Eventarc API

  • Cloud Build API

  • Artifact Registry API

  • Cloud Pub/Sub API

  • Cloud Logging API



Set IAM permissions (UI)

  1. Go to
    IAM & Admin → IAM

  2. Find the service account:

PROJECT_NUMBER-compute@developer.gserviceaccount.com

  1. Add role:

-Eventarc Event Receiver

-Pub/Sub Publisher


Create the BigQuery dataset (UI)

  1. Open Google BigQuery

  2. Click Create Dataset

Fill:

Dataset ID: loadavroCreate.


Create the Cloud Function (UI)

 → Cloud Functions -> NodeJS 
Name: loadbigqueryfromavro
Runtime: Node.js 20

Trigger type ->‌ Cloud Storage

Event type -> Create

google.cloud.storage.object.v1.finalized

the code:

const functions = require('@google-cloud/functions-framework');
const { Storage } = require('@google-cloud/storage');
const { BigQuery } = require('@google-cloud/bigquery');

const storage = new Storage();
const bigquery = new BigQuery();

functions.cloudEvent('loadBigQueryFromAvro', async (cloudEvent) => {
try {
console.log('FULL EVENT:', JSON.stringify(cloudEvent, null, 2));

const data = cloudEvent.data;

if (!data || !data.bucket || !data.name) {
throw new Error('Invalid event data');
}

const bucketName = data.bucket;
const fileName = data.name;

const datasetId = 'loadavro';
const tableId = fileName.replace('.avro', '');

const options = {
sourceFormat: 'AVRO',
autodetect: true,
createDisposition: 'CREATE_IF_NEEDED',
writeDisposition: 'WRITE_TRUNCATE',
};

const [job] = await bigquery
.dataset(datasetId)
.table(tableId)
.load(storage.bucket(bucketName).file(fileName), options);

console.log(`Job ${job.id} completed. Created table ${tableId}`);
} catch (error) {
console.error('Error:', error);
throw error;
}
});

and package.json:

{
"name": "load-bigquery-from-avro",
"version": "1.0.0",
"main": "index.js",
"scripts": {
"start": "functions-framework --target=loadBigQueryFromAvro --signature-type=cloudevent"
},
"dependencies": {
"@google-cloud/functions-framework": "^3.0.0",
"@google-cloud/storage": "^7.0.0",
"@google-cloud/bigquery": "^7.0.0"
}
}



Go to bucket -> settings -> copy the service account -> 

-> Go to IAM -> give role: roles/pubsub.publisher 


Download https://storage.googleapis.com/cloud-training/dataengineering/lab_assets/idegc/campaigns.avro


and then upload it into the bucket we created.



Comments

Post a Comment

Popular posts from this blog

Generate Realistic Cat Driving and crashing Car with AI

AI tool: https://vivago.ai/video-generation   Two photos needed for start and end: Or you can generate your own image with prompt: use this reference image: Prompts: - add a cat to the car image holding the wheel and standing there like the cat driving the car . - now change the angle of camera to be at back of the cat and showing the road through the front window - now add a truck in front of the car in the road comming towards the car and is about to crash the car And the prompt for Video Generation in https://vivago.ai:  Vertical 9:16 aspect ratio, low-quality found-footage smartphone video frame, filmed from the passenger seat of a moving car. Handheld feel with slight camera shake, imperfect off-center framing, mild motion blur from vehicle movement. A chubby gray tabby cat sits upright in the driver’s seat, one paw resting naturally on the steering wheel as if driving. The cat appears calm, focused, and serious, eyes fixed forward on the road, creatin...
Post a Comment
Read more

How simply use Manim to create animated charts and graphs

Links: https://3b1b.github.io/manim/getting_started/example_scenes.html Install: brew install ffmpeg pip install manim Create a new project : manim init project my-project --default or you can just start writing the code in a Python file. then you should have this default code in main.py from manim import * class DefaultTemplate ( Scene ): def construct ( self ): circle = Circle () # create a circle circle . set_fill ( PINK , opacity = 0.5 ) # set color and transparency square = Square () # create a square square . flip ( RIGHT ) # flip horizontally square . rotate ( - 3 * TAU / 8 ) # rotate a certain amount self . play ( Create ( square )) # animate the creation of the square self . play ( Transform ( square , circle )) # interpolate the square into the circle self . play ( FadeOut ( square )) # fade out animation if don't have it just create main.py and put that code in it.  now run this comm...
Post a Comment
Read more