Skip to main content

Posts

Use Cloud Run Functions to Load BigQuery

Use Cloud Run Functions to Load BigQuery   Main article: https://www.skills.google/paths/1336/course_templates/1157/labs/603516 Enable these APIs: Cloud Functions API Cloud Run Admin API Eventarc API Cloud Build API Artifact Registry API Cloud Pub/Sub API Cloud Logging API Set IAM permissions (UI) Go to IAM & Admin → IAM Find the service account: PROJECT_NUMBER-compute@developer.gserviceaccount.com Add role: -Eventarc Event Receiver - Pub/Sub Publisher Create the BigQuery dataset (UI) Open Google BigQuery Click Create Dataset Fill: Dataset ID: loadavroCreate. Create the Cloud Function (UI) → Cloud Functions -> NodeJS Name: loadbigqueryfromavro Runtime: Node.js 20 Trigger type ->‌ Cloud Storage Event type -> Create google.cloud.storage.object.v1.finalized the code: const functions = require ( '@google-cloud/functions-framework' ); const { Storage } = require ( '@google-cloud/storage' ); const { BigQ...
Recent posts
  The actual example link: https://www.skills.google/paths/1336/course_templates/1157/labs/603507 1- Go to BigQuery -> create new dataset name:  taxirides   -> create a table name: realtime -> In Schema section, click on edit as text -> use this schema: ride_id :string, point_idx :integer, latitude : float , longitude : float , timestamp : timestamp , meter_reading : float , meter_increment : float , ride_status :string, passenger_count :integer -> Then create. 2-Download and upload the data into bucket Source of data: https://opendata.cityofnewyork.us/ Download the three files from following link: https://drive.google.com/drive/folders/1TKLesdtIHNb197ouyXOyXXwVJUpD-J6E?usp=sharing -> Go to buckets -> Create a new bucket called:  taxirides-data -> then upload those three files you downloaded there. 3- Go to DataFlow -> Create Job From Template -> Enable to Use -> name for the flow:  streaming-taxi-pipeline -> templ...

Generate Realistic Cat Driving and crashing Car with AI

AI tool: https://vivago.ai/video-generation   Two photos needed for start and end: Or you can generate your own image with prompt: use this reference image: Prompts: - add a cat to the car image holding the wheel and standing there like the cat driving the car . - now change the angle of camera to be at back of the cat and showing the road through the front window - now add a truck in front of the car in the road comming towards the car and is about to crash the car And the prompt for Video Generation in https://vivago.ai:  Vertical 9:16 aspect ratio, low-quality found-footage smartphone video frame, filmed from the passenger seat of a moving car. Handheld feel with slight camera shake, imperfect off-center framing, mild motion blur from vehicle movement. A chubby gray tabby cat sits upright in the driver’s seat, one paw resting naturally on the steering wheel as if driving. The cat appears calm, focused, and serious, eyes fixed forward on the road, creatin...

How simply use Manim to create animated charts and graphs

Links: https://3b1b.github.io/manim/getting_started/example_scenes.html Install: brew install ffmpeg pip install manim Create a new project : manim init project my-project --default or you can just start writing the code in a Python file. then you should have this default code in main.py from manim import * class DefaultTemplate ( Scene ): def construct ( self ): circle = Circle () # create a circle circle . set_fill ( PINK , opacity = 0.5 ) # set color and transparency square = Square () # create a square square . flip ( RIGHT ) # flip horizontally square . rotate ( - 3 * TAU / 8 ) # rotate a certain amount self . play ( Create ( square )) # animate the creation of the square self . play ( Transform ( square , circle )) # interpolate the square into the circle self . play ( FadeOut ( square )) # fade out animation if don't have it just create main.py and put that code in it.  now run this comm...

A very simple Terraform example with GCP

Install Terrafrom and gcloud. Step 1  A very simple Terraform example with GCP Step 2 After making sure you have Terraform installed, create a simple file bigquery.tf then put these inside: provider "google" { project = "shiritech-488519" region = "europe-north1" } resource "google_bigquery_dataset" "dataset" { dataset_id = "test_dataset2" friendly_name = "Example Test Dataset" description = "Dataset created with Terraform" } *Remember that you can have several tf files in the same directory and Terraform will combine the together as a single file. Step 3 For Terraform to connect to GCP we need to login with gcloud first using: gcloud auth application-default login Select the correct user who has the permissions to the project.   Step 4   Now we need to run the following commands: terraform init #Connect to the backend terraform validate #Make sure config is healthy terraform plan ...

A simple example to show how Kubernetes works in practice | Deploy A Flask app running on AKS

This article makes more sense when you watch the actual video: In this video, I’m going to show you how to setup kubernatives to run a flask application on two different nodes so that when one fails the other one takes place and keeps the app running.  One important thing that I’ll try to show in the video is the main functionality of Kubernatives and see how it works in practice. So before diving into the actual setup let me talk a little bit about Kubernatives and the big picture workflow.  There are a lot of benefits that come from using container orchestration systems like Kubernetes but one of the most important reasons is load balancing.  Imagine a website running on a server, if the server crashes for any reason, you’re site will be done. One solution is having a duplicate version of the app in a completely different server and use the second server when the first stops working for any reason.  But having this setup is harder than said. When using docker to ru...