Query BigQuery with Python

This week, I wanted to connect with BigQuery using Python. BigQuery is a service from Google, which helps to store and process large volumes of data quickly. We are using it to store Google billing data and wanted to generate some reports by executing different SQL queries.

from google.cloud import bigquery
from google.oauth2 import service_account

auth_json = Your JSON for authentication

credentials = service_account.Credentials.from_service_account_info(auth_json)

client = bigquery.Client(project="project name", credentials=credentials)

query_job = client.query("""your query inside""")

results = query_job.result()  # Waits for job to complete.

for row in results:
    print("{}".format(row.id, any other fields))


 

The above code shows, how we can connect and query the bigquery. It returns an iteratable, which can allow looping over the rows and access each column as shown in the code.

I hope it would help you in your big projects.

Resize Disk of an Openstack’s Instance

In my current job, we are using Openstack to test and deploy our projects locally. Last week, I got an email from my colleague, and he told me that he cannot install any software in his Openstack’s instance.

When I checked the instance using the following command:

df -h –total

I found out that its root disk is full. That is why, he cannot install any software. After taking help from my friend GOOGLE. I found the following article (https://access.redhat.com/solutions/2151971) to resize cinder volume of an Openstack instance. I followed the following steps:

  1. Stop the instance (nova stop <isntance id>)
  2. Reset state of the volume to available (cinder reset-state –state available <cinder volume id>)
  3. Extend volume size (cinder extend <cinder volume id> <size in GB>)
  4. Change the status of volume to in-use (cinder reset-state –state in-use <cinder volume id>)
  5. Start the instance (nova start <isntance id>)

After doing that, I was able to see disk size has increased but Ubuntu OS was still showing full disk. Then I found one more article (https://devops.ionos.com/tutorials/increase-the-size-of-a-linux-root-partition-without-rebooting/) to extend the root disk in Ubuntu OS. I did the following steps:

  1. Use fdisk utility on your root disk (fdisk /dev/vda)
  2. Delete your primary disk by pressing “d
  3. Recreate it using “n
  4. First sector, choose default
  5. Last sector, give maximum size using “+size
  6. Write the changes by pressing “w
  7. Now run “df -h –total” to see the effect

 

 

CloudStack KVM Host Installation

CloudStack Version 4.0
OS Version Ubuntu Server 12.04, 64 bit

 

  1. Install Ubuntu Server 12.04, 64 bit, you can download it from http://stingray.cyber.net.pk/pub/ubuntu-releases//precise/ubuntu-12.04.2-server-amd64.iso
  2. Set root password using these commands
    1. a.      sudo su
    2. b.      Enter your password
    3. c.       passwd
    4. d.      Enter and confirm your password for root user
  3. Assign static IP address to Ethernet card with these commands
    1. a.      vim /etc/network/interfaces
    2. b.      Press insert button to change the editor to insert mode
    3. c.       Change

                                                  i.      auto eth0

                                                ii.      iface eth0 inet dhcp

  1. d.      To

                                                  i.      auto eth0

                                                ii.      iface eth0 inet static

                                              iii.      address xxx.xxx.xxx.xxx

                                               iv.      netmask xxx.xxx.xxx.xxx

                                                 v.      gateway xxx.xxx.xxx.xxx

  1. e.       Restart network service

                                                  i.      /etc/init.d/networking restart

  1. Install SSH server to allow ssh access to this machine
    1. a.      apt-get install ssh
  2. Configure repository for CloudStack
    1. vim /etc/apt/sources.list.d/cloudstack.list
    2. b.      Write to file

                                                  i.      deb http://cloudstack.apt-get.eu/ubuntu precise 4.0

  1. c.       Save the file
  2. d.      wget -O – http://cloudstack.apt-get.eu/release.asc | apt-key add –
  3. e.       apt-get update
  4. Install NTP server
    1. a.      apt-get install openntpd
  5. Install cloud agent
    1. a.      apt-get install cloud-agent
  6. Configure libvirt
    1. a.      vim /etc/libvirt/libvirtd.conf, and uncomment the below lines

                                                   i.      listen_tls = 0

                                                 ii.      listen_tcp = 1

                                               iii.      tcp_port = 16059

                                                iv.      auth_tcp = “none”

                                                  v.      mdns_adv = 0

  1. b.      Save file and close it
  2. c.       vim /etc/init/libvirt-bin.conf

                                                   i.      Modify libvirtd_opts to “-d –l”

  1. d.      Save file and close it
  2. e.       service libvirt-bin restart
  3. 9.      vim /etc/idmap.conf
    1. a.      Uncomment Domain and make it same to the management server
  4. Configure Firewall
    1. a.      dpkg –list ‘apparmor’
    2. b.      ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
    3. c.       ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
    4. d.      apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
    5. e.       apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper
  5. Open Ports
    1. a.      ufw allow proto tcp from any to any port 22
    2. b.      ufw allow proto tcp from any to any port 1798
    3. c.       ufw allow proto tcp from any to any port 16509
    4. d.      ufw allow proto tcp from any to any port 5900:6100
    5. e.       ufw allow proto tcp from any to any port 49152:49216
  6. System Reboot

Just access cloud stack management interface and add the host.

CloudStack Management Server Installation

CloudStack Version 4.0
OS Version Ubuntu Server 12.04, 64 bit
  1. Install Ubuntu Server 12.04, 64 bit, you can download it from http://stingray.cyber.net.pk/pub/ubuntu-releases//precise/ubuntu-12.04.2-server-amd64.iso
  2. Set root password using these commands
    1. a.      sudo su
    2. b.      Enter your password
    3. c.       passwd
    4. d.      Enter and confirm your password for root user
  3. Assign static IP address to Ethernet card with these commands
    1. a.      vim /etc/network/interfaces
    2. b.      Press insert button to change the editor to insert mode
    3. c.       Change

                                                  i.      auto eth0

                                                ii.      iface eth0 inet dhcp

  1. d.      To

                                                  i.      auto eth0

                                                ii.      iface eth0 inet static

                                              iii.      address xxx.xxx.xxx.xxx

                                               iv.      netmask xxx.xxx.xxx.xxx

                                                 v.      gateway xxx.xxx.xxx.xxx

  1. e.       Restart network service

                                                  i.      /etc/init.d/networking restart

  1. Install SSH server to allow ssh access to this machine
    1. a.      apt-get install ssh
  2. Configure repository for CloudStack
    1. vim /etc/apt/sources.list.d/cloudstack.list
    2. b.      Write to file

                                                  i.      deb http://cloudstack.apt-get.eu/ubuntu precise 4.0

  1. c.       Save the file
  2. d.      wget -O – http://cloudstack.apt-get.eu/release.asc | apt-key add –
  3. e.       apt-get update
  4. Install NTP server
    1. a.      apt-get install openntpd
  5. wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util
  6. Copy vhd-util
    1. a.      mkdir -p /usr/lib/cloud/common/scripts/vm/hypervisor/xenserver/
    2. b.      cp  vhd-util /usr/lib/cloud/common/scripts/vm/hypervisor/xenserver/vhd-util
  7. 9.     apt-get install cloud-client
  8. Install mysql database
    1. a.      apt-get install mysql-server
    2. b.      Mysql will ask password, write it when prompt
    3. c.       ufw allow mysql
    4. d.      vim /etc/mysql/my.cnf
    5. e.       At the end of file, add the following lines

                                                   i.      innodb_rollback_on_timeout=1

                                                 ii.      innodb_lock_wait_timeout=600

                                               iii.      max_connections=350

                                                iv.      log-bin=mysql-bin

                                                  v.      binlog-format = ‘ROW’

  1. f.         Save file
  2. g.       Restart mysql

                                                   i.      service mysql restart

  1. Grant Rights
    1. a.      mysql –u root –p
    2. b.      Enter password
    3. c.       GRANT all privileges on *.* to ‘root’@’%’ identified by ‘password’;
    4. d.      flush privileges;
    5. e.       quit;
  2. Deploy database using
    1. a.      cloud-setup-databases cloud:password@localhost –deploy-as=root:password
  3. Setup NFS
    1. a.      mkdir -p /export/primary
    2. b.      mkdir -p /export/secondary
    3. c.       apt-get install nfs-kernel-server
    4. d.      vim /etc/exports
    5. e.       Write

                                                   i.      /export *(rw,async,no_root_squash)

  1. f.        Save file
  2. g.      exportfs –a
  3. h.      vim /etc/idmapd.conf

                                                   i.      uncomment Domain, this domain should be same on both management server and host

  1. i.        Save file
  2. Reboot machine
  3. cloud-setup-management to start management server
  4. For XenServer
    1. a.      /usr/lib/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.vhd.bz2 -h xenserver -F
  5. For KVM
    1. a.      /usr/lib/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2 -h kvm -F
  6. Congratulations, your CloudStack management server is ready