Wednesday, 24 September 2014

Unix Interview Prep Tutorial Part2- Simple unix scripts (Usage of IF, FOR Loops, While, Case, parameters, and importing variables from a file)

Unix Interview Prep Tutorial Part2- Simple unix scripts (Usage of IF, FOR Loops, While, Case, parameters, and importing variables from a file)


1) For usage for IF command in unix script, see the below sample script

#!/usr/bin/ksh
$num=3
if [ "$num" -gt "0" ];
  then
  echo "$num greater than 0"
  else
  echo "$num less than 0"
 fi



2) For usage of FOR loop to read filenames from a filelist and move those files to a target directory, see the below example: (You can edit the $SOURCEDIR and $inputfilelist to the path you need for your purpose)

#!/usr/bin/ksh
$SOURCEDIR=/home/directorypath
starttime=`date +'%Y-%m-%d %H:%M:%S'`
$inputfilelist=myinputfilelist.txt
for file in $(<$SOURCEDIR/$inputfilelist)
do
if [[ $file == "#"* ]];
then
echo "Moving of $file ignored"
else
starttime=`date +'%Y-%m-%d %H:%M:%S'`
echo "Starting moving $file at $starttime......"
mv $file /targetfiledirectory/
echo "$file moved to target directory"
fi
done



3) To check number of parameters sent to a script use the below commands in a script:

#!/usr/bin/ksh
if [[ $# -ne 1 ]]; then
   echo "Invalid parameters.\nUsage: $0 <filelist>"
   exit 1
fi


4) To import the variables defined in other file you can use the below commands at the beginning of the script:


. ./file.conf   /* works on unix and import variables from file.conf */
source directory_path/file.conf  /* works on linux and imports variable names from file.conf */

5) Usage of while statement in unix is shown below:#!/usr/bin/ksh
count=5                                   # Initialise count to 5
while [ $count -gt 0 ]                     # while count is greater than 0 do
do
   echo "the value of count is $count"
   count=$(expr $count -1)                 # decrement count by 1
done

6) Usage of string array in unix is shown below:


#!/usr/bin/ksh
namelist="Joe John Peter Tom Bill"
for name in $namelist
do
   if [ "$name" = "Joe" ]
   then
      echo "My name is $name"
   else
      echo "My other names are   ${name}s"
   fi
done
     
7) Usage of case statement in unix is shown below:


#!/usr/bin/ksh
case $1 #value from first parameter
in
1) echo 'One';;
2) echo 'Two';;
*) echo 'Three';;
esac 



How to create XML parser in Informatica using XMLSpy and .dtd?


How to create XML parser in Informatica?


Let’s say you are reading an XML and want to create an XML parser transformation to parse the XML. If the XML is not very complex, then a very simple way to generate the XML parser transformation might be to use XML spy to generate the .dtd file using the XML. Steps below:

How to generate .dtd in XML spy:

1)      Go to File Menu- Open the XML file

2)      Go to DTD/Schema and click on Generate DTD/Schema. It will ask do you want to associate the dtd with the xml file? Click yes and save the dtd file.

3)      Go to XML menu- Click validate. If any issues with the DTD , it will show up when you validate the XML. For the sample xml, I was working with I had to remove one of the child item in the .dtd to make it the validation successful. You can figure it out from the message the validation process throws at you.

Once you have generated the .dtd you can go to Informatica designer-transformation menu and create xml parser transformation using the .dtd schema you just generated. If you need to make any changes to the XML view, you can use the XML Editor that you can access from the midstream XML parser tab of the XML parser.

Java program to add or substract dates

Java program to add or substract dates

Below is the java program for date addition. In example below, you are adding 3 days to date 20140101. You can change the format of the date by changing the format mentioned in the code in red below. 


-----dateadd.java code------------

import java.io.*;
import java.util.*;
import java.text.SimpleDateFormat;
import java.text.DateFormat;
import java.text.ParseException;
public class dateadd {
public static void main (String[] args) {
DateFormat formatter = new SimpleDateFormat("yyyymmdd");
Date dt=new Date();
try
            {
 dt = formatter.parse(args[0]);
Calendar c = Calendar.getInstance();
c.setTime(dt);
c.add(Calendar.DATE,Integer.parseInt(args[1]) );
dt = c.getTime();
System.out.println(formatter.format(dt));
            } catch (ParseException e)
            {
                e.printStackTrace();
            }
}
}


Also check out the commonly used date functions in netezza:
http://dwbitechguru.blogspot.ca/2014/09/commonly-used-netezza-date-functions.html

Monday, 8 September 2014

Commonly used netezza date functions and how to do data manipulations?

How to use date functions in Netezza?

--to select the current date and timestamp use:


select current_date
select current_timestamp 

--to add months and convert date to integer use:

select to_number(to_char(add_months(current_date, -1),'YYYYMMDD'),'99999999') --to get date id

select to_number(to_char(add_months(current_date, -1),'YYYYMM'),'9999999') --to get month id

select date(current_date -  cast('1 month' as interval))

--to convert timestamp to yyyy-mm-dd format:


select TO_CHAR(TO_DATE('2011-11-24 22:42:28','YYYY-MM-DD HH24:MI:SS'),'YYYY-MM-DD')
select TO_CHAR(current_timestamp,'YYYY-MM-DD')

--to extract year part from the current date

select extract (year from current_date)
select date_trunc('year', current_date)
to_number(to_char(current_date,'YYYY'),'9999')  --integer format


--to extract day and month from the current date

select extract (day from current_date)

--to get first and last day of the year:


select to_date((extract (year from current_date)) || '0101','YYYYMMDD')
select to_date((extract (year from current_date)) || '1231','YYYYMMDD')

--to get first of the month given a date

select date_trunc('month', current_timestamp)

--to get last month end date

select date_trunc('month', current_timestamp)-1


--to add 7 days to current date


select date(current_date +  cast('7 days' as interval))

--to get timestamp in a specific format

select to_timestamp( current_timestamp, 'YYYY-MM-DD HH24:MI:SS')
select to_timestamp( '2014-12-01 23:12:12', 'YYYY-MM-DD HH24:MI:SS')

--to add hours to a date

 select TO_CHAR(date((current_date +  cast('1 hour' as interval))),'YYYY-MM-DD') 

--To calculate difference in days , weeks between two dates

select datediff(week,'2012-01-01','2015-12-31') as numweeks;

select datediff(day,'2009-01-01','2015-12-31') as numdays;

Also check:
http://dwbitechguru.blogspot.ca/2014/09/jave-program-to-add-or-substract-dates.html

For loading and migration of data in  netezza, check:
http://dwbitechguru.blogspot.ca/2014/11/extract-load-migrate-filesdata-from.html

For SQL server date functions check:
http://dwbitechguru.blogspot.ca/2014/11/sql-server-important-date-manipulation.html

How to generate SSH keys and use it for Sftp / SSH/ SCP?

How to generate SSH keys and use it for Sftp / SSH/ SCP?

SSH keys are commonly used during sftp and scp to authourize access to a host machine. Instead of using a password, ssh keys are used to identify and authourize a machine to login to a target machine. It is more safe and hard to breakin.

In brief, the steps to set up ssh authetication would be as follows:

Step a: On any source/client machine, use the command below to generate the ssk keys:

ssh-keygen -t rsa -f <nameofthekeyfile>

Example: ssh-keygen -t rsa -f mykeys-for-targetmachine

The command will prompt for keys size, expiry date, passphase, etc. You can enter the values you deem fit for your purpose.The above command will create a private and pub key with the key name mykeys-for-targetmachine you have provided in the command. The public key file will have .pub extention.

Step b: Now open the public key file and copy the contents of the key file.

Example:
cat mykeys-for-targetmachine.pub

The output will look something like this:
ssh-rsaAAAAB3NzaC1yc2ASAGSAHGSJAHSGJASGAJSG38fxq8VHDwNRP/asJHGJHGJGaasasaJGJJGJHGJHJx305gH3XKZA3
asdasdaLLHJLLJLJLJLJLLKJLJLasdsadsadjlkjlajdsadl= myuser@targetmachine.com

Step c: Login to the target machine that you want to access and open the authorised_keys file. This file should be in your home directory.Paste the contents of the public file that you copied in step b to this file.

Step d: Back to the source/client machine, change the permission of the keys to make it secure.
For the above example, it would be:

chmod 600 mykeys-for-targetmachine


Step e: Now try to SSH to target machine using your new keys to test the connection: The command would be something like the below (replace the <myuser> with the user name requierd on the target machine and the <targetmachine> with the name of the target machine that you want to login i.e. the machine to with the authourized key file that you just updated in step c:

ssh -i mykeys-for-targetmachine <myuser>@<targetmachine>

it will prompt first to add the target machine to the list of known host, press yes and you should be now logged into the machine. If it prompts for a password then you haven't done something right. Check the above steps and do the required correction.

How to sftp and scp using ssh keys? Below are the commands to use:

sftp -i mykeys-for-targetmachine <myuser>@<targetmachine>
scp -i  mykeys-for-targetmachine <myuser>@<targetmachine>



More information can be found in the below link:
https://help.ubuntu.com/community/SSH/OpenSSH/Keys

Wednesday, 27 August 2014

Computer Hacking - Things you need to know- How and Prevention

Computer Hacking - Things you need to know- How and Prevention


This blog is about the general network/data security knowledge that everyone working in an office setup with desktops and laptops should know. In an Internet age, with every computer hooked to internet, it has become very easy for any one to gain access and cause havoc. Some awareness of internet security can go a long way in preventing such security breaches.

Damages from Internet/Data security breach and computer hacking:

a) Data theft/Loss:  Lets says your credit card number is stolen, then someone can hack into bank account. If your computer password in stolen, someone can delete or steal data from your computer.

b) Loss of customer trust: If you are working for an organization or running an organization that stores customer data, then any theft or loss of customer data can damage the customer trust, bring law suits, and big financial loss.

c) Data/System availability: If someone hacks into your computing device then the hacker can bring the whole system down, install virus/backdoor programs and make the system unusable.


Common ways data security breeches and computer hacking occur:

a) Installing virus or Trojan programs to steal data by sending the virus programs through emails or making users download it automatically when they visit certain web sites.

b) Social engineering- Using the information available on internet or other sources, hackers can try to call the people impersonating as someone else to gather more information required to steal an organization or personal information.

c) Insider job- Majority of the data security breaches or intrusion happen with help of a person from within the organization. The person working in an organization has access to computing assets that can be used to steal data or whatever malicious purpose.

d) Hackers: These are folks who know to hack into IT networks and try various sophisticated ways to get into an organization data network. They can remotely monitor an organizations or personal network set up and find entry points to get into the network.


Prevention of Internet/Data security breach:

Personal users:
a) Keep all your computer passwords secure and strong. Don't give away your password to anyone. Always set up a password for your computer, phones, and tablets. Keep it strong by  making it hard to predict. Don't use your names as password. Use special characters such as #$% in your passwords and use mix of numbers, upper and lower case letters.

b) If you are working always lock your computer/phone before leaving your desk. Do not leave your phone on the desk. Do not write passwords on sheet of paper or yellow sticky and leave it on your desk.

c) Do not install programs from popups and websites that are fishy and not well known. Don't open email attachments that have executable programs. Always use anti virus and malware detection programs. Microsoft defender and security essential is a free anti virus program.

d) Apply the operating system security patches whenever available and prompted by computers to do so. Create a recovery CD of your operating system and if your computer is infected then do a fresh install of OS by reinstalling it from the recovery CD.


Organizations:
a) Every organizations these days have IT security polices and security devices to protect their organizational IT assets. Every employee should follow those security policies.

b) If you are a start up, then better to get the security software at the earliest and protect your network. Common security devices used by companies are Firewalls, Intrusion detections systems, security auditing programs, virus and malware scanners and detectors,etc. If you cannot affords costly devices, then use the open source programs available. 

c) Use VPN (virtual private networks) to connect to office network. VPNs encrypt data during transmission from outside to inside network. Use VPNs if you are a mobile user and connect frequently to office network. Always encrypt emails and any data sent outside the network.

d) Set proper folder permissions on your network so that users not belonging to your organization cannot access it.

e) If you are having any website, make it secure by hosting it on servers behind firewalls. Firewalls are security programs that blocks connection to certain ports of your server/network.

g) Security audit the computer network to identify programs that are more vulnerable to hacking, and find all the ways to secure your network.

f) Keep back up of data on data centers and have disaster recovery plans.

Saturday, 9 August 2014

How to migrate/deploy code from one environment to another in Informatica


There are mainly three ways to migrate/deploy code from one environment to another in Informatica. Those three ways are:

Method 1- Deployment groups- This involves creating a deployment group from the repository manager with all the informatica objects that need to be migrated to the destination machine. This most likey is used wherever there is version control used in the destination machine since the deployed objects can be rolled back to the previous version. The steps to create deployment group is below:

Go to Repository Manager.
Under Tools>Deployment Groups > Create
Provide a name for the deployment group and add the desired permissions.
Select the informatica objects that you want to migrate , right click and select add to deployment group or you can also copy the objects and paste it to the deployment group
Now copy this deployment group from source repository to the destination repository using normal copy and paste.
The deployment wizard will start and you have to follow the on screen steps


Method 2- Import/Export using XML- In this method all the objects (source/target definitions, mappings, workflows, etc) are exported as xml to local hard disk from the source repository using either workflow manager or repositor manager. The exported XMLs are imported into the destination machine. Make sure to import the source/target definitions before the mappings.

Method 3- Drap and drop/copy and paste method- This is the most simplistic one. Open the target and source repository in repository manager. You can do it with workflow manager too but repository manager is more better option. Copy first all the source/targets then you want to migrate and paste it to the destination machine. Next do the same with mappings, sessions, workflows in this order. The shortcuts need to be created in the designer.

Check more about migration issues in this article:
http://dwbitechguru.blogspot.ca/2014/08/informatica-migrationimplementation-in.html