Oracle SQL Hints

/*+ hint */
/*+ hint(argument) */
/*+ hint(argument-1 argument-2) */

Difference Between UNION and UNION ALL

The Main Difference is

UNION returns the data without duplicates
UNION ALL returns the data with duplicates if there any.

UNION is costly as it has to check for redundancy
UNION ALL is faster.

Difference Between Delete,Truncate and Drop

Delete :

• Delete is a DML command.
• Delete command uses undo table space for roll back.
• deletes can be undo by using Rollback.
• Delete command can be used to delete few rows from table or all rows from table.
• After the Delete command need to use the  Commit or Rollback command to permanent the transaction.

Truncate ;

• Truncate is a DDL command.
• Truncate removes all rows from a table.
• truncate wont use any table space and triggers will be filred.
• It cannot be Rolled Back.
• It's faster than Delete.

Drop :

• Drop removes the data and the table structure from the Database.
• Operation cannot be rolled back.
• No DML trigger will be fired.

How to run the UNIX Processes in background

using "&" symbol (ampersand) at the end of the command we can run the processes in the background.

Ex :  sleep 100 &

it will give the PID and leave the terminal for other work.we can see the background process by using the "ps" command.

Copy Stage - Force Option

Copy Stage :  Stage->Properties have option called Force

Force : true or False.

True to specify that DataStage should not try to optimize the job by removing the Copy operation.

False to specify that Datastage should try to optimize the job (it might remove the copy operator or might not).

Conductor Node in Datastage

Below is the sample APT CONFIG FILE ,see in bold to mention conductor node.


{
node "node0"
{
fastname "server1"
pools "conductor"
resource disk "/datastage/Ascential/DataStage/Datasets/node0" {pools "conductor"}
resource scratchdisk "/datastage/Ascential/DataStage/Scratch/node0" {pools ""}
}
node "node1"
{
fastname "server2"
pools ""
resource disk "/datastage/Ascential/DataStage/Datasets/node1" {pools ""}
resource scratchdisk "/datastage/Ascential/DataStage/Scratch/node1" {pools ""}
}
node "node2"
{
fastname "server2"
pools ""
resource disk "/datastage/Ascential/DataStage/Datasets/node2" {pools ""}
resource scratchdisk "/datastage/Ascential/DataStage/Scratch/node2" {pools ""}
}
}

Please find the below different answers :
------
For every job that starts there will be one (1) conductor process (started on the conductor node), there will be one (1) section leader for each node in the configuration file and there will be one (1) player process (may or may not be true) for each stage in your job for each node. So if you have a job that uses a two (2) node configuration file and has 3 stages then your job will have

1 conductor
2 section leaders (2 nodes * 1 section leader per node)
6 player processes (3 stages * 2 nodes)

Your dump score may show that your job will run 9 processes on 2 nodes.

This kind of information is very helpful when determining the impact that a particular job or process will have on the underlying operating system and system resources.

-----
Conductor Node :
It is a main process to

1.  Start up jobs
2.  Resource assignments
3.  Responsible to create Section leader (used to create & manage player player process which perform actual job execution).
4.  Single coordinator for status and error messages.
5.  manages orderly shutdown when processing completes in the event of fatal error.

-----
Jobs developed with DataStage EE and QualityStage are independent of the actual hardware and degree of parallelism used to run the job. The parallel Configuration File provides a mapping at runtime between the job and the actual runtime infrastructure and resources by defining logical processing nodes.


To facilitate scalability across the boundaries of a single server, and to maintain platform independence, the parallel framework uses a multi-process architecture.

The runtime architecture of the parallel framework uses a process-based architecture that enables scalability beyond server boundaries while avoiding platform-dependent threading calls. The actual runtime deployment for a given job design is composed of a hierarchical relationship of operating system processes, running on one or more physical servers

• Conductor Node (one per job): the main process used to startup jobs, determine resource assignments, and create Section Leader processes on one or more processing nodes. Acts as a single coordinator for status and error messages, manages orderly shutdown when processing completes or in the event of a fatal error. The conductor node is run from the primary server
• Section Leaders (one per logical processing node): used to create and manage player processes which perform the actual job execution. The Section Leaders also manage communication between the individual player processes and the master Conductor Node.
• Players: one or more logical groups of processes used to execute the data flow logic. All players are created as groups on the same server as their managing Section Leader process.

-----

When the job is initiated the primary process (called the “conductor”) reads the job design, which is a generated Orchestrate shell (osh) script. The conductor also reads the parallel execution configuration file specified by the current setting of the APT_CONFIG_FILE environment variable.

Once the execution nodes are known (from the configuration file) the conductor causes a coordinating process called a “section leader” to be
started on each; by forking a child process if the node is on the same machine as the conductor or by remote shell execution if the node is on a
different machine from the conductor (things are a little more dynamic in a grid configuration, but essentially this is what happens). Each section
leader process is passed the score and executes it on its own node, and is visible as a process running osh. Section leaders’ stdout and stderr are
redirected to the conductor, which is solely responsible for logging entries from the job.


The score contains a number of Orchestrate operators. Each of these runs in a separate process, called a “player” (the metaphor clearly is one of an
orchestra). Player processes’ stdout and stderr are redirected to their parent section leader. Player processes also run the osh executable.

Communication between the conductor, section leaders and player processes in a parallel job is effected via TCP.

Difference between scratch disk and resource scratch disk

The Only difference is :

• Scratch Disk is for Temporary storage (Like RAM in our PC)

Ex : Files created during the process between the source and targtet such as Sort,Remove duplicate,Aggregator etc..

• Resource Scratch Disk is for Permanent storage (like a Hard Drice in our PC)

Ex : Data sets,files sets, Loookup file sets etc..

Common Errors,warnings in Datastage

• Warning ; A sequential operator cannot preserve the partitioning of input data set on input port 0

          Sol : Clear the partitoning

• Warning : Agg_stg: When checking operator: When binding input interface field “column1” to field “ column1 ”: Implicit conversion from source type “string[5]” to result type “dfloat”: Converting string to number.

          Sol: use data type conversion

• Warning ; oci_oracle_source: When checking operator: When binding output interface field “column1” to field “column1”: Converting a nullable source to a non-nullable result;

         Sol : Use Null functions

Initial Load and Delta Load

Difference Between Initial Load and Delta Load :

Initial Load :

Ø       Occurs Once

Ø       Large amount of Data

Delta Load :

If the data service has the capability to return the data modified only after a specified date and time, the ETL process will load only the data modified after the last successful load. This is called delta load

Ø       Occurs regularly

Ø       Adjustments to Initial load

Ø       Small amount of data

Information Analyzer Introduction

sed unix

awk unix

NR = Number or Records read so far
NF = Number of Fields in current record
FS = the Field Separator
RS = the Record Separator
BEGIN = a pattern that's only true before processing any input
END = a pattern that's only true after processing all input.

UNIX Commands

COPY THE FIRST 10 LINES FROM ONE FILE TO ANOTHER FILE :
sed -e "10q" GDK2120_D1.TXT > GDK2120_D2.TXT

-----------------------------------
COUNT THE NUMBER OF FILES IN THE FOLDER :

 ls -1|wc -l    [-ONE & -L]
-----------------------------------
CHECKING DISK SPACE :
df -h
or
df -k
or
du -sh

-----------------------------------
MIGRATE FOLDERS FROM ONE ENVIRONMENT TO ANOTHER ENVIRONMENT(EX: DEV TO QA)
scp -rp /home/data user@dev.com:/home/data

-----------------------------------
CHANGE THE OWNER OF THE FOLDER :
sudo chown -R user /home/data
-----------------------------------
TO CHECK FOR THE FILE(GREP)
ls /home/data | grep ^CED_DELTA.*TXT$
ls /home/data | grep CED_DELTA.*TXT$
-----------------------------------
to check the process of my own
ps -f -l -uusername
-------------------------------------
to convert multiple lines into one single line
awk '{printf("%s",$0)}' HPS_D1120807_142128.TXT > HPS_D1120807_142128_1.TXT
--------------------------------------------------------------------------------
to convert one single row with delimeters into multiple rows (~ is  a delimeter here)
awk -F"~" '{for(i=1 ;i <= NF;i++ ) {print $i;}}' HPS_D1120807_142128_1.TXT >HPS_D1120807_142128_2.TXT
--------------------------------------------------------------------------------
to remove the last line from the file
sed '$d' EDISAMPLE_3.TXT > EDISAMPLE_4.TXT

-------------------------------------------------------------------------
rename file with one pattern to another pattern
rename  ATTXJZZ  ATTXJGD *.ksh

-------------------------------------------------------------------
touch to change the timestamp of the file
touch -a -m -t 201301181205.09 filename.txt

Change Data Capture Stage - Properties&Behaviour

Change mode determines how keys and values are specified :

• Explicit Keys & Values    : means that key & value columns must be explicitly defined. 
• All Keys, Explicit Values : means that value columns must be defined, but all other columns are key columns unless they are excluded. 
• Explicit Keys, All Values : means that key columns must be defined, but all other columns are value columns unless they are excluded.

Join stage input/output link Partitioning types

Lookup stage input/output link Partitioning types

how many input and output links can Merge Staga have

Database Join or Datastage Join,which is better

Find and replace in UNIX

how many input and output links can Join Stage have

Beacause people asking this question,i really tried connecting around 70 input links to Join stage and it took.

So the conclusion is, it can have any number of input links as per the ibm developer guide.

Any Number of Input Links
One Output Link
Zero Reject Links.

Difference Between Primary key and Unique key

Unique Key :

Unique key is a single column or set of columns that can uniquely identify a row in table.

It Doesn't allow duplicates

It Allows Null values

Primary Key :
Primary key doesn't allow Null values
It doesnt allow duplicates

Finding duplicates SQL

SELECT item_id,
 COUNT(item_id) AS NumOccurrences
FROM item
GROUP BY item_id
HAVING ( COUNT(item_id) > 1 )

Different padding functions in Datastage

Padding with spaces or zeroes:

• Str(" ", 50 - Len(inputcolumn)) : inputcolumn

in above example,50 is the total lenghth of the Target filed
if the input value have 10 charaters so the above function will replace the remaining 40 with spaces
Output Looks like : "xxxxxxxxxx                                             "

• Str("0", 50 - Len(inputcolumn)) : inputcolumn

in above example,50 is the total lenghth of the Target filed
if the input value have 10 charaters so the above function will replace the remaining 40 with zeroes

Output Looks like : "xxxxxxxxxx000000000000000000000000000"

• Right(STR('0',18):inputcolumn,18)
• Right('0000':inputcolumn,4)

how to see the processes you own in AIX (UNIX)

To Display All the processes :
>> ps -e -f

It will display all the process information.

To display the the processes you own :

>>ps -f -l -uname

Output :
----------------------------------------------------------------------------------------------------------
 F  S      UID     PID    PPID   C PRI NI ADDR    SZ    WCHAN    STIME    TTY  TIME CMD
------------------------------------------------------------------------------------------------------------
240001 A   name  376928  504028   0  60 20 40d05400  1308 f10001003177efb8 16:31:09  pts/0  0:00 cmdlog -ksh
   40001 A   name  504028  434330   0  60 20 1f0a5e400   988          16:31:09      -  0:00 sshd: n74950@pts/0
  240001 A   name  536690  376928   0  60 20 c0d0d400   768          16:31:09  pts/1  0:00 -ksh

Hide Password Entry in UNIX script:

We can use 'stty' to hide the password entry in UNIX command prompt, which is like entering password when you login to the UNIX.
stty:
'stty' sets options for your terminal
Ex :

• stty erase \^h --- set the backspace as erase key
• stty sane
• stty

Example Script:
#!/bin/ksh
echo "Please enter the pwd:"
stty -echo
read pwd
stty echo
if [ "$pwd" == "secret" ]; then
echo "u have access"
else
  echo "access denied"
fi

Sort Utility in Datastage

The type of sort the stage will carryout:

• Datastage: This used built-in DataStage sorter.do not require any additional software.
• UNIX : unix sort command is used to perform sort.

Stable Sort :

• Applies to sort utility type of Datastage,the default is true.It is set to True to guarantee that this sort operation will not rearrange records that are already in sorted dataset.
• If it is set to False no prior ordering of records is guaranteed.

How do we know whether we running Datastage jobs on SMP or MPP System:

Look in the Configuration file if the "fastname" has the same value for all given nodes then you are running on SMP.
If there are different "fastname's" then its MPP.
For Server jobs it won't matter if it is SMP or MPP because Server jobs execute only on the machine where Datastage installed.
See below in Bold.
Ex :
main_program: APT configuration file: /ibmiscode/Configurations/ETL/apt_config_max
{ 
  node "DEV1" { 
    fastname "DEV1" 
    pools "" 
    resource disk "/is_resource1/isent" {pools ""} 
    resource scratchdisk "/scratch1/sent" {pools ""} 
  } 


  node "local2" { 
    fastname "DEV1" 
    pools "" 
    resource disk "/is_resource2/isent" {pools ""} 
    resource scratchdisk "/scratch2/sent" {pools ""} 
  } 


  node "local3" { 
    fastname "DEV1" 
    pools "" 
    resource disk "/is_resource3/isent" {pools ""} 
    resource scratchdisk "/scratch3/sent" {pools ""} 
  } 


  node "local4" { 
    fastname "DEV1" 
    pools "" 
    resource disk "/is_resource4/isent" {pools ""} 
    resource scratchdisk "/scratch4/sent" {pools ""} 
  } 
}

CUT command unix/linux

CUT command :
i'm going to take below sample file for the examples shown.
-> cat test.txt
201200001|201200001|LIFE|A
201200002|201200002|STD|B
201200003|201200003|LTD|C
201200004|201200004|LTD|X
201200005|201200005|LTD|E

cut command will be used with the following options

• cut -c
• cut -d with -f

1) cut -c :

-> cut -c 11 test.txt
 the above command will give the 11th character from each line of the file
2
2
2
2
2

-> cut -c 11,12 test.txt
the above command will give the 11th and 12th characters from each line
20
20
20
20
20

-> cut -c 1-10 test.txt
the above command will give characters from 1 to 10 from each line
201200001|
201200002|
201200003|
201200004|
201200005|

-> cut -c -8 test.txt
the above command will return charaters from 1 to 8 from each like similary like above result
20120000
20120000
20120000
20120000
20120000

-> cut -c 10- test.txt

the above command will return charaters from 10th to end of the line from each line.
|201200001|LIFE|A
|201200002|STD|B
|201200003|LTD|C
|201200004|LTD|X
|201200005|LTD|E

2) cut -d with -f:
character followed by -d in the unix command is a delimeter either we can use in the quotes     (-d "|") or we can use with back slash after the -d ( -d\).

-f specifies the filed number to return

-> cut -d "|" -f3 test.txt
or we can use
-> cut -d\| -f3 test.txt
the above command returns 3rd filed in each line which have the pipe(|) as the delimeter
LIFE
STD
LTD
LTD
LTD
-> cut -d\| -f3,4 test.txt
the above command will return the 3rd and 4th filed from each line

LIFE|A
STD|B
LTD|C
LTD|X
LTD|E

-> cut -d\| -f3,5 test.txt

the above command only gives 3rd filed bcz there is no 5th filed

LIFE

STD

LTD

LTD

LTD

-> cut -d\| -f1- test.txt

the above command will return data from filed1 to the end of each line.

201200001|201200001|LIFE|A
201200002|201200002|STD|B
201200003|201200003|LTD|C
201200004|201200004|LTD|X
201200005|201200005|LTD|E
-> cut -d\| -f-3 test.txt
the above command will return data from field1 to field3 from each line.

201200001|201200001|LIFE
201200002|201200002|STD
201200003|201200003|LTD
201200004|201200004|LTD
201200005|201200005|LTD

Find If the file is in Unix format or DOS format :

We can find by using the 'od' command and 'grep' for carriage returns('\r') that usually exist in DOS format files.
Example : od -c filename.txt| grep'\\r'
if it won't show up anything then it is in Unix format.
Or we can use the following method also :
Example : grep -c '^M$' filename 
Note that ^M has to be typed with control-v and enter(Not by typing shift 6 and M)

How to Find the changes made by particular USER in DataStage :

How to Find the changes made by particular USER in Datastage :

Here is how -

•  Go to Data stage Designer
• from the Menu bar  Tools --> Advanced Find

Then Select filelds based upon your choice:

• Name To find : * or part of the job name
• Folder to search : select the particular folder you are looking for
• Modified by User: Enter the user name of the datastage user.
• Date  : select the date type

Then click Find.

It will give you the list of jobs modified by that particular USER.