Working with Lists in Python

Lists in Python

Like an array, List is a collection of data (of different data types). Lists are specified using square brackets. 

Lists are very helpful when working with data in Python. Below are some of the features of Lists. 

• Lists can hold data of different data types. 
• Lists can be amended.
• Data in the list can be accessed by using index.
• Lists can hold duplicate data. 

In this post, we will see how to access, append or remove the data in Lists and how to use different list methods. 

Creating List in Python

Before we go on to different methods of lists, Let us see How to create a List in Python?

Like any other variable in python, List is created when the data is assigned with in square brackets. 

In the above example, 'a_list' is created with the data and 'b_list' is created with no data. 

Accessing the data in a List

How to access the individual elements of Lists? Elements in the Lists can be accessed using index. Index starts with '0' and incremented by '1'. 

Elements of a List can be accessed from the last element with the negative index value. As we can see above, Index of last element would be '-1'. 

If we need to retrieve the second element ("TWO"), index '1' or '-4' needs to be mentioned within square brackets. 

E.g.: 

print(a_list[1]) #prints the second element of a list

print(a_list[-4]) #prints the fourth element from end of a list

Retrieve Index of an element

We have seen how to access the elements using an index, If we know the element and need to know the index, Method 'index()' would be helpful. One thing to note is, it always returns the first index if the element is present multiple times in the list. 

In the above example, '2' is present twice in the list. But, only one index (first occurrence '1') will be returned.

Adding data to a List

One of the features of a List is the data can be amended. Using append() method, we can add the data to a list. data that needs to be added is to be passed as a parameter to append() method. 

In the above example, we are passing "FOUR" to the method 'append()'. This should be added towards end of the list. 

If we notice, "FOUR" is already present in the list. Python allows duplicate data in the list and "FOUR" should be added successfully and printed like the below. 

In the above example, we are only adding one element at a time. If we have another list that needs to be added, this needs to be done with in a loop to add all of these elements using append. What if the list have large number of elements? 

Method 'extend()' can be used to append one list to the other. This method accepts a list (that is to be added) as an argument.

In the above example, data in b_list would be added to a_list. 

Removing data from a List

Data can be removed from a list by using remove() method. Similar to the append() method, data that is passed as a parameter would be removed from a List. 

In this example, we have two entries with 'FOUR'. And, first occurrence would be deleted from a list. 

Copy data from one List to other

Like copying data from one variable to other, copying one list to other becomes essential. 

Is copying data from one list to other as simple as copying data from one variable to other? 

Simple answer is 'No'. 

In the above example, 

• Line - 3: 'b_list = a_list', a_list doesn't actually copied over to b_list. b_list would be referencing a_list and any changes to a_list or b_list would reflect for both these. 
• Lines - 5 & 6: We are appending '6' to b_list and 'SEVEN' to a_list. 
• Lines - 8 & 9: Both lists should print the same result. 

To avoid this issue, using copy() method would help to copy the data from one list to other. Let's see how this works using the same example. 

• Line - 3: we are using 'a_list.copy()' to copy the data, copy() method doesn't require any arguments.

One other way to do this is by using function list(). This can be to copy the data from a list, tuple or by passing the data directly. 

In the above example, Lines from 5 to 8 should work in the same way and create list with the data passed.

Clear data from a List

Clearing data from a list can be done using 'clear()' method. This method doesn't require any arguments to be passed and clears the referencing list. 

Retrieve the number of occurrences of an element

We know that Python allows duplicate values in the list. If we need to know the number of occurrences of an element in a list, we can use method 'count()'. We need to pass the element value as an argument to this method.

In the above example, '2' is present '3' times. 

Retrieve & Delete the value by Index

In the beginning we have seen how to access the values in a list by using index. Let's say if we need to retrieve the value by index and delete this from the list, 'pop()' method is helpful for doing this. 

• 'pop()' accepts index as an argument and it returns the value in the specific index. 
• Once the value is deleted, Python would automatically adjust the index, value in the next position would now be referred with this index. 

In the above example,

• Line - 4: value '2' (at index 1) should be returned and deleted from the list.
• Line - 5: value '3' would now be at index '1' and should be returned and deleted from the list.

Sorting a List

Lists doesn't necessarily hold the data in the order (ascending or descending). But, if we need to sort the data in a list, Method 'sort()' is useful. With default parameters, this method sorts the data in ascending order and by passing 'reverse=True' data is updated in reverse order. And, method reverse() is useful to sort the data in reverse order.

In the above example, 

• Line-3: With no arguments passed, list would be sorted in ascending order.
• Line-8: with 'reverse=True', list would be sorted in descending order. passing 'reverse=False' would work like default and data is sorted in reverse order.
• Line-14: Data would be sorted in reverse order of index.

One thing to note here is, sort() would only work with data of same data type in a list. If there are different data types present (like int and str), sort() would fail.

The above 'sort()' method updates the same list. If we need to sort the data and move it to the new list (and leave the original as is), we can use the function 'sorted()'. This accepts a list as an argument and returns sorted list. There is an optional parameter 'reverse', by passing 'True' to this parameter would sort the data in descending order.

Hope the above details were a bit of help to you in understanding more about Lists in Python.

If you have any Suggestions or Feedback, Please leave a comment below or use Contact Form.

Create/Write Data to IFS file from SQL - IBM i

IFS Stream File

IFS (Integrated File System) stream files are major part of any (or most of) IBM i applications. 

There are various ways to create stream files like CPYFRMIMPF, CPYFRMSTMF, EDTF...

Creating the IFS file and writing the data has been made much easier with SQL procedure IFS_WRITE (IFS_WRITE_BINARY for writing the data in Binary format and IFS_WRITE_UTF8 for writing the data in UTF8 format). 

In this post, we will see how to write the data into Stream file directly from SQL using IFS_WRITE procedure. 

E.g.: 

To start with, What we need is 'Path Name (IFS Directory & File Name)' and 'Data to be written'. 

• First parameter 'PATH_NAME' accepts the IFS path name along with the file name.
• In this example, '/home/REDDYP' is the directory and 'NewTextFile.txt' is the file name. 
• Directory should be present by the time the procedure is called. If not, Procedure call would be failed. 
• File name doesn't necessarily need to be present. If the is not present, File would be created. If already present, Data will appended to the file by default.
• Second parameter 'LINE' accepts the data that is to be written into the file. 

In this example, file 'NewTextFile.txt' isn't already present. So, the file is created once the procedure is run. 

This looks fine when we look at the file from 5250 session. But, if we download the file and open in the Text editor, it appears like the below.

The reason behind this is CCSID used. By default IFS_WRITE creates the file with Job's default CCSID. If we need the file to be created with specific CCSID, we need to pass this using 'FILE_CCSID' parameter. 

• Parameter 'FILE_CCSID' accepts an integer value (CCSID) and will be used to create the stream file.
• This is only used when creating the new stream file and has no effect if the file is already present. 
• Default value for this parameter is '0' for IFS_WRITE. File will be created with Job's CCSID if '0' is specified (or default).
• Default value for this parameter is '1208' for IFS_WRITE_UTF8 and '0' for IFS_WRITE_BINARY.

This looks fine from 5250 session, let's see how this would look like if we download the file. 

So, If we need to create the file with the data (and replace if the file already exist), we can make use of the parameter 'OVERWRITE'. 

• Parameter 'OVERWRITE' determines how the data passed (in LINE parameter) to be added to the file. Below are the valid values for this parameter.
• APPEND - This is the default value for this parameter. Data is appended if the file already exist and File would be created if not already exist.
• REPLACE - Replace the data in the existing stream file with the data passed. If file isn't already exist, file would be created. 
• NONE - This is to be used if the new file needs to be created always. Procedure will fail if the file already present and new file would be created if the file is not present.

Passing 'APPEND' parameter or not passing this parameter would append the data to the file.

In the above example, New line has been written with the data passed. If we need to append the data on to the same line rather than creating new line, we can make use of 'END_OF_LINE' parameter. However, this parameter needs to be used for the line which needs data is added on to (rather than in the query that needs to be added). 

E.g.: 

If the data needs to be appended to Third Record, END_OF_LINE parameter needs to be passed with 'NONE' while writing this record. So that following line can be appended to the same record. 

• Parameter 'END_OF_LINE' determines what needs to be written at the end of the line. Below are the valid values for this parameter.
• CRLF - This is the default value for IFS_WRITE (and IFS_WRITE_UTF8). Carriage return and Line feed are appended to the end of the line.
• CR - Carriage return is appended to the end of the line.
• LFCR - Line feed and Carriage return are appended to the end of the line. 
• LF - Line feed is appended to the end of the line.
• NONE - No end of the line characters are appended i.e., Data appended will be added to the same record. This is the default value for IFS_WRITE_BINARY.

When the IFS_WRITE is called next time data is appended to the same line. 

Below is the Syntax of this procedure with all the parameters. 

CALL QSYS2/IFS_WRITE

     (PATH_NAME => 'IFS Path with directory and File Name', 

      LINE => 'Data to be written',

      FILE_CCSID => CCSID(Integer),

      OVERWRITE => 'Overwrite Option',

      END_OF_LINE => 'End of Line Option') 

All the above examples showing the data line by line into the file. Let's see how we can write the data from a file into the Stream file using IFS_WRITE. This would a bit of coding in SQL. 

In the above query, we are calling IFS_WRITE twice. Once to create the empty file (or replace with blank if file is already present) and Second time, it is called inside FOR loop to write each line from Table. Let's see each of the parameter in detail.

• Line - 4: Parameter 'LINE' with no data is to create the blank file. Next time IFS_WRITE is called is in the loop, so it is better to create the blank file first and append the data. 
• Line - 5: Parameter 'OVERWRITE' with 'REPLACE', This is to replace any data in the current file if the file is already present. 
• Line - 6: Parameter 'END_OF_LINE' with 'NONE', This is to append the data from the first line.
• Line - 9: FOR loop with SELECT query of table 'REDDYP1/IFSFILE'.
• Line - 10: IFS_WRITE with data to be written. This procedure is called for each row in the table and the data is added to the new row (with default value on OVERWRITE).

IFS_WRITE_BINARY and IFS_WRITE_UTF8 works in the same way only difference is data to be passed in Binary and UTF8 formats respectively. 

If you have any Suggestions or Feedback, Please leave a comment below or use Contact Form.

Read data in IFS file from SQL - IBM i

IFS Stream File

IFS (Integrated File System) Stream files are major part of any (or most of) IBM i applications. Having to read data from the Stream files become necessary. 

We can read the IFS file from Programs in two different ways. 

• By copying the data from IFS file to Database file (CPYFRMSTMF or CPYFRMIMPF).
• By using 'open' & 'read' procedures written in 'C' language. Click Here to see more about how to read IFS file using these procedures from RPGLE.

One other way to do this is by using SQL table function 'IFS_READ'. There are couple of more table functions 'IFS_READ_BINARY' and 'IFS_READ_UTF8' to read the data and return in Binary and UTF8 formats accordingly. 

In this post, we will see how to read IFS file using 'IFS_READ' and little about the other two table functions towards the end.

E.g.: 

Above query is simple and straight forward, All we need to do is pass the required file name (along with full path) against parameter 'PATH_NAME'.

This table function has got only two columns. 

• 'LINE_NUMBER' - Indicates the position of the line in file.
• 'LINE' - Actual data in the file.

Let's say if there is a limit on the number of characters to receive, We can control this using MAXIMUM_LINE_LENGTH parameter. 

• Parameter 'MAXIMUM_LINE_LENGTH' accepts numeric values.

If we look at the above result, each row contains only 12 digits and any additional characters has been returned as new line. 

Line number 1, 4 & 7 has length of 12 digits and has returned only one row. Line numbers 2 & 6 are exceeding 12 digits, so only 12 digits are returned and the rest are returned as new lines (3 & 6).

There is another important parameter 'END_OF_LINE' this is used to determine the end of line in Stream file. Let's have a look at how it makes difference before we see the list of valid values. 

In the above query, 'NONE' is passed as 'END_OF_LINE', Query will not consider any End of Line characters. 

All the records has been returned in a single row. Number of digits allowed in a single line can be controlled using 'MAXIMUM_LINE_LENGTH'. If nothing is specified, by default 2GB is returned in a single line and the next is returned in a new line. 

Below are the possible options for END_OF_LINE parameter.

• CRLF - A Carriage return and Line feed indicate End of line.
• LFCR - A Line feed and Carriage return indicate End of line.
• CR - A Carriage return indicate End of line.
• LF - A Line feed indicate End of line.
• ANY - Any of the above four indicate End of line (Default for IFS_READ & IFS_READ_UTF8).
• NONE - No End of line characters are considered (Default and only allowed value for IFS_READ_BINARY). 

Let's see a simple example of how to get the data in Binary format using IFS_READ_BINARY. 

Parameters are same as IFS_READ except 'END_OF_LINE' which always accepts 'NONE'. This implies that IFS_READ_BINARY always returns a single row (MAXIMUM_LINE_LENGTH restrictions apply).

In the same way, data can be retrieved in UTF8 format using IFS_READ_UTF8. 

Below is the syntax of IFS_READ with all the parameters.

SELECT * FROM

         TABLE(QSYS2/IFS_READ(

               PATH_NAME => 'ifs_path_name',

               MAXIMUM_LINE_LENGTH => maximum_length(in decimal),

               END_OF_LINE => 'end_of_line_characters'))

In all of the examples shared above returns a valid data. So, What happens if an incorrect path is passed? No data is received and query is completed with SQL Code '100'. 

If you have any Suggestions or Feedback, Please leave a comment below or use Contact Form.

Operators in Python

Operators

Operator is a symbol that is used to perform specific operations like mathematical, relational or logical operations. 

There are different types of operators. Below are some the most common operators

• Mathematical Operators.
• Assignment Operators.
• Relational Operators.
• Logical Operators.

Apart from these, there are couple of more types. 

• Identity Operators.
• Membership Operators.

Let's have a look at each of these. 

Mathematical Operators

Mathematical operators are used to perform mathematical operations (as name states) with numeric values. 

These are operators like '+' Addition, '-' Subtraction, '*' Multiplication, '/' Division, '%' Modulus, '//' Floor Division and '**' Exponentiation.

a = 10

b = 5

c = a + b 

# Addition of two variables, Result would be c = 15

c = a - b 

# Subtraction of second variable from first variable, Result would be c = 5

c = a * b 

# Multiplication of two variables, Result would be c = 50

c = a / b 

# Division, Divides first variable with second variable, Result would be c = 2.0. Result for the division operation would always be float.

c = a % b 

# Modulus, Returns the reminder after dividing first variable with second variable. Result would be c = 0. If a = 11, b = 5, then Result would be c = 1

c = a // b 

# Floor Division, Divides first variable (a) with second variable (b) and returns the whole number only. Result would be c = 2

c = a ** b 

# Exponentiation, Calculating the first variable (a) with the power of second variable (b), Result would be c = 10000.

Assignment Operators

Assignment Operators are used to assign value to a variable. 

One of the assignment operators is '=' (equal to) to assign a value to a variable. In python, a variable is declared when a value is assigned. 

a = 10 

# This defines a integer variable 'a' and assigns a value '10'

Below are the few other assignment operators. 

*Considering the current value of 'a' as '10' for all of these examples.

'+=' Adds the value on the right side to the current value of the variable. 

a += 5 

# adds 5 to the current value of 'a' (i.e., 10) and assigns it to 'a'. This is same as 'a = a + 5'.

'-=' Subtracts the value on the right side from the current value of the variable. 

a -= 5 

# subtracts 5 from the current value of 'a' (i.e., 10) and assigns it to 'a'. This is same as 'a = a - 5'.

  

'*=' Multiplies the value on the right side with the current value of the variable. 

a *= 5 

# multiplies 5 with the current value of 'a' (i.e., 10) and assigns it to 'a'. This is same as 'a = a * 5'.

'/=' Divides the current value of the variable with the value on the right side. 

a /= 5

# divides the current value of 'a' (i.e., 10) with 5 and assigns to 'a'. This is same as 'a = a / 5'.

'%=' Divides the current value of the variable with the value on the right side to calculate the remainder. 

a %= 5 

# divides the current value of 'a' (i.e., 10) with 5 and assigns the remainder to 'a'. This is same as 'a = a % 5'.

'//=' Divides the current value of the variable with the value on the right side and assign the resulting whole number. 

a //= 5 

# (Floor) divide the current value of 'a' (i.e., 10) with '5' and assigns it to 'a'. This is same as 'a = a // 5'.

'**=' Calculate the exponent of the variable to the power of value on the right side. 

a **= 5 

# Calculate the current value of 'a' (i.e., 10) to the power of 5 and assigns it to 'a'. This is same as 'a = a ** 5'.

Relational Operators

Relational Operators are used to compare the relation between two variables (e.g. greater than, less than, equal to...).

Below are the Relational operators. 

• Equal To (==), used to check if the variables on either sides are equal (a == b). Returns True if equal and False if not equal.
• Not Equal To (!=), used to check if the variables on either sides are not equal (a != b). Returns True if not equal and False if equal.
• Greater than (>), used to check if the left side variable is greater than the right side variable (a > b). Returns True if greater and False if not greater. 
• Greater than or Equal to (>=), used to check if the left side variable is either greater or equal to right side variable (a >= b). Returns True if greater than or equal and False if not.
• Less than (<), used to check if the left side variable is less than the right side variable (a < b). Returns True if less and False if not less. 
• Less than or Equal to (<=), used to check if the left side variable is either less or equal to right side variable (a <= b). Returns True if less than or equal and False if not.

Logical Operators

Logical operators are used to combine multiple conditions and returns if the result is True or False. 

• 'and', returns True if conditions on either sides are True and returns False if at least one of them is False.

(a > b) and (c > d)

# if both conditions (a > b) and (c > d) are True, then the result would be True. If at least one of them is False, then result would be False.

• 'or', returns True if one of the condition is True and returns False if both of them are False. 

(a > b) or (c > d)

# if at least one of the condition (a > b) or (c > d) are True, then result would be True. If both conditions are False, then result would be False. 

• 'not', returns the opposite value of the current condition. If the condition is True, not would return False and vice versa.

not (a > b) 

# if (a > b) is True, then the result would False. if (a > b) is False, then result would be True.

Identity Operators

Identity Operators are used to identify if a value belongs to particular class or type. 

• 'is', returns True if the type (or value) on the left side matches with the type (or value) mentioned on the right side. 

E.g.:

a = 10

print(type(a) is int) # prints True, as the data type of 'a' is int and the condition is satisfied. 

a = 10

b = 11 

print(a is b) # prints False, as the values of 'a' and 'b' aren't same.

• 'is not', returns False if the type (or value) on the left side doesn't match with the type (or value) mentioned on the right side.

E.g.:

a = 10

print(type(a) is not int) # prints False, as the data type of 'a' is int and the condition is not satisfied. 

a = 10

b = 11 

print(a is not b) # prints True, as the values of 'a' and 'b' aren't same.

Membership Operators

Membership operators are used to check if the value on the left side is part of the sequence (list or tuples). 

• 'in', returns True if the left side value is part of sequence mentioned on the right side. returns False if not part of sequence mentioned on the right side.

E.g.:

a = [1, 2, 3]

b = 2

print(b in a) # prints True as '2' is part of list 'a'

• 'not in', returns False if the left side value is part of sequence mentioned on the right side. returns True if not part of sequence mentioned on the right side.

E.g.:

a = [1, 2, 3]

b = 2

print(b not in a) # prints False as '2' is part of list 'a' 

If you have any Suggestions or Feedback, Please leave a comment below or use Contact Form.

Retrieve Objects present in IFS Directory from SQL - IBM i

Objects in IFS Directory

Working with IFS (Integrated File System) is essential and almost part of day to day work for most of the IBM i developers.  

In my experience below are the couple tasks I had to do frequently. 

• Check if an object is present in IFS directory. 
• Retrieve list of objects present in IFS directory. 

This has been made easier to do these now (and much more) from SQL directly with 'IFS_OBJECT_STATISTICS' table function. 

What does IFS_OBJECT_STATISTICS table function return? It returns the attributes of an IFS Object for every object (including directory) present in the IFS Path provided. 

Let's now have a look at how to retrieve the objects present for a specified path. 

In the above query,

• Line - 1: Column 'PATH_NAME' returns the IFS Path (or Object) the row is returned for. 
• Line - 2: Column 'OBJECT_TYPE' returns the type of object retrieved. 
• Line - 3: Column 'CREATE_TIMESTAMP' returns the timestamp when the object was created. 
• Line - 5: 'IFS_OBJECT_STATISTICS' is the table function that retrieves the data. 
• Line - 6: Parameter 'START_PATH_NAME' is to pass the IFS Directory or Object Name. 

Looking at the above result, 

• PATH_NAME contains,
• Path passed in the parameter 'home/REDDYP'.
• All Stream files under directory 'home/REDDYP' (accessibmi.py, Sample1.csv etc).
• Sub Directories under the path passed (/home/REDDYP/SubDirectory).
• Files present in the Subdirectory (/home/REDDYP/SubDirectory/Sample_Textfile.txt). 
• OBJECT_TYPE contains the type of the Object returned i.e., *DIR (for directory), *STMF (stream file) etc. 
• CREATE_TIMESTAMP contains the timestamp the Object or Directory is created on.
• There are many other columns IFS_OBJECT_STATISTICS would return. You can explore the full list of columns by running the query with '*' instead of column names. 

The above query returned sub directories and objects inside sub directory. It is helpful to know all the objects present in the sub directories as well if there aren't many sub directories and objects in sub directories. 

With the use of 'SUBTREE_DIRECTORIES' parameter, we can control if we need to display the objects of sub directory or not. This would become helpful if there are many objects and sub directories. 

In the above query, Additional parameter SUBTREE_DIRECTORIES accepts two parameters.

• YES (Default) - This would retrieve all the objects inside Sub Directory.
• NO - This would not retrieve the objects inside Sub Directory (Would list any Sub Directory under the path specified).

There is another important parameter I would like to mention is 'OBJECT_TYPE_LIST'. This parameter can be used to verify or retrieve any the IFS Objects of specific type. 

In the above example, We can see there are two different object types *DIR and *STMF. Let's say If I'm only interested in *STMF and not the directories. then this parameter is very much helpful. 

Above query would only return objects of type '*STMF'. So, what if we need to select from multiple types? All the required object types can be specified by separating with space.

OBJECT_TYPE_LIST => '*STMF *DIR'

This parameter accepts some special values to indicate group of object types along with specific object types. 

• *ALLDIR - All directory types (*LIB, *DIR, *FLR, *FILE and *DDIR)
• *ALLSTMF - All stream file types (*MBR, *DOC, *STMF, *DSTMF and *USRSPC)
• *MBR - All database file member types.
• *NOQDLS - All object types except QDLS object types.
• *NOQOPT - All object types except QOPT and QNTC object types.
• *NOQSYS - All object types except QSYS.LIB object types.

There are couple of more parameters and below is the brief about these parameters.

OMIT_LIST

List of path names to exclude from processing. All objects and sub directories under this path would be excluded as well. Multiple path names can be specified by separating with blanks. There are couple of exceptions to this.

• If there are spaces exist in the path name then path name to be enclosed in apostrophes or quotes. 

    OMIT_LIST => '/home/REDDYP/Sub Directory'

• If there is an apostrophes or quotes in the path name, the embedded character needs to be doubled. 

       OMIT_LIST => '"/home/REDDYP/Memeber''s Data" "/home/REDDYP/Sub Directory"'

       In the above example,

• First path '/home/REDDYP/Memeber's Data' has apostrophe. So, this needs to be doubled (Member''s Data)
• Both the paths are enclosed in quotes. Using the quotes is the best practice when dealing with multiple paths, it is easy to understand and maintain.

IGNORE_ERRORS

This is more of a error handling, Advising the system on what to do in case of an error while executing the query. 

• NO - An error is returned.
• YES (default) - A warning is returned and no data is returned when error is occurred. 

Query looks like below if we use all the parameters.

SELECT * FROM TABLE(IFS_OBJECT_STATISTICS(

                    START_PATH_NAME => 'Start Path Name'

                    SUBTREE_DIRECTORIES => 'YES/NO',

                    OBJECT_TYPE_LIST => 'List of Object Types',

                    OMIT_LIST => 'Paths to be Omitted',

                    IGNORE_ERRORS => 'YES/NO'))  

There is one thing I haven't mentioned is "How to check if an object is present in IFS directory?" Well, this isn't the actual purpose of the function. But, we can achieve this with a query like the below.

This query returns '1' if the object (/home/REDDYP/hello.py) is present and returns '0' if not present. 

If you have any Suggestions or Feedback, Please leave a comment below or use Contact Form.