Puzzled !! Maximum DataStage jobs can be created in a Project?

Puzzled !! Maximum DataStage jobs can be created in a Project?

Have you ever puzzled about number of Jobs that can fit in a DataStage project – Here is the answer on How much space is required to accommodate as well

DataStage runs on a Host Operating System and runs a file system

Said that, Filesystems have a hard limit on number of sub directories that can be contained in a directory, for example around 32 K for Journaled File System on AIX, EXT3 on Linux and VXFS/HFS on HP-UX.

For every job, DataStage will create hash files as sub directories in the project directory. Here is a list of the sub directories created for every job. In this list “XX” represents the number (or id) of the job:

DS_TEMPXX
RT_CONFIGXX
RT_STATUSXX
RT_LOGXX
RT_BPXX
RT_BPXX.O
RT_SCXX (only for parallel jobs)

As the number of sub directories in a directory is limited to 32766, the maximum number of jobs for a project will be, in theory, approximately 4600 parallel jobs (32766/7 – 80 where 80 is for overhead of project default directory) or 5380 server jobs (32766/6 – 80). In practice, the number might be smaller due to sub directories not counted above, such as directories created by QualityStage jobs.

When the maximum number of sub directories is reached in the project directory, an error similar to below can occur when trying to save a newly created job:

Error On CREATE.FILE command
Creating file “RT_CONFIGXX ** as type 30, mkdbfile: connot create file
RT_CONFIG4817. Unable to create operating system file “RT_CONFIG4817”

** XX is the job number of the newly created job.

Solution:

To identify if the limit of sub directories in the project directory has been reached, run the command below.

mkdir filetest

if above command fails, you reached max limit

  1. If the maximum number of sub directories is reached, consider:

Exporting jobs in the project category by category, reallocate jobs into new projects;

  • To be frank – Managing a project with 4000+ jobs is really a Hard task :)

Please reach out to me if you still have questions

Author: Kuntamukkala Ravi

ETL Consultant by Profession, Webmaster by Passion

Leave a Reply

Your email address will not be published. Required fields are marked *