InsightIQ can export two kinds of statistics:

  • cluster performance statistics
    using the command iiq_data_export perf export
  • file system analytic’s
    using the command: iiq_data_export fsa export

In this exercise we will focus on FSA statistics

Note: the following commands where tested on InsightIQ 3.1

iiq_data_export fsa list

iiq_data export fsa list usage/switches:

iiq_data_export fsa list {–clusters | –data-modules | –breakouts | –filters | –reports }

options:

-h, –help
Show this help message and exit.

Mutually Exclusive Options:
–clusters
Displays the names of all clusters that InsightIQ is monitoring.

–breakouts
Displays the names of all breakouts that InsightIQ supports for file system analytic data modules for the specified cluster. Each data module supports a subset of breakouts.

–data-modules
Displays the names of all data modules that InsightIQ supports for file system analytic reports. Each data module supports a subset of breakouts.

–filters
Displays the names of all filters that have been created for the specified cluster.

–reports
Displays the names of all available file system analytic reports that have been created for the specified cluster.

Example output:
The following command displays the names of all file-system reports created for cluster1:

iiq_data_export fsa list –reports cluster1

Available Reports for: cluster1 Time Zone: EST

ID   FSA Job Start                     FSA Job End                    Size
—————————————————————————————————-
001  Sep 01 2015, 10:00 PM  Sep 01 2015, 11:21 PM  150.12M
002  Sep 02 2015, 10:00 PM  Sep 02 2015, 10:36 PM  152.12M
003  Sep 03 2015, 10:00 PM  Sep 03 2015, 11:06 PM  151.88M
004  Sep 04 2015, 10:00 PM  Sep 04 2015, 10:55 PM  154.12M
005  Sep 05 2015, 10:00 PM  Sep 05 2015, 11:12 PM  153.00M
006  Sep 06 2015, 10:00 PM  Sep 06 2015, 11:10 PM  152.50M
007  Sep 07 2015, 10:00 PM  Sep 07 2015, 01:17 AM  156.12M
008  Sep 08 2015, 10:00 PM  Sep 08 2015, 11:03 PM  152.00M
—————————————————————————————————-

iiq_data_export fsa export

iiq_data export fsa export usage/switches:

iiq_data_export fsa export [-h] -c -d -o [-b ] [-m ] [-n ] [-p ] [-r [: [: …]]] [-z ]

Required switches:

-c , –cluster
Exports data about the specified cluster.

-d , –data-module
Exports the specified data module.
The following module values are valid:

Module description Module Value
File Count by Logical Size file_count_by_logical_size
File Count by Last Modified file_count_by_modified_time
File Count by Physical Size file_count_by_physical_size
Top Directories top_directories
Top Files top_files

-o , –report
Exports data contained in the specified report.

Option switches:

-h, –help
Show this help message and exit.

-b , –breakout-by
The following breakout values are valid:

Breakout description Breakout Value
Accessed Time atime
Created Time btime
Modified Time ctime
Directory directory
Disk Pool disk_pool
File Extension path_ext
Logical Size log_size
Physical Size phys_size
User Attribute attribute

-m , –number-breakouts
Specifies the number of breakout components to include for each breakout. For example, if you broke out a data module by directory, specifying –number-breakouts 5 would export data about the top 5 directories. The default number is 12.

-n , –name
Specifies a name for the .csv file.
A time stamp is appended to the specified name. If no name is specified, the file is named according to the following pattern:
_[_][-]_.csv

-p , –path
Specifies a directory to store the .csv file.

-z , –comp-report
Instead of exporting the data contained in the report specified by the –report option, exports the comparison between the older report and the newer report specified by the –report and –comp-report options.

-r [: [: …]], –filter-rule [: [: …]] Applies the specified filter rule to the exported data. To specify a filter, you must specify the breakout that the filter applies to
The following filter values are valid:

Breakout description Breakout value Filter values
Accessed Time atime -60
Specifies 0:00:00 – 0:01:00
-3600
Specifies 0:01:00 – 1:00:00
-86400
Specifies 1:00:00 – 1 day
-604800
Specifies 1 day – 7 days
-2592000
Specifies 7 days – 30 days
-5184000
Specifies 30 days – 60 days
-7776000
Specifies 60 days – 90 days
-15552000
Specifies 90 days – 180 days
-31536000
Specifies 180 days – 365 days
-63072000
Specifies 365 days – 730 days
-1328431946
Specifies 730 days – Infinite
Example:
–filter-rule atime:-3600
Modified Time ctime -60
Specifies 0:00:00 – 0:01:00
-3600
Specifies 0:01:00 – 1:00:00
-86400
Specifies 1:00:00 – 1 day
-604800
Specifies 1 day – 7 days
-2592000
Specifies 7 days – 30 days
-5184000
Specifies 30 days – 60 days
-7776000
Specifies 60 days – 90 days
-15552000
Specifies 90 days – 180 days
-31536000
Specifies 180 days – 365 days
-63072000
Specifies 365 days – 730 days
-1328431946
Specifies 730 days – Infinite
Example:
–filter-rule ctime:-3600
Directory directory The path of a directory starting with /ifs
Example:
–filter-rule directory:/ifs/dir1
Disk Pool disk_pool The name of a disk pool
Example:
–filter-rule disk_pool:pool1
File Extension path_ext The name of a file extension
Example:
–filter-rule path_ext:(pdf)
Logical Size log_size A range of sizes in the following format:

0
Specifies 0 byes – 8 KB
8192
Specifies 8 KB – 128 KB
131072
Specifies 128 KB – 1 MB
1048576
Specifies 1 MB – 10 MB
10485760
Specifies 10 MB – 100 MB
104857600
Specifies 100 MB – 1 GB
1073741824
Specifies 1 GB – 10 GB
10737418240
Specifies 10 GB – 100 GB
107374182400
Specifies 100 GB – 1 TB
1099511627776
Specifies 1 TB – Infinite
Example:
–filter-rule log_size:1GB-10GB
Physical Size phys_size A range of sizes in the following format:

0
Specifies 0 byes – 8 KB
8192
Specifies 8 KB – 128 KB
131072
Specifies 128 KB – 1 MB
1048576
Specifies 1 MB – 10 MB
10485760
Specifies 10 MB – 100 MB
104857600
Specifies 100 MB – 1 GB
1073741824
Specifies 1 GB – 10 GB
10737418240
Specifies 10 GB – 100 GB
107374182400
Specifies 100 GB – 1 TB
1099511627776
Specifies 1 TB – Infinite
Example:
–filter-rule phys_size: 1GB-10GB
User Attribute attribute The name of a user attribute defined on the cluster.
All characters are valid.

Example of exporting InsightIQ file system analytics (FSA) modules to CSV:

Module: File Count query
#iiq_data_export fsa export –cluster Cluster1–data-module file_count_by_physical_size –report 002

Output:
Successfully exported data to:
file_count_by_physical_size_Cluster1_002_1442429501.csv

#cat file_count_by_physical_size_Cluster1_002_1442429501.csv

[no_filters],Totals,0 – 8192 bytes (file_count),8192 – 131072 bytes (file_count),131072 – 1048576 bytes (file_count),1048576 – 10485760 bytes (file_count),10485760 – 104857600 bytes (file_count),104857600 – 1073741824 bytes (file_count),1073741824 – 10737418240 bytes (file_count),10737418240 – 107374182400 bytes (file_count),107374182400 – 1099511627776 bytes (file_count),1099511627776 – Infinite bytes (file_count),report_date: 1442368827,ALL,4487319,649,56929,349275,3971841,34802,24204,27153,18833,3623,10

Output after formatting:

Totals 4,487,319
0 – 8192 bytes (file_count) 649
8192 – 131072 bytes (file_count) 56,929
131072 – 1048576 bytes (file_count) 349,275
1048576 – 10485760 bytes (file_count) 3,971,841
10485760 – 104857600 bytes (file_count) 34,802
104857600 – 1073741824 bytes (file_count) 24,204
1073741824 – 10737418240 bytes (file_count) 27,153
10737418240 – 107374182400 bytes (file_count) 18,833
107374182400 – 1099511627776 bytes (file_count) 3,623
1099511627776 – Infinite bytes (file_count) 10

Module: Top directories size
(by default it list 12 directories only unless you use –m then you can specify up to 1,000 directories)
# iiq_data_export fsa export –cluster Cluster1–data-module top_directories –report 002

Output:
Successfully exported data to:
top_dirs_btime_Cluster1_002_1442430835.csv

#cat top_dirs_btime_Cluster1_002_1442430835.csv

path:[no_filters],atime (unix),btime (unix),ctime (unix),report_date: 1442368827 Cluster1/dir01/dir01,1441783031,1415174737,1441783031
Cluster1/dir01/dir02,1441781797,1415174364,1441781797
Cluster1/dir01/dir03,1441778365,1415174022,1441778365
Cluster1/dir01/dir04,1441767600,1415173808,1441767600
Cluster1/dir01/dir05,1441778832,1415173760,1441778832
Cluster1/dir01/dir06,1441773114,1415173755,1441773114
Cluster1/dir01/dir07,1441782034,1415173381,1441782034
Cluster1/dir01/dir08,1441777596,1415173261,1441777596
Cluster1/dir01/dir09,1441770935,1415171817,1441770935
Cluster1/dir01/dir10,1441781832,1415170648,1441781832
Cluster1/dir01/dir11,1442366094,1415170066,1442366094
Cluster1/dir01/dir12,1441765803,1415169922,1441765803

Output after formatting:

Directory (Sorted descending by Create time) access time (atime) atime (converted to date) atime (days) created time (btime) btime (converted to date) btime (days) modify time (ctime) ctime (converted to date) ctime (days)
Cluster1/dir01/dir01 1441783031 9/9/15 7:17 AM 6.70 1415174737 11/5/14 8:05 AM 314.66 1441783031 9/9/15 7:17 AM 6.70
Cluster1/dir01/dir02 1441781797 9/9/15 6:56 AM 6.71 1415174364 11/5/14 7:59 AM 314.67 1441781797 9/9/15 6:56 AM 6.71
Cluster1/dir01/dir03 1441778365 9/9/15 5:59 AM 6.75 1415174022 11/5/14 7:53 AM 314.67 1441778365 9/9/15 5:59 AM 6.75
Cluster1/dir01/dir04 1441767600 9/9/15 3:00 AM 6.88 1415173808 11/5/14 7:50 AM 314.67 1441767600 9/9/15 3:00 AM 6.88
Cluster1/dir01/dir05 1441778832 9/9/15 6:07 AM 6.74 1415173760 11/5/14 7:49 AM 314.67 1441778832 9/9/15 6:07 AM 6.74
Cluster1/dir01/dir06 1441773114 9/9/15 4:31 AM 6.81 1415173755 11/5/14 7:49 AM 314.67 1441773114 9/9/15 4:31 AM 6.81
Cluster1/dir01/dir07 1441782034 9/9/15 7:00 AM 6.71 1415173381 11/5/14 7:43 AM 314.68 1441782034 9/9/15 7:00 AM 6.71
Cluster1/dir01/dir08 1441777596 9/9/15 5:46 AM 6.76 1415173261 11/5/14 7:41 AM 314.68 1441777596 9/9/15 5:46 AM 6.76
Cluster1/dir01/dir09 1441770935 9/9/15 3:55 AM 6.84 1415171817 11/5/14 7:16 AM 314.70 1441770935 9/9/15 3:55 AM 6.84
Cluster1/dir01/dir10 1441781832 9/9/15 6:57 AM 6.71 1415170648 11/5/14 6:57 AM 314.71 1441781832 9/9/15 6:57 AM 6.71
Cluster1/dir01/dir11 1442366094 9/16/15 1:14 AM -0.05 1415170066 11/5/14 6:47 AM 314.72 1442366094 9/16/15 1:14 AM -0.05
Cluster1/dir01/dir12 1441765803 9/9/15 2:30 AM 6.90 1415169922 11/5/14 6:45 AM 314.72 1441765803 9/9/15 2:30 AM 6.90

Note:
* To convert unix time to data use the following formula (unixtime/(60*60*24)+”1/1/1970″
* to calculate the number of days we deducted the unix date from this report date which is 9/16/2015

Module: Top files size
(by default it list 12 files only unless you use –m then you can specify up to 1,000 files)
#iiq_data_export fsa export –cluster Cluster1–data-module top_files –report 002
Output:
Successfully exported data to:
top_files_btime_Cluster1_002_1442433280.csv

#cat top_files_btime_Cluster1_002_1442433280.csv

path:[no_filters],atime (unix),btime (unix),ctime (unix),log_size (bytes),phys_size (bytes),report_date: 1442368828 Cluster1/dir01/dir01/file01.pdf,1305942888,1421848937,1421862576,207378,780800
Cluster1/dir01/dir01/file02.pdf,1305939742,1421848937,1421862576,253591,821760
Cluster1/dir01/dir01/file03.pdf,1305941284,1421848936,1421862576,262503,838144
Cluster1/dir01/dir01/file04.pdf,1305904200,1421848936,1421862576,1241200,1812992
Cluster1/dir01/dir01/file5.pdf,1305906624,1421848936,1421862576,2680124,3779072
Cluster1/dir01/dir01/file06.pdf,1421113620,1421848935,1421862576,710,43520
Cluster1/dir01/dir01/file07.pdf,1305942306,1421848935,1421862576,207378,780800
Cluster1/dir01/dir01/file08.pdf,1417556492,1421848935,1421862576,770,43520
Cluster1/dir01/dir01/file09.pdf,1417556361,1421848935,1421862576,633,43520
Cluster1/dir01/dir01/file10.pdf,1305907374,1421848934,1421862576,207378,780800
Cluster1/dir01/dir01/file11.pdf,1305944776,1421848934,1421862577,2748940,3844608
Cluster1/dir01/dir01/file12.pdf,1305944776,1421848933,1421862577,262503,838144

Output after formatting:

Directory (Sorted descending by Create time) access time (atime) atime (converted to date) atime (days) created time (btime) btime (converted to date) btime (days) modify time (ctime) ctime (converted to date) ctime (days) log_size (bytes) phys_size (bytes)
Cluster1/dir01/dir01/file01.pdf 1305942888 5/21/11 1:54 AM 1578.92 1421848937 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 207,378 780,800
Cluster1/dir01/dir01/file02.pdf 1305939742 5/21/11 1:02 AM 1578.96 1421848937 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 253,591 821,760
Cluster1/dir01/dir01/file03.pdf 1305941284 5/21/11 1:28 AM 1578.94 1421848936 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 262,503 838,144
Cluster1/dir01/dir01/file04.pdf 1305904200 5/20/11 3:10 PM 1579.37 1421848936 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 1,241,200 1,812,992
Cluster1/dir01/dir01/file05.pdf 1305906624 5/20/11 3:50 PM 1579.34 1421848936 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 2,680,124 3,779,072
Cluster1/dir01/dir01/file06.pdf 1421113620 1/13/15 1:47 AM 245.93 1421848935 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 710 43,520
Cluster1/dir01/dir01/file07.pdf 1305942306 5/21/11 1:45 AM 1578.93 1421848935 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 207,378 780,800
Cluster1/dir01/dir01/file08.pdf 1417556492 12/2/14 9:41 PM 287.10 1421848935 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 770 43,520
Cluster1/dir01/dir01/file09.pdf 1417556361 12/2/14 9:39 PM 287.10 1421848935 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 633 43,520
Cluster1/dir01/dir01/file10.pdf 1305907374 5/20/11 4:02 PM 1579.33 1421848934 1/21/15 2:02 PM 237.42 1421862576 1/21/15 5:49 PM 237.26 207,378 780,800
Cluster1/dir01/dir01/file11.pdf 1305944776 5/21/11 2:26 AM 1578.90 1421848934 1/21/15 2:02 PM 237.42 1421862577 1/21/15 5:49 PM 237.26 2,748,940 3,844,608
Cluster1/dir01/dir01/file12.pdf 1305944776 5/21/11 2:26 AM 1578.90 1421848933 1/21/15 2:02 PM 237.42 1421862577 1/21/15 5:49 PM 237.26 262,503 838,144

Note:
* To convert unix time to data use the following formula (unixtime/(60*60*24)+”1/1/1970″
* to calculate the number of days we deducted the unix date from this report date which is 9/16/2015