Pulling in Qualys data with python and Talend Studio, so it can be dashboarded in excel :(

First you enable ticket policy.
For some reason somebody forgot to include tickets in version 2 of the API, you can only see ticket data with version 1 of the API, and you cannot set the truncate limit. That means you need to program extra logic to iterate over the entire dataset burning up API requests. Every smart person I spoke to regarding this cannot understand this illogical step in the improvement of their product. To top it off, try and download the KB data, and the API may say no… but the web portal says yes.
Well I was left with no choice but to do the horrible act of webscraping, thankfully I was able to use the python extension in visual studio and take advantage of some intellisence. So after installing selenium python libraries and firefox, I was able to make some progress and build up a job for grabbing the ticket and KB data in one request’ish.
I’m not a data analyst in fact it’s not in my personality, I can do it, but it’s just not fun. So thanks to Andrew for the recommendation to use Talend Studio to do the legwork, I mean every time I use it, I realise just how awesome it is.
So then the plan is to use my python script to webscrape some data, then in talend studio I call that script, make a webrequest to the API for some other data, and then with some data mapping and a custom IP to host mapping file I can pull all this data together into a nice big flatfile for Management. Of course I’m a few KPI short but not bad for a start.
My folder structure is;
D:\Data\Qualys
\Xml
\maps
\ExternalJob
\Documentation



The Python code is below;
# Requirements
#
# Install Python 2.7.9 or greater, added to path variable.
# Install additional python libraries;
# C:\Python27\Scripts\pip.exe install selenium
# C:\Python27\Scripts\pip.exe install bcrypt
# Install firefox 43.0.1 or higher
# Invoke this script from talend studio job; C:\Python27\python.exe D:\Data\Qualys\ExternalJob\QualysDownloadSelenium.py
# Created 22-02-2016
# Last Tested 09-03-2016
# Script likely to break upon password expiry and or changes to the qualys site
# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import unittest, time, re, os
import base64
SAVE_TO_DIRECTORY = "D:\\Data\\Qualys\\Xml\\" # change this to the correct path
userAccount = "Z29kbW9kZQ==" # change for real account, base64 to stop shoulder snoops
Password = "dTgwMDhwYXNzd2Rub3RsaWtlbHl0b2JlaGVyZS4uLi4uMTIz" # change for real password, base64 to stop shoulder snoops
tmpAcc = base64.b64decode(userAccount)
spltUser = tmpAcc.split(r'-')
tmpAcc = ""
tmpKBfile = SAVE_TO_DIRECTORY+"DL_vulnerabilities_"+spltUser[0]+"_"+spltUser[1]+"_"+time.strftime("%Y%m%d")+".xml"
tmpTLfile = SAVE_TO_DIRECTORY+"DL_tickets_"+spltUser[0]+"_"+spltUser[1]+"_"+time.strftime("%Y%m%d")+".xml"
docName = "2tmpVulns"
docName2 = "2tmpTickets"
# check for files we don't want to trip over
if os.path.isfile(SAVE_TO_DIRECTORY+docName+".xml"):
os.remove(SAVE_TO_DIRECTORY+docName+".xml")
if os.path.isfile(SAVE_TO_DIRECTORY+docName2+".xml"):
os.remove(SAVE_TO_DIRECTORY+docName2+".xml")
if os.path.isfile(tmpKBfile):
os.remove(tmpKBfile)
if os.path.isfile(tmpTLfile):
os.remove(tmpTLfile)
profile = webdriver.FirefoxProfile()
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.dir", SAVE_TO_DIRECTORY)
profile.set_preference("browser.download.manager.alertOnEXEOpen", False);
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "text/xml, application/msword, application/csv, application/ris, text/csv, image/png, application/pdf, text/html, text/plain, application/zip, application/x-zip, application/x-zip-compressed, application/download, application/octet-stream" )
profile.set_preference("browser.helperApps.neverAsk.openFile", "text/xml, application/msword, application/csv, application/ris, text/csv, image/png, application/pdf, text/html, text/plain, application/zip, application/x-zip, application/x-zip-compressed, application/download, application/octet-stream" )
profile.set_preference("browser.download.manager.showWhenStarting", False);
profile.set_preference("browser.download.manager.focusWhenStarting", False);
profile.set_preference("browser.download.useDownloadDir", True);
profile.set_preference("browser.helperApps.alwaysAsk.force", False);
profile.set_preference("browser.download.manager.alertOnEXEOpen", False);
profile.set_preference("browser.download.manager.closeWhenDone", True);
profile.set_preference("browser.download.manager.showAlertOnComplete", False);
profile.set_preference("browser.download.manager.useWindow", False);
profile.set_preference("services.sync.prefs.sync.browser.download.manager.showWhenStarting", False);
profile.set_preference("pdfjs.disabled", True);
class QualysSeleniumScript(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox(firefox_profile=profile)
self.driver.implicitly_wait(30)
self.base_url = "https://qualysguard.qualys.eu"
self.verificationErrors = []
self.accept_next_alert = True
def test_qualys_selenium_script(self):
driver = self.driver
driver.get(self.base_url + "/")
driver.find_element_by_id("myform_UserLogin").clear()
driver.find_element_by_id("myform_UserLogin").send_keys(base64.b64decode(userAccount))
driver.find_element_by_id("myform_UserPasswd").clear()
driver.find_element_by_id("myform_UserPasswd").send_keys(base64.b64decode(Password))
driver.find_element_by_name("_form_action1").click()
driver.get(self.base_url + "/fo/tools/kbase.php")
time.sleep(3)
driver.find_element_by_id("ext-gen202").click()
driver.find_element_by_id("ext-gen248").click()
for i in range(60):
try:
if self.is_element_present(By.ID, "format_handle"): break
except: pass
time.sleep(1)
else: self.fail("time out")
driver.find_element_by_id("select_download_format_xml").click()
driver.find_element_by_id("dload_btn").click()
# should loop around watching for finsihed download KB
# DL_vulnerabilities_mdand_mk_20160309.xml
# DL_vulnerabilities_mdand_mk_20160309.xml.part
X = 0
while True:
time.sleep(5)
X += 1
if os.path.isfile(tmpKBfile):
if os.stat(tmpKBfile).st_size > 0:
break
if X > 60:
break # 60 iterations of 5 seconds will be 5 minutes
# rename downloaded file for KB.
os.chdir(SAVE_TO_DIRECTORY)
if os.path.isfile(docName+".xml"):
os.remove(docName+".xml")
files = filter(os.path.isfile, os.listdir(SAVE_TO_DIRECTORY))
files = [os.path.join(SAVE_TO_DIRECTORY, f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
newest_file = files[-1]
os.rename(newest_file, docName+".xml")
#Now we pull down the tickets because the API is poo poo
driver.get(self.base_url + "/fo/remedy/index.php")
time.sleep(3)
driver.find_element_by_id("ext-gen68").click() #search button
time.sleep(3)
driver.find_element_by_id("dl[search][state_search][OPEN]").click()
driver.find_element_by_id("dl[search][state_search][RESOLVED]").click()
driver.find_element_by_id("dl[search][state_search][CLOSED]").click()
driver.find_element_by_id("dl[search][state_search][IGNORED]").click()
driver.find_element_by_id("search_btn").click()
time.sleep(6)
driver.find_element_by_id("ext-gen65").click() #new button
driver.find_element_by_id("ext-gen114").click() #new menu download...
driver.find_element_by_id("select_download_format_xml").click()
driver.find_element_by_id("dload_btn").click()
# need to do loop looking for completed download
# DL_tickets_mdand_mk_20160309.xml
# DL_tickets_mdand_mk_20160309.xml.part
X = 0
while True:
time.sleep(5)
X += 1
if os.path.isfile(tmpTLfile):
if os.stat(tmpTLfile).st_size > 0:
break
if X > 60:
break # 60 iterations of 5 seconds will be 5 minutes #check file is greater than 0kb
#>>to do rename downloaded file.
os.chdir(SAVE_TO_DIRECTORY)
if os.path.isfile(docName2+".xml"):
os.remove(docName2+".xml")
files = filter(os.path.isfile, os.listdir(SAVE_TO_DIRECTORY))
files = [os.path.join(SAVE_TO_DIRECTORY, f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
newest_file = files[-1]
os.rename(newest_file, docName2+".xml")
#finish off by logging off
driver.find_element_by_id("ext-gen78").click()
time.sleep(5)
def is_element_present(self, how, what):
try: self.driver.find_element(by=how, value=what)
except NoSuchElementException as e: return False
return True
def is_alert_present(self):
try: self.driver.switch_to_alert()
except NoAlertPresentException as e: return False
return True
def close_alert_and_get_its_text(self):
try:
alert = self.driver.switch_to_alert()
alert_text = alert.text
if self.accept_next_alert:
alert.accept()
else:
alert.dismiss()
return alert_text
finally: self.accept_next_alert = True
def tearDown(self):
self.driver.quit()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()


Now it’s time to document the Talend Studio job, which just a few click…. see below;

clip_image002 Job Documentation
Generated by Talend Open Studio for Big Data
Project Name Qualys GENERATION DATE 09-Mar-2016 16:06:01
AUTHOR user@talend.com Talend Open Studio VERSION 6.1.1.20151214_1327

Summary

Project Description
Description
Preview Picture
Settings
Context List
Component List
Components Description

Project Description

Properties Values
Name Qualys
Language java
Description

Description

Properties Values
Name QualysDataProcessor
Author user@talend.com
Version 0.1
Purpose XML TO CSV
Status
Description
Creation 14-Jan-2016 14:52:57
Modification 09-Mar-2016 16:06:00

Preview Picture



Settings

Extra settings
Name Value
COMP_DEFAULT_FILE_DIR
Multi thread execution false
Implicit tContextLoad false
Status & Logs
Name Value
Use statistics (tStatCatcher) false
Use logs (tLogCatcher) false
Use volumetrics (tFlowMeterCatcher) false
On Console false
On Files false
On Databases false
Catch components statistics false
Catch runtime errors true
Catch user errors true
Catch user warnings true

Context List

ContextDefault
Name Prompt Need Prompt? Type Value Source
Password Password? false id_Password **** ContextVars

Component List

Component Name Component Type
tFileInputDelimited_1 tFileInputDelimited
tFileInputXML_1 tFileInputXML
tFileInputXML_2 tFileInputXML
tFileInputXML_5 tFileInputXML
tFileOutputDelimited_2 tFileOutputDelimited
tHttpRequest_2 tHttpRequest
tSystem_1 tSystem
tSystem_2 tSystem
tXMLMap_1 tXMLMap

Components Description

Component   tFileInputDelimited
clip_image007 UNIQUE NAME tFileInputDelimited_1 INPUT(S) none
LABEL IP2DeptMap OUTPUT(S) tXMLMap_1
Component Parameters:
Properties Values
Unique Name tFileInputDelimited_1
Component Name tFileInputDelimited
Version 0.102 (ALPHA)
Family File/Input
Start false
Startable true
SUBTREE_START false
END_OF_FLOW false
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tFileInputDelimited
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
!!!FILENAMETEXT.NAME!!! "When the input source is a stream or a zip file,footer and random shouldn't be bigger than 0."
File name/Stream "D:/Data/Qualys/maps/Host-to-Department.csv"
CSV options false
Row Separator "\n"
CSV Row Separator "\n"
Field Separator ","
Escape char """
Text enclosure """
Header 1
Footer 0
Limit
Skip empty rows false
Uncompress as zip file false
Die on error false
REPOSITORY_ALLOW_AUTO_SWITCH false
Schema
!!!SCHEMA_REJECT.NAME!!! null:errorCode errorMessage
!!!TEMP_DIR.NAME!!! "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace"
Advanced separator (for numbers) false
Thousands separator ","
Decimal separator "."
Extract lines at random false
Number of lines 10
Trim all columns false
Check column to trim [{TRIM=false, SCHEMA_COLUMN=IP}, {TRIM=false, SCHEMA_COLUMN=CIDR}, {TRIM=false, SCHEMA_COLUMN=Zone}, {TRIM=false, SCHEMA_COLUMN=Department}, {TRIM=false, SCHEMA_COLUMN=Function}]
Check each row structure against schema false
Check date false
Encoding "UTF-8"
Split row before field false
Permit hexadecimal (0xNNN) or octal (0NNNN) for numeric types false
Decode table [{DECODE=false, SCHEMA_COLUMN=IP}, {DECODE=false, SCHEMA_COLUMN=CIDR}, {DECODE=false, SCHEMA_COLUMN=Zone}, {DECODE=false, SCHEMA_COLUMN=Department}, {DECODE=false, SCHEMA_COLUMN=Function}]
!!!DESTINATION.NAME!!!
Min column number of optimize code 100
Label format IP2DeptMap
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for metadata :
Column Key Type Length Precision Nullable Comment
IP false String 11 true
CIDR false String 14 true
Zone false String 8 true
Department false String 3 true
Function false String 15 true

Original Function Parameters:
Component   tFileInputXML
clip_image009 UNIQUE NAME tFileInputXML_1 INPUT(S) tHttpRequest_2
LABEL WsTicketList OUTPUT(S) tXMLMap_1

Component Parameters:
Properties Values
Unique Name tFileInputXML_1
Component Name tFileInputXML
Version 0.102 (ALPHA)
Family File/Input|XML
Start false
Startable true
SUBTREE_START true
END_OF_FLOW false
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tFileInputXML
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH false
!!!SCHEMA.NAME!!!
File name/Stream "D:/Data/Qualys/Xml/2tmpTickets.xml"
Loop XPath query "/DATALIST/LIST/RECORD"
Mapping [{QUERY="KEY[@name=\"TICKET_#\"]", NODECHECK=, SCHEMA_COLUMN=NUMBER}, {QUERY="KEY[@name=\"STATE\"]", NODECHECK=, SCHEMA_COLUMN=STATE}, {QUERY="KEY[@name=\"DUE_DATE\"]", NODECHECK=, SCHEMA_COLUMN=DUE_DATE}, {QUERY="KEY[@name=\"IP\"]", NODECHECK=, SCHEMA_COLUMN=IP}, {QUERY="KEY[@name=\"PORT_#\"]", NODECHECK=, SCHEMA_COLUMN=PORT}, {QUERY="KEY[@name=\"DNS_HOSTNAME\"]", NODECHECK=, SCHEMA_COLUMN=DNS_HOSTNAME}, {QUERY="KEY[@name=\"SEVERITY\"]", NODECHECK=, SCHEMA_COLUMN=SEVERITY}, {QUERY="KEY[@name=\"QID\"]", NODECHECK=, SCHEMA_COLUMN=QID}, {QUERY="KEY[@name=\"VULNERABILITY_TITLE\"]", NODECHECK=, SCHEMA_COLUMN=VULNERABILITY_TITLE}, {QUERY="KEY[@name=\"MODIFIED\"]", NODECHECK=, SCHEMA_COLUMN=MODIFIED}, {QUERY="KEY[@name=\"CREATED\"]", NODECHECK=, SCHEMA_COLUMN=CREATED}, {QUERY="KEY[@name=\"RESOLVED\"]", NODECHECK=, SCHEMA_COLUMN=RESOLVED}]
Limit -1
Die on error false
!!!SCHEMA_REJECT.NAME!!! null:errorCode errorMessage
Advanced separator (for numbers) false
Thousands separator ","
Decimal separator "."
Ignore the namespaces false
Ignore DTD file false
Generate a temporary file "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml"
Use Separator for mode Xerces false
Field Separator ","
Generation mode Dom4j
Validate date false
Encoding "UTF-8"
Min column number of optimize code 100
Label format WsTicketList
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for metadata :
Column Key Type Length Precision Nullable Comment
NUMBER false Integer 6 true
STATE false String 12 true
DUE_DATE false String 20 true
IP false String 15 true
PORT false String 5 true
DNS_HOSTNAME false String 37 true
SEVERITY false String 46 true
QID false Integer 6 true
VULNERABILITY_TITLE false String 197 true
MODIFIED false String 28 true
CREATED false String 28 true
RESOLVED false String 28 true

Original Function Parameters:
Component   tFileInputXML
clip_image009[1] UNIQUE NAME tFileInputXML_2 INPUT(S) none
LABEL AssetList OUTPUT(S) tXMLMap_1

Component Parameters:
Properties Values
Unique Name tFileInputXML_2
Component Name tFileInputXML
Version 0.102 (ALPHA)
Family File/Input|XML
Start false
Startable true
SUBTREE_START false
END_OF_FLOW false
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tFileInputXML
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH false
!!!SCHEMA.NAME!!!
File name/Stream "D:/Data/Qualys/Xml/tmpAssetList.xml"
Loop XPath query "/HOST_LIST_OUTPUT/RESPONSE/HOST_LIST/HOST"
Mapping [{QUERY="ID", NODECHECK=, SCHEMA_COLUMN=Qualys_Asset_ID}, {QUERY="IP", NODECHECK=, SCHEMA_COLUMN=IP}, {QUERY="TRACKING_METHOD", NODECHECK=, SCHEMA_COLUMN=TRACKING_METHOD}, {QUERY="NETWORK_ID", NODECHECK=, SCHEMA_COLUMN=NETWORK_ID}, {QUERY="OS", NODECHECK=, SCHEMA_COLUMN=OS}, {QUERY="LAST_COMPLIANCE_SCAN_DATETIME", NODECHECK=, SCHEMA_COLUMN=LAST_COMPLIANCE_SCAN_DATETIME}, {QUERY="LAST_VULN_SCAN_DATETIME", NODECHECK=, SCHEMA_COLUMN=LAST_VULN_SCAN_DATETIME}, {QUERY="NETBIOS", NODECHECK=, SCHEMA_COLUMN=NETBIOS}, {QUERY="DNS", NODECHECK=, SCHEMA_COLUMN=DNS}]
Limit -1
Die on error false
!!!SCHEMA_REJECT.NAME!!! null:errorCode errorMessage
Advanced separator (for numbers) false
Thousands separator ","
Decimal separator "."
Ignore the namespaces false
Ignore DTD file false
Generate a temporary file "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml"
Use Separator for mode Xerces false
Field Separator ","
Generation mode Dom4j
Validate date false
Encoding "UTF-8"
Min column number of optimize code 100
Label format AssetList
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for metadata :
Column Key Type Length Precision Nullable Comment
Qualys_Asset_ID false Integer 9 true
IP false String 15 true
TRACKING_METHOD false String 11 true
NETWORK_ID false Integer 5 true
OS false String 80 true
LAST_COMPLIANCE_SCAN_DATETIME false java.util.Date 20 true
LAST_VULN_SCAN_DATETIME false java.util.Date 20 true
NETBIOS false String 15 true
DNS false String 45 true

Original Function Parameters:
Component   tFileInputXML
clip_image009[2] UNIQUE NAME tFileInputXML_5 INPUT(S) none
LABEL VulnerabilityList OUTPUT(S) tXMLMap_1

Component Parameters:
Properties Values
Unique Name tFileInputXML_5
Component Name tFileInputXML
Version 0.102 (ALPHA)
Family File/Input|XML
Start false
Startable true
SUBTREE_START false
END_OF_FLOW false
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tFileInputXML
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH false
!!!SCHEMA.NAME!!!
File name/Stream "D:/Data/Qualys/Xml/2tmpVulns.xml"
Loop XPath query "/DATALIST/LIST/RECORD"
Mapping [{QUERY="KEY[@name=\"QID\"]", NODECHECK=, SCHEMA_COLUMN=QID}, {QUERY="KEY[@name=\"TITLE\"]", NODECHECK=, SCHEMA_COLUMN=TITLE}, {QUERY="KEY[@name=\"CATEGORY\"]", NODECHECK=, SCHEMA_COLUMN=CATEGORY}, {QUERY="KEY[@name=\"CVE_ID\"]", NODECHECK=, SCHEMA_COLUMN=CVE_ID}, {QUERY="KEY[@name=\"CVSS_BASE\"]", NODECHECK=, SCHEMA_COLUMN=CVSS_BASE}, {QUERY="KEY[@name=\"BUGTRAQ_ID\"]", NODECHECK=, SCHEMA_COLUMN=BUGTRAQ_ID}, {QUERY="KEY[@name=\"MODIFIED\"]", NODECHECK=, SCHEMA_COLUMN=MODIFIED}, {QUERY="KEY[@name=\"PUBLISHED\"]", NODECHECK=, SCHEMA_COLUMN=PUBLISHED}]
Limit -1
Die on error false
!!!SCHEMA_REJECT.NAME!!! null:errorCode errorMessage
Advanced separator (for numbers) false
Thousands separator ","
Decimal separator "."
Ignore the namespaces false
Ignore DTD file false
Generate a temporary file "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml"
Use Separator for mode Xerces false
Field Separator ","
Generation mode Dom4j
Validate date false
Encoding "UTF-8"
Min column number of optimize code 100
Label format VulnerabilityList
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for metadata :
Column Key Type Length Precision Nullable Comment
QID false Integer 6 true
TITLE false String 241 true
CATEGORY false String 27 true
CVE_ID false String 2855 true
CVSS_BASE false String 3 true
BUGTRAQ_ID false String 263 true
MODIFIED false String 28 true
PUBLISHED false String 28 true

Original Function Parameters:
Component   tFileOutputDelimited
clip_image011 UNIQUE NAME tFileOutputDelimited_2 INPUT(S) tXMLMap_1,  tFileInputXML_2,  tFileInputXML_5,  tFileInputDelimited_1
LABEL __UNIQUE_NAME__ OUTPUT(S) tSystem_2

Component Parameters:
Properties Values
Unique Name tFileOutputDelimited_2
Component Name tFileOutputDelimited
Version 0.101 (ALPHA)
Family File/Output
Startable false
SUBTREE_START false
END_OF_FLOW true
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tFileOutputDelimited
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
Use Output Stream false
Output Stream outputStream
File Name "D:/Data/Qualys/VTM.csv"
Row Separator "\r\n"
Use OS line separator as row separator when CSV Row Separator is set to CR,LF or CRLF. true
CSV Row Separator "\r\n"
Field Separator ","
Append false
Include Header true
Compress as zip file false
REPOSITORY_ALLOW_AUTO_SWITCH false
Schema
Advanced separator (for numbers) false
Thousands separator ","
Decimal separator "."
CSV options true
Escape char """
Text enclosure """
Create directory if does not exist true
Split output in several files false
Rows in each output file 1000
Custom the flush buffer size false
Row number 1
Output in row mode false
Encoding "ISO-8859-15"
Don't generate empty file false
Min column number of optimize code 90
Label format __UNIQUE_NAME__
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for VTM :
Column Key Type Length Precision Nullable Comment
QUALYS_TICKET_NUMBER false Integer 10 true
TICKET_CREATION_DATETIME false String 20 true
TICKET_RESOLVED_DATETIME false String 28 true
TICKET_MODIFIED_DATETIME false String 28 true
TICKET_DUE_DATETIME false String 20 true
TICKET_CURRENT_STATE false String 6 true
IP false String 15 true
PORT false String 4 true
QUALYS_SEVERITY false String 255 true
QID false Integer 6 true
TRACKING_METHOD false String 11 true
OS false String 80 true
NETBIOS false String 15 true
DNS false String 45 true
LAST_COMPLIANCE_SCAN_DATETIME false java.util.Date 20 true
LAST_VULN_SCAN_DATETIME false java.util.Date 20 true
TITLE false String 241 true
CATEGORY false String 27 true
CVE_ID false String 2855 true
CVSS_BASE false String 3 true
BUGTRAQ_ID false String 263 true
QID_MODIFIED false String 28 true
QID_PUBLISHED false String 28 true
CIDR false String 14 true
Zone false String 8 true
Department false String 3 true
Function false String 15 true

Original Function Parameters:
Component   tHttpRequest
clip_image013 UNIQUE NAME tHttpRequest_2 INPUT(S) tSystem_1
LABEL __UNIQUE_NAME__ OUTPUT(S) tFileInputXML_1

Component Parameters:
Properties Values
Unique Name tHttpRequest_2
Component Name tHttpRequest
Version 0.101 (ALPHA)
Family Internet
Start false
Startable true
SUBTREE_START true
END_OF_FLOW true
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tHttpRequest
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
REPOSITORY_ALLOW_AUTO_SWITCH false
Property null:ResponseContent
URI "https://qualysapi.qualys.eu/api/2.0/fo/asset/host/?action=list&truncation_limit=1000000&details=All&vm_scan_since=2015-01-01"
Method POST
Post parameters from file
Write response content to file true
"D:/Data/Qualys/Xml/tmpAssetList.xml"
Create directory if not exists false
Headers [{HEADER_NAME="X-Requested-With", HEADER_VALUE="Talend"}, {HEADER_NAME="Authorization", HEADER_VALUE="Basic Z29kbW9kZTp1ODAwOHBhc3N3ZG5vdGxpa2VseXRvYmVoZXJlLi4uLi4xMjM="}]
Need authentication false
user ""
password **
Die on error false
Label format __UNIQUE_NAME__
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for tHttpRequest_2 :
Column Key Type Length Precision Nullable Comment
ResponseContent false String true

Original Function Parameters:
Component   tSystem
clip_image015 UNIQUE NAME tSystem_1 INPUT(S) none
LABEL __UNIQUE_NAME__ OUTPUT(S) tHttpRequest_2

Component Parameters:
Properties Values
Unique Name tSystem_1
Component Name tSystem
Version 0.101 (ALPHA)
Family System
Start true
Startable true
SUBTREE_START true
END_OF_FLOW true
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tSystem
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
Use Home Directory false
Home Directory "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace"
Use Single Command true
Command "C:\\Python27\\python.exe D:\\Data\\Qualys\\ExternalJob\\QualysDownloadSelenium.py"
Use Array Command false
Command []
Standard Output OUTPUT_TO_CONSOLE
Error Output OUTPUT_TO_CONSOLE
REPOSITORY_ALLOW_AUTO_SWITCH false
!!!SCHEMA.NAME!!! null:
Environment variables []
Label format __UNIQUE_NAME__
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for tSystem_1 :
Column Key Type Length Precision Nullable Comment

Original Function Parameters:
Component   tSystem
clip_image015[1] UNIQUE NAME tSystem_2 INPUT(S) tFileOutputDelimited_2
LABEL __UNIQUE_NAME__ OUTPUT(S) none

Component Parameters:
Properties Values
Unique Name tSystem_2
Component Name tSystem
Version 0.101 (ALPHA)
Family System
Start false
Startable true
SUBTREE_START true
END_OF_FLOW true
Activate true
DUMMY false
tStatCatcher Statistics false
Help org.talend.help.tSystem
Update components true
IREPORT_PATH
JAVA_LIBRARY_PATH C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
Use Home Directory false
Home Directory "C:/install/TOS_BD-20151214_1327-V6.1.1/workspace"
Use Single Command true
Command "cmd /c del D:\\Data\\Qualys\\Xml\\*.xml"
Use Array Command false
Command []
Standard Output OUTPUT_TO_CONSOLE
Error Output OUTPUT_TO_CONSOLE
REPOSITORY_ALLOW_AUTO_SWITCH false
!!!SCHEMA.NAME!!! null:
Environment variables []
Label format __UNIQUE_NAME__
Hint format <b>__UNIQUE_NAME__</b><br>__COMMENT__
Connection format row
Show Information false
Comment
Use an existing validation rule false
Validation Rule Type

Schema for VTM :
Column Key Type Length Precision Nullable Comment
QUALYS_TICKET_NUMBER false Integer 10 true
TICKET_CREATION_DATETIME false String 20 true
TICKET_RESOLVED_DATETIME false String 28 true
TICKET_MODIFIED_DATETIME false String 28 true
TICKET_DUE_DATETIME false String 20 true
TICKET_CURRENT_STATE false String 6 true
IP false String 15 true
PORT false String 4 true
QUALYS_SEVERITY false String 255 true
QID false Integer 6 true
TRACKING_METHOD false String 11 true
OS false String 80 true
NETBIOS false String 15 true
DNS false String 45 true
LAST_COMPLIANCE_SCAN_DATETIME false java.util.Date 20 true
LAST_VULN_SCAN_DATETIME false java.util.Date 20 true
TITLE false String 241 true
CATEGORY false String 27 true
CVE_ID false String 2855 true
CVSS_BASE false String 3 true
BUGTRAQ_ID false String 263 true
QID_MODIFIED false String 28 true
QID_PUBLISHED false String 28 true
CIDR false String 14 true
Zone false String 8 true
Department false String 3 true
Function false String 15 true

Original Function Parameters:
Component   tXMLMap
clip_image017 UNIQUE NAME tXMLMap_1 INPUT(S) tXMLMap_1,  tFileInputXML_2,  tFileInputXML_5,  tFileInputDelimited_1
LABEL __UNIQUE_NAME__ OUTPUT(S) tFileOutputDelimited_2

Component Parameters:
Properties         Values
Activate          true
tStatCatcher Statistics          false
Map Editor:
Keep order for document          false
Show Information          false
Comment
Use an existing validation rule          false

Schema for VTM :
Column Key Type Length Precision Nullable Comment
QUALYS_TICKET_NUMBER false Integer 10 true
TICKET_CREATION_DATETIME false String 20 true
TICKET_RESOLVED_DATETIME false String 28 true
TICKET_MODIFIED_DATETIME false String 28 true
TICKET_DUE_DATETIME false String 20 true
TICKET_CURRENT_STATE false String 6 true
IP false String 15 true
PORT false String 4 true
QUALYS_SEVERITY false String 255 true
QID false Integer 6 true
TRACKING_METHOD false String 11 true
OS false String 80 true
NETBIOS false String 15 true
DNS false String 45 true
LAST_COMPLIANCE_SCAN_DATETIME false java.util.Date 20 true
LAST_VULN_SCAN_DATETIME false java.util.Date 20 true
TITLE false String 241 true
CATEGORY false String 27 true
CVE_ID false String 2855 true
CVSS_BASE false String 3 true
BUGTRAQ_ID false String 263 true
QID_MODIFIED false String 28 true
QID_PUBLISHED false String 28 true
CIDR false String 14 true
Zone false String 8 true
Department false String 3 true
Function false String 15 true



P.S. for the astute, no I did not leave any credentials of use :)  it's garbage...

Comments

Popular Posts