SCRIPTING_04 - Jython stage does not bypass proxy leading to timeout

Problem

1. The reason why this may occur is because the Python requests library does not support wildcards in the no_proxy environment variable.

2. StreamSets Platform copies the http.nonProxyHosts over to the no_proxy variable automatically when the deployment starts.

3. However, the Python requests library does not respect the same syntax, and wildcard characters such as * are not interpreted correctly.

4. This is why specifying nonProxyHosts like *.host.com or 10.0.0.* can be problematic.

1. You can disable the use of proxies in your Jython script by creating a session and setting session.trust_env to False. ex:
   
   

   url = "http://test.com"
   session = requests.Session()
   session.trust_env = False
   try:
       response = session.get(url)
       
       record = sdc.createRecord('test')
       record.value = response.text
       
       
       
       cur_batch.add(record)
   
   except Exception as e:
       cur_batch.addError(record, str(e))
       cur_batch.process(entityName, str(offset))

1. The requests library has shown to support specifying CIDR ranges and subdomains for the no_proxy variable in the following format:

   • .host.com

   • 10.0.0.0/8

   • ex: http.nonProxyHosts=.host.com, 10.0.0.0/8

2. Account for the different types of syntax and include them in your deployment’s http.nonProxyHosts list.