ns.config Encryption
From build 13.0–76.31, upon deployment, NetScaler automatically generates a unique master key (to encrypt) to protect certain security parameters stored in the NetScaler ns.conf file, such as service accounts passwords and SSL Private Key passwords. In the past these were encrypted with a hardcoded master key using AES256-CBC encryption method in the firmware making it shared between all appliances. This was quite useful for tranfering config lines from one appliance to another during migrations as it didn't require decryption of the CLI.
Now, that part of the config is protected by unique appliance generated key indicated by the kek flag in the ns.conf:
You can still copy CLI files, but you need to copy the keys files as well (F1.key & F2.key) located in /nsconfig/:
If you wanna know more visit the following blog posts: ferroque dozer.nz
What about file level security?
A bad practice that we see, three times out of five, customers store their certificates' Private keys unencrypted on the NetScaler appliances. Data protection practices should extends to the file system level of the FreeBSD OS. Protecting your critical cryptographic keys is esential and the bare minimum is storying them not in plaintext files. FIPS models use HSM (hardware security module) which stores the keys in cryptographic module (encrypted with FIPS approved ciphers) protecting them from physical and software attacks. The NetScaler also supports Thales Luna network HSM for MPX, VPX & SDX models not supporting FIPS. These are non-exportable variants.
How to find your unecrypted keys?
- You can use Xpand Config Analyzer. It's completely free and it will flag any certificate stored in plaintext.
NOTE! Your configuration is loaded and analyzed only in your local browser session. XConfig does not store the ns.config file, unless you choose to do so! Even upon saving a strict privacy settings apply like retantion period, masking all sensitive info as IPs, encrypted CLI string, etc.
- You can manually go over the configuration file and look for ssl certkey lines without a password.
No password:
add ssl certKey CERT_CONSTOSO_COM_WILD -cert CERT_CONSTOSO_COM_WILD_PUBLIC.pem -key CERT_CONSTOSO_COM_WILD_KEY.pem -expiryMonitor DISABLED
Password protected:
add ssl certKey API_SW_KERNETIX_LOCAL_CERT -cert API_SW_KERNETIX_LOCAL_2023.cer -key API_SW_KERNETIX_LOCAL_2023.key 3887f4bed4d9a74b665fed690d6df59ec32c521627a1d70515fc47e9200647bd -encrypted -encryptmethod ENCMTHD_3 -kek -suffix 2021_04_21_18_24_38
- You can confirm by analyzing the key files:
RSA Encrypted Key VS Plaintext Key
Encrypting the Private Keys
Below you can find openssl cli to encrypt your keys on the NetScaler appliance itself. I don't like keeping .pfx files, but rather certificate-key pairs. You can delete the .pfx afterwards. It works absolutly same for single PEM files or CRT.
Existing CLI
add ssl certKey CERT_PAIR_NAME -cert CERT_PAIR_FILE_PUBLIC.cer -key CERT_PAIR_FILE_PLAINTEXT_PRIVATE.key
FreeBSD CLI:
openssl pkcs12 -inkey /nsconfig/ssl/ CERT_PAIR_FILE_PLAINTEXT_PRIVATE.key -in /nsconfig/ssl/CERT_PAIR_FILE_PUBLIC.cer -export -out /nsconfig/ssl/CERT_PAIR_PFX_FILE.pfx
openssl pkcs12 -in /nsconfig/ssl/CERT_PAIR_PFX_FILE.pfx -nocerts -out /nsconfig/ssl/CERT_PAIR_FILE_ENCRYPTED_PRIVATE.key
openssl pkcs12 -in /nsconfig/ssl/CERT_PAIR_PFX_FILE.pfx -clcerts -nokeys -out /nsconfig/ssl/CERT_PAIR_FILE_PUBLIC.cer
Update CLI:
update ssl certKey CERT_PAIR_NAME -cert CERT_PAIR_FILE_PUBLIC.cer -key CERT_PAIR_FILE_ENCRYPTED_PRIVATE.key -password *******************
Best practices for Private Key management
- Do not generate keys outside the server that's going to use them. Create a CSR (Certificate signing request) on the NetScaler itself. This way a key cannot be compromised during exchange.
- Enforce Password Complexity requirement for NetScaler imports
- Store the keys in hardware HSM or services like Azure Key Vault or AWS CloudHSM.
- Have in mind that servers (including the NetScaler) needs the private key only upon restart. These are loaded into memory and not read until next reboot.
- Use CIPHERS with Perfect Forward Secrecy, so any system compromise (stolen key or server) does not lead to data leak (past recorded traffic will not be decrypted).
- Keep low TLS Cert Life Span (Google already proposed to reduce the life span of digital certificates used to secure websites and other online communications to just 90 days. Currently, public Transport Layer Security (TLS) certificates have a maximum validity of 13 months, or 398 days.)